I have worked with several tensor languages / GPU domain specific languages (Trition, Hidet, futhark, XLA/Jax), and read the papers about several more (Tiramisu, tvm, tensor comprehensions). However, every one of them have limitations that I think fail to address the problems faced by performance engineers at ML companies.
I have been wanting to learn about polyhedral compilation for the longest time, and finally had some time recently to wrap my head around it. I wanted to see its applicability into tensor programming DSLs, particular for my planned language texpr, the idea of which is not full automatic parallelization like tensor comprehensions, which does use polyhedral techniques, but make CUDA programming easier and more composable, ideally with a hierarchical IR which exposes incrementally more hardware details the lower down you go.
I think that there are two orthogonal concerns when it comes to the design of a type system for memory management, that of efficiency and correctness. Efficiency is concerned about modeling lower-level details on the type level as to prevent unsafe uses, while correctness is concerned about making mutability a contract as to ensure better software design, in terms of “metrics” such as composability or local reasoning. I think the two concerns are different, although they are addressed in the same manner by Rust’s ownership model, for example. It attempts to both model the concept of a stack (or more generally inline allocations), which limits the expressiveness of its memory model, which necessitates escape hatches like interior mutability that subverts the goal of local mutable reasoning.
The Hessian of the KL divergence is the fisher information matrix
