-
Backpropagation is The Adjoint Method
Backpropagation is often introduced as something developed within the field of machine learning. However, the story is that backpropagation really is just a special case of the adjoint method. This note is in two parts. In the first part we review the adjoint method while in the second part we describe how backpropagation is a special case of the adjoint method with a structure that result in scalable computations algorithms.
-
Associative Scan
and its relation to state-space models
-
Structured Masked Attention
and its relation to semiseparable matrices and state-space models
-
Block Tridiagonal Matrices
and their relation to semiseparable matrices and state-space models
-
Continuous Matrix Factorizations
and their relation to kernel approximations