Backpropagation as a Linear Solve

5 October 2026 · Download PDF

In the previous post on backpropagation, we showed that backpropagation is a special case of the adjoint method. In this post we skip the adjoint description and instead derive the same result directly from the chain rule.

1. The forward relation

We denote the data by $𝑢_{0}$ and the parameters by $𝜃 \in ℝ^{𝑝}$ . The network has $𝑁$ layers, each of which is a function $Φ_{𝑖}$ that takes as input outputs from previous layers ( $𝑢_{𝑖 - 1}, \dots, 𝑢_{0}$ ) as well as the parameters ( $𝜃$ ), and produces the next layer output $𝑢_{𝑖}$ . The full forward pass, including all layer outputs, can be expressed as

\underset{𝑢 (𝜃)}{\underset{⏟}{[\begin{matrix} 𝑢_{1} \\ 𝑢_{2} \\ ⋮ \\ 𝑢_{𝑁} \end{matrix}]}} = \underset{Φ (𝑢 (𝜃), 𝜃)}{\underset{⏟}{[\begin{matrix} Φ_{1} (𝑢_{0}, 𝜃) \\ Φ_{2} (𝑢_{1}, 𝑢_{0}, 𝜃) \\ ⋮ \\ Φ_{𝑁} (𝑢_{𝑁 - 1}, \dots, 𝑢_{0}, 𝜃) \end{matrix}]}} \in ℝ^{𝑑},

(1)

where $𝑑 = \sum_{𝑖 = 1}^{𝑁} 𝑛_{𝑖}$ and $𝑛_{𝑖}$ is the size of $𝑢_{𝑖}$ .

We introduce the projection $𝑃 \in ℝ^{𝑟 \times 𝑑}$ that takes out relevant layer-outputs, i.e.¹¹ Often $𝑃$ just picks out the last layer, i.e. $𝑢_{𝑃} = 𝑢_{𝑁} = 𝑃 𝑢 (𝜃), 𝑃 = (\begin{matrix} 0 & \dots & 0 & 𝐼_{𝑟} \end{matrix}) .$

𝑢_{𝑃} (𝜃) = 𝑃 𝑢 (𝜃) .

(2)

Using this the loss function can be written as

𝒥︀ (𝜃) = ℓ (𝑢_{𝑃} (𝜃)) + 𝜇 ℛ︀ (𝜃),

(3)

where $ℓ$ is the data loss and $ℛ︀$ is a regularization term.

2. Differentiating the network

Applying the chain rule to (1), we have

\frac{d 𝑢}{d 𝜃} = \frac{𝜕 Φ}{𝜕 𝑢} \frac{d 𝑢}{d 𝜃} + \frac{𝜕 Φ}{𝜕 𝜃}, hence \frac{d 𝑢}{d 𝜃} = {(\underset{𝐿}{\underset{⏟}{𝐼 - \frac{𝜕 Φ}{𝜕 𝑢}}})}^{- 1} \underset{𝑀^{⊤}}{\underset{⏟}{\frac{𝜕 Φ}{𝜕 𝜃}}} .

(4)

Applying the projection $𝑃$ , it further follows that

\frac{d 𝑢_{𝑃}}{d 𝜃} = 𝑃 𝐿^{- 1} 𝑀^{⊤} .

(5)

At this point it is worth looking at the structure of the matrices. As the $𝑖$ th layer depends only on previous outputs, the matrix $𝐿$ is block lower-triangular with identity blocks on the diagonal²² The sparsity pattern of the strictly lower/upper part of a (block) triangular matrix is a directed acyclic graph (DAG), which is often how backpropagation is introduced.. The triangular structure makes it efficient to solve the linear system

𝐿^{⊤} Λ = 𝑃^{⊤}, Λ \in ℝ^{𝑑 \times 𝑟} .

(6)

Using $Λ$ it follows that

\frac{d 𝑢_{𝑃}}{d 𝜃} = Λ^{⊤} 𝑀^{⊤} .

(7)

In practice, as seen in the next section, one rarely needs to form $\frac{d 𝑢_{𝑃}}{d 𝜃}$ as a full matrix; it suffices to define products with it.

3. A scalar loss

Applying the chain rule to (3), we have

\nabla_{𝜃} 𝒥︀ = {(\frac{d 𝑢_{𝑃}}{d 𝜃})}^{⊤} \nabla_{𝑢_{𝑃}} ℓ + 𝜇 \nabla_{𝜃} ℛ︀ = 𝑀 𝐿^{- ⊤} 𝑃^{⊤} \nabla_{𝑢_{𝑃}} ℓ + 𝜇 \nabla_{𝜃} ℛ︀ .

(8)

Given that $\nabla_{𝑢_{𝑃}} ℓ$ is a vector of length $𝑟$ we have that the product $𝑃^{⊤} \nabla_{𝑢_{𝑃}} ℓ$ is a vector of length $𝑑$ . It is therefore of interest to first form $𝑃^{⊤} \nabla_{𝑢_{𝑃}} ℓ$ and then solve $𝐿^{⊤} 𝜆 = 𝑃^{⊤} \nabla_{𝑢_{𝑃}} ℓ$ , so that

\nabla_{𝜃} 𝒥︀ = 𝑀 𝜆 + 𝜇 \nabla_{𝜃} ℛ︀,

(9)

which is equivalent to the final result of the adjoint method (see the previous post on backpropagation). Note that $𝜆$ here is the negative of the adjoint variable of the previous post, which absorbs the sign of $\frac{𝜕 𝑓}{𝜕 𝜃} = - 𝑀^{⊤}$ there.

Cite this post

bibtex


@misc{paltorp2026backpropagationlinearsolve,
  author = {Paltorp, Mikkel},
  title = {{Backpropagation as a Linear Solve}},
  year = {2026},
  month = oct,
  url = {https://mipals.github.io/pdf/2026/10-05-backpropagation-linear-solve/backpropagation-linear-solve.pdf},
  note = {Blog post}
}