Частные производные и матричная арифметикаPartial derivatives and matrix arithmetic

Entry: one function, two routes

Take the matrix $A = \begin{pmatrix}1&3\\1&2\end{pmatrix}$ and the function of two variables $f(x) = x^tAx$, where $x = (x^1,x^2)^t$. Expand the product: $Ax = (x^1+3x^2,\ x^1+2x^2)^t$, and

The question of the lesson: how fast does $f$ change at the point $x_0 = (1,1)^t$ if only $x^1$ moves? And if only $x^2$ moves? The answer is two numbers, $6$ and $8$. By the end of the lesson you get them by two independent routes: partial derivatives of the formula, and a single matrix product $(A+A^t)x_0$. On the way you need sizes in a product, the transpose rule, the trace, and partial derivatives — exactly what lectures math-01 and math-02 stand on.

Sizes and the matrix product

The picture: the entry of $AB$ in row $i$, column $j$ is the dot product of row $i$ of $A$ with column $j$ of $B$. For it to exist, the length of a row of $A$ must equal the height of a column of $B$.

Example. $A = \begin{pmatrix}1&2\\0&-1\\3&1\end{pmatrix}\in M_{3\times2}$ and $x = (2,1)^t\in M_{2\times1}$. Sizes: $(3\times2)(2\times1) = 3\times1$, so the answer is a column of three numbers:

The product $xA$ is undefined: in $(2\times1)(3\times2)$ the inner sizes $1$ and $3$ do not match.

A row is a $1\times N$ matrix, a column is an $N\times1$ matrix. So for $x,y\in\R^N$ the product $x^ty$ has size $(1\times N)(N\times1) = 1\times1$ — it is one number $x^1y^1+\dots+x^Ny^N$, while $xy^t$ has size $(N\times1)(1\times N) = N\times N$ — it is a whole matrix, the outer product.

Transpose of a product

The transpose reflects the table across the main diagonal: $(A^t)_{ij} = a_{ji}$, rows become columns. If $A\in M_{m\times n}$, then $A^t\in M_{n\times m}$.

Why does the order change for a product? The sizes give the answer before the formula does. Let $A\in M_{m\times n}$, $B\in M_{n\times k}$; then $(AB)^t\in M_{k\times m}$. From $A^t\in M_{n\times m}$ and $B^t\in M_{k\times n}$ a matrix of size $k\times m$ can only be obtained as $B^tA^t$; the product $A^tB^t$ is not even defined when $m\ne k$.

Why $(AB)^t = B^tA^t$. Compare the entries in row $i$, column $j$:

The third equality swaps two numbers in each term; the fourth writes $b_{si} = (B^t)_{is}$ and $a_{js} = (A^t)_{sj}$.

A number equals its own transpose. A $1\times1$ matrix does not change under transposition. The lecturer uses this all the time: for $A\in M_{D\times N}$, $u\in\R^N$, $v\in\R^D$

Example. $u = (1,2)^t$, $v = (1,0,1)^t$, and $A\in M_{3\times2}$ is the same as in the example above. Then $Au = (1+4,\ 0-2,\ 3+2)^t = (5,-2,5)^t$, and $v^tAu = 5+0+5 = 10$. On the other side, $A^tv = (1+3,\ 2+1)^t = (4,3)^t$, and $u^tA^tv = 4+6 = 10$ ✓.

Trace and the cyclic property

The picture: a square matrix has a diagonal from the top-left corner to the bottom-right one. The trace is the sum of the numbers on it. The unexpected property of the trace is that it does not care about the order of two factors, even though the products themselves do.

The cyclic property was stated in lecture math-01 (board 15). Its proof is item 4 of homework HW-M01, so there is no proof here: the lesson teaches you to use the property and to check it on numbers.

Example. $A = \begin{pmatrix}1&2&0\\0&1&1\end{pmatrix}\in M_{2\times3}$, $B = \begin{pmatrix}1&0\\2&1\\0&3\end{pmatrix}\in M_{3\times2}$. The trace needs only the diagonal entries. For $AB$ ($2\times2$): $(AB)_{11} = 1\cdot1+2\cdot2+0\cdot0 = 5$, $(AB)_{22} = 0\cdot0+1\cdot1+1\cdot3 = 4$, so $\tr(AB) = 9$. For $BA$ ($3\times3$): $(BA)_{11} = 1\cdot1+0\cdot0 = 1$, $(BA)_{22} = 2\cdot2+1\cdot1 = 5$, $(BA)_{33} = 0\cdot0+3\cdot1 = 3$, so $\tr(BA) = 9$ ✓.

Three consequences that are used more often than the property itself.

1. A number equals its own trace: for a $1\times1$ matrix, $\tr c = c$.
2. Factors can be rotated around the cycle: $\tr(ABC) = \tr(CAB) = \tr(BCA)$. The first equality is $\tr(XY) = \tr(YX)$ with $X = AB$, $Y = C$; the second is the same with $X = CA$, $Y = B$: $\tr\big((CA)B\big) = \tr\big(B(CA)\big)$. Factors cannot be permuted arbitrarily: $\tr(ABC)$ and $\tr(ACB)$ are different in general.
3. A vector at the end can be moved to the front: for $u,v\in\R^N$, $\tr(uv^t) = \tr(v^tu) = v^tu$. On the left is the trace of an $N\times N$ matrix, on the right a single dot product.

This is how lecture math-01 got the answer for $f(X) = a^tXb$: the matrix $U$ is “trapped” in the middle, and it is moved to the end:

Derivative of a composite function of one variable

The picture: $y$ depends on $u$, and $u$ depends on $x$. Move $x$ by a small $h$: $u$ moves by about $g'(x_0)\,h$, and $y$ by about $f'(u_0)\cdot g'(x_0)\,h$. The rates multiply.

Example. $y = (1+\sin x)^3$ at the point $x = 0$. Layers: $y = u^3$, $u = 1+\sin x$. The value of the inner layer: $u_0 = 1+\sin0 = 1$. Derivatives: $\frac{dy}{du} = 3u^2 = 3$ at $u_0 = 1$ and $\frac{du}{dx} = \cos x = 1$ at $x = 0$. Result: $\frac{dy}{dx} = 3\cdot1 = 3$. With three layers there are three factors, and each derivative is taken at the value of its own inner layer.

Partial derivatives and the gradient

The picture: the graph $z = f(x^1,x^2)$ is a surface over the plane. Cut it with the vertical plane $x^2 = \text{const}$: you get a curve, and the slope of that curve is the partial derivative with respect to $x^1$. One coordinate moves, the others stay put.

Example from the entry. $f = (x^1)^2 + 4x^1x^2 + 2(x^2)^2$. In $x^1$, treating $x^2$ as a number: $\frac{\partial f}{\partial x^1} = 2x^1 + 4x^2$, which is $6$ at $(1,1)^t$. In $x^2$, treating $x^1$ as a number: $\frac{\partial f}{\partial x^2} = 4x^1 + 4x^2$, which is $8$ at the point.

Example with a composite function. $f(x^1,x^2) = \ln\big(1+(x^1)^2x^2\big)$. In $x^1$ this is the one-variable chain rule, with $x^2$ standing as a number: $\frac{\partial f}{\partial x^1} = \frac{2x^1x^2}{1+(x^1)^2x^2}$; at $(1,1)^t$ this is $\frac{2}{2} = 1$.

The lecturer stressed in math-01: if $Df(x_0)$ exists, its entries are the partial derivatives, but the mere existence of all partial derivatives is weaker than the existence of $Df(x_0)$.

Gradient of a quadratic form. Why did the entry example give $(A+A^t)x_0$? Expand $f(x_0+hu) = (x_0+hu)^tA(x_0+hu)$, using $(x_0+hu)^t = x_0^t + hu^t$:

The number $x_0^tAu$ equals its own transpose $u^tA^tx_0$, so the coefficient of $h$ is $u^tA^tx_0 + u^tAx_0 = u^t(A+A^t)x_0$. By the proposition on directional derivatives from lecture math-01, the limit of $\frac{f(x_0+hu)-f(x_0)}{h}$ as $h\to0$ equals $Df(x_0)\,u$, so $Df(x_0)\,u = \big((A+A^t)x_0\big)^tu$ for every $u$, and

Check on the entry: $A + A^t = \begin{pmatrix}2&4\\4&4\end{pmatrix}$, and $(A+A^t)(1,1)^t = (6,8)^t$ — the same $6$ and $8$ ✓.

Second and mixed derivatives

The partial derivative $\partial f/\partial x^i$ is again a function of the same variables, and it can be differentiated once more.

Example. $f(x,y) = x^2y^3$. First derivatives: $\frac{\partial f}{\partial x} = 2xy^3$, $\frac{\partial f}{\partial y} = 3x^2y^2$. Second: $\frac{\partial^2 f}{\partial x^2} = 2y^3$, $\frac{\partial^2 f}{\partial y^2} = 6x^2y$, $\frac{\partial^2 f}{\partial x\,\partial y} = \frac{\partial}{\partial x}(3x^2y^2) = 6xy^2$ and $\frac{\partial^2 f}{\partial y\,\partial x} = \frac{\partial}{\partial y}(2xy^3) = 6xy^2$. The mixed derivatives coincide.

Two constructions made of second derivatives appear in the course right away. The Hessian is the matrix $H$ with entries $H_{ij} = \frac{\partial^2 f}{\partial x^i\,\partial x^j}$. For the entry function $\frac{\partial^2 f}{\partial (x^1)^2} = 2$, $\frac{\partial^2 f}{\partial x^1\,\partial x^2} = 4$, $\frac{\partial^2 f}{\partial (x^2)^2} = 4$, that is, $H = \begin{pmatrix}2&4\\4&4\end{pmatrix} = A + A^t$. The Laplacian of a function of two variables is $\Delta f = \frac{\partial^2 f}{\partial x^2} + \frac{\partial^2 f}{\partial y^2}$, the sum of the diagonal of the Hessian. For example, $\Delta(x^2-y^2) = 2-2 = 0$, and $\Delta(x^2+y^2) = 2+2 = 4$.

Multivariable chain rule and change of variables

The picture: $t$ moves by $\Delta t$, each $u^j$ moves by about $g_j'(t_0)\,\Delta t$, and each such move changes $f$ by about $\frac{\partial f}{\partial u^j}\,g_j'(t_0)\,\Delta t$. The contributions add up.

The right-hand side of this formula is not on the board photo; the lesson notes reconstruct it as the standard formula, and lecture math-02 proved it in this same form.

Example. $y = (u^1)^2u^2$, $u^1 = \cos t$, $u^2 = \sin t$, the point $t = 0$. Values: $u_0 = (1,0)^t$. Partial derivatives at $u_0$: $\frac{\partial f}{\partial u^1} = 2u^1u^2 = 0$, $\frac{\partial f}{\partial u^2} = (u^1)^2 = 1$. Derivatives of the layers: $g_1'(0) = -\sin0 = 0$, $g_2'(0) = \cos0 = 1$. Result: $\frac{dy}{dt} = 0\cdot0 + 1\cdot1 = 1$. Check by substitution: $y = \cos^2t\sin t$, $y' = -2\cos t\sin^2t + \cos^3t$, which is $1$ at $t = 0$ ✓.

Change of variables. If $x = x(u,v)$, $y = y(u,v)$, and $g(u,v) = f\big(x(u,v),y(u,v)\big)$, then the partial derivative in $u$ is the same chain rule with $v$ frozen:

For polar coordinates $x = r\cos\theta$, $y = r\sin\theta$ these are the formulas from the board of lecture math-02 (there, as the lecturer does, one letter $f$ denotes both the function of $x,y$ and the same function of $r,\theta$):

Second derivatives after a change of variables — the central point of the lesson. The formula for $\frac{\partial g}{\partial u}$ contains $\frac{\partial f}{\partial x}$ and $\frac{\partial f}{\partial y}$ evaluated at the point $\big(x(u,v),y(u,v)\big)$. These are again composite functions of $u$ and $v$. So when you differentiate in $u$ a second time, two rules work at once:

1. The chain rule again for each partial derivative of $f$: $\frac{\partial}{\partial u}\Big(\frac{\partial f}{\partial x}\Big) = \frac{\partial^2 f}{\partial x^2}\frac{\partial x}{\partial u} + \frac{\partial^2 f}{\partial y\,\partial x}\frac{\partial y}{\partial u}$.
2. The product rule, if the coefficients $\frac{\partial x}{\partial u}$, $\frac{\partial y}{\partial u}$ themselves depend on $u$: then one more term appears, $\frac{\partial f}{\partial x}\frac{\partial^2 x}{\partial u^2}$ (and the same one for $y$).

The model problem below shows how this looks on a concrete function.

What should stay with you

1. $(m\times n)(n\times k) = m\times k$: write the sizes under every factor.
2. $(AB)^t = B^tA^t$; a number equals its own transpose, so $x^tAu = u^tA^tx$.
3. $\tr(AB) = \tr(BA)$ also for non-square $A$, $B$; a number equals its own trace; $\tr(uv^t) = v^tu$.
4. Chain rule: the derivative of the outer layer at the value of the inner one, times the derivative of the inner one.
5. Partial derivative: freeze the other variables; the gradient is a column; $\nabla(x^tAx) = (A+A^t)x$.
6. $\frac{\partial^2 f}{\partial x^j\,\partial x^i}$ means first in $x^i$, then in $x^j$; with continuous second derivatives the order does not matter.
7. After a change of variables a partial derivative of $f$ is again a composite function: the second differentiation is the chain rule again plus the product rule.