GRADIENT VECTOR

For a multivariate function \(f(x)\)  where \(x=\begin{bmatrix} x_1\\ \vdots\\x_n \end{bmatrix}\), the vector with enteries as partial derivatives with respect to \(x_1,x_2,\dots,\) and \(x_n\) is called the gradient vector and it is denoted as:

\[\partial_x f= \begin{bmatrix} \partial f/\partial x_1\\ \vdots\\ \partial f/\partial x_n \end{bmatrix}.\]

For example, let's assume  \(x=\begin{bmatrix} x_1\\ x_2\\x_3 \end{bmatrix}\) and \(f(x)=x_1^2+3x_2+\cos(x_3)\). Then the gradient vector of \(f(x)\) is:\[\partial_x f= \begin{bmatrix} 2x_1\\3 \\-\sin(x_3)  \end{bmatrix}.\]

GRADIENT OF LINEAR FUNCTIONS

If \(b\) and \(x\) are the column vectors with the same size, the gradient of \(f(x)=b^Tx=x^Tb\) with respect to \(x\)  is derived as:\[\partial_x f=b. \]

If \(b\) is a \(m\times1\) vector, \(x\) is a \(n\times1\), and \(A\) is a matrix of size \(m\times n\), the gradient of \(f(x)=b^TAx=x^TA^Tb\) with respect to \(x\)  is derived as:\[\partial_x f=A^Tb. \]  

GRADIENT OF QUADRATIC FUNCTIONS

If \(x\) is the column vector of size \(n\times 1\) , and \(B\) is an square matrix of size \(n\times n\),  the gradient of \(f(x)=x^TBx=x^TB^Tx\)  with respect to \(x\)  is derived as:\[\partial_x f=(B+B^T)x. \]

Note that if \(B\) is a symmetric matrix, then \(B^T=B\), and so the gradient of \(f(x)=x^TBx=x^TB^Tx\)  with respect to \(x\)  is derived as: \[\partial_x f=2Bx. \]