Review of Multivariate Calculus

 

Review of Multivariate Calculus (for Optimization & Machine Learning)

1. Functions of Several Variables

A multivariate function maps vectors to scalars:

$$f:\mathbb{R}^n \rightarrow \mathbb{R}, \quad f(x_1,x_2,\dots,x_n)$$

Examples

• $f(x,y)=x^2+y^2$
  

• $f(x,y,z)=x^2+y^2+z^2$
  

• Loss functions in ML: $f(\mathbf{w})$
  

---

2. Limits and Continuity (Brief Review)

• A function is continuous at $\mathbf{x}_0$ if:

$$\lim_{\mathbf{x}\to\mathbf{x}_0} f(\mathbf{x}) = f(\mathbf{x}_0)$$

📌 In multiple dimensions, limits must be independent of path.

---

3. Partial Derivatives

Definition

The partial derivative measures change with respect to one variable while holding others constant.

$$\frac{\partial f}{\partial x_i}$$

Example

$$f(x,y)=x^2y+y^3$$
$$\frac{\partial f}{\partial x}=2xy, \quad \frac{\partial f}{\partial y}=x^2+3y^2$$

---

4. Gradient Vector

Definition

$$\nabla f = \begin{bmatrix} \frac{\partial f}{\partial x_1}\\ \vdots\\ \frac{\partial f}{\partial x_n} \end{bmatrix}$$

Properties

• Direction of steepest ascent

• Orthogonal to level curves

• Used in gradient descent

Example

$$f(x,y)=x^2+y^2 \Rightarrow \nabla f = [2x,2y]^T$$

---

5. Directional Derivative

Definition

Rate of change of $f$ in direction $\mathbf{u}$ (unit vector):

$$D_{\mathbf{u}}f(\mathbf{x})=\nabla f(\mathbf{x})\cdot\mathbf{u}$$

📌 Maximum directional derivative occurs in the gradient direction.

---

6. Hessian Matrix (Second Derivatives)

Definition

$$\nabla^2 f = \begin{bmatrix} \frac{\partial^2 f}{\partial x_1^2} & \cdots & \frac{\partial^2 f}{\partial x_1\partial x_n}\\ \vdots & \ddots & \vdots\\ \frac{\partial^2 f}{\partial x_n\partial x_1} & \cdots & \frac{\partial^2 f}{\partial x_n^2} \end{bmatrix}$$

Importance

• Determines curvature

• Used in Newton’s method

• Classifies critical points

---

7. Taylor Expansion (Multivariate)

Second-Order Approximation

$$f(\mathbf{x}) \approx f(\mathbf{x}_0) + \nabla f(\mathbf{x}_0)^T(\mathbf{x}-\mathbf{x}_0) + \frac{1}{2}(\mathbf{x}-\mathbf{x}_0)^T \nabla^2 f(\mathbf{x}_0)(\mathbf{x}-\mathbf{x}_0)$$

📌 Central in optimization theory.

---

8. Critical Points

Definition

$$\nabla f(\mathbf{x}^\ast)=0$$

Types

• Local minimum

• Local maximum

• Saddle point

---

9. Classification Using Hessian

Hessian Type	Nature
Positive definite	Local minimum
Negative definite	Local maximum
Indefinite	Saddle point
Semi-definite	Inconclusive

---

10. Convexity in Multivariate Functions

Definition

$$f(\lambda x+(1-\lambda)y)\le \lambda f(x)+(1-\lambda)f(y)$$

Hessian Condition

• $f$ is convex ⇔ Hessian is positive semidefinite

📌 Convex ⇒ every local minimum is global.

---

11. Constrained Optimization (Preview)

Lagrange Multipliers

$$\min f(x,y)\quad \text{s.t. } g(x,y)=0$$

Condition:

$$\nabla f=\lambda \nabla g$$

---

12. Geometry of Multivariate Functions

Concept	Interpretation
Level sets	Contours of equal value
Gradient	Normal to level sets
Hessian	Curvature of surface
Saddle point	Mixed curvature

---

13. Connection to Machine Learning

Concept	ML Use
Gradient	Backpropagation
Hessian	Second-order methods
Convexity	Guarantees global optimum
Saddle points	Optimization challenges