# Optimisation

## Multidimensional Taylor Series

The scalar case of the taylor series is an expansion of the function about the point :

This can be generalised to a matrix case. Let be a scalar function of a column vector . The taylor series expansion of about the point is:

This result is a scalar. Consider the first three terms:

is a row vector with it's gradient evaluated at the point :

is the matrix of second derivatives, called the Hessian matrix, evaluated at point

• The Hessian matrix is generally symmetric
• Matrix of mixed partial derivatives

Taylor series can be used to approximate multidimensional functions:

• Let be a scalar function of an vector ,
• The first term is a linear form
• Second term a quadratic form
• Higher order terms are ignored

## Multidimensional Optimisation

Optimisation tasks involve finding such that is at an extremum (max/min).

Consider a continuous function , expanded about the point , with a vector as the displacement from :

The point is an extremum if the gradient vector when . The homogenous nonlinear equation therefore defines an extremum .

If , then:

Therefore:

This is the important result that defines the extremum of a function

To determine the nature of the extremum, the sign of the must be determined. By the spectral resolution, this is determined by the eigenvalues of . It is said that the sign definiteness of is determined by

Let be a quadratic form m, where is a symmetric matrix. The eigenvalues of are . The definiteness is determined by all of the eigenvalues:

Definiteness of Nature of Point
Positive DefiniteMinimum
Positive SemidefiniteProbable Valley
Negative SemidefiniteProbable Ridge
Negative DefiniteMaximum

### Extrema of a Multivariate Quadratic

For a quadratic , the extremum is at the point where :

A maximum/minimum exists at the point if the matrix is positive/negative definite.

## Example 1

Find the extremum (and it's nature) of the quadratic function:

Put into matrix form:

The extremum of this quadratic form exists at the point where:

To determine the nature, find the eigenvalues of :

The eigenvalues lie either side of zero, which makes indefinite, and the extremum is therefore a saddle point.

## Example 2

Find the stationary points (and their nature) for the function:

The stationary points lie where :

The solutions are therefore:

Two solutions:

To determine the nature of the extremum, we need the hessian matrix:

The eigenvalues at each point will give the nature. For :

The hessian matrix is positive semidefinite, so the point is probably a valley, but further analysis is required to determine the nature of the point.

For :

The hessian matrix is indefinite, so the point is a saddle point.