Constrained Optimization

minxRnf(x)minxRnf(x)

subject to: ci(x)=0,iE(equality constraints) ci(x)0,iI(inequality constraints)

Feasible Set $\Omega = { x_i | c_i(x) = 0, i \in E \text{ and } c_i(x) \geq 0, i \in I}$.

Case 1

constrained_1

$\min_{x \in R^n} f(x)$ subject to $c_1 (x) = 0$

x is local minimum if x + s $\notin \Omega$ or f(x+s) $\geq$ f(x).

Now, assume the opposite. Suppose s is a small step, $c_1(x+s) = c_1(x)$ and $f(x+s) < f(x)$

Using Taylor Series Expansion:

c1(x+s)c1(x)+c1(x)Ts=0

So

c1(x)Ts=0

Also, since $f(x+s) < f(x)$, then

0>f(x+s)f(x)f(x)Ts

Therefore, $\exists s$ when

c1(x)Ts=0f(x)Ts<0

However, this is not possible when $\nabla f(x) = \lambda \nabla c(x)$. If $\nabla f(x)$ and $\nabla c(x)$ lie parallely, s will be perpendicular to $\nabla f(x)$ as well, making dot product 0 instead of <0.

$\therefore$ the optimality condition are $\nabla f(x) = \lambda \nabla c(x)$.

Case 2

Subcase 2.a

constrained_2a

$\min_{x \in R^n} f(x)$ subject to $c_1 (x) \geq 0$

Suppose $\exists$ a small step s, s.t. $c_1(x+s) \geq 0$ and $f(x+s) < f(x)$.

Then

c1(x+s)c1(x)+c1(x)Ts0 (a)

f(x+s)=f(x)+f(x)Ts<f(x) (b)

Any s satisfy the equation (a), and $\exists$ s that satisfy (b) unless $\nabla f(x) = 0$.

$\therefore$ optimality conditions: $\nabla f(x) = 0, c_1(x) > 0$.

Subcase 2.b

constrained_2b

$c_1(x) = 0$

If x is not local optimal, then $\exists$ s that $c_1(x+s) \geq 0$ and $f(x+s) < f(x)$.

Again by Taylor series expandsion

c1(x+s)c1(x)+c1(x)Ts0 (a)

f(x+s)=f(x)+f(x)Ts<f(x) (b)

Because $c_1(x) = 0$, then we have

c1(x)Ts0 (a)

f(x)Ts<0 (b)

$\exists$ no s that satisfy both a and b iff $\nabla f(x) = \lambda \nabla c_1(x)$ for $\lambda \geq 0$

The Lagrangian function:

L(x,λ)=f(x)iϵIλici(x)

where $\lambda$ is called Lagrangian variable.

Karush-Kahn-Tucker (KKT) Conditions

At a local solution $x^* $, under some conditions, $\exists$ a Lagrange multiplier $\lambda^* $ s.t.

L(x,λ)=0ci(x)=0,iEci(x)0,iIλi0,iIλici(x)=0,iEI

The last is called complementary conditions (either $\lambda_i = 0$ or $c_i(x^*) = 0$ when $c_i(x^*) > 0$) based on rule 3 and 2.

Example

constrained_example