Constrained Optimization
Constrained Optimization
minx∈Rnf(x)minx∈Rnf(x)
subject to: ci(x)=0,i∈E(equality constraints) ci(x)≥0,i∈I(inequality constraints)
Feasible Set $\Omega = { x_i | c_i(x) = 0, i \in E \text{ and } c_i(x) \geq 0, i \in I}$.
Case 1
$\min_{x \in R^n} f(x)$ subject to $c_1 (x) = 0$
x is local minimum if x + s $\notin \Omega$ or f(x+s) $\geq$ f(x).
Now, assume the opposite. Suppose s is a small step, $c_1(x+s) = c_1(x)$ and $f(x+s) < f(x)$
Using Taylor Series Expansion:
c1(x+s)≈c1(x)+∇c1(x)Ts=0
So
∇c1(x)Ts=0
Also, since $f(x+s) < f(x)$, then
0>f(x+s)−f(x)≈∇f(x)Ts
Therefore, $\exists s$ when
∇c1(x)Ts=0∇f(x)Ts<0
However, this is not possible when $\nabla f(x) = \lambda \nabla c(x)$. If $\nabla f(x)$ and $\nabla c(x)$ lie parallely, s will be perpendicular to $\nabla f(x)$ as well, making dot product 0 instead of <0.
$\therefore$ the optimality condition are $\nabla f(x) = \lambda \nabla c(x)$.
Case 2
Subcase 2.a
$\min_{x \in R^n} f(x)$ subject to $c_1 (x) \geq 0$
Suppose $\exists$ a small step s, s.t. $c_1(x+s) \geq 0$ and $f(x+s) < f(x)$.
Then
c1(x+s)≈c1(x)+∇c1(x)Ts≥0 (a)
f(x+s)=f(x)+∇f(x)Ts<f(x) (b)
Any s satisfy the equation (a), and $\exists$ s that satisfy (b) unless $\nabla f(x) = 0$.
$\therefore$ optimality conditions: $\nabla f(x) = 0, c_1(x) > 0$.
Subcase 2.b
$c_1(x) = 0$
If x is not local optimal, then $\exists$ s that $c_1(x+s) \geq 0$ and $f(x+s) < f(x)$.
Again by Taylor series expandsion
c1(x+s)≈c1(x)+∇c1(x)Ts≥0 (a)
f(x+s)=f(x)+∇f(x)Ts<f(x) (b)
Because $c_1(x) = 0$, then we have
∇c1(x)Ts≥0 (a)
∇f(x)Ts<0 (b)
$\exists$ no s that satisfy both a and b iff $\nabla f(x) = \lambda \nabla c_1(x)$ for $\lambda \geq 0$
The Lagrangian function:
L(x,λ)=f(x)−∑i∈ϵ∪Iλici(x)
where $\lambda$ is called Lagrangian variable.
Karush-Kahn-Tucker (KKT) Conditions
At a local solution $x^* $, under some conditions, $\exists$ a Lagrange multiplier $\lambda^* $ s.t.
∇L(x∗,λ∗)=0ci(x∗)=0,∀i∈Eci(x∗)≥0,∀i∈Iλ∗i≥0,∀i∈Iλici(x∗)=0,∀i∈E∩I
The last is called complementary conditions (either $\lambda_i = 0$ or $c_i(x^*) = 0$ when $c_i(x^*) > 0$) based on rule 3 and 2.