The dual simplex method with bounds
Linear programming basis. Let a linear programming problem be given by
min cTx
s.t. Ax = b
` ≤ x ≤ u
x ∈ Rn,
(P)
where we assume A ∈ Rm×n to be full row rank (we will see in the section “Starting basis” how to make sure that
this assumption holds). We first introduce the concept of a basis:
• There are n variables xj for j = {0, . . . , n− 1}.
• A basis of (P) is a partition of {0, . . . , n− 1} into three disjoint index subsets B, L and U , such that if B is
the matrix formed by taking the columns of A indexed by B, then B is square and invertible.
Thus, we always have |B| = m, and there are at most ( nm ) different bases, possibly less than that since some of the
combinations may yield a singular B matrix. Given a specific basis, we establish some notation:
• For all j ∈ B the variable xj is called a basic variable, and the corresponding jth column of A is called a basic
column.
• For all j ∈ L∪U the variable xj is called a nonbasic variable, and the corresponding jth column of A is called
a nonbasic column.
• By convention, the vector formed by taking together all the basic variables is denoted xB. Similarly, cB, `B
and uB are formed by taking together the same indices of c, ` and u, respectively. The same notation is also
used for the indices in L and U , giving cL, cU , `L, `U , uL, and uU . We already defined B as taking together
the basic columns of A. The remaining (nonbasic) columns form the submatrices L and U . Thus, there is a
permutation of the columns of A that is given by [B | L | U ]. For conciseness, we will write A = [B | L | U ],
although it is an abuse of notation.
• B, L and U are sets, so the order of the indices does not matter. However, it must be consistent in the vectors
defined above. For example, (i) cB1 must be the objective function coefficient associated with the variable
xB1, (ii) `B1 and uB1 must be the bounds on that same variable, and (iii) the first column of B is the column
of A that corresponds to that same variable.
The concept of basis is useful because of the following construction:
• We construct a solution x̄ of (P) as follows. Let us fix the components of x̄ in L or U at their lower or upper
bound, respectively: x̄L = `L and x̄U = uU . Given that these components are fixed, we can now compute the
unique value of x̄B such that the equality constraints Ax = b of (P) are satisfied. Indeed, using the abuse of
1
notation described earlier, we have
[B | L | U ] x̄ = b
Bx̄B + Lx̄L + Ux̄U = b
Bx̄B + L`L + UuU = b
Bx̄B = b− L`L − UuU
x̄B = B
−1 (b− L`L − UuU ) .
• The solution x̄ constructed above is uniquely defined by the partition B,L,U (i.e., by the basis). We now see
why B was required to be an invertible matrix.
• Any solution x that can be constructed as above for some partition B,L,U is called a basic solution.
• If a basic solution x̄ satisfies ` ≤ x̄ ≤ u, then it is called a basic feasible solution. Indeed, it satisfies all the
constraints of (P). Note that the bound constraints are automatically satisfied for x̄L and x̄U , so it is enough
to verify that `B ≤ x̄B ≤ uB.
• The feasible region of (P) is a polyhedron, and it has been shown that x̄ is a basic feasible solution if and
only if it is a vertex of that feasible region. In other words, basic feasible solutions and vertices are defined
differently, but they are the same thing in the context of linear programming.
• Clearly, vertices are only a subset of all the feasible solutions to (P). However, in the context of optimization,
it sufficient to look at vertices because of the following: If (P) has an optimal solution, then at least one
optimal solution of (P) is a vertex.
Tableau. A tableau is an equivalent reformulation of (P) that is determined by a given basis. It lets us easily
assess the impact of changing the current basis (making a pivot) on (a) the objective function value, and (b) primal
or dual feasibility.
• A tableau is given by
min c̄Tx
s.t. Āx = b̄
` ≤ x ≤ u
x ∈ Rn.
• c̄T := cT −cTBB
−1A are called the reduced costs corresponding to the basis B,L,U . They have the property
that c̄B = 0, so c̄ expresses the direction of the objective function only in terms of the nonbasic variables.
• b̄ := B−1b.
• Ā := B−1A. If we use the partition Ā = [B̄ | L̄ | Ū ], we have that B̄ = B−1B = I. As a consequence, the
2
tableau can be written
min c̄TLxL + c
T
UxU
s.t. xB + L̄xL + ŪxU = b̄
` ≤ x ≤ u
x ∈ Rn.
• We already saw above that the (primal) basic solution corresponding to a basis (and hence to a tableau) is
given by x̄B = B
−1
(
b− L¯̀L − UūU
)
. It is feasible if `B ≤ x̄B ≤ uB. In that case, we say that the basis is
primal feasible.
Duality.
• The dual of (P) is
min −bTπ − `Tλ − uTµ
s.t. ATπ + Iλ + Iµ = c
π free λ ≥ 0, µ ≤ 0.
(D)
• A basis of (D) is a partition of {1, . . . , 3n}. However, (D) has a special structure with two identity matrices
in the constraints and no bounds on π. This yields a characterization of the bases of (D) that follows.
• A basis of (D) needs n basic variables, as many as there are equality constraints in (D). The π variables do
not have bounds, so they are always basic. We now need to select n−m basic variables among the λ and µ.
• For any given j, the variables λj and µj cannot be both basic, because if they were, the basis matrix would
contain twice the same identity column, and thus would not be invertible (see Figure 1). So we have the
following: Consider the n possible indices j ∈ {0, . . . , n− 1}. For n−m of them, either λj or µj is basic (but
not both). For m of them, neither λj nor µj is basic.
[AT | I | I] =
· · · · ·
· · · · ·
· · · · ·
· · · · ·
· · · · ·
· · · · · ↓ · · · · ↓ · ·
· · · · · j · · · · j · ·
Figure 1: The constraint matrix of (D)
· · · · ·
• • • · ·
· · · · ·
• • • · ·
• • • · ·
· · · · ↓ · ↓ ↓ · ↓ · ↓ ↓
· · · · nonbasic · nonbasic
→ basis matrix
· · · ·
• • •
· · · ·
• • •
• • •
· · · · ·
· · · · ·
Figure 2: A basis of (D)
3
• Let use consider one of the latter values of j, i.e., one of the m values of j such that neither λj nor µj is
basic. Then, the only nonzeros in the jth row of the basis matrix come from the jth row of AT (Figure 2).
Now, considering all the m such value of j, they give a square submatrix of AT . That submatrix is also a
submatrix of the basis matrix, with only zeros to its right. Since the basis matrix must be invertible, that
square m×m submatrix of AT must be invertible too.
• We now see that a basis of (D) is uniquely defined by a partition of {0, . . . , n− 1} into three disjoint sets B,
L, U such that |B| = m and if BT is the matrix formed by taking the rows of AT indexed by B, then BT
is invertible. For every j ∈ L, λj is basic, and for every j ∈ U , µj is basic. Observe that the conditions for
B,L,U to form a basis of (D) are exactly the same as the conditions for B,L,U to form a basis of (P).
• In order to construct a basic solution of (D), we partition A into [B | L | U ]. Knowing that λj = 0 for all
j /∈ L and µj = 0 for all j /∈ U , we rewrite the constraints as
BTπ = cB
LTπ + λL = cL
UTπ + µU = cU
UTπ free λ ≥ 0, µ ≤ 0.
The π variables are all basic and their values can be computed directly as π̄T = cTBB
−1. Then, the basic λ
variables have values λ̄TL = c
T
L−π
TL = cTL− c
T
BB
−1L and the basic µ variables have values µ̄TU = c
T
U −π
TU =
cTL − c
T
BB
−1U . For the basic solution (π̄T , λ̄T , µ̄T ) to be feasible in (D), we need λ̄ ≥ 0 and µ̄ ≤ 0. The
basis is then called dual feasible. Let c̄ be the reduced costs in the corresponding primal tableau, i.e.,
c̄T := cT − cTBB
−1A. It is easy to verify that (π̄T , λ̄T , µ̄T ) is feasible if and only if c̄j ≥ 0 for all j ∈ L and
c̄j ≤ 0 for all j ∈ U . Observe that these are the optimality condition of the simplex method on the primal (P).
• To derive the reduced costs of (D) for a given basis B,L,U , we need to express the objective function in
terms of λB, λU , µB, µL only (the nonbasic variables). Let us write a partitioned version of (D) again, this
time without discarding the nonbasic variables:
min −bTπ − `TBλB − `
T
LλL − `
T
UλU − u
T
BµB − u
T
LµL − u
T
UµU
s.t. BTπ + λB + µB = cB
LTπ + λL + µL = cL
UTπ + λU + µU = cU
UTπ free λ ≥ 0, µ ≤ 0.
This gives us
π = B−T (cB − λB − µB) = (B−T cB)−B−T (λB + µB)
λL = cL − LTπ − µL = (cL − LTB−T cB) + LTB−T (λB + µB)− µL
µU = cU − UTπ − λU = (cU − UTB−T cB) + UTB−T (λB + µB)− λU
where the first term of each right-hand side is constant and can be ignored in an objective function. After
rewriting the objective function and simplifying the result, we get
min (x̄B − `B)λB + (uU − `U )λU − (uB − x̄B)µB − (uL − `L)µL
4
where x̄B = B
−1(b − L`L − UuU ) corresponds to the primal basic solution associated with B,L,U . The
optimality conditions of the simplex method in (D) are that the reduced costs for λ must be nonnegative and
the reduced costs for µ must be nonpositive. Observe that we can always assume (uU − `U ) ≥ 0 otherwise the
problem is trivially infeasible. The conditions become x̄B ≥ `B and x̄B ≤ uB. Observe that they correspond
exactly to the feasibility of the primal basic solution x̄ associated with B,L,U .
Pivoting. We can now apply the simplex method to (D). We need to start (and maintain) a basis B,L,U that
is dual feasible, so we need B invertible, c̄j ≥ 0 for all j ∈ L and c̄j ≤ 0 for all j ∈ U .
At the beginning of each iteration, we select a dual nonbasic variable λB or µB with a negative reduced cost to
become a basic variable. If no such variable can be found, then we have reached optimality, and we can stop.
Otherwise, let j ∈ B be the index of such a dual variable. Then at the next iteration, we will have either j ∈ L′
(if it was a component of λB with a negative reduced cost) or j ∈ U ′ (if it was a component of µB with a negative
reduced cost), i.e., that variable will be basic. That dual variable is said to enter the dual basis.
In a primal view, this corresponds to finding a component j ∈ B of the primal basic solution x̄ that is infeasible.
At the next iteration, we will have j ∈ L′ or j ∈ U ′. When adopting a primal view, the very same operation is
described as the primal variable xj leaving the primal basis.
The next step is to choose a primal entering variable. We will choose this variable carefully in order to to maintain
an invertible B matrix and reduced costs of the appropriate sign.
Assume that the primal leaving variable xj is currently basic in row i (it corresponds to the basic variable xBi).
Let us consider the objective and ith row of the current tableau:
e ∈ L f ∈ L g ∈ U h ∈ U
min c̄exe + c̄fxf + c̄gxg + c̄hxh
s.t.
…
xj + āiexe + āifxf + āigxg + āihxh = b̄i
…
` ≤ x ≤ u
x ∈ Rn
where āie, āig > 0 and āif , āih < 0. The four indices e, f, g, h represent the four possible configurations: variable at upper or lower bound, and āik positive or negative. We only use the notation e, f, g, h for simplicity: there can be zero or more than one variable in each configuration. All variables in one given configuration are treated similarly. Any āik = 0 can be ignored. They do not interfere with the computations below, and it can be shown that the B′ matrix of the next iteration will be invertible if and only if we do not consider the corresponding columns as candidate entering columns. Since e, f ∈ L and g, h ∈ U , we currently have c̄e, c̄f ≥ 0 and c̄g, c̄h ≤ 0. • If xj leaves to its lower bound, we will need c̄′j ≥ 0 at the next iteration, while maintaining zero reduced 5 costs for all other indices in B. Any such new objective function can be achieved by adding a nonnegative multiple t of the ith row of the tableau to the current objective function. The multiplier t will be called the dual step length. - We know that c̄e will become c̄ ′ e = c̄e + t āie, which is guaranteed to always meet c̄ ′ e ≥ 0 because āie > 0.
– Instead, since āif < 0, we will have c̄ ′ f = c̄f + t āif ≥ 0 if and only if t ≤ c̄f/(−āif ). - For c̄′g = c̄g + t āig ≤ 0, we need t ≤ −c̄g/āig. - Finally, c̄′h = c̄h + t āih ≤ 0 is guaranteed to always be met. • If xj leaves to its upper bound, we will need c̄′j ≤ 0 at the next iteration, while maintaining zero reduced costs for all other indices in B. Any such new objective function can be achieved by subtracting a nonnegative multiple t of the ith row of the tableau to the current objective function. - The condition c̄′e = c̄e − t āie ≥ 0 requires t ≤ c̄e/āif . - The condition c̄′f = c̄f − t āif ≥ 0 is always satisfied. - The condition c̄′g = c̄g − t āig ≤ 0 is always satisfied. - The condition c̄′h = c̄h − t āih ≤ 0 requires t ≤ (−c̄h)/(−āih). If the signs of the c̄k and āik coefficients are such that no conditions are imposed on t, it can be shown that (D) is unbounded, which corresponds to (P) being infeasible (note that, because of the finite bounds ` and u, (P) is never unbounded). Each of the above conditions defines an upper bound tk on t, i.e., t ≤ tk for all k ∈ L ∪ U . The most restrictive condition can be selected by computing t = mink tk. If k is a value of k that yields the minimum, we will have c̄′k = 0 and k can be our entering variable, i.e., we can set B ′ = B \ {j} ∪ {k}. Finding k is called the ratio test. Figure 3 summarizes how to compute tk depending on the signs of āik and c̄k. j ∈ B k ∈ L ∪ U āik tk k ∈ L > 0
j ∈ L′ < 0 c̄k/(−āik)
(x̄j < `j) k ∈ U > 0 (−c̄k)/āik
< 0
k ∈ L > 0 c̄k/āik
j ∈ U ′ < 0
(x̄j > uj) k ∈ U > 0
< 0 (−c̄k)/(−āik)
Figure 3: Computing the upper bounds tk on the dual step length t in the ratio test.
Starting basis. Before we can apply the dual simplex method, we need to have a dual feasible basis. First, this
means that we need a set of column indices B such that B is invertible. A simple way to obtain that is to add m
6
artificial variables z fixed to zero, as demonstrated in (P+):
min cTx+ 0T z
s.t. Ax+ Iz = b
` ≤ x ≤ u
0 ≤ z ≤ 0
x ∈ Rn, z ∈ Rm
(P+)
We can do that as a very first step before starting the dual simplex method. Then, it is easier to let n := n+m,
cT := [cT 0T ], `T := [`T 0T ], uT := [uT 0T ] and A := [A I], so that you can forget about the z variables and have
a problem of the form (P), but with the guarantee that the last m columns of A form an identity matrix (which is
invertible: I−1 = I). Note that having an m×m identity in A also ensures that A is full row rank.
Once we have B, it is straightforward to construct L and U such that B,L,U is dual feasible. Having B is enough
to compute the reduced costs c̄T = cT − cTBB
−1A. For all j /∈ B, we can assign j to L if c̄j ≥ 0 or to U if c̄j ≤ 0.
This way, c̄j will always have the appropriate sign to ensure dual feasibility.
Summary. We can now give, in Figure 4, a precise description of the operations in the dual simplex method
with bounds. We can also make a few observation that will prove useful in implementing the dual simplex method.
At Step 1, in most cases, there will be multiple candidate values of i such that x̄Bi violates its bounds. Choosing
one to become the leaving variable is called a pricing rule. In theory, any candidate would work, but in practice it
is a good idea to choose a candidate with a large bound violation, for example one with the largest violation.
There are a few useful invariants in the dual simplex method that we can use to verify that our implementation is
working as intended. First, we have the matrix B formed with the columns of A with indices in B. This matrix
must always stay invertible. If B becomes singular, then the ratio test is not working properly. Specifically, we
are choosing an entering variable k such that the tableau element āik is zero. Second, there is dual feasibility. We
must always have c̄j ≥ 0 for all j ∈ L and c̄j ≤ 0 for all j ∈ U . If we lose dual feasibility, it also means that the
ratio test is not working. In this case, we chose a wrong value for tk, not actually mink{tk}, something larger.
Finally, recall that at any given iteration of the simplex method, we can compute the corresponding basic solution
by letting x̄B = B
−1(b − L`L − UuU ), x̄L = `L and x̄U = uU . In the dual simplex method, x̄ will not be feasible
(until the last iteration, at which point we stop). However, we can still compute the corresponding dual obective
function value: z̄ = cT x̄. As the dual simplex method makes progress, this objective should be nondecreasing:
from one iteration to the next, it either stays the same (when t = 0), or increases. If z̄ decreases, it means that we
made a mistake in the choice of the leaving variable.
7
Initialization
Add m variables to the problem, fixed to zero by their bounds.
From now on, only consider the enlarged problem:
n := n+m, cT := [cT 0T ], `T := [`T 0T ], uT := [uT 0T ] and A := [A I],
where 0T is a row vector of size m with all components set to zero.
Build the starting basis:
Set B := {n, . . . , n+m− 1}.
Form the corresponding basis matrix B.
Compute c̄T = cT − cTBB
−1A.
For all j ∈ {0, . . . , n− 1},
if c̄j > 0, set j ∈ L,
if c̄j < 0, set j ∈ U ,
if c̄j = 0, we can arbitrarily select either j ∈ L or j ∈ U .
Step 1 (leaving variable)
Form the basis matrix B (from the columns of A indexed by B).
Compute c̄T = cT − cTBB
−1A.
Compute x̄B = B
−1(b− L`L − UuU ).
Find a component i of xB such that either x̄Bi < `Bi or x̄Bi > uBi.
If no such i exists, we reached optimality. Stop.
Let j be such that xj corresponds to xBi.
Step 2 (entering variable)
Compute the ith row of B−1A.
Perform the ratio test: compute k = argmink∈L∪U{tk}, where tk is defined as in Figure 3.
If there is no bound tk, the problem is infeasible. Stop.
Step 3 (pivoting)
Leaving variable:
B := B \ {j}
If x̄Bi < `Bi, then L := L ∪ {j}.
If x̄Bi > uBi, then U := U ∪ {j}.
Entering variable:
If k ∈ L, then L := L \ {k}.
If k ∈ U , then U := U \ {k}.
B := B ∪ {k}
Go to Step 1.
Figure 4: Summary of the dual simplex method with bounds.
8