程序代写代做代考 flex arm Excel assembly Java finance data structure database scheme chain algorithm AI compiler Fortran matlab information retrieval Erlang Formulas from Algebra

Now
200 Q=⎣0 3 0⎦
002
⎡100⎤
⎡1−1 0⎤
In the Jacobi iteration, Q is taken to be the diagonal of A: ⎡⎤
8.2 Iterative Solutions of Linear Systems 325
⎢2⎥⎢2⎥ Q−1 =⎣0 1 0⎦, Q−1A=⎣−1 1 −1 ⎦
333
001 0−1 1 22
The Jacobi iterative matrix and constant vector are
⎡010⎤ ⎡1⎤
⎢2⎥⎢2⎥ B = I − Q−1 A = ⎣ 1 0 1 ⎦, h = Q−1b = ⎣ 8 ⎦
010 −5 22
One can see that Q is close to A, Q−1 A is close to I, and I − Q−1 A is small. We write the Jacobi method as
x(k) = Bx(k−1) + h
(Gauss-Seidel iteration) Repeat the preceding example using the Gauss-Seidel iteration.
The idea of the Gauss-Seidel iteration is simply to accelerate the convergence by incorpo-
rating each vector as soon as it has been computed. Obviously, it would be more efficient
333
EXAMPLE 2
Solution
in the Jacobi method to use the updated value x(k) in the second equation instead of the old 1
value x(k−1). Similarly, x(k) could be used in the third equation in place of x(k−1). Using the 122
new iterates as soon as they become available, we have the Gauss-Seidel method:
x(k) = 1x(k−1) + 1 1222
x(k) = 1x(k) + 1x(k−1) + 8 231333
x(k) = 1x(k) − 5 3222
Starting with the initial vector zero, some of the iterates are
x(0) =[0,0,0]T
x(1) =[0.5000,2.8333,−1.0833]T x(2) =[1.9167,2.9444,−1.0278]T
.
x(9) =[2.0000,3.0000,−1.0000]T
In this example, the convergence of the Gauss-Seidel method is approximately twice as fast as that of the Jacobi method. ■
In the iterative algorithm that goes by the name Gauss-Seidel, Q is chosen as the lower triangular part of A, including the diagonal. Using the data from the previous example, we

326
Chapter 8
Additional Topics Concerning Systems of Linear Equations
now find that
⎡⎤
200 Q = ⎣ −1 3 0 ⎦
The usual row operations give us ⎡100⎤
⎡1−1 0⎤
0 −1 2
⎢2⎥⎢2⎥ Q−1 =⎣ 1 1 0⎦, Q−1A=⎣0 5 −1 ⎦
63 63 111 0−15 12 6 2 12 6
Again, we emphasize that in a practical problem we would not compute Q−1. The Gauss- Seidel iterative matrix and constant vector are
⎡010⎤ ⎡1⎤
⎢2⎥⎢2⎥ L = I − Q−1 A = ⎣0 1 1 ⎦, h = Q−1b = ⎣ 17 ⎦
636 011 −13 12 6 12
EXAMPLE 3
Solution
We write the Gauss-Seidel method as
x(k) = Lx(k−1) + h
(SOR iteration) Repeat the preceding example using the SOR iteration with ω = 1.1. Introducing a relaxation factor ω into the Gauss-Seidel method, we have the SOR method:
􏰖1 12221
x(k) = ω
x(k−1) +
+ (1 − ω)x(k−1)
1􏰗
􏰖1 1 8􏰗
x(k) = ω x(k) + x(k−1) + + (1 − ω)x(k−1) 2313332
􏰖1 32223
5􏰗
Starting with the initial vector of zeros and with ω = 1.1, some of the iterates are
x(0) =[0,0,0]T
x(1) =[0.5500,3.1350,−1.0257]T x(2) =[2.2193,3.0574,−0.9658]T
.
x(7) =[2.0000,3.0000,−1.0000]T
In this example, the convergence of the SOR method is faster than that of the Gauss-Seidel method. ■
In the iterative algorithm that goes by the name successive overrelaxation (SOR), Q is chosen as the lower triangular part of A including the diagonal, but each diagonal element ai j is replaced by ai j /ω, where ω is the so-called relaxation factor. (Initial work on the SOR method was done by Southwell [1946] and Young [1950].) From the previous example,
x(k) =ω
x(k) −
+(1−ω)x(k−1)

this means that
⎡20 00⎤
⎥ 0 ⎦
20 11
⎡11 00⎤
⎢20 ⎥ ⎢10 20
Now
⎡11−11 Q−1A=⎣ 11 539
0⎤
8.2 Iterative Solutions of Linear Systems 327
⎢ 11
Q = ⎣ −1 30
11 0 −1
Q−1=⎣121 11 0⎦, 600 30
1331 121 11 12000 600 20
300 600 121 671
6000 12000
⎥ −11 ⎦
30 539 600
The SOR iterative matrix and constant vector are
⎡−1110⎤ ⎡11⎤ ⎢1020⎥ ⎢20⎥
Lω = I − Q−1 A = ⎣ − 11 300
−121 6000
61 600
− 671 12000
11 ⎦, 30
61 600
h = Q−1b = ⎣
627 ⎦
200 −4103 4000
We write the SOR method as
Pseudocode
x(k) =Lωx(k−1) +h
We can write pseudocode for the Jacobi, Gauss-Seidel, and SOR methods assuming that the linear system (1) is stored in matrix-vector form:
procedure Jacobi( A, b, x)
realkmax ←100,δ←10−10,ε← 1 ×10−4
2 integer i, j, k, kmax, n; real diag, sum
real array ( A)1:n×1:n , (b)1:n , (x)1:n , ( y)1:n n ← size(A)
fork =1tokmax do
y←x
for i = 1 to n do
sum ← bi
diag ← aii
if |diag| < δ then output “diagonal element too small” return end if for j = 1 to n do if j ̸=i then sum ← sum − ai j y j end if end for xi ← sum/diag end for output k, x 328 Chapter 8 Additional Topics Concerning Systems of Linear Equations if∥x− y∥<εthen output k, x return end if end for output “maximum iterations reached” return end Jacobi Here, the vector y contains the old iterate values, and the vector x contains the updated ones. The values of kmax, δ, and ε are set either in a parameter statement or as global variables. The pseudocode for the procedure Gauss Seidel( A, b, x) would be the same as that for the Jacobi pseudocode above except that the innermost j -loop would be replaced by the following: for j = 1 to i − 1 do sum ← sum − ai j x j end for for j = i + 1 to n do sum ← sum − ai j x j end for The pseudocode for procedure SOR( A, b, x, ω) would be the same as that for the Gauss- Seidel pseudocode with the statement following the j-loop replaced by the following: xi ← sum/diag xi ←ωxi +(1−ω)yi In the solution of partial differential equations, iterative methods are frequently used to solve large sparse linear systems, which often have special structures. The partial derivatives are approximated by stencils composed of relatively few points, such as 5, 7, or 9. This leads to only a few nonzero entries per row in the linear system. In such systems, the coefficient matrix A is usually not stored since the matrix-vector product can be written directly in the code. See Chapter 15 for additional details on this and how it is related to solving elliptic partial differential equations. Convergence Theorems For the analysis of the method described by System (2), we write x(k) = Q−1􏰠(Q − A)x(k−1) + b􏰡 or x(k) = Gx(k−1) + h (7) EXAMPLE 4 Solution The conclusion of this theorem can also be written as ρ(I − Q−1 A) < 1 where ρ is the spectral radius function: For any n × n matrix G, having eigenvalues λi,ρ(G)=max1􏶞i􏶞n |λi|. Determine whether the Jacobi, Gauss-Seidel, and SOR methods (with ω = 1.1) of the previous examples converge for all initial iterates. For the Jacobi method, we can easily compute the eigenvalues of the relevant matrix B. The steps are 8.2 Iterative Solutions of Linear Systems 329 where the iteration matrix and vector are G = I − Q−1 A, h = Q−1b Notice that in the pseudocode, we do not compute Q−1. The matrix Q−1 is used to facilitate the analysis. Now let x be the solution of System (1). Since A is nonsingular, x exists and is unique. We have, from Equation (7), x(k) − x = (I − Q−1 A)x(k−1) − x + Q−1b = (I − Q−1 A)x(k−1) −(I − Q−1 A)x = (I − Q−1 A)(x(k−1) − x) One can interpret e(k) ≡ x(k) − x as the current error vector. Thus, we have e(k) = (I − Q−1 A)e(k−1) (8) We want e(k) to become smaller as k increases. Equation (8) shows that e(k) will be smaller than e(k−1) if I − Q−1 A is small, in some sense. In turn, that means that Q−1 A should be close to I. Thus, Q should be close to A. (Norms can be used to make small and close precise.) SPECTRAL RADIUS THEOREM In order that the sequence generated by Qx(k) = ( Q − A)x(k−1) + b to converge, no matter what starting point x(0) is selected, it is necessary and sufficient that all eigenvalues of I − Q−1 A lie in the open unit disc, |z| < 1, in the complex plane. ■ THEOREM1 ⎡−λ 1 0⎤ ⎢2⎥11 det(B−λI)=det⎣ 1 −λ 1 ⎦=−λ3 + λ+ λ=0 3366 0 1 −λ 2 􏶣 The eigenvalues are λ = 0, ± 1/3 ≈ ±0.5774. Thus, by the preceding theorem, the Jacobi iteration succeeds for any starting vector in this example. 330 Chapter 8 Additional Topics Concerning Systems of Linear Equations from ⎡−λ11 0⎤􏰔􏰕 For the Gauss-Seidel method, the eigenvalues of the iteration matrix L are determined ⎢20 ⎥121 det(L−λI)=det⎣ 0 1 −λ 1 ⎦=−λ −λ + λ=0 6 3 6 36 The eigenvalues are λ = 0, 0, 1 ≈ 0.333. Hence, the Gauss-Seidel iteration will also 3 succeed for any initial vector in this example. For the SOR method with ω = 1.1, the eigenvalues of the iteration matrix Lω are determined from ⎡ − 1 − λ 11 0 ⎤ ⎢10 20 ⎥ 011−λ 12 6 det(Lω−λI)=det⎣ −11 300 61 −λ 600 671 11 ⎦ 30 61 −λ 600 −121 6000 = − 10 − λ 600 − λ 11 11 􏰔 61 􏰕 􏰔 1 􏰕 671 11 􏰔 12000 1 􏰕􏰔61 􏰕2 121 1111 − 6000 30 20 +20300 600−λ − −10−λ 1200030 =−1 +31λ+31λ2−λ3=0 1000 3000 3000 The eigenvalues are λ ≈ 0.1200,0.0833,−0.1000. Hence, the SOR iteration will also succeed for any initial vector in this example. ■ A condition that is easier to verify than the inequality ρ(I − Q−1 A) < 1 is the dominance of the diagonal elements over the other elements in the same row. As defined in Section 7.3, we can use the property of diagonal dominance 􏰃n j=1 j ̸=i |ai i | >
to determine whether the Jacobi and Gauss-Seidel methods converge via the following
theorem.
Notice that this is a sufficient but not a necessary condition. Indeed, there are matrices that are not diagonally dominant for which these methods converge.
Another important property follows:
|ai j |
JACOBI AND GAUSS-SEIDEL CONVERGENCE THEOREM
If A is diagonally dominant, then the Jacobi and Gauss-Seidel methods converge for any starting vector x(0).
■ THEOREM2
SYMMETRIC POSITIVE DEFINITE
Matrix A is symmetric positive definite (SPD) if A = AT and xT Ax > 0 for all nonzero real vectors x.
■ DEFINITION1

■ THEOREM3
8.2 Iterative Solutions of Linear Systems 331 For a matrix A to be SPD, it is necessary and sufficient that A = AT and that all eigenvalues
of A are positive.
SOR CONVERGENCE THEOREM
Suppose that the matrix A has positive diagonal elements and that 0 < ω < 2. The SOR method converges for any starting vector x(0) if and only if A is symmetric and positive definite. Matrix Formulation For the formal theory of iterative methods, we split the matrix A into the sum of a nonzero diagonal matrix D, a strictly lower triangular matrix CL, and a strictly upper triangular matrix CU such that A = D − CL − CU Here, D = diag(A), CL = (−aij)i>j, and CU = (−aij)i0

We can derive the following:
8.2 Iterative Solutions of Linear Systems 333
for all nonzero vectors x ∈ Rn. In general, expressions such as ⟨u,v⟩ and ⟨u,v⟩A reduce to 1 × 1 matrices and are treated as scalar values. A quadratic form is a scalar quadratic function of a vector of the form
f(x)= 1⟨x,x⟩A −⟨b,x⟩+c 2
Here, A is a matrix, x and b are vectors, and c is a scalar constant. The gradient of a quadratic form
f′(x)=􏰠∂f(x)/∂x , ∂f(x)/∂x , ···, ∂f(x)/∂x 􏰡T 12n
f ′(x) = 1 AT x + 1 Ax − b 22
If A is symmetric, this reduces to
f′(x)= Ax−b
Setting the gradient to zero, we obtain the linear system to be solved, Ax = b. Therefore, the solution of Ax = b is a critical point of f (x). If A is symmetric and positive definite, then f (x) is minimized by the solution of Ax = b. So an alternative way of solving the linear system Ax = b is by finding an x that minimizes f (x).
We want to solve the linear system
Ax = b
where the n × n matrix A is symmetric and positive definite.
Suppose that { p(1), p(2), . . . , p(k), . . . , p(n)} is a set containing a sequence of n mutually
conjugate direction vectors. Then they form a basis for the space Rn . Hence, we can expand the true solution vector x∗ of Ax = b into a linear combination of these basis vectors:
x∗ =α1p(1)+α2p(2)+···+α(k)p(k)+···+αnp(n) where the coefficients are given by
αk =⟨p(k),b⟩/⟨p(k),p(k)⟩A
This can be viewed as a direct method for solving the linear system Ax = b: First find the sequence of n conjugate direction vectors p(k), and then compute the coefficients αk. However, in practice, this approach is impractical because it would take too much computer time and storage.
On the other hand, if we view the conjugate gradient method as an iterative method, then we could solve large sparse linear systems in a reasonable amount of time and storage. The key is carefully choosing a small set of the conjugate direction vectors p(k) so that we do not need them all to obtain a good approximation to the true solution vector.
Start with an initial guess x(0) to the true solution x∗. We can assume without loss of generality that x(0) is the zero vector. The true solution x∗ is also the unique minimizer of
f(x)= 1⟨x,x⟩A −⟨x,x⟩= 1xT Ax−xTx 22
for x ∈ Rn. This suggests taking the first basis vector p(1) to be the gradient of f at x = x(0), which equals −b. The other vectors in the basis are now conjugate to the gradient—hence

334 Chapter 8
Additional Topics Concerning Systems of Linear Equations
the name conjugate gradient method. The kth residual vector is r(k) = b − Ax(k)
The gradient descent method moves in the direction r(k). Take the direction closest to the gradient vector r(k) by insisting that the direction vectors p(k) be conjugate to each other. Putting all this together, we obtain the expression
p(k+1) = r(k) − 􏰠􏶳 p(k), r(k)􏶴 􏶭􏶳 p(k), p(k)􏶴 􏰡 pk AA
After some simplifications, the algorithm is obtained for solving the linear system Ax = b, where the coefficient matrix A is real, symmetric, and positive definite. The input vector x(0) is an initial approximation to the solution or the zero vector.
In theory, the conjugate gradient iterative method solves a system of n linear equations
in at most n steps, if the matrix A is symmetric and positive definite. Moreover, the nth
iterative vector x(n) is the unique minimizer of the quadratic function q(x) = 1 xT Ax−xT b. 2
When the conjugate gradient method was introduced by Hestenes and Stiefel [1952], the initial interest in it waned once it was discovered that this finite-termination property was not obtained in practice. But two decades later, there was renewed interest in this method when it was viewed as an iterative process by Reid [1971] and others. In practice, the solution of a system of linear equations can often be found with satisfactory precision in a number of steps considerably less than the order of the system.
Here is a pseudocode for the conjugate gradient algorithm:
k ← 0; x ← 0; r ← b − Ax; δ ← ⟨r, r⟩
while 􏰉√δ>ε√⟨b,b⟩􏰈and􏰉ki
􏰃n ⎪⎬
(−a /a )x(k−1) −(b/a ) +(1−ω)x(k−1)
x(k) =ω
The SOR method reduces to the Gauss-Seidel method when ω = 1.
(3) For a matrix formulation, we split the matrix A:
A = D − CL − CU
where D is a nonzero diagonal matrix, CL is a strictly lower triangular matrix, and CU is a strictlyuppertriangularmatrix.Here,D=diag(A),CL =(−aij)i>j,andCU =(−aij)ii
h = D−1b

For the Gauss-Seidel method, we have
Q = D − CL
L = (D − CL)−1CU
For the SOR method, we have ω
8.2 Iterative Solutions of Linear Systems 337
h = ( D − C L )−1 b Q= 1(D−ωCL)
Lω =(D−ωCL)−1[ωCU +(1−ω)D] h = ω ( D − ω C L )−1 b
(4) An iterative method converges for a specific matrix A if and only if ρ(I − Q−1 A) < 1 If A is diagonally dominant, then the Jacobi and Gauss-Seidel methods converge for any x(0). The SOR method converges, for 0 < ω < 2 and any x(0), if and only if A is symmetric and positive definite with positive diagonal elements. 1. GiveanalternativesolutiontoExample4. 2. WritethematrixformulafortheGauss-Seideloverrelaxationmethod. a3. (Multiplechoice)InsolvingasystemofequationsAx=b,itisoftenconvenienttouse an iterative method, which generates a sequence of x(k) vectors that should converge to a solution. The process is stopped when sufficient accuracy has been attained. A general procedure is to obtain x(k) by solving Qx(k) = (Q − A)x(k−1) + b. Here, Q is a certain matrix that is usually connected somehow to A. The process is repeated, starting with any available guess, x(0). What hypothesis guarantees that the method works, no matter what starting point is selected? a. ||Q||<1 b. ||QA||<1 c. ||I−QA||<1 d. ||I−Q−1A||<1 4. (Multiplechoice)Fromavectornorm,wecancreateasubordinatematrixnorm.Which e. Noneofthese. Hint: The spectral radius is less than or equal to the norm. relation is satisfied by every subordinate matrix norm? a. ||Ax||􏶟 ||A||||x|| b. ||I|| = 1 c. ||AB||􏶟 ||A||||B|| d. ||A+B||􏶟 ||A||+||B|| e. None of these. a 5. (Multiple choice) The condition for diagonal dominance of a matrix A is: a. |aii| < 􏰍nj=1 |aij| b. |aii|􏶟 􏰍nj=1 |aij| c. |aii| < 􏰍nj=1 |aij| j ̸=i j ̸=i d. |aii|>􏰍nj=1|aij|
e. None of these.
Problems 8.2

338 Chapter 8
Additional Topics Concerning Systems of Linear Equations
6.
(Multiplechoice)Anecessaryandsufficientconditionforthestandarditerationformula x(k) =Gx(k−1)+htoproduceasequencex(k)thatconvergestoasolutionoftheequation (I − G)x = h is that:
a. ThespectralradiusofGisgreaterthan1. b. The matrix G is diagonally dominant.
c. ThespectralradiusofGislessthan1.
d. G is nonsingular.
e. Noneofthese.
(Multiple choice) A sufficient condition for the Jacobi method to converge for the linear system Ax = b.
7.
8.
a9.
10.
11.
a. b. c. d. e.
A − I is diagonally dominant. A is diagonally dominant.
G is nonsingular. ThespectralradiusofGislessthan1. Noneofthese.
(Multiple choice) A sufficient condition for the Gauss-Seidel method to work on the linear system Ax = b.
a. b. c. d. e.
A is diagonally dominant.
A − I is diagonally dominant.
The spectral radius of A is less than 1. G is nonsingular.
Noneofthese.
(Multiple choice) Necessary and sufficient conditions for the SOR method, where 0<ω<2,toworkonthelinearsystem Ax=b. b. ρ ( A) < 1. d. x(0) = 0. 123 ⎢3054⎥ ⎢23456⎥ a.⎣0 5 4⎦ b.⎢⎣1 1 1 2⎥⎦ c.⎢0 1 0 1 0⎥ 213 1322 ⎣34343⎦ 55555 Determine the condition numbers κ(A) of these matrices: ⎡⎤⎡⎤ a. A is diagonally dominant. c. A is symmetric positive definite. e. None of these. 􏶢􏰍n 􏰍n = i =1 j =1 􏶨􏶨 2 􏶨􏶨 ai j The Frobenius norm is given by || A||F used because it is so easy to compute. Find the value of this norm for these matrices: which is frequently ⎡ ⎤ ⎡0 0 1 2⎤ ⎡⎢1 1 1 1 1⎤⎥ −2 1 0 a.⎣1 −2 1⎦ 0 1 −2 0 0 1 b.⎣0 1 0⎦ 1 1 1 ⎡3 0 0⎤ c. ⎣0 2 0⎦ 001 ⎡⎢−2 −1 2 −1⎤⎥ d. ⎢ 1 2 1 −2⎥ ⎣2−12 1⎦ 0201 8.2 Iterative Solutions of Linear Systems 339 1. Redo several or all of Examples 1–5 using the linear system involving one of the following coefficient matrix and right-hand side vector pairs: 􏰖􏰗􏰖􏰗 a.A= 5−1, b=7 ⎡−1 3 ⎤ ⎡4⎤ 5−10 7 b.A=⎣−1 3−1⎦,b=⎣4⎦ ⎡0−12⎤ ⎡5⎤ 2−10 1 c.A=⎣−1 6−2⎦,b=⎣3⎦ 4−38 9 ⎡⎤⎡⎤ 7 3 −1 3 d. A=⎣ 3 8 1⎦, b=⎣−4⎦ −114 2 2. Using the Jacobi, Gauss-Seidel, and SOR (ω = 1.1) iterative methods, write and exe- cute a computer program to solve the following linear system to four decimal places (rounded) of accuracy: ⎡ ⎤⎡⎤⎡⎤ 7 1 −1 2 x1 3 ⎢1 8 0 −2⎥⎢x2 ⎥ = ⎢−5⎥ ⎣−1 0 4 −1⎦⎣x3⎦ ⎣ 4⎦ 2 −2 −1 6 x4 −3 Compare the number of iterations needed in each case. Hint: The exact solution is x = (1,−1,1,−1)T . 3. Using the Jacobi, Gauss-Seidel, and the SOR (ω = 1.4) iterative methods, write and run code to solve the following linear system to four decimal places of accuracy: ⎡ ⎤⎡⎤⎡⎤ 7 3 −1 2 x1 −1 ⎢3 8 1 −4⎥⎢x2⎥=⎢ 0⎥ ⎣−1 1 4 −1⎦⎣x3⎦ ⎣−3⎦ 2 −4 −1 6 x4 1 Compare the number of iterations in each case. Hint: Here, the exact solution is x = (−1, 1, −1, 1)T . 4. (Continuation) Solve the system using the SOR iterative method with values of ω = 1(0.1)2. Plot the number of iterations for convergence versus the values of ω. Which value of ω results in the fastest convergence? Computer Problems 8.2 340 Chapter 8 Additional Topics Concerning Systems of Linear Equations 5. Program and run the Jacobi, Gauss-Seidel, and SOR methods for the system of Example 1 a. using equations involving the splitting matrix Q. b. using the equation formulations in Example 4. c. usingthepseudocodeinvolvingmatrix-vectormultiplication. 6. (Continuation) Select one or more of the systems in Computer Problem 1, and rerun these programs. 7. Considerthelinearsystem ⎡12−64 ⎢3 ⎢−46−4 ⎢1−46 ⎤⎡y⎤⎡b⎤ ⎥⎢1⎥⎢1⎥ ⎢ ⎢ ⎢ ⎢ ⎢⎣ 6 −4 1 1 −4 6 −4 1 −4 6 1 −4 4 3 ⎥⎢ y4 ⎥ ⎢ . ⎥⎢y ⎥⎢ n−3 ⎥⎦ ⎢⎣ y n − 2 yn−1 ⎥ ⎢b4 ⎥ ⎥ = ⎢ . ⎥ 􏰖􏰗􏰖􏰗􏰖􏰗 9 −3 x1 = 6 −2 8 x2 −4 Using Maple or Matlab, compare solving it by using the Jacobi method and the Gauss- Seidel method starting with x(0) = (0, 0)T . 8. (Continuation) a. Changethe(1,1)entryfrom9to1sothatthecoefficientmatrixisnolongerdiag- onally dominant and see whether the Gauss-Seidel method still works. Explain why or why not. b. Thenchangethe(2,2)entryfrom8to1aswellandtest.Againexplaintheresults. 9. Usetheconjugategradientmethodtosolvethislinearsystem: ⎡ ⎤⎡⎤⎡⎤ 2.0 −0.3 −0.2 x1 7 ⎣−0.3 2.0 −0.1⎦⎣x2 ⎦ = ⎣5⎦ −0.2 −0.1 2.0 x3 3 10. (Euler-Bernoulli beam) A simple model for a bending beam under stress involves the Euler-Bernoulli differential equation. A finite difference discretization converts it into a system of linear equations. As the size of the discretization decreases, the linear system becomes larger and more ill-conditioned. a. For a beam pinned at both ends, we obtain the following banded system of linear equations with a bandwidth of five: 1 −4 1 ⎥⎢y2⎥⎢b2⎥ ⎥⎢y3⎥⎢b3⎥ 1 −4 ... ... ... ... ... ... ⎥ ⎢b ⎥ ⎥ ⎢ n−3 ⎥ 1 −4 1 ⎥⎦ ⎢⎣bn−2 ⎥⎦ bn−1 6 −4 6−12yn bn The right-hand side represents forces on the beam. Set the right-hand side so that there is a known solution, such as a sag in the middle of the beam. Using an iterative ... ... ... ... ... ... 1 −4 6 −4 1 ⎥⎢ . ⎥⎢y ⎥ ⎢b ⎥ 8.2 Iterative Solutions of Linear Systems 341 method, repeatedly solve the system by allowing n to increase. Does the error in the solution increase when n increases? Use mathematical software that computes the condition number of the coefficient matrix to explain what is happening. b. Thelinearsystemofequationsforacantileverbeamwithafreeboundarycondition at only one end is ⎡12−6 4 ⎤⎡y⎤⎡b⎤ 311 ⎢−4 6 −4 1 ⎥⎢ y2 ⎢ 1 −4 6 −4 1 ⎥⎢ y3 ⎥⎢b2 ⎥ ⎥ ⎢ b3 ⎥ ⎥⎢b4 ⎥ ⎢ ⎢ ⎢ ⎢ ⎢⎣ 1−46−41 ⎥⎢y4 1 −4 1 6 −4 −93 111 25 25 ⎥ = ⎢ . ⎥ 1⎥⎢ n−3⎥ ⎢ n−3⎥ −43 ⎥⎦⎢⎣yn−2⎥⎦ ⎢⎣bn−2⎥⎦ 25 yn−1 bn−1 Repeat the numerical experiment for this system. See Sauer [2006] for additional details. 11. Consider this sparse linear system: ⎡ 3 −1 ⎢ −1 3 −1 1⎤⎡x ⎤ ⎡2.5⎤ 1 1 2⎥⎢x2⎥⎢1.5⎥ 2 ⎥⎢x3⎥⎢1.5⎥ ⎥⎢ . ⎥ ⎢.⎥ ⎥⎢.⎥⎢.⎥ ⎥⎢ . ⎥ = ⎢1.0⎥ 12 24 12 25 25 25 yn bn ⎢−1 3−1 ⎢−13−1 1 ⎢....2 ⎢ .. .. .. .. ⎢ ... ... ... ... ⎥⎢.⎥⎢.⎥ ⎢ 1 −13 ⎥⎢ . ⎥ ⎢1.5⎥ ⎢⎣ 12 2 ⎥⎦⎢xn−2⎥⎢⎣⎥⎦ −13xn 1.5 −1 3 −1 ⎣xn−1⎦ 1.5 −1 The true solution is x = [1,1,1,...,1,1,1]T . Use an iterative method to solve this 1 2 system for increasing values of n. 12. Consider the sample two-dimensional linear system Ax = b, where A = 3 2 , 􏰖􏰗 􏰖􏰗 26 b = 2 , and c = 0. Plot graphs to show the following: −8 a. Thesolutionliesattheintersectionoftwolines. b. Graph of the quadratic form F(x) = c + bT x + 1 xT Ax showing that the minimum 2 point of this surface is the solution of Ax = b. c. Contours of the quadratic form so each ellipsoidal curve has a constant value. d. Gradient F′(x) of the quadratic form. Show that for every x, the gradient points in the direction of the steepest increase of F(x) and is orthogonal to the contour lines. (See Section 16.2.) 342 Chapter 8 Additional Topics Concerning Systems of Linear Equations 8.3 Eigenvalues and Eigenvectors EXAMPLE1 Solution EXAMPLE 2 Solution Let A be an n × n matrix. We ask the following natural question about A: Are there nonzero vectors v for which Av is a scalar multiple of v? Although we pose this question in the spirit of pure curiosity, there are many situations in scientific computation in which this question arises. The answer to our question is a qualified Yes! We must be willing to consider complex scalars, as well as vectors with complex components. With that broadening of our viewpoint, such vectors always exist. Here are two examples. In the first, we need not bring in complex numbers to illustrate the situation, while in the second, the vectors and scalar factors must be complex. 􏰖􏰗 Let A= 3 2 .Findanonzerovectorvforwhich Avisamultipleofv. 7 −2 One easily verifies that A 􏰖􏰗􏰖􏰗􏰖􏰗 A 1 = 5 =5 1 􏰖􏰗􏰖􏰗 􏰖􏰗 2 = −8 =−4 2 −7 28 −7 We have two different answers (but we have not revealed how to find them). ■ 􏰖􏰗 Repeat the preceding example with the matrix A = 1 1 . −2 3 As in Example 1, it can be verified that 􏰖􏰗 􏰖􏰗 A 1 =(2+i) 1 􏰖1+i􏰗 􏰖1+i􏰗 A 1 =(2−i) 1 1−i 1−i In these equations, i = √−1. Surprisingly, we find answers involving complex numbers even though the matrix does not contain any complex entries! ■ When the equation Ax = λx is valid and x is not zero, we say that λ is an eigenvalue of A and x is an accompanying eigenvector. Thus, in Example 1, the matrix has 5 as an eigenvalue with accompanying eigenvector [1, 1]T , and −4 is another eigenvalue with accompanying eigenvector [2, −7]T . Example 2 emphasizes that a real matrix may have complex eigenvalues and complex eigenvectors. Notice that an equation A0 = λ0 and an equation A0 = 0x say nothing useful about eigenvalues and eigenvectors of A. Many problems in science lead to eigenvalue problems in which the principal question usually is: What are the eigenvalues of a given matrix, and what are the accompanying eigenvectors? An outstanding application of this theory is to systems of linear differential equations, about which more will be said later. Notice that if Ax = λx and x ≠ 0, then every nonzero multiple of x is an eigenvector (with the same eigenvalue). If λ is an eigenvalue of an n × n matrix A, then the set {x: Ax = λx} is a subspace of Rn called an eigenspace. It is necessarily of dimension at least 1. Calculating Eigenvalues and Eigenvectors Given a square matrix A, how does one discover its eigenvalues? Begin by observing that the equation Ax = λx is equivalent to (A − λI)x = 0. Since we are interested in nonzero solutions to this equation, the matrix A − λ I must be singular (noninvertible), and therefore, Det( A − λ I ) = 0. This is how (in principle) we can find all the eigenvalues of A. Specifically, form the function p by the definition p(λ) = Det( A − λ I ), and find the zeros of p. It turns out that p is a polynomial of degree n and must have n zeros, provided that we allow complex zeros and count each zero a number of times equal to its multiplicity. Even if the matrix A is real, we must be prepared for complex eigenvalues. The polynomial just described is called the characteristic polynomial of the matrix A. If this polynomial has a repeated factor, such as (λ − 3)k , then we say that 3 is a root of multiplicity k . Such roots are still eigenvalues, but they can be troublesome when k > 1.
To illustrate the calculation of eigenvalues, let us use the matrix in Example 1, namely,
􏰖􏰗
A=32 7 −2
The characteristic polynomial is
p(λ)=Det(A−λI)=Det 3−λ 2 =(3−λ)(−2−λ)−14
The eigenvalues are 5 and −4.
We can carry out this calculation with one or two commands in Matlab, Maple, or
Mathematica. We can determine the characteristic polynomial and subsequently compute its zeros. This gives us the two roots of of the characteristic polynomial, which are the eigenvalues 5 and −4. These mathematical software systems also have single commands to produce a list of eigenvalues, computed in the best possible way, which is usually not to determine the characteristic polynomial and subsequently compute its zeros!
In general, an n × n matrix will have a characteristic polynomial of degree n, and its roots are the eigenvalues of A. Since the calculation of zeros of a polynomial is numeri- cally challenging if not unstable, this straightforward procedure is not recommended. (See Computer Problem 8.3.2 for an experiment pertaining to this situation.) For small values of n, it may be quite satisfactory, however. It is called the direct method for computing eigenvalues.
Once an eigenvalue λ has been determined for a matrix A, an eigenvector can be computed by solving the system (A − λI)x = 0. Thus, in Example 1, we must solve (A − 5I)x = 0, or
􏰖􏰗􏰖􏰗􏰖􏰗
−2 2 x1 = 0 7−7×2 0
8.3 Eigenvalues and Eigenvectors 343
􏰖􏰗
7 −2−λ
=λ2 −λ−20=(λ−5)(λ+4)

344 Chapter 8
Additional Topics Concerning Systems of Linear Equations
Of course, this matrix is singular, and the homogeneous equation has nontrivial solutions, such as [1, 1]T . The other eigenvalue is treated in the same way, leading to an eigenvector [2, −7]T . Any scalar multiple of an eigenvector is also an eigenvector.
This work can be done by using mathematical software to find an eigenvector for each eigenvalue λ via the null space of the matrix A − λI. Also, we can use a single command to compute all the eigenvalues directly or request the calculation of all the eigenvalues and eigenvectors at once. The Matlab command [V,D] = eig(A) produces two arrays, V and D. The array V has eigenvectors of A as its columns, and the array D contains all the eigenvalues of A on its diagonal. The program returns a vector of unit length such as [0.7071, 0.7071]T . That vector by itself provides a basis for the null space of A − 5 I .
Notice that the eigenvalue-eigenvector problem is nonlinear. The equation Ax = λx has two unknowns, λ and x. They appear in the equation multiplied together. If either x or λ were known, finding the other would be a linear problem and very easy.
Mathematical Software
A typical, mundane use of mathematical software such as Matlab might be to compute the eigenvalues and eigenvalues of a matrix with a command such as [V,D] = eig(A) for the matrix
⎡⎤
1 3 −7 A = ⎣ −3 4 1 ⎦
2 −5 3
Matlab responds instantly with the eigenvectors in the array V and the eigenvalues in the diagonal array D. The real eigenvalue is 0.0214 and the complex pair of eigenvalues are 3.9893 ± 5.5601i . Behind the scenes, much complicated computing may be taking place. The general procedure has these components: First, by means of similarity transforma- tions, A is put into lower Hessenberg form. This means that all elements below the first subdiagonal are zero. Thus, the new A = (aij) satisfies aij = 0 when i > j + 1. Sim- ilarity transformations ensure that the eigenvalues are not disturbed. If A is real, further similarity transformations put A into a near-diagonal form in which each diagonal element is either a single real number or a 2 × 2 real matrix whose eigenvalues are a pair of con- jugate complex numbers. Creating the additional zeros just below the diagonal requires some iterative process, because after all, we are in effect computing the zeros of a poly- nomial. The iterative process is reminiscent of the power method that will be described in Section 8.4.
Maple can be used to compute the eigenvalues and eigenvectors. The quantities are computed in exact arithmetic and then converted to floating-point. In some versions of Maple and Matlab, one can use some of the commands from one of these packages in the other. In Mathematica, we can use commands to obtain similar results.
The best advice for anyone who is confronted with challenging eigenvalue problems is to use the software in the package LAPACK. Special eigenvalue algorithms for various types of matrices are available there. For example, if the matrix in question is real and symmetric, one should use an algorithm tailored for that case. There are about a dozen categories available to choose from in LAPACK. Matlab itself employs some of the programs in LAPACK.

Properties of Eigenvalues
A theorem that summarizes the special properties of a matrix that impinge on the computing of its eigenvalues follows.
8.3 Eigenvalues and Eigenvectors 345
MATRIX EIGENVALUE PROPERTIES
The following statements are true for any square matrix A:
1. IfλisaneigenvalueofA,thenp(λ)isaneigenvalueofp(A),foranypolynomial
p. In particular, λk is an eigenvalue of Ak .
2. If A is nonsingular and λ is an eigenvalue of A, then p(1/λ) is an eigenvalue of
p(A−1), for any polynomial p. In particular, λ−1 is an eigenvalue of A−1.
3. If A is real and symmetric, then its eigenvalues are real.
4. If A is complex and Hermitian, then its eigenvalues are real.
5. If A is Hermitian and positive definite, then its eigenvalues are positive.
6. If P is nonsingular, then A and P A P −1 have the same characteristic polynomial (and the same eigenvalues).
■ THEOREM1
EIGENVALUES OF SIMILAR MATRICES
Similar matrices have the same eigenvalues.
■ THEOREM2
Recall that a matrix A is symmetric if A = AT , where AT = (aji ) is the transpose of A = (aij). On the other hand, a complex matrix A is Hermitian if A = A∗, where A∗ = AT = (a j i ). Here A∗ is the conjugate transpose of the matrix A. Using the syntax of programming, we can write AT (i, j) = A( j, i) and A∗(i, j) = A( j, i). Recall also that A
is positive definite if xT Ax > 0 for all nonzero vectors x.
Two matrices A and B are similar to each other if there exists a nonsingular matrix P
such that B = P AP−1. Similar matrices have the same characteristic polynomial
Det(B − λI) = Det(P AP−1 − λI) = Det(P(A − λI)P−1)
= Det( P ) · Det( A − λ I ) · Det( P −1 ) = Det( A − λ I )
Thus, we have an important theorem.
This theorem suggests a strategy for finding eigenvalues of A. Transform the matrix A to a matrix B using a similarity transformation B = P A P −1 in which B has a special structure, and then find the eigenvalues of matrix B. Specifically, if B is triangular or diagonal, the eigenvalues of B (and those of A) are simply the diagonal elements of B.
Matrices A and B are said to be unitarily similar to each other if B = U∗ AU for some unitary matrix U . Recall that a matrix U is unitary if U U ∗ = I . This brings us naturally to another important theorem and two corollaries.

346 Chapter 8
Additional Topics Concerning Systems of Linear Equations
SCHUR’S THEOREM
Every square matrix is unitarily similar to a triangular matrix.
■ THEOREM3
MATRIX SIMILAR TO A TRIANGULAR MATRIX
Every square real matrix is similar to a triangular matrix.
■ COROLLARY1
In this theorem, an arbitrary complex n × n matrix A is given, and the assertion made is that a unitary matrix U exists such that:
UAU∗ =T
where UU∗ = I and T is a triangular matrix.
The proof of Schur’s Theorem can be found in Kincaid and Cheney [2002] and Golub
and Van Loan [1996].
Thus the factorization
PAP−1 =T
is possible, where T is triangular, P is invertible, and A is real.
We illustrate Schur’s Theorem by finding the decomposition of this 2 × 2 matrix: 􏰖􏰗
A= 3 −2 83
From the characteristic equation det( A − λ I ) = λ2 − 6λ + 25 = 0, the eigenvalues are 3 ± 4i . By solving A − λ I = 0 with each of these eigenvalues, the corresponding eigen- vectors are v1 = [i, 2]T and v2 = [−i, 2]T . Using the Gram-Schmidt orthogonalization process, we obtain u1 = v1 and u2 = v2 −[v∗2u1/u∗1u1]u1 = [−2,−i]T . After normalizing these vectors, we obtain the unitary matrix
1 􏰖i −2􏰗 U=√5 2 −i
which satisfies the property UU∗ = I, Finally, we obtain the Schur form 􏰖􏰗
UAU∗=3+4i −6 0 3−4i
which is an upper triangular matrix with the eigenvalues on the diagonal. ■
EXAMPLE 3
Solution
HERMITIAN MATRIX UNITARILY SIMILAR TO A DIAGONAL MATRIX
Every square Hermitian matrix is unitarily similar to a diagonal matrix.
■ COROLLARY2

■ THEOREM4
In the second corollary, a Hermitian matrix, A, is factored as A = U∗ DU
where D is diagonal and U is unitary.
Furthermore, U∗ AU = T and U∗ A∗U = T∗ and A = A∗, so T = T∗, which must
be a diagonal matrix.
Most numerical methods for finding eigenvalues of an n × n matrix A proceed by
determining such similarity transformations. Then one eigenvalue at a time, say, λ, is com- puted, and a deflation process is used to produce an (n − 1) × (n − 1) matrix 􏶗A whose eigenvalues are the same as those of A, except for λ. Any such procedure can be repeated with the matrix 􏶗A to find as many eigenvalues of the matrix A as desired. In practice, this strategy must be used cautiously because the successive eigenvalues may be infected with roundoff error.
Gershgorin’s Theorem
Sometimes it is necessary to determine in a coarse manner where the eigenvalues of a matrix are situated in the complex plane C. The most famous of these so-called localization theorems is the following.
The matrix A can have either real or complex entires. The region containing the eigenvalues of A can be written
􏶵n 􏶵n􏶮 􏶯 Ci = z∈C:|z−aii|􏶞ri
i=1 i=1
wheretheradiiareri =􏰍nj=1 |aij|. j ≠ i
TheeigenvaluesofAandAT arethesamebecausethecharacteristicequationinvolves the determinant, which is the same for a matrix and its transpose. Therefore, we can apply theGershgorinTheoremtoAT andobtainthefollowingusefulresult.
8.3 Eigenvalues and Eigenvectors 347
GERSHGORIN’S THEOREM
All eigenvalues of an n × n matrix A = (ai i ) are contained in the union of the n discs Ci =Ci(aii,ri)inthecomplexplanewithcenteraii andradiiri givenbythesumof the magnitudes of the off-diagonal entries in the ith row.
MORE GERSHGORIN DISCS
All eigenvalues of an n × n matrix A = (ai i ) are contained in the union of the n discs Di = Di (aii , si ) in the complex plane having center at aii and radii si given by the sum of the magnitudes of the columns of A.
■ COROLLARY3

348 Chapter 8
Additional Topics Concerning Systems of Linear Equations
Consequently, the region containing the eigenvalues of A can be written as 􏶵n 􏶵n􏶮 􏶯
Di = z∈C:|z−aii|􏶞si i=1
i=1
where the radii are si = 􏰍ni =1 |ai j |. Finally, the region containing the eigenvalues of A is
i̸=j
􏶠 􏶵n i=1
􏶡 􏶶 􏶠 􏶵n 􏶡 Di
Ci
i=1
This may contain tighter bounds on the eigenvalues in some case. Also, a useful localization
result is
For a matrix A, the union of any k Gerschgorin discs that do not intersect the remaining n − k circles contains exactly k (counting multiplicities) of the eigenvalues of A.
■ COROLLARY4
For a strictly diagonally dominant matrix, zero cannot lie in any of its Gershgorin discs, so it must be invertible. Consequently, we obtain the following results.
Every strictly diagonally dominant matrix is nonsingular.
■ COROLLARY5
EXAMPLE 4
Solution
Consider the matrix
Draw the Gershgorin discs.
⎡⎤
4−i2i
A = ⎣ −1 2i 2 ⎦
1 −1 −5
Using the rows of A, we find that the Gershgorin discs are C1(4 − i, 3), C2(2i, 3), and C3(−5, 2). By using the columns of A, we obtain more Gershgorin discs: D1(4 − i, 2), D2(2i, 3), and D3(−5, 3). Consequently, all the eigenvalues of A are in the three discs D1, C2, and C3, as shown in Figure 8.1. By other means, we compute the eigenvalues of A as λ1 = 3.7208 − 1.05461i, λ2 = 4.5602 + −0.2849i, and λ3 = −0.1605 + 2.3395i. In Figure 8.1, the center of the discs are designated by dots • and the eigenvalues by ∗. ■
Singular Value Decomposition
This subsection requires of the reader some further knowledge of linear algebra, in particular the diagonalization of symmetric matrices, eigenvalues, eigenvectors, rank, column space,

Im(z) 6
4 2
D3 C3
C2, D2 *
8.3 Eigenvalues and Eigenvectors 349
C1 0* D1
FIGURE 8.1
Gershorgin discs
−2 −4
*
−6 −4 −2 0 2 4 6
and norms. See Appendix D for a brief review of these topics. (In the discussion below, we assume that the Euclidean norm is being used.)
The singular value decomposition is a general-purpose tool that has many uses, par- ticularly in least-squares problems (Chapter 12). It can be applied to any matrix, whether square or not. We begin by stating that the singular values of a matrix A are the nonnegative square roots of the eigenvalues of AT A.
Furthermore, the diagonal matrix D contains the eigenvalues of AT A on its diagonal. This follows from the fact that AT A Q = Q D, so the columns of Q are eigenvectors of AT A. If λ is an eigenvalue of AT A and if x is a corresponding eigenvector, then AT Ax = λx whence
||Ax||2 = (Ax)T (Ax) = xT AT Ax = xT λx = λ||x||2
This equation shows that λ is real and nonnegative. We can order the eigenvalues as
λ 􏶟 λ 􏶟 · · · 􏶟 λ 􏶟 0. (Reordering the eigenvalues requires reordering the columns of Q.)
􏶣
12n
The numbers σj = + λj are the singular values of A.
Re(z)
MATRIX SPECTRAL THEOREM
Let A be m × n. Then AT Ais an n × n symmetric matrix and it can be diagonalized by an orthogonal matrix, say, Q:
AT A= QDQ−1 whereQQT =QTQ=IandDisadiagonaln×nmatrix.
■ THEOREM5

350 Chapter 8
Additional Topics Concerning Systems of Linear Equations
Since Q is an orthogonal matrix, its columns form an orthonormal base for Rn . They are unit eigenvectors of AT A, so if vj is the jth column of Q, then AT Avj = λjvj. Some of the eigenvalues of AT A can be zero. Define r by the condition
λ1􏶟λ2􏶟 ···􏶟λr >0=λr+1 =···=λn
For a review of concepts such as rank, orthogonal basis, orthonormal basis, column
space, null space, and so on, see Appendix D.
ORTHOGONAL BASIS THEOREM
If the rank of A is r, then an orthogonal basis for the column space of A is {Avj: 1􏶞 j􏶞r}.
■ THEOREM6
Proof
Observe that
This establishes the orthogonality of the set {Avj:1􏶞 j􏶞n}. By letting k = j, we get
||Av ||2 =λ .Hence, Av ≠ 0ifandonlyif1􏶞 j􏶞r.Ifwisanyvectorinthecolumn jjj 􏰍n
(Avk)T (Avj) = vkT AT Avj = vkT λjvj = λjδkj
space of A, then w = Ax for some x in Rn. Putting x =
j=1 cjvj, we get cjAvj
􏰃n w=Ax=
􏰃r j=1
cjAvj = and therefore, w is in the span of {Av1, Av2,…, Avr}.
j=1
The preceding theorem gives a reasonable way of computing the rank of a numerical matrix. First, compute its singular values. Any that are very small can be assumed to be zero. The remaining ones are strongly positive, and if there are r of them, we take r to be the numerically computed rank of A.
A singular value decomposition of an m × n matrix A is any representation of A in the form
A = U DV T
where U and V are orthogonal matrices and D is an m × n diagonal matrix having nonneg- ative diagonal entries that are ordered d11 􏶟 d22 􏶟 · · · 􏶟 0. Then from Problem 4, it follows that the diagonal elements dii are necessarily the singular values of A. Note that the matrix U is m × m and V is n × n. A nonsquare matrix D is nevertheless said to be diagonal if the only elements that are not zero are among those whose two indices are equal.
One singular value decomposition of A (there are many of them) can be obtained from the work described above. Start with the vectors v1, v2, . . . , vr . Normalize the vectors Av j to get vectors u j . Thus, we have
uj = Avj/||Avj|| (1􏶞 j􏶞r)
Extend this set to an orthonormal base for Rm . Let U be the m ×m matrix whose columns are u1,u2,…,um.Define Dtobethem×nmatrixconsistingofzerosexceptforσ1,σ2,…,σr on its diagonal. Let V = Q, where Q is as above.
■

EXAMPLE 5
Solution
There will be certain small values of ε for which A has rank 3 and AT A has rank 1 (in the computer).
In an example in Section 1.1 (p. 4), we encountered this matrix:
􏰖􏰗
A = 0.1036 0.2122 0.2081 0.4247
Determine its eigenvalues, singular values, and condition number.
By using mathematical software, it is easy to find the eigenvalues λ1(A) ≈ −0.0003 and
λ2(A) ≈ 0.5286. We can form the matrix 􏰖􏰗
AT A = 0.0540 01104 0.1104 0.2254
and find its eigenvalues λ ( AT A) ≈ 0.3025 × 10−4 and λ ( AT A) ≈ 0.2794. Therefore, the 1􏶣2􏶣
singularvaluesareσ1(A)= |λ1(AT A)|≈0.0003andσ2(A)= |λ2(AT A)|≈0.5286. Also, we can obtain the singular values directly as σ1 ≈ 0.0003 and σ2 ≈ 0.5286 using mathematical software. Consequently, the condition number is κ(A) = σ2/σ1 ≈ 1747.6. Because of this large condition number, we now understand why there was difficultly in solving a linear system of equations with this coefficient matrix! ■
8.3 Eigenvalues and Eigenvectors 351 ToverifytheequationA=UDVT,firstnotethatσj =||Avj||2andthatσjuj =Avj.
Then compute U D. Since D is diagonal, this is easy. We get
UD = [u1,u2,…,um]D = [σ1u1,σ2u2,…,σrur,0,…,0]
= [Av1, Av2,…, Avr,…, Avn] = AQ = AV
This implies that
A = U DV T
The condition number of a matrix can be expressed in terms of its singular values
􏶫 σmax κ(A) = σmin
since || A||2 = ρ( AT A) = σmax( A) and || A−1||2 = ρ( A−T A−1|| = σmin( A). Numerical Examples of Singular Value Decomposition
The numerical determination of a singular value decomposition is best left to the available high-quality software. Such programs can be found in Matlab, Maple, LAPACK, and other software packages. The high-quality programs do not form AT A and seek its eigenvalues. One wishes to avoid using AT A in numerical work because its condition number may be much worse than that of A. This phenomenon is easily illustrated by the matrices
⎡1 1 1⎤ A = ⎢ ⎢⎣ ε 0 0 ⎥ ⎥⎦ ,
0ε0 00ε
⎡1+ε2 1 1 ⎤
A T A = ⎣ 1 1 + ε 2 1
1 1 1+ε2
⎦

352
Chapter 8
Additional Topics Concerning Systems of Linear Equations
EXAMPLE 6
Solution
Calculate the singular value decomposition of the matrix
⎡⎤
11
A=⎣0 1⎦ (1)
10 Here,thematrixAism×nandm=3andn=2.First,wefindthattheeigenvaluesof
the matrix
􏰖􏰗
ATA= 2 1 12
arrangedindescendingorderareλ1 =3andλ1 =1.Thenumberofnonzeroeigenvaluesof
the matrix AT A is 2. Next, we determine that the eigenvectors of the matrix AT A are [1, 1]T
for λ = 3 and [1, −1]T for λ = 1. Consequently, the orthonormal set of eigenvectors 1􏰠√√􏰡T2 􏰠√√􏰡T
of AT Aare 1 2,1 2 forλ1 =3and 1 2,−1 2 .Thenwearrangetheminthe 22 22
same order as the eigenvalues to form the column vectors of the n × n matrix V : 􏰙1√ 1√􏰚
V=􏰠vv􏰡=2√2 2√2 12 12−12
22
Now we form a diagonal matrix D, placing on the leading diagonal the singular values: σi =√λi.Sinceσ1 =√3andσ2 =1,them×nsingularvaluematrixis
⎡√⎤ ⎢ 3 √0⎥
D=⎣01⎦ 00
Here, on the leading diagonal are the square roots of the eigenvalues of AT A in descending
order, and the rest of the entries of the matrix D are zeros. Next, we compute vectors
u =σ−1Av fori=1andformthecolumnvectorsofthem×mmatrixU.Inthiscase, iii
we find
and
⎡1 1⎤􏰙√􏰚 ⎡1√6⎤ 1√⎢ ⎥12 ⎢3√⎥
u=σ−1Av= 3⎣01⎦2√ =⎣1 6⎦ 1113126√
102 16 6
u
⎡11⎤􏰙1√􏰚⎡0⎤ = σ − 1 A v = ⎢⎣ 0 1 ⎥⎦ 2 √ 2 = ⎢⎣ − 1 √ 2 ⎥⎦
222 −122√ 10212
Finally, we add to the matrix U the rest of the m − r vectors using the Gram-Schmidt orthogonalization process. So we make the vector u3 perpendicular to u1 and u2:
⎡1⎤ 􏰉􏰈􏰉􏰈⎢3⎥
􏶗u 3 = e 1 − u 1T e 1 u 1 − u 1T e 2 u 2 = ⎣ − 1 ⎦ 3
2
−1 3

Normalizing the vector u3, we get
So we have the matrix
⎡ 1√3⎤
⎢3√⎥ u3 =⎣−1 3⎦
8.3 Eigenvalues and Eigenvectors 353
3√ −1 3
⎡ 1 √6
0 1 √3 ⎤
3
􏰠􏰡⎢3√√3√⎥ U= u1 u2 u′3 =⎣1 6 1 2 −1 3⎦
6√2√3√ 1 6 −1 2 −1 3
A = UDVT
⎡1 1⎤ ⎡1√6 0 1√3⎤⎡√3 0⎤􏰙 √ √ 􏰚
623 The singular value decomposition of the matrix A is
⎢⎥⎢3√√3√⎥⎢√⎥1212 ⎣0 1⎦=⎣1 6 1 2 −1 3⎦⎣ 0 1⎦ 2√ 2√
6√ 2√ 3√ 12−12 1016−12−13002 2
623
So there we have it! Fortunately, there is mathematical software for doing all of this instantly! We can verify the results by computing the diagonal matrix and the matrix A from the factorization. ■
See Chapters 12 and 16, for some important applications of the singular value decom- position. Further examples are given there and in the problems of those chapters.
Application: Linear Differential Equations
The application of eigenvalue theory to systems of linear differential equations will be briefly explained here. Let us start with a single linear differential equation with one dependent variable x. The independent variable is t and often represents time. We write x′ = ax, or in more detail (d/dt)x(t) = ax(t). There is a family of solutions, namely, x(t) = ceat, where c is an arbitrary real parameter. If an initial value x(0) is prescribed, we shall need parameter c to get the initial value right.
A pair of linear differential equations with two dependent variables, x1 and x2 will
look like this:
􏰎
x1′ =a11x1+a12x2 x2′ =a21x1+a22x2
The general form of a system of n linear first-order differential equations, with constant coefficients, is simply x′ = Ax. Here, A is an n × n numerical matrix, and the vector x has n components, x j , each being a function of t . Differentiation is with respect to t . To solve this, we are guided by the easy case of n = 1, discussed above. Here, we try x(t) = eλtv, where v is a constant vector. Taking the derivative of x, we have x′ = λeλtv. Now the system of equations has become λeλt v = Aeλt v, or λv = Av. This is how eigenvalues come into the process. We have proved the following result.

354 Chapter 8
Additional Topics Concerning Systems of Linear Equations
LINEAR DIFFERENTIAL EQUATIONS
If λ is an eigenvalue of the matrix A and if v is an accompanying eigenvector, then one solution of the differential equation x′ = Ax is x(t) = eλt v.
■ THEOREM7
FIGURE 8.2
Two-mass vibration problem
Application: A Vibration Problem
Eigenvalue-eigenvector analysis can be utilized for a variety of differential equations. Con- sider the system of two masses and three springs shown in Figure 8.2. Here, the masses are constrained to move only in the horizontal direction.
From this situation, we write the equations of motion in matrix-vector form:
􏰙′′􏰚 􏰖 􏰗􏰖 􏰗
x1 =−β α x1 x′′=Ax
x′′ α −β x2 2
By assuming that the solution is purely oscillatory (no damping), we have
In matrix form, we get
By differentiation, we obtain and
x = veiωt
􏰖􏰗􏰖􏰗
x1 = v1 eiωt x2 v2
x′′ = −ω2veiωt = −ω2x 􏰖􏰗
−β α x = −ω2 x α −β
This is the eigenvalue problem
where λ = −ω2. Eigenvalues can be found from the characteristic equation:
Ax = λx
􏰖ω2 −β α􏰗
Thisis(ω2 −β)2 −α2 =ω4 −2βω2 +(β2 −α2)=0,and 21􏰛􏶣􏰜
det(A+ω2I)=det α ω2−β =0
ω =
2β± 4β2 −4(β2 −α2) =β±α
2
For simplicity, we now assume unit masses and unit springs so that β = 2 and α = 1. Then
we obtain
􏰖􏰗
A=−2 1 1 −2

Thentherootsofthecharacteristicequationsareω12 =β+α=3andω2 =β−α=1. Next, we can find the eigenvectors. For the first eigenvalue, we obtain
􏰖􏰗􏰖􏰗
(A+ω12I)v1 =0 1 1 v11 =0 1 1 v12
Since v11 = −v12, we obtain the first eigenvector 􏰖􏰗
v1= 1 −1
For the second eigenvector, we have
(A+ω2I)v2 =0 −1 1 v21 =0
􏰖􏰗
v2= 1
The general solution for the equations of motion for the two-mass system is x(t) = c1v1eiω1t + c2v1e−iω1t + c3v2eiω2t + c4v2e−iω2t
Because the solution was for the square of the frequency, each frequency is used twice (one positive and one negative). We can use initial conditions to solve for the unknown coefficients.
Summary
(1) An eigenvalue λ and eigenvector x satisfy the equation Ax = λx. The direct method to compute the eigenvalues is to find the roots of the characteristic equation p(λ) = det(A − λI) = 0. Then, for each eigenvalue λ, the eigenvectors can be found by solv- ing the homogeneous system ( A − λ I )x = 0. There are software packages for finding the eigenvalue-eigenvector pairs using more sophisticated methods.
(2) There are many useful properties for matrices that influence their eigenvalues. For ex- ample, the eigenvalues are real when A is symmetric or Hermitian. The eigenvalues are positive when A is symmetric or Hermitian positive definite.
(3) Many eigenvalue procedures involve similarity or unitary transformations to produce triangular or diagonal matrices.
(4) Gershgorin’s discs can be used to localize the eigenvalues by finding coarse estimates of them.
(5) The singular value decomposition of an m × n matrix A is A = U DV T
where D is an m × n diagonal matrix whose diagonal entries are the singular values, U is an m × m orthogonal matrix, and V is an n × n orthogonal matrix. The singular values of
1 −1 v22 Since v21 = −v22, we obtain the first eigenvector
8.3 Eigenvalues and Eigenvectors 355
􏰖􏰗􏰖􏰗
A are the nonnegative square roots of the eigenvalues of AT A.

356 Chapter 8
Additional Topics Concerning Systems of Linear Equations
1. Are[i,−1+i]T and[−i,−1−i]T eigenvectorsofthematrixinExample2?
2. Prove that if λ is an eigenvalue of a real matrix with eigenvector x, then λ is also an eigenvalue with eigenvector x. (For a complex number z = x + iy, the conjugate is defined by z = x − iy.)
3. Let
a 9.
3 −1 1 a. ⎣2 4 −2⎦
3 1 2 b. ⎣−1 4 −1⎦
1−i 1 i c. ⎣ 0 2i 2⎦
􏰖􏰗
A= cosθ −sinθ sin θ cos θ
Account for the fact that the matrix A has the effect of rotating vectors counterclockwise through an angle θ and thus cannot map any vector into a multiple of itself.
4. LetAbeanm×nmatrixsuchthatA=UDVT,whereUandVareorthogonaland D is diagonal and nonnegative. Prove that the diagonal elements of D are the singular
values of A.
5. Let A,U, D,andV beasinthesingularvaluedecomposition: A=UDVT.Letr be asdescribedinthetext.DefineUr toconsistofthefirstrcolumnsofU.LetVr consist of the first r columns of V , and let Dr be the r × r matrix having the same diagonal as
D. Prove that A = Ur Dr VrT . (This factorization is called the economical version of the singular value decomposition.)
6. A linear map P is a projection if P2 = P. We can use the same terminology for an n × n matrix: A2 = A is the projection property. Use the Pierce decomposition, I = A+(I − A),toshowthateverypointinRn isthesumofavectorintherangeof
A and a vector in the null space of A. What are the eigenvalues of a projection?
7. Find all of the Gershgorin discs for the following matrices. Indicate the smallest
region(s) containing all of the eigenvalues:
⎡⎤⎡⎤⎡⎤
1 −2 9
8. (Multiplechoice)LetAbeann×ninvertible(nonsingular)matrix.Letxbeanonzero
3 −1 9
vector. Suppose that Ax = λx. Which equation does not follow from these hypotheses?
a. Akx=λkx b. λ−kx=(A−1)kxfork􏶟0
c. p(A)x = p(λ)x foranypolynomial p d. Akx =(1−λ)kx e. Noneofthese.
(Multiple choice) For what values of s will the matrix I − svv∗ be unitary, where v is
a column vector of unit length?
√
1 0 2
a. 0,1 b. 0,2 c. 1,2 d. 0, 2 e. Noneofthese.
10. (Multiple choice) Let U and V be unitary n × n matrices, possibly complex. Which conclusion is not justified?
Problems 8.3

8.3 Eigenvalues and Eigenvectors 357 a. U + V is unitary. b. U∗ is unitary. c. UV is unitary.
d. U − vv∗ is unitary when ||v|| = √2 and v is a column vector. a11. (Multiplechoice)Whichassertionistrue?
a. Every n × n matrix has n distinct (different) eigenvalues.
b. Theeigenvaluesofarealmatrixarereal.
c. If U is a unitary matrix, then U∗ = UT
d. Asquarematrixanditstransposehavethesameeigenvalues.
12. (Multiple choice) Consider the symmetric matrix ⎡⎤
1 3 4 −1 A=⎢3 7−6 1⎥ ⎣ 4 −6 3 0⎦
e. None of these.
e. Noneofthese.
−1 1 0 5
What is the smallest interval derived from Gershgorin’s Theorem such that all eigen-
values of the matrix A lie in that interval?
a. [−7,9] b. [−7,13] c. [3,7] d. [−3,17] e. Noneofthese.
13. (Trueorfalse)Gershgorin’sTheoremassertsthateveryeigenvalueλofann×nmatrix A must satisfy one of these inequalities:
􏰃n j=1
j ̸=i
factored as A = P T P −1 , where P is a nonsingular matrix and T is upper triangular.
15. (Trueorfalse)AconsequenceofSchur’sTheoremisthatevery(real)symmetricmatrix
A can be factored in the form A = P DP−1, where P is unitary and D is diagonal.
16. Explain why ||U B||2 = ||B||2 for any matrix B when UT U = I.
⎢2⎥
17. Consider the matrix A = ⎣ 3 5 −3 ⎦. Plot the Gershgorin discs in the complex
013 2
planeforAandAT aswellasindicatethelocationsoftheeigenvalues.
18. (Continuation) Let B be the matrix obtained by changing the negative entries in A to
positive numbers. Repeat the process for B. ⎡⎤
4 0 −2
19. (Continuation) Repeat for C = ⎣ 1 2 0 ⎦.
119
􏰖􏰗
20. Find the Schur decomposition of A = 5 7 . −2 −4
|aij| for 1􏶞i􏶞n.
14. (Trueorfalse)AconsequenceofSchur’sTheoremisthateverysquarematrixAcanbe
|λ−aii|􏶞
⎡4−1 0⎤
55

358 Chapter 8
Additional Topics Concerning Systems of Linear Equations
1. Use Matlab, Maple, Mathematica, or other computer programs available to you to compute the eigenvalues and eigenvectors of these matrices:
􏰖􏰗
a.A=1 7 ⎡2−5 ⎤
4 −7 3 ⎢1 6 11 b.⎢5 −5 −2 ⎣9−31 3 2 5
2 3 −1 2⎥ −4 1⎥
65⎦ −5 1
c. Letn=12,aij =i/jwheni􏶞j,andaij =j/iwheni>j.Findtheeigenvalues.
d. Createann×nmatrixwithatridiagonalstructureandnonzeroelements(−1,2,−1) in each row. For n = 5 and 20, find all of the eigenvalues, and verify that they are 2 − 2 cos( jπ/(n + 1)).
e. For any positive integer n, form the symmetric matrix A whose upper triangular part is given by
⎡nn−1n−2n−3···21⎤
n−1 n−2 n−3 ··· 2 1⎥ n−2 n−3 ··· 2 1⎥
1
The eigenvalues of A are 1/{2 − 2 cos[(2i − 1)π/(2n + 1)]}. (See Frank [1958] and
Gregory and Karney [1969].) Numerically verify this result for n = 30.
2. Use Matlab to compute the eigenvalues of a random 100 × 100 matrix by direct use of the command eig and by use of the commands poly and roots. Use the timing functions to determine the CPU time for each.
3. Let p be the polynomial of degree 20 whose roots are the integers 1, 2, . . . , 20. Find the usual power form of this polynomial so that p(t) = t20 +a19t19 +a18t18 +···+a0. Next, form the so-called companion matrix, which is 20 × 20 and has zeros in all positions except all 1’s on the superdiagonal and the coefficients −a0 , −a1 , . . . , −a19 as its bottom row. Find the eigenvalues of this matrix, and account for any difficulties encountered.
4. (Student research project) Investigate some modern methods for computing eigen- values and eigenvectors. For the symmetric case, see the book by Parlett [1997]. Also, read the LAPACK User’s Guide. (See Anderson, et al. [1999].)
5. (Student research project) Experiment with the Cayley-Hamilton Theorem, which asserts that every square matrix satisfies its own characteristic equation. Check this
⎢
⎢
⎢
⎢
⎢ ⎣
… ··· . .⎥ .⎥
.. 2 1⎥ 21⎦
Computer Problems 8.3

numerically by using Matlab or some other mathematical software system. Use matrices of size 3, 6, 9, 12, and 15, and account for any surprises. If you can use higher-precision arithmetic do so—Matlab works with 15 digits of precision.
6. (Studentresearchproject)ExperimentwiththeQRalgorithmandthesingularvalue decomposition of matrices—for example, using Matlab. Try examples with four types of equations Ax = b—namely, (a) the system has a unique solution; (b) the system has many solutions; (c) the system is inconsistent but has a unique least-squares solution; (d) the system is inconsistent and has many least-squares solutions.
7. Using mathematical software such as Matlab, Maple, or Mathematica on each of the following matrices, compute the eigenvalues via the characteristic polynomial, compute the eigenvectors via the null space of the matrix, and compute the eigenvalues and eigenvectors directly:
􏰖3 2􏰗 a.7−1
⎡ ⎤
1 3−7 b.⎣−3 4 1⎦
2 −5 3
8. Using mathematical software such as Matlab, Maple, or Mathematica, determine the execution time for computing all eigenvalues of a 1000 × 1000 matrix with random entries.
9. Using mathematical software such as Matlab, Maple, or Mathematica, compute the Schur factorization of these complex matrices, and verify the results according to Schur’s Theorem and its corollaries:
􏰖􏰗􏰖􏰗􏰖􏰗
a. 3−i 2−i b. 2+i 3+i c. 2−i 2+i 2+i 3+i 3−i 2−i 3−i 3+i
10. Using mathematical software such as Matlab, Maple, or Mathematica, compute the singular value decomposition of these matrices, and verify that each result satisfies the
equation A=UDVT: ⎡1 1⎤
a. ⎣0 1⎦ 1 0
Create the diagonal matrix D = UT AV to
⎤
−2 5⎥
4⎦ −2
check the results (always recommended).
⎣1142⎦ 1124
⎡
1 3 b. ⎢ 2 7
⎣−2 −3 5 −3
One can see the effects of roundoff errors in these calculations, for the off-diagonal
elements in D are theoretically zero. ⎡⎤
5411
a11. Consider A = ⎢ 4 5 1 1 ⎥. Find the eigenvalues and accompanying eigenvectors
of this matrix, from Gregory and Karney [1969], without using software. Hint: The answers can be integers.
12. Find the singular value decomposition of these matrices:
􏰠􏰡􏰖3􏰗􏰠5√5√􏰡 a. 2 1 −2 b. 4 c. −2+3 3 2 3+3
8.3 Eigenvalues and Eigenvectors 359

360
Chapter 8
Additional Topics Concerning Systems of Linear Equations
⎡2222⎤ ⎡7−13√67+13√6⎤ ⎢ ⎥⎢26√26√⎥
d.⎣17 1 −17 −1⎦ e.⎣−7−13 6−7+13 6⎦ 10 10 10 10 2 6√ 2 6√
8.4
3 9 −3 −9 −13 6 13 6 55⎡55 ⎤6 6
−149 −50 −154
13. Consider B = ⎣ 537 180 546⎦. Find the eigenvalues, singular values, and
−27 −9 −25 condition number of the matrix B.
Power Method
A procedure called the power method can be employed to compute eigenvalues. It is an example of an iterative process that, under the right circumstances, will produce a sequence converging to an eigenvalue of a given matrix.
Suppose that A is an n × n matrix, and that its eigenvalues (which we do not know) have the following property:
|λ1|>|λ2|􏶟|λ3|􏶟 ···􏶟|λn|
Notice the strict inequality in this hypothesis. Except for that, we are simply ordering the eigenvalues according to decreasing absolute value. (This is only a matter of notation.) Each eigenvalue has a nonzero eigenvector u(i) and
Au(i) =λiu(i) (i =1,2,…,n) (1)
We assume that there is a linearly independent set of n eigenvectors {u(1), u(2), . . . , u(n)}. It is necessarily a basis for Cn .
We want to compute the single eigenvalue of maximum modulus (the dominant eigen- value) and an associated eigenvector. We select an arbitrary starting vector, x(0) ∈ Cn and express it as a linear combination of u(1), u(2), . . . , u(n):
x(0) =c1u(1) +c2u(2) +···+cnu(n)
In this equation, we must assume that c1 ≠ 0. Since the coefficients can be absorbed into
the vectors u(i), there is no loss of generality in assuming that
x(0) = u(1) +u(2) +···+u(n) (2)
Then we repeatedly carry out matrix-vector multiplication, using the matrix A to produce a sequence of vectors. Specifically, we have
⎧⎪ x(1) = Ax(0)
⎪x(2) = Ax(1) = A2x(0) ⎪⎨x(3) = Ax(2) = A3x(0)
⎪ .
⎪ x(k) = Ax(k−1) = Ak x(0)
⎩ .

In general, we have
x(k) = Ak x(0) (k = 1, 2, 3, . . .) Substituting x(0) in Equation (2), we obtain
x(k) = Ak x(0)
= Aku(1) + Aku(2) + Aku(3) +···+ Aku(n) = λk1u(1) +λk2u(2) +λk3u(3) +···+λknu(n)
by using Equation (1). This can be written in the form
􏰙 􏰔λ 􏰕k 􏰔λ 􏰕k 􏰔λ 􏰕k u(1) + 2 u(2) + 3 u(3) +···+ n
􏰚
u(n)
Since|λ |>|λ |for j >1,wehave|λ /λ |<1and􏰉λ /λ 􏰈k →0ask→∞.Tosimplify x(k) =λk1 1jj1j1 8.4 Power Method 361 λ1 λ1 λ1 the notation, we write the above equation in the form x(k) =λk􏰠u(1) +ε(k)􏰡 (3) 1 where ε(k) → 0 as k → ∞. We let φ be any complex-valued linear functional on Cn such that φ(u(1)) ≠ 0. Recall that φ is a linear functional if φ(ax + by) = aφ(x) + bφ(y) for scalars a and b and vectors x and y. For example, φ(x) = xj for some fixed j (1􏶞 j 􏶞 n) is a linear functional. Now, looking back at Equation (3), we apply φ to it: φ􏰉x(k)􏰈 = λk 􏰠φ􏰉u(1)􏰈 + φ􏰉ε(k)􏰈􏰡 1 Next, we form ratios r1, r2, . . . as follows: φ􏰉x(k+1)􏰈 􏰙φ􏰉u(1)􏰈 + φ􏰉ε(k+1)􏰈􏰚 rk ≡ φ􏰉x(k)􏰈 =λ1 φ􏰉u(1)􏰈+φ􏰉ε(k)􏰈 →λ1 as k →∞ Hence, we are able to compute the dominant eigenvalue λ1 as the limit of the sequence {rk}. With a little more care, we can get an accompanying eigenvector. In the definition of the vectors x(k) in Equation (2), we see nothing to prevent the vectors from growing or converging to zero. Normalization will cure this problem, as in one of the pseudocodes below. Power Method Algorithms Here we present pseudocode for calculating the dominant eigenvalue and an associated eigenvector for a prescribed matrix A. In each algorithm, φ is a linear functional chosen by the user. For example, one can use φ(x) = x1 (the first component of the vector). 362 Chapter 8 Additional Topics Concerning Systems of Linear Equations Power Method Algorithm integer k, kmax, n; real r real array ( A)1:n×1:n , (x)1:n , ( y)1:n external function φ output 0, x fork =1tokmax do y ← Ax r ← φ(y)/φ(x) x←y output k, x, r end do FIGURE 8.3 In 2D, power method illustration We use a simple 2×2 matrix such as A = 3 1 to give a geometric illustration of the 13 power method as shown in Figure 8.3. Clearly, the eigenvalues are λ1 = 2 and λ2 = 4 with eigenvectors v(1) = [−1, 1]T and v(2) = [1, 1]T , respectively. Starting with x(0) = [0, 1]T , the power method repeatedly multiplies the matrix A by a vector. It produces a sequence of vectors x(1), x(2), and so on that move in the direction of the eigenvector v(2), which corresponds to the dominant eigenvalue λ2 = 4. v(2) x(0) x(1) x(2) v(1) –1 0 1 We can easily modify this algorithm to produce normalized eigenvectors by using the infinity vector norm ||x||∞ = max1 􏶞 j 􏶞 n |x j |, as in the following code: Modified Power Method Algorithm with Normalization 􏰖􏰗 integer k, kmax, n; real r real array ( A)1:n×1:n , (x)1:n , ( y)1:n external function φ output 0, x fork =1tokmax do y ← Ax r ← φ(y)/φ(x) x ← y/||y||∞ output k, x, r end do EXAMPLE 1 Solution Aitken Acceleration From a given sequence {rk }, we can construct another sequence {sk } by means of the Aitken acceleration formula (rk −rk−1)2 sk = rk − r − 2r + r (k 􏶟 3) k k−1 k−2 If the original sequence {rk } converges to r and if certain other conditions are satisfied, then the new sequence {sk } will converge to r more rapidly than the original one. (For details, see Kincaid and Cheney [2002].) Because subtractive cancellation may eventually spoil the results, the Aitken acceleration process should be stopped soon after the values become apparently stationary. Use the modified power method algorithm and Aitken acceleration to find the dominant eigenvalue and an eigenvector of the given matrix A, with vector x(0) and φ(x) given as follows: ⎡⎤⎡⎤ we obtain the following results: x(0) x (1) x (2) x (3) x (4) x (5) x (6) = [−1.0000, −0.4074, −0.4074]T r2 = = [−1.0000, −0.6049, −0.6049]T r3 = = [−1.0000, −0.7366, −0.7366]T r4 = = [−1.0000, −0.8244, −0.8244]T r5 = . . = [−1.0000, −0.9931, −0.9931]T r13 = 22.0000 8.9091 7.3061 6.7151 s3 = 13.5294 s4 = 7.0825 s5 = 6.3699 6 5 −5 A=⎣2 6 −2⎦, −1 x(0) =⎣ 1⎦, φ(x)=x2 2 5 −1 After coding and running the modified power method algorithm with Aitken acceleration, 1 =[−1.0000,1.0000,1.0000]T = [−1.0000, 0.3333, 0.3333]T r0 = 2.0000 = [−1.0000, −0.1111, −0.1111]T r1 = −2.0000 x (14) The Aitken-accelerated sequence, sk, converges noticeably faster than the sequence {rk}. 6.0208 The actual dominant eigenvalue and an associated eigenvector are λ1 =6 u(1) =[1,1,1]T ■ The coding of the modified power method is very simple, and we leave the actual imple- mentation as an exercise. We also use the simple infinity-norm for normalizing the vectors. The final vectors and estimates of the eigenvalue are displayed with 15 decimals digits. In such a problem, one should always seek an independent verification of the purported answer. Here, we simply compute Ax to see whether it coincides with s14x. The last few commands in the code are doing this rough checking, taking s14 as probably the best estimate of the eigenvalue and the last x-vector as the best estimate of an eigenvector. The results after 14 steps are not very accurate. For better accuracy, take 80 steps! 8.4 Power Method 363 . s13 = 6.0005 364 Chapter 8 Additional Topics Concerning Systems of Linear Equations EXAMPLE 2 Solution and the vector x(k+1) can be more easily computed by solving this last linear system. To do this, we first find the LU factorization of A, namely, A = LU. Then we repeatedly update the right-hand side and back solve: Ux(k+1) = L−1x(k) to obtain x(1), x(2), . . . . Compute the smallest eigenvalue and an associated eigenvector of the following matrix: ⎡⎤ 1 −154 528 407 A=3⎣ 55 −144 −121⎦ −132 396 318 using the following initial vector and linear function: x(0) = [1,2,3]T , φ(x) = x2 We decide to take the easy route and use the inverse of A for producing the successive x vectors. We leave the actual implementation as an exercise. The ratios rk are saved, and once it is complete, the Aitken accelerated values, sk , are computed. Notice that at the end, we will want the reciprocal of the limiting ratio. Hence, it is easier to use reciprocals at every step in the code. Thus, you see rk = x2/y2 rather than y2/x2, and these ratios should Inverse Power Method It is possible to compute other eigenvalues of a matrix by using modifications of the power method. For example, if A is invertible, we can compute its eigenvalue of smallest magnitude by noting this logical equivalence: Ax=λx ⇐⇒ x=A−1(λx) ⇐⇒ A−1x=1x λ Thus, the smallest eigenvalue of A in magnitude is the reciprocal of the largest eigenvalue of A−1. We compute it by applying the power method to A−1 and taking the reciprocal of the result. Suppose that there is a single smallest eigenvalue of A. With our usual ordering, this will be λn : |λ1|􏶟|λ2|􏶟|λ3|􏶟 ···􏶟|λn−1|>|λn|>0
It follows that A is invertible. (Why?) The eigenvalues of A−1 are λ−1 for 1 􏶞 j 􏶞 n. There-
fore, we have
|λ−1|>|λ−1 |􏶟 ···􏶟|λ−1|>0 n n−1 1
We can use the power method on the matrix A−1 to compute its dominant eigenvalue λ−1. n
The reciprocal of this is the eigenvalue of A that we sought. Notice that we need not compute A−1 because the equation
is equivalent to the equation
x(k+1) = A−1x(k) Ax(k+1) = x(k)
j

8.4 Power Method 365 converge to the smallest eigenvalue of A. The final results after 80 steps are these:
x = [0.26726101285547, −0.53452256017715, 0.80178375118802]T s80 = 3.33333333343344
We can divide each entry in x by the first component and arrive at
x = [1.0, −2.00000199979120, 3.00000266638827]T
The eigenvalue is actually 10 , and the eigenvector should be [1, −2, 3]T . The discrepancy 3
between Ax and s80 x is about 2.6 × 10−6. ■ Software Examples: Inverse Power Method
Using mathematical software on a small example,
⎡⎤
6 5 −5
A=⎣26 2⎦ (4)
2 5 −1
we can first get A−1 and then use the power method. (We have changed one entry in the matrix A from Example 1 to solve a different problem.) We leave the implementation of the code as an exercise. In the code, r is the reciprocal of the quantity r in the original power method. Thus, at the end of the computation, r should be the eigenvalue of A that has the smallest absolute value. After the prescribed 30 steps, we find that r = 0.214 and x = [0.7916, 0.5137, 0.3308]T . As usual, we can verify the result independently by computing Ax and rx, which should be equal. The method just illustrated is called the inverse power method. On larger examples, the successive vectors should be computed not via A−1 but rather by solving the equation A y = x for y. In mathematical software systems such as Matlab, Maple, and Mathematica, this can be done with a single command. Alternatively, one can get the LU factorization of A and solve Lz = x and U y = z.
In this example, two eigenvalues are complex. Since the matrix is real, they must be conjugate pairs of the form α + βi and α − βi. They have the same magnitude; thus, the hypothesis |λ1| > |λ2| needed in the convergence proof of the power method is violated. What happens when the power method is applied to A? The values of r for k = 26 to 30 are 0.76, −53.27, 8.86, 2.69, and −9.42. We leave the implementation of the code as a computer problem.
Shifted (Inverse) Power Method
Other eigenvalues of a matrix (besides the largest and smallest) can be computed by exploiting the following logical equivalences:
Ax=λx⇐⇒(A−μI)x=(λ−μ)x⇐⇒(A−μI)−1x= 1 x λ−μ
If we want to compute an eigenvalue of A that is close to a given number μ, we can apply the inverse power method to A − μ I and take the reciprocal of the limiting value of r . This should be λ − μ.

366 Chapter 8
Additional Topics Concerning Systems of Linear Equations
We can also compute an eigenvalue of A that is farthest from a given number μ. Suppose that for some eigenvalue λj of matrix A, we have
|λj −μ|>ε and 0<|λi −μ|<ε foralli ≠ j Consider the shifted matrix A − μI. Applying the power method to the shifted matrix A − μ I , we compute ratios rk that converge to λ j − μ. This procedure is called the shifted power method. If we want to compute the eigenvalue of A that is closest to a given number μ, a variant of the above procedure is needed. Suppose that λ j is an eigenvalue of A such that 0<|λj −μ|<ε and |λi −μ|>ε foralli ≠ j
Consider the shifted matrix A − μI. The eigenvalues of this matrix are λi − μ. Applying the inverse power method to A − μ I gives an approximate value for (λ j − μ)−1 . We can use the explicit inverse of A − μI or the LU factorization A − μI = LU. Now we repeatedly solve the equations
(A−μI)x(k+1) =x(k)
bysolvinginsteadUx(k+1) = L−1x(k).Sincetheratiosrk convergeto(λj −μ)−1,wehave
􏰒􏰓−1 1 λj=μ+ limrk =μ+lim
k→∞ k→∞ rk This algorithm is called the shifted inverse power method.
Example: Shifted Inverse Power Method
To illustrate the shifted inverse power method, we consider the following matrix:
⎡⎤
137
A=⎣2−4 5⎦ (5)
3 4 −6
and use mathematical software to compute the eigenvalue closest to −6. The code we use takes ratios of y2/x2, and we are therefore expecting convergence of these ratios to λ + 6. After eight steps, we have r = 0.9590 and x = [−0.7081, 0.6145, 0.3478]T . Hence, the eigenvalue should be λ = 0.9590 − 6 = −5.0410. We can ask Matlab to confirm the eigenvalue and eigenvector by computing both Ax and λx to be approximately [3.57, −3.10, −1.75]T .
Summary
(1) We have considered the following methods for computing eigenvalues of a matrix. In the power method, we approximate the largest eigenvalue λ1 by generating a sequence of points using the formula
x(k+1) = Ax(k)
and then forming a sequence rk = φ(x(k+1))/φ(x(k)), where φ is a linear functional. Under the right circumstances, this sequence, rk , will converge to the largest eigenvalue of A.

(2) In the inverse power method, we find the smallest eigenvalue λn by using the preceding process on the inverse of the matrix. The reciprocal of the largest eigenvalue of A−1 is the smallest eigenvalue of A. We can also describe this process as one of computing the sequence so that
Ax(k+1) = x(k)
(3) In the shifted power method, we find the eigenvalue that is farthest from a given number μ by seeking the largest eigenvalue of A − μI. This involves an iteration to produce a sequence
x(k+1) =(A−μI)x(k)
(4) In the shifted inverse power method, we find the eigenvalue that is closest to μ by
applying the inverse power method to A − μ I . This requires solving the equation (A−μI)x(k+1) =x(k) (A−μI=LU)
Additional References
For supplemental reading and study, see Anderson Bai, Bischof, Blackford, Demmel, Dongarra, Du Croz, Greenbaum, Hammarling, and McKenney [1999]; Axelsson [1994]; Bai, Demmel, Dongarra, Ruhe, and van der Vorst [2000]; Barrett, Berry, Chan, Demmel, Donato, Dongarra, Eijkhout, Pozo, Romine, and van der Vorst [1994]; Davis [2006]; Dekker and Hoffmann [1989]; Dekker, Hoffmann, and Potma [1997]; Demmel [1997]; Dongarra et al. [1990]; Elman, Silvester, and Wathen [2004]; Fox [1967]; Gautschi [1997]; Greenbaum [1997]; Hageman and Young [1981]; Heroux, Raghavan, and Simon [2006]; Jennings [1977]; Kincaid and Young [1979, 2000]; Lynch [2004]; Meurant [2006]; Noble and Daniel [1988]; Ortega [1990b]; Parlett [2000]; Saad [2003]; Schewchuck [1994]; Southwell [1946]; Stewart [1973]; Trefethen and Bau [1997]; Van der Vorst [2003]; Watkins [1991]; Wilkinson [1988]; and Young [1971].
􏰖􏰗
a1. Let A = 5 2 . The power method has been applied to the matrix A. The result is 47
a long list of vectors that seem to settle down to a vector of the form [h, 1]T , where |h| < 1. What is the largest eigenvalue, approximately, in terms of that number h? a. 4h+7 b. 5h+2 c. 1/h d. 5h+4 e. Noneofthese. 2. Whatistheexpectedfinaloutputofthefollowingpseudocode? integer n, kmax; real r real array ( A−1)1:n×1:n , (x)1:n , ( y)1:n fork =1to30do y ← A−1x r ← y1/x1 (first components of y and x) x ← y/||y|| output r, x end do 8.4 Power Method 367 Problems 8.4 368 Chapter 8 Additional Topics Concerning Systems of Linear Equations a. ristheeigenvalueofAlargestinmagnitude,andxisanaccompanyingeigenvector. b. r = 1/λ, where λ is the smallest eigenvalue of A, and x is such that Ax = λx. c. A vector x such that Ax = r x, where r is the eigenvalue of A having the smallest magnitude. d. r is the largest (in magnitude) eigenvalue of A and x is a corresponding eigen- vector of A. e. Noneofthese. 3. Briefly describe how to compute the following: a. Thedominanteigenvalueandassociateeigenvector. b. Thenextdominanteigenvalueandassociatedeigenvector. c. Theleastdominanteigenvalueandassociatedeigenvector. d. Aneigenvalueotherthanthedominantorleastdominanteigenvalueandassociated eigenvectors. ⎡⎤ 2 −1 0 4. Let A = ⎣ −1 2 −1 ⎦ Carry out several iterations of the power method, starting 0 −1 2 with x(0) = (1, 1, 1). What is the purpose of this procedure? ⎡⎤ −2 −1 0 5. Let B = A−4I = ⎣−1 −2 −1⎦. Carry out some iterations of the power method 0 −1 −2 applied to B, starting with x(0) = (1, 1, 1). What is the purpose of this procedure? ⎡⎤ 321 6. LetC = A−1 = 1 ⎣2 4 2⎦.Carryoutafewiterationsofthepowermethodapplied to C, starting with x(0) = (1, 1, 1). What is the purpose of this procedure? 7. The Rayleigh quotient is the expression ⟨x, x⟩A/⟨x, x⟩ = xT Ax/xT x. How can the Rayleigh quotient be used when Ax = λx? 1. Use the power method, the inverse power method, and their shifted forms as well as Aitken’s acceleration to find some or all of the eigenvalues of the following matrices: 4 123 ⎡5411⎤ ⎡234⎤ a.⎢4 5 1 1⎥ ⎣1 1 4 2⎦ 1124 ⎡⎤ −2 1 0 0 0 ⎢ 1 −2 1 0 0⎥ c.⎢0 1−2 1 0⎥ ⎣0 0 1−2 1⎦ b.⎣7 −1 3⎦ 1 −1 5 0 0 0 1 −2 Computer Problems 8.4 8.4 Power Method 369 2. Redotheexamplesinthissection,usingeitherMatlab,Maple,orMathematica. 3. Modify and test the pseudocode for the power method to normalize the vector so that the largest component is always 1 in the infinity-norm. This procedure gives the eigenvector and eigenvalue without having to compute a linear functional. 4. Findtheeigenvaluesofthematrix A=⎣ 20 −53 −44⎦ −48 144 115 that are close to −4, 2, and 8 by using the inverse power method. 5. UsingmathematicalsoftwaresuchasMatlab,Maple,orMathematica,writeandexecute code for implementing the methods in Section 8.4. Verify that the results are consistent with those described in the text. a. Example 1 using the modified power method. b. Example 2 using the inverse power method with Aitken acceleration. c. Matrix(4)usingtheinversepowermethod. d. Matrix (5) using the shifted power method. ⎡⎤ ⎢2⎥ 6. Considerthematrix A=⎣1 1 1 ⎦ 4 2 a. Use the normalized power method starting with x(0) = [1, 1, 1]T , and find the dominant eigenvalue and eigenvector of the matrix A. b. Repeat, starting with the initial value x(0) = [−0.64966116, 0, 74822116, 0]T . Explain the results. See Ralston [1965, p. 475–476]. ⎡⎤ −4 14 0 7. Let A = ⎣ −5 13 0 ⎦. Code and apply each of the following: −1 0 2 a. The modified power algorithm starting with x(0) = [1, 1, 1]T as well as the Aitken’s acceleration process. b. Theinversepoweralgorithm. c. Theshiftedpoweralgorithm. d. Theshiftedinversepoweralgorithm. ⎡⎤ 4 −1 1 8. (Continuation) Let B = ⎣ −1 3 −2 ⎦. Repeat the previous problem starting with 111 ⎡⎤ −57 192 148 11 24 x(0) =[1,0,0]T. 1 −2 3 ⎡⎤ −8 −5 8 9. (Continuation) Let C = ⎣ 6 3 −8 ⎦. Use x(0) = [1, 1, 1]T . Repeat the previous −3 1 9 problem starting with x(0) = [1, 0, 0]T . 370 Chapter 8 Additional Topics Concerning Systems of Linear Equations 10. Bymeansofthepowermethod,findaneigenvalueandassociatedeigenvectorofthese matrices from the historical books by Fox [1957] and Wilkinson [1965]. Verify your results by using mathematical software such as Matlab, Maple, or Mathematica. 􏰖􏰗 a. 0.9901 0.002 starting with x(0) = [1, 0.9]T −0.0001 0.9904 8 −1 −5 b. ⎣ −4 4 −2 ⎦ starting with x(0) = [1, 0.8, 1]T 18 −5 −7 ⎡⎤ 113 c. ⎣1 −2 1⎦ starting with x(0) = [1,1,1]T ⎡313⎤ −2 −1 4 d. ⎣ 2 1 −2 ⎦ starting with x(0) = [3, 1, 2]T without normalization and with −1 −1 3 normalization 11. FindalloftheeigenvaluesandassociatedeigenvectorsofthesematricesfromFox[1957] and Wilkinson [1965] by means of the power method and variations of it. Verify your ⎡⎤ results by using mathematical software such as Matlab, Maple, or Mathematica. a. c. e. 􏰖􏰗􏰖􏰗 2 1 b. 0.4812 0.0023 4 2 −0.0024 0.4810 ⎡⎤⎡⎤ 1 10 ⎣−1+10−8 3 0⎦ 0 11 ⎡⎤ 0.987 0.400 −0.487 ⎣ −0.079 0.500 −0.479 ⎦ 0.082 0.400 0.418 5−1−2 d. ⎣−1 3 −2⎦ −2−25 9 Approximation by Spline Functions FIGURE 9.1 By experimentation in a wind tunnel, an airfoil is constructed by trial and error so that it has certain desired characteristics. The cross section of the airfoil is then drawn as a curve on coordinate paper (see Figure 9.1). To study this airfoil by analytical methods or to manufacture it, it is essential to have a formula for this curve. To arrive at such a formula, one first obtains the coordinates of a finite set of points on the curve. Then a smooth curve called a cubic interpolating spline can be constructed to match these data points. This chapter discusses general polynomial spline functions and how they can be used in various numerical problems such as the data-fitting problem just described. y Airfoil cross section x 9.1 First-Degree and Second-Degree Splines The history of spline functions is rooted in the work of draftsmen, who often needed to draw a gently turning curve between points on a drawing. This process is called fairing and can be accomplished with a number of ad hoc devices, such as the French curve, made of plastic and presenting a number of curves of different curvature for the draftsman to select. Long strips of wood were also used, being made to pass through the control points by weights laid on the draftsman’s table and attached to the strips. The weights were called ducks and the strips of wood were called splines, even as early as 1891. The elastic nature of the wooden strips allowed them to bend only a little while still passing through the prescribed points. The wood was, in effect, solving a differential equation and minimizing the strain energy. The latter is known to be a simple function of the curvature. The mathematical theory of these curves owes much to the early investigators, particularly Isaac Schoenberg in the 1940s and 1950s. Other important names associated with the early development of the subject (i.e., prior to 1964) are Garrett Birkhoff, C. de Boor, J. H. Ahlberg, E. N. Nilson, 371 372 Chapter 9 Approximation by Spline Functions FIGURE 9.2 First-degree spline function x H. Garabedian, R. S. Johnson, F. Landis, A. Whitney, J. L. Walsh, and J. C. Holladay. The first book giving a systematic exposition of spline theory was the book by Ahlberg, Nilson, and Walsh [1967]. First-Degree Spline A spline function is a function that consists of polynomial pieces joined together with certain smoothness conditions. A simple example is the polygonal function (or spline of degree 1), whose pieces are linear polynomials joined together to achieve continuity, as in Figure9.2.Thepointst0,t1,...,tn atwhichthefunctionchangesitscharacteraretermed knots in the theory of splines. Thus, the spline function shown in Figure 9.2 has eight knots. Knots: forced to write where t1 t2 t3 t4 ⎧⎪ ⎪ ⎪ S 0 ( x ) ⎨S1(x) t5 t6 x ∈ [ t 0 , t 1 ] x ∈[t1,t2] t7 􏶾 b S0 S1 S2 S4 S3 S6 S5 a 􏶾 t0 Such a function appears somewhat complicated when defined in explicit terms. We are S(x)=⎪ . ⎪⎩. . . (1) Sn−1(x) x ∈ [tn−1,tn] Si (x) = ai x + bi (2) because each piece of S(x) is a linear polynomial. Such a function S(x) is piecewise linear. Iftheknotst0,t1,...,tn weregivenandifthecoefficientsa0,b0,a1,b1,...,an−1,bn−1 were all known, then the evaluation of S(x) at a specific x would proceed by first determining the interval that contains x and then using the appropriate linear function for that interval. If the function S defined by Equation (1) is continuous, we call it a first-degree spline. It is characterized by the following three properties. SPLINE OF DEGREE 1 A function S is called a spline of degree 1 if: 1. ThedomainofSisaninterval[a,b]. 2. Siscontinuouson[a,b]. 3. Thereisapartitioningoftheintervala=t0 b.
Continuity of a function f at a point s can be defined by the condition lim f(x)= lim f(x)= f(s)
x→s+ x→s−
Here, limx→s+ means that the limit is taken over x values that converge to s from above s; that is, (x − s) is positive for all x values. Similarly, limx→s− means that the x values converge to s from below.
Determine whether this function is a first-degree spline function:
9.1 First-Degree and Second-Degree Splines 373
⎧ ⎪⎨ x
S ( x ) = ⎪⎩ 1 − x 2x − 2
x ∈ [−1, 0] x ∈ ( 0 , 1 ) x ∈ [1, 2]
The function is obviously piecewise linear but is not a spline of degree 1 because it is discontinuous at x = 0. Notice that limx→0+ S(x) = limx→0(1 − x) = 1, whereas limx→0− S(x) = limx→0 x = 0. ■
The spline functions of degree 1 can be used for interpolation. Suppose the following table of function values is given:
x t0 t1 ··· tn
y y0 y1 ··· yn
There is no loss of generality in supposing that t0 < t1 < · · · < tn because this is only a matter of labeling the knots. The table can be represented by a set of n + 1 points in the plane, (t0, y0), (t1, y1), . . . , (tn , yn ), and these points have distinct abscissas. Therefore, we can draw a polygonal line through the points without ever drawing a vertical segment. This polygonal line is the graph of a function, and this function is obviously a spline of degree 1. What are the equations of the individual line segments that make up this graph? By referring to Figure 9.3 and using the point-slope form of a line, we obtain Si(x)=yi +mi(x−ti) (3) on the interval [ti , ti +1 ], where m i is the slope of the line and is therefore given by the formula mi = yi+1 − yi ti+1 −ti FIGURE 9.3 First-degree spline: linear Si (x) Si (x) (ti, yi) ti (ti􏶽1, yi􏶽1) x ti􏶽1 374 Chapter 9 Approximation by Spline Functions Notice that the function S that we are creating has 2n parameters in it: the n coefficients ai and the n constants bi in Equation (2). On the other hand, exactly 2n conditions are being imposed, since each constituent function Si must interpolate the data at the ends of its subinterval. Thus, the number of parameters equals the number of conditions. For the higher-degree splines, we shall encounter a mismatch in these two numbers; the spline of degree k will have k − 1 free parameters for us to use as we wish in the problem of interpolating at the knots. The form of Equation (3) is better than that of Equation (2) for the practical evaluation of S(x) because some of the quantities x − ti must be computed in any case simply to determinewhichsubintervalcontainsx.Ift0􏶞x􏶞tn thentheinterval[ti,ti+1]containing x is characterized by the fact that x −ti is the first of the quantities x −tn−1,x −tn−2,..., x − t0 that is nonnegative. The following is a function procedure that utilizes n + 1 table values (ti , yi ) in linear arrays(ti)and(yi),assumingthata=t0 tn, then S(x) = yn−1 + mn−1(x − tn−1).
real function Spline1(n, (ti ), (yi ), x)
integer i,n; real x; real array (ti)0:n,(yi)0:n fori =n−1to0 step−1do
ifx−ti 􏶟0thenexitloop end for
Spline1←yi +(x−ti)[(yi+1−yi)/(ti+1−ti)] end function Spline1
Modulus of Continuity
To assess the goodness of fit when we interpolate a function with a first-degree spline, it is useful to have something called the modulus of continuity of a function f . Suppose f is defined on an interval [a, b]. The modulus of continuity of f is
ω(f;h)=sup{|f(u)− f(v)|:a􏶞u􏶞v􏶞b,|u−v|􏶞h}
Here, sup is the supremum, which is the least upper bound of the given set of real numbers. The quantity ω( f ; h) measures how much f can change over a small interval of width h. If f is continuous on [a, b], then it is uniformly continuous, and ω( f ; h) will tend to zero as h tends to zero. If f is not continuous, ω( f ; h) will not tend to zero. If f is differentiable on (a, b) (in addition to being continuous on [a, b]) and if f ′(x) is bounded on (a, b), then the Mean Value Theorem can be used to get an estimate of the modulus of continuity: If u
and v are as described in the definition of ω( f ; h), then
|f(u)− f(v)|=|f′(c)(u−v)|􏶞M1|u−v|􏶞M1h
Here, M1 denotes the maximum of | f ′(x)| as x runs over (a, b). For example, if f (x) = x3 and [a, b] = [1, 4], then we find that ω( f ; h) 􏶞 48h.

Hence,
f(x)− p(x)= Then we have
􏰔x −a􏰕
b−a [f(x)− f(b)]+
􏰔b−x􏰕
b−a [f(x)− f(a)]
9.1 First-Degree and Second-Degree Splines 375
FIRST-DEGREE POLYNOMIAL ACCURACY THEOREM
If p is the first-degree polynomial that interpolates a function f at the endpoints of an interval [a, b], then with h = b − a, we have
| f (x) − p(x)| 􏶞 ω( f ; h) (a 􏶞 x 􏶞 b)
■ THEOREM1
Proof
The linear function p is given explicitly by the formula
􏰔x −a􏰕 􏰔b−x􏰕 p(x)= b−a f(b)+ b−a f(a)
􏰔 x − a 􏰕
b−a |f(x)− f(b)|+ b−a |f(x)− f(a)|
􏰔 b − x 􏰕 􏰔x −a􏰕 􏰔b−x􏰕
􏰖􏰔x−a􏰕 􏰔b−x􏰕􏰗
= b − a + b − a ω( f ; h) = ω( f ; h)
|f(x)− p(x)| 􏶞
􏶞 b−a ω(f;h)+ b−a ω(f;h)
■
FIRST-DEGREE SPLINE ACCURACY THEOREM
Let p be a first-degree spline having knots a = x0 < x1 <···0,letδhavethepropertythat|f(x)− f(y)|<εwhenever|x−y|<δ(uniformcontinuityprinciple).Letn >1+(b−a)/δ. Show that there is a first-degree spline S having n knots such that | f (x) − S(x)| < ε on [a, b]. Hint: Use Problem 5. Problems 9.1 382 Chapter 9 Approximation by Spline Functions a9. If the function f(x) = sin(100x) is to be approximated on the interval [0,π] by an interpolating spline of degree 1, how many knots are needed to ensure that |S(x) − f (x)| < 10−8? Hint: Use Problem 7. a10. Let t0 < t1 < ··· < tn. Construct first-degree spline functions G0,G1,...,Gn by requiring that Gi vanish at t0,t1,...,ti−1,ti+1,...,tn but that Gi(ti) = 1. Show that thefirst-degreesplinefunctionthatinterpolates f att0,t1,...,tn is􏰍n f(ti)Gi(x). i=0 11. Show that the trapezoid rule for numerical integration (Section 5.2) results from approximating f by a first-degree spline S and then using 􏶩b 􏶩b f(x)dx ≈ S(x)dx aa a12. Provethatthederivativeofaquadraticsplineisafirst-degreespline. 13. Iftheknotsti happentobetheintegers0,1,...,n,findagoodwaytodeterminethe index i for which ti 􏶞 x < ti+1. (Note: This problem is deceptive, for the word good can be given different meanings.) 14. Show that the indefinite integral of a first-degree spline is a second-degree spline. 15. Define f(x)=0ifx <0and f(x)=x2 ifx􏶟0.Showthat f and f′ arecontinuous. Showthatanyquadraticsplinewithknotst0,t1,...,tn isoftheform n−1 􏰃 i=1 􏰎 g(x) = Provethateveryfirst-degreesplinefunctionthathasknotst0,t1,...,tn canbewritten in the form n−1 􏰃 ci g(x − ti ) a17. Findaquadraticsplineinterpolantforthesedata: x −1 0 1 1 2 5 22 y210123 Assume that z0 = 0. 18. (Continuation)ShowthatnoquadraticsplineQinterpolatesthetableofthepreceding problem and satisfies Q′(t0) = Q′(t5). a19. What equations must be solved if a quadratic spline function Q that has knots t0, t1,...,tn isrequiredtotakeprescribedvaluesatpoints 1(ti +ti+1)for0􏶞i􏶞n−1? 2 ax2+bx+c+ 16. Define a function g by the equation di f(x−ti) ax + b + 0 (t0 􏶞 x 􏶞 0) (0 􏶞 x 􏶞 tn ) x i=1 9.1 First-Degree and Second-Degree Splines 383 20. Arethesefunctionsquadraticsplines?Explainwhyorwhynot. 􏰎 aa. Q(x)= 􏰎 ab. Q(x)= ⎧ ⎪⎨ x ac. Q(x)=⎪⎩x2 0.1x2 9.3x2 −18.4x+9.2 (0 􏶞 x 􏶞 1) (1􏶞x􏶞1.3) −x2 x (−100 􏶞 x 􏶞 0) (0 􏶞 x 􏶞 100) (−50 􏶞 x 􏶞 1) (1􏶞x􏶞2) (2 􏶞 x 􏶞 50) 22. VerifythatFormula(5)hasthethreepropertiesQi(ti)=yi,Qi′(ti)=zi,andQi′(ti+1)= zi+1. 23. (Continuation) Impose the continuity condition on Q and derive the system of Equa- tion (6). 24. ShowbyinductionthattherecursiveFormula(6)togetherwithEquation(5)produces an interpolating quadratic spline function. 25. Verify the correctness of the equations in the text that pertain to Subbotin’s spline interpolation process. 26. Analyze the Subbotin interpolation scheme in this alternative manner. First, let vi = Q(ti ). Show that where 27. (Continuation) When continuity conditions on Q′ are imposed, show that the result is the following equation, in which i = 1,2,...,n − 1: hi vi−1 + 3(hi + hi+1)vi + hi−1vi+1 = 4hi−1 yi + 4hi yi−1 28. (Student research project) It is commonly accepted that Schoenberg’s [1946] paper is the first mathematical reference in which the word spline is used in connection with smooth, piecewise polynomial approximations. However, the word spline as a thin strip of wood used by a draftsman dates back to the 1890s at least. Many of the ideas used in spline theory have their roots in work done in various industries such as the building of aircraft, automobiles, and ships in which splines are used extensively. Research and write a paper on the history of splines. (See books on mathematical history. For a discussion of the history of splines in the automobile industry, see the NA Digest, Volume 98, Issue 26, July 19, 1998.) 4 a21. Is S(x) = |x| a first-degree spline? Why or why not? Qi(x)= Ai(x−ti)2 +Bi(x−ti+1)2 +Ci Ci =2yi −1vi −1vi+1, Bi =vi −Ci 22 hi2 Ai = vi+1 −Ci hi =ti+1 −ti hi2 Hint:Showthat Qi(ti)=vi,Qi(ti+1)=vi+1,and Qi(τi)= yi. 384 Chapter 9 Approximation by Spline Functions 1. Rewrite procedure Spline1 so that ascending subintervals are considered instead of descending ones. Test the code on a table of 15 unevenly spaced data points. 2. RewriteprocedureSpline1sothatabinarysearchisusedtofindthedesiredinterval. Test the revised code. What are the advantages and/or disadvantages of a binary search compared to the procedure in the text? A binary search is similar to the bisection method inthatwechoosetk withk=(i+j)/2ork=(i+j+1)/2anddeterminewhether x is in [ti,tk] or [tk,tj]. 3. Apiecewisebilinearpolynomialthatinterpolatespoints(x,y)specifiedinarectan- gular grid is given by p(x,y)= (lijzi+1, j+1 +li+1, j+1zij)−(li+1, jzi, j+1 +li, j+1zi+1, j) (xi+1 −xi)(yj+1 −yj) wherelij =(xi −x)(yj −y).Herexi􏶞x􏶞xi+1 andyj􏶞y􏶞yj+1.Thegivengrid (xi,yj) is specified by strictly increasing arrays (xi) and (yj) of length n and m, respectively.Thegivenvalueszij atthegridpoints(xi,yj)arecontainedinthen×m array (zi j ), shown in the figure below. Write real function Bi Linear((xi ), n, (yj ), m, (zi j ), x, y) to compute the value of p(x, y). Test this routine on a set of 5 × 10 unequally spaced data points. Evaluate Bi Linear at four grid points and five nongrid points. zij yj yj􏶽1 xi xi􏶽1 4. Write an adaptive spline interpolation procedure. The input should be a function f , an interval [a,b], and a tolerance ε. The output should be a set of knots a = t0 < t1 < ··· < tn = b and a set of function values yi = f(ti) such that the first-degree spline interpolating function S satisfies |S(x) − f (x)| 􏶞 ε whenever x is any point xij =ti + j(ti+1 −tj)/10for0􏶞i􏶞n−1and0􏶞 j􏶞9. 5. Write procedure Spline2 Coef (n, t, (yi ), (zi )) that computes the (zi ) array in the quadratic spline interpolation process (interpolation at the knots). Then write real function Spline2 Eval(n, (ti ), (yi ), (zi ), x) that computes values of Q(x). 6. CarryouttheprogrammingprojectoftheprecedingcomputerproblemfortheSubbotin quadratic spline. Computer Problems 9.1 9.2 Natural Cubic Splines Introduction The first- and second-degree splines discussed in the preceding section, though useful in certain applications, suffer an obvious imperfection: Their low-order derivatives are discon- tinuous. In the case of the first-degree spline (or polygonal line), this lack of smoothness is immediately evident because the slope of the spline may change abruptly from one value to another at each knot. For the quadratic spline, the discontinuity is in the second derivative and is therefore not so evident. But the curvature of the quadratic spline changes abruptly at each knot, and the curve may not be pleasing to the eye. The general definition of spline functions of arbitrary degree is as follows. 9.2 Natural Cubic Splines 385 SPLINE OF DEGREE k A function S is called a spline of degree k if: 1. ThedomainofSisaninterval[a,b]. 2. S, S′, S′′, . . . , S(k−1) are all continuous functions on [a, b]. 3. Therearepointsti (theknotsofS)suchthata=t0 h1 >0.Ifui−1 >hi−1, then ui > hi because
ui =2(hi +hi−1)− hi2−1 >2(hi +hi−1)−hi−1 >hi ui−1
Then by induction, ui > 0 for i = 1,2,…,n − 1.
Equation (5) is not the best computational form for evaluating the cubic polynomial
Si (x ). We would prefer to have it in the form
Si(x)= Ai +Bi(x−ti)+Ci(x−ti)2 +Di(x−ti)3 (11)
because nested multiplication can then be utilized.
Notice that Equation (11) is the Taylor expansion of Si about the point ti . Hence,
A =S(t), B =S′(t), C = 1S′′(t), D = 1S′′′(t) iii iii i2ii i6ii
Therefore, Ai = yi and Ci = zi/2. The coefficient of x3 in Equation (11) is Di, whereas the coefficient of x3 in Equation (5) is (zi+1 − zi )/6hi . Therefore,
Di= 1(zi+1−zi) 6hi

392 Chapter 9
Approximation by Spline Functions
Finally, Equation (6) provides the value of Si′ (ti ), which is
Bi =−hizi+1−hizi + 1(yi+1−yi)
6 3 hi
􏰔 􏰔zi 1 􏰕􏰕
Thus, the nested form of Si (x ) is
Si(x)=yi+(x−ti) Bi+(x−ti) 2+6h(x−ti)(zi+1−zi) (12)
i
We now write routines for determining a natural cubic spline based on a table of values and for evaluating this function at a given value. First, we use Algorithm 1 for directly solving the tridiagonal System (10). This procedure, called Spline3 Coef , takes n + 1 table values (ti , yi ) in arrays (ti ) and (yi ) and computes the zi ’s, storing them in array (zi ). Intermediate (working) arrays (hi ), (bi ), (ui ), and (vi ) are needed.
Pseudocode for Natural Cubic Splines
procedure Spline3 Coef (n, (ti ), (yi ), (zi ))
integer i, n; real array (ti )0:n , (yi )0:n , (zi )0:n
allocate real array (hi )0:n−1, (bi )0:n−1, (ui )1:n−1, (vi )1:n−1 fori =0ton−1do
hi ←ti+1−ti
bi ←(yi+1−yi)/hi end for
u1 ← 2(h0 + h1)
v1 ← 6(b1 − b0) fori =2ton−1do
ui ← 2(hi + hi−1) − hi2−1/ui−1
vi ← 6(bi − bi−1) − hi−1vi−1/ui−1 end for
zn ←0
fori =n−1to1 step−1do
zi ←(vi −hizi+1)/ui end for
z0 ← 0
deallocate array (hi ), (bi ), (ui ), (vi ) end procedure Spline3 Coef
Now a procedure called Spline3 Eval is written for evaluating Equation (12), the natural cubic spline function S(x), for x a given value. The procedure Spline3 Eval first determines the interval [ti , ti+1] that contains x and then evaluates Si (x) using the nested form of this cubic polynomial:
real function Spline3 Eval(n, (ti ), (yi ), (zi ), x) integer i; real h,tmp
real array (ti )0:n , (yi )0:n , (zi )0:n
fori =n−1to0 step−1do
ifx−ti 􏶟0thenexitloop

9.2 Natural Cubic Splines 393
end for
h ← ti+1 − ti
tmp←(zi/2)+(x−ti)(zi+1 −zi)/(6h) tmp←−(h/6)(zi+1 +2zi)+(yi+1 −yi)/h+(x−ti)(tmp) Spline3 Eval ← yi + (x − ti )(tmp)
end function Spline3 Eval
EXAMPLE 3
Solution
The function Spline3 Eval can be used repeatedly with different values of x after one call to procedure Spline3 Coef . For example, this would be the procedure when plotting a natural cubic spline curve. Since procedure Spline3 Coef stores the solution of the tridiagonal sys- tem corresponding to a particular spline function in the array (zi ), the arguments n, (ti ), (yi ), and (zi ) must not be altered between repeated uses of Spline3 Eval.
Using Pseudocode for Interpolating and Curve Fitting
To illustrate the use of the natural cubic spline routines Spline3 Coef and Spline3 Eval, we rework an example from Section 4.1.
Write pseudocode for a program that determines the natural cubic spline interpolant for sin x at ten equidistant knots in the interval [0, 1.6875]. Over the same interval, subdivide each subintervalintofourequallyspacedparts,andfindthepointwherethevalueof|sinx−S(x)| is largest.
Here is a suitable pseudocode main program, which calls procedures Spline3 Coef and Spline3 Eval:
procedure Test Spline3 integeri; reale,h,x
real array (ti )0:n , (yi )0:n , (zi )0:n integer n ← 9
real a ← 0, b ← 1.6875
h ← (b − a)/n
for i = 0 to n do
ti ←a+ih
yi ←sin(ti) end for
call Spline3 Coef (n, (ti ), (yi ), (zi )) temp ← 0
for j = 0 to 4n do
x ← a + jh/4
e ← | sin(x) − Spline3 Eval(n, (ti ), (yi ), (zi ), x)| if e > temp then temp ← e
output j,x,e
end for
end Test Spline3
From the computer, the output is j = 19, x = 0.890625, and d = 0.930 × 10−5. ■

394
Chapter 9
Approximation by Spline Functions
We can use mathematical software such as in Matlab to plot the cubic spline curve for this data, but the Matlab routine spline uses the not-a-knot end condition, which is different from the natural end condition. It dictates that S′′′ be a single constant in the first two subintervals and another single constant in the last two subintervals. First, the original data are generated. Next, a finer subdivision of the interval [a, b] on the x-axis is made, and the corresponding y-values are obtained from the procedure spline. Finally, the original data points and the spline curve are plotted.
We now illustrate the use of spline functions in fitting a curve to a set of data. Consider the following table:
x 0.0 0.6 1.5 1.7 1.9 2.1 2.3 2.6 2.8 3.0
y −0.8 −0.34 0.59 0.59 0.23 0.1 0.28 1.03 1.5 1.44
3.6 4.7 5.2 5.7 5.8 6.0 6.4 6.9 7.6 8.0 0.74 −0.82 −1.27 −0.92 −0.92 −1.04 −0.79 −0.06 1.0 0.0
These 20 points were selected from a wiggly freehand curve drawn on graph paper. We intentionally selected more points where the curve bent sharply and sought to reproduce the curve using an automatic plotter. A visually pleasing curve is provided by using the cubic spline routines Spline3 Coef and Spline3 Eval. Figure 9.8 shows the resulting natural cubic spline curve.
y
2 1.5 1 0.5
y 􏶾 S(x)
FIGURE 9.8
Natural cubic spline curve
0x 12345678
– 0.5 –1 –1.5 –2
Alternatively, we can use mathematical software such as Matlab, Maple, or Mathemat- ica to plot the cubic spline function for this table.
Space Curves
In two dimensions, two cubic spline functions can be used together to form a parametric representation of a complicated curve that turns and twists. Select points on the curve and

x0
x1
···
EXAMPLE 4
9.2 Natural Cubic Splines 395 label them t = 0,1,…,n. For each value of t, read off the x- and y-coordinates of the
point, thus producing a table:
t01···n x xn y y0 y1 ··· yn
Then fit x = S(t) and y = S(t), where S and S are natural cubic spline interpolants. The two functions S and S give a parametric representation of the curve. (See Computer Problem 9.2.6.)
Select 13 points on the well-known serpentine curve given by y=x
1/4 + x2
So that the knots will not be equally spaced, write the curve in parametric form:
􏰎
x = 1 tanθ 2
y = sin 2θ
and take θ = i (π/12), where i = −6, −5, . . . , 5, 6. Plot the natural cubic spline curve and
the interpolation polynomial in order to compare them.
This is example of curve fitting using both the polynomial interpolation routines Coef and Eval from Chapter 4 and the cubic spline routines Spline3 Coef and Spline3 Eval. Figure 9.9 shows the resulting cubic spline curve and the high-degree polynomial curve (dashed line) from an automatic plotter. The polynomial becomes extremely erratic after the fourth knot from the origin and oscillates wildly, whereas the spline is a near perfect fit.
Solution
–1
y
8 6 4 2
–2 –4 –6 􏶼8
Polynomial curve
Cubic spline curve
x
–2 –1.5
–0.5
0 0.5
1 1.5 2
FIGURE 9.9
Serpentine curve
■

396
Chapter 9
Approximation by Spline Functions
EXAMPLE 5
Solution
Use cubic spline functions to produce the curve for the following data: t01234567
y 1.0 1.5 1.6 1.5 0.9 2.2 2.8 3.1 It is known that the curve is continuous but its slope is not.
A single cubic spline is not suitable. Instead, we can use two cubic spline interpolants, the first having knots 0, 1, 2, 3, 4 and the second having knots 4, 5, 6, 7. By carrying out two separate spline interpolation procedures, we obtain two cubic spline curves that meet at the point (4, 0.9). At this point, the two curves have different slopes. The resulting curve is shown in Figure 9.10.
y
3 2.5 2 1.5 1 0.5
y 􏶾 Sˆ ( x )
̃
y 􏶾 S(x)
FIGURE 9.10
Two cubic splines
0x 1234567
Smoothness Property
■
Why do spline functions serve the needs of data fitting better than ordinary polynomials? To answer this, one should understand that interpolation by polynomials of high degree is often unsatisfactory because polynomials may exhibit wild oscillations. Polynomials are smooth in the technical sense of possessing continuous derivatives of all orders, whereas in this sense, spline functions are not smooth.
Wild oscillations in a function can be attributed to its derivatives being very large. Consider the function whose graph is shown in Figure 9.11. The slope of the chord that
p
q
r
FIGURE 9.11
Wildly oscillating function

■ THEOREM1
Proof
To verify the assertion about [S′′(x)]2, we let
g(x)= f(x)−S(x)
so that g(ti ) = 0 for 0 􏶞 i 􏶞 n, and
joins the points p and q is very large in magnitude. By the Mean-Value Theorem, the slope of that chord is the value of the derivative at some point between p and q. Thus, the derivative must attain large values. Indeed, somewhere on the curve between p and q, there is a point where f ′(x) is large and negative. Similarly, between q and r, there is a point where f ′(x) is large and positive. Hence, there is a point on the curve between p and r where f ′′(x) is large. This reasoning can be continued to higher derivatives if there are more oscillations. This is the behavior that spline functions do not exhibit. In fact, the following result shows that from a certain point of view, natural cubic splines are the best functions to use for curve fitting.
Now
f ′′ = S′′ + g′′ 􏶩b􏶩b􏶩b􏶩b
(f′′)2 dx = (S′′)2 dx + (g′′)2 dx +2 aaaa
If the last integral were 0, we would be finished because then
􏶩b􏶩b􏶩b􏶩b ( f ′′)2 dx = (S′′)2 dx + (g′′)2 dx 􏶟
(S′′)2 dx
We apply the technique of integration by parts to the integral in question to show that it
is 0.∗ We have
∗The formula for integration by parts is
aaaa
􏶩b 􏶨􏶨b 􏶩b 􏶩b
S′′g′′ dx = S′′g′􏶨 − S′′′g′ dx = − S′′′g′ dx
aaaa
9.2 Natural Cubic Splines 397
CUBIC SPLINE SMOOTHNESS THEOREM
If S is the natural cubic spline function that interpolates a twice-continuously differ- entiablefunction f atknotsa=t0 2hi + 3hi−1 in the algorithm for determining the cubic spline 2
interpolant.
a32. Byhandcalculation,findthenaturalcubicsplineinterpolantforthistable:
x12345
y01010
a33. Find a cubic spline over knots −1,0, and 1 such that the following conditions are
satisfied: S′′(−1) = S′′(1) = 0, S(−1) = S(1) = 0, and S(0) = 1.
34. This problem and the next two lead to a more efficient algorithm for natural cubic spline interpolation in the case of equally spaced knots. Let hi = h in Equation (5), andreplacetheparameterszi byqi =h2zi/6.ShowthatthenewformofEquation(5) is then
􏰔 x − t 􏰕3 􏰔 t − x 􏰕3 􏰔 x − t 􏰕 S(x)=q i+qi+1 +(y−q)i
i i+1 h i h i+1 i+1 h 􏰔ti+1 −x􏰕
+(yi −qi) h
35. (Continuation) Establish the new continuity conditions:
q0 =qn =0 qi−1+4qi +qi+1 =yi+1−2yi +yi−1 (1􏶞i􏶞n−1)
36. (Continuation) Show that the parameters qi can be determined by backward recursion as follows:
qn =0 qn−1 =βn−1 qi =αiqi+1 +βi (i =n−2,n−3,…,0)

wherethecoefficientsαi andβi aregeneratedbyascendingrecursionfromtheformulas α0 = 0 αi = −(αi−1 + 4)−1 (1 􏶞 i 􏶞 n)
β0=0 βi =−αi(yi+1−2yi+yi−1−βi−1) (1􏶞i􏶞n)
(This stable and efficient algorithm is due to MacLeod [1973].)
37. Prove that if S(x) is a spline of degree k on [a, b], then S′(x) is a spline of degree k −1.
a38. Howmanycoefficientsareneededtodefineapiecewisequartic(fourth-degree)function with n + 1 knots? How many conditions will be imposed if the piecewise quartic function is to be a quartic spline? Justify your answers.
a39. Determinewhetherthisfunctionisanaturalcubicspline:
􏰎
x3 + 3×2 + 7x − 5 (−1 􏶞 x 􏶞 0) −x3 + 3×2 + 7x − 5 (0 􏶞 x 􏶞 1)
x3 + x − 1 (0 􏶞 x 􏶞 1) −(x −1)3 +3(x −1)2 +4(x −1)+1 (1 􏶞 x 􏶞 2)
41. Showthatthenaturalcubicsplinegoingthroughthepoints(0,1),(1,2),(2,3),(3,4), and (4, 5) must be y = x + 1. (The natural cubic spline interpolant to a given data set is unique, because the matrix in Equation (10) is diagonally dominant and nonsingular, as proven in Section 7.3.)
1. Rewrite and test procedure Spline3 Coef using procedure Tri from Chapter 7. Use the symmetry of the (n − 1) × (n − 1) tridiagonal system.
2. Theextrastoragerequiredinstep1ofthealgorithmforsolvingthenaturalcubicspline tridiagonal system directly can be eliminated at the expense of a slight amount of extra computation—namely, by computing the hi ’s and bi ’s directly from the ti ’s and yi ’s in the forward elimination phase (step 2) and in the back substitution phase (step 3). Rewrite and test procedure Spline3 Coef using this idea.
3. Using at most 20 knots and the cubic spline routines Spline3 Coef and Spline3 Eval, plot on a computer plotter an outline of your:
a. school’smascot. b. signature. c. profile.
4. Let S be the cubic spline function that interpolates f (x) = (x2 + 1)−1 at 41 equally spacedknotsintheinterval[−5,5].EvaluateS(x)− f(x)at101equallyspacedpoints on the interval [0, 5].
5. Draw a free-form curve on graph paper, making certain that the curve is the graph of a function. Then read values of your function at a reasonable number of points, say, 10–50, and compute the cubic spline function that takes those values. Compare the freely drawn curve to the graph of the cubic spline.
S(x) =
40. Determine whether this function is or is not a natural cubic spline having knots 0, 1,
and 2:
􏰎
f (x) =
9.2 Natural Cubic Splines 403
Computer Problems 9.2

404
Chapter 9
Approximation by Spline Functions
6. Drawaspiral(orothercurvethatisnotafunction)andreproduceitbywayofparametric spline functions. (See the figure below.)
y
7
3
9.3
7. Writeandtestproceduresthatareassimpleaspossibletoperformnaturalcubicspline interpolation with equally spaced knots. Hint: See Problems 9.3.34–9.3.36.
8. Write a program to estimate 􏶧 b f (x) dx, assuming that we know the values of f at a
only certain prescribed knots a = t0 < t1 < ··· < tn = b. Approximate f first by an interpolating cubic spline, and then compute the integral of it using Equation (5). 9. Write a procedure to estimate f ′(x) for any x in [a, b], assuming that we know only thevaluesof f atknotsa=t0 0 x ∈ ( t i , t i + k + 1 ) ( k 􏶟 0 )
By Equation (2), this assertion is true when k = 0. If it is true for index k − 1, then Bk−1(x) > 0 on (t , t ) and Bk−1(x) > 0 on (t , t ). In Equation (3), the factors that
i i i+k i+1 i+1 i+k+1
multiply Bk−1(x) and Bk−1(x) are positive when t < x < t . Thus, Bk(x) > 0 on this
i i+1 i i+k+1
Figure 9.14 shows the first four B splines plotted on the same axes.
i
y
1 B0i B1i
B 2i
B 3i
FIGURE 9.14
First four B-splines
x
ti􏶽3 ti􏶽4
kth-degree splines that have the same knot sequence. Thus, linear combinations
􏰃∞
ci Bik
i =−∞
are important objects of study. (We use ci for fixed k and Cik to emphasize the degree k of the corresponding B splines.) Our first task is to develop an efficient method to evaluate a function of the form
ti ti􏶽1 ti􏶽2
The principal use of the B splines Bik(i = 0,±1,±2,…) is as a basis for the set of all
f (x) in the form
f (x) = Ck i
i ti+k −ti
Bk−1(x) + i
i+k+1 Bk−1(x)
􏰃∞ i =−∞
Cik Bik(x) (4) Using Definition (3) and some simple series manipulations, we have
f (x) =
under the supposition that the coefficients Cik are given (as well as the knot sequence ti ).
􏰃∞ 􏰖􏰔x−t􏰕 􏰔t −x􏰕 􏰗
i=−∞􏰖 􏰔
􏰃∞ x−ti ti+k−x
􏰕􏰗
= Ck
+Ck
i−1 ti+k −ti
Bk−1(x) i
i =−∞ 􏰃∞
i ti+k −ti Ck−1 Bk−1(x)
􏰕
􏰔
=
where Ck−1 is defined to be the appropriate coefficient from the line preceding Equation (5).
ii i =−∞
(5)
i
This algebraic manipulation shows how a linear combination of Bik (x ) can be expressed
as a linear combination of Bk−1(x). Repeating this process k−1 times, we eventually express i
f (x) =
􏰃∞ i =−∞
Ci0 Bi0(x) (6)
ti+k+1 −ti+1
i+1

408 Chapter 9
Approximation by Spline Functions
If t 􏶞x < t , then f(x) = C0. The formula by which the coefficients Cj−1 are ob- mm+1m i tained is 􏰔x−t􏰕 􏰔t −x􏰕 Cj−1 =Cj i +Cj i+j (7) i it−ti−1t−t i+j i i+j i A nice feature of Equation (4) is that only the k + 1 coefficients Cmk , Cmk −1, . . . , Cmk −k are needed to compute f (x) if tm 􏶞 x < tm+1 (see Problem 9.3.6). Thus, if f is defined by Equation (4) and we want to compute f (x), we use Equation (7) to calculate the entries in the following triangular array: Ck Ck−1···C0 mmm ... Although our notation does not show it, the coefficients in Equation (4) are independent of x, whereas the C j−1’s calculated subsequently by Equation (7) do depend on x. It is now a simple matter to establish that 􏰃∞ Bik(x) = 1 for all x and all k 􏶟 0 i =−∞ Ifk = 0,wealreadyknowthis.Ifk > 0,weuseEquation(4)withCik = 1foralli.
By Equation (7), all subsequent coefficients Ck,Ck−1,Ck−2,…,C0 are also equal to 1 iiii
(induction is needed here!). Thus, at the end, Equation (6) is true with Ci0 = 1, and so f (x) = 1. Therefore, from Equation (4), the sum of all B splines of degree k is unity.
The smoothness of the B splines Bik increases with the index k. In fact, we can show by induction that Bik has a continuous k − 1st derivative.
The B splines can be used as substitutes for complicated functions in many mathematical situations. Differentiation and integration are important examples. A basic result about the derivatives of B splines is
i
Ck Ck−1 m−1 m−1
. … C mk − k
d 􏰔k􏰕 􏰔k􏰕
Bk(x) = Bk−1(x) − Bk−1(x)
(8)
dx i ti+k −ti i ti+k+1 −ti+1 i+1
This equation can be proved by induction using the recursive Formula (3). Once Equation (8)
is established, we get the useful formula
where
d􏰃∞ 􏰃∞
c Bk(x) = d Bk−1(x) (9)
dxii ii i =−∞ i =−∞
􏰔c−c 􏰕 di=ki i−1
ti+k −ti

The verification is as follows. By Equation (8),
d 􏰃∞
dx
ci Bik(x)
􏰃∞ d k
9.3 B Splines: Interpolation and Approximation 409
i =−∞
= ci dx Bi (x)
i=−∞􏰖􏰔 􏰕 􏰔 􏰕 􏰗 􏰃∞ k k−1 k k−1
= ci t −t Bi (x)− t −t Bi+1(x) i=−∞ i+k i i+k+1 i+1
= =
ti+k −ti d Bk−1(x)
−
Bk−1(x) ti+k −ti i
􏰃∞ 􏰖􏰔 cik 􏰕 􏰔ci−1k􏰕􏰗
i=−∞ 􏰃∞
ii i =−∞
For numerical integration, the B splines are also recommended, especially for indefinite integration. Here is the basic result needed for integration:
􏶩x 􏰔t −t􏰕􏰃∞
Bk(s)ds = i+k+1 i Bk+1(x) (10)
−∞i k+1j j=i
This equation can be verified by differentiating both sides with respect to x and simplifying by the use of Equation (9). To be sure that the two sides of Equation (10) do not differ by a constant, we note that for any x < ti , both sides reduce to zero. The basic result (10) produces this useful formula: where 􏶩 x 􏰃∞ 􏰃∞ c Bk(s)ds = e Bk+1(x) (11) ii ii −∞ i=−∞ i=−∞ 1 􏰃i ei =k+1 It should be emphasized that this formula gives an indefinite integral (antiderivative) of any function expressed as a linear combination of B splines. Any definite integral can be obtained by selecting a specific value of x . For example, if x is a knot, say, x = tm , then 􏶩 t m 􏰃∞ 􏰃∞ cBk(s)ds= cj(tj+k+1−tj) j =−∞ 􏰃m ii iim iim eBk+1(t )= −∞ i=−∞ i=−∞ i=m−k−1 eBk+1(t ) Matlab has a Spline Toolbox, developed by Carl de Boor, that can be used for many tasks involving splines. For example, there are routines for interpolating data by splines with diverse end conditions and routines for least-squares fits to data. There are many demon- stration routines in this Toolbox that exhibit plots and provide models for programming Matlab M-files. These demonstrations are quite instructive for visualizing and learning the concepts in spline theory, especially B splines. Maple has a BSpline package for constructing B spline basis functions of degree k from a given knot list, which may include multiple knots. It is based on a divided-difference 410 Chapter 9 Approximation by Spline Functions implementation found in Bartels, Beatty, and Barskey [1987]. It can be downloaded from the Maple Application Center at www.maplesoft.com. Interpolation and Approximation by B Splines We developed a number of properties of B splines and showed how B splines are used in various numerical tasks. The problem of obtaining a B spline representation of a given function was not discussed. Here, we consider the problem of interpolating a table of data; later, a noninterpolatory method of approximation is described. A basic question is how to determine the coefficients in the expression S(x) = i =−∞ so that the resulting spline function interpolates a prescribed table: We mean by interpolate that x t0 t1 ··· tn y y0 y1 ··· yn S(ti ) = yi (0 􏶞 i 􏶞 n) (13) 􏰃∞ i i−k The natural starting point is with the simplest splines, corresponding to k = 0. Since 􏰘 has the interpolation property (13). y B1 (x) has the interpolation property (13). So Ai = yi again. Bi0(tj)=δij = 1 (i = j) 0 (i≠j) the solution to the problem is immediate: Just set Ai = yi for 0 􏶞 i 􏶞 n. All other coefficients in Equation (12) are arbitrary. In particular, they can be zero. We arrive then at this result: The zero-degree B spline B1 (t)=δ i−1j ij Hence, the following is true: The first-degree B spline i=0 inturn,requirefortheirdefinitionknotst−1,t0,t1,...,t4.Knotst−1 andt4 canbearbitrary. Figure 9.15 shows the graphs of the four B1-splines. In such a problem, if t−1 and t4 are not prescribed, it is natural to define them in such a way that t0 is the midpoint of the interval [t−1, t1] and t3 is the midpoint of [t2, t4]. In both elementary cases considered, the unknown coefficients A0, A1, . . . , An in Equation (12) were uniquely determined by the interpolation conditions (13). If terms were yi Bi0(x) The next case, k = 1, also has a simple solution. We use the fact that S(x) = If the table has four entries (n = 3), for instance, we use B1 S(x) = 􏰃n i=0 A Bk (x) (12) 􏰃n i i−1 , B1, B1, and B1. They, −101 2 FIGURE 9.15 Bi1 splines B1􏶼1 B10 B1 B12 t􏶼1 t0 t1 t2 t3 t4 x 9.3 B Splines: Interpolation and Approximation 411 present in Equation (12) corresponding to values of i outside the range {0, 1, . . . , n}, then they would have no influence on the values of S(x) at t0, t1, ...,tn. For higher-degree splines, we shall see that some arbitrariness exists in choosing coefficients. In fact, none of the coefficients is uniquely determined by the interpolation conditions. This fact can be advantageous if other properties are desired of the solution. In the quadratic case, we begin with the equation 􏰃∞ 1􏰠 􏰡 AB2 (t)= i i−2 j t −t j+1 A(t −t)+A (t−t ) (14) j j+1 j j+1 j j−1 j−1 i=−∞ Its justification is left to Problem 9.3.26. If the interpolation conditions (13) are now imposed, we obtain the following system of equations, which gives the necessary and sufficient conditions on the coefficients: Aj(tj+1 −tj)+ Aj+1(tj −tj−1)= yj(tj+1 −tj−1) (0􏶞 j􏶞n) (15) This is a system of n + 1 linear equations in n + 2 unknowns A0, A1,..., An+1. One way to solve Equation (15) is to assign any value to A0 and then use Equation (15) to compute for A1, A2, . . . , An+1, recursively. For this purpose, the equations could be rewritten as Aj+1 =αj +βjAj where these abbreviations have been used: ⎧􏰔􏰕 ⎪⎨α=y tj+1−tj−1 j j tj −tj−1 ⎪ ⎪ ⎪⎩ β j = t j − t j + 1 (0􏶞 j􏶞n) (16) (0􏶞 j􏶞n) tj −tj−1 To keep the coefficients small in magnitude, we recommend selecting A0 such that the expression n+1 􏰃 􏶛= Ai2 i=0 will be a minimum. To determine this value of A0, we proceed as follows: By successive substitution using Equation (16), we can show that Aj+1 =γj +δjA0 (0􏶞 j􏶞n) (17) 412 Chapter 9 Approximation by Spline Functions where the coefficients γj and δj are obtained recursively by this algorithm: 􏰘 Then 􏶛 is a quadratic function of A0 as follows: 􏶛 = A 20 + A 21 + · · · + A 2n + 1 γ0 = α0 δ0 = β0 γj =αj +βjγj−1 δj =βjδj−1 (1􏶞 j􏶞n) (18) = A20 +(γ0 +δ0A0)2 +(γ1 +δ1A0)2 +···+(γn +δnA0)2 To find the minimum of 􏶛, we take its derivative with respect to A0 and set it equal to zero: d􏶛 =2A0 +2(γ0 +δ0A0)δ0 +2(γ1 +δ1A0)δ1 +···+2(γn +δnA0)δn =0 dA0 This is equivalent to q A0 + p = 0, where 􏰎 Pseudocode and a Curve-Fitting Example A procedure that computes coefficients A0, A1, . . . , An+1 in the manner outlined above is given now. In its calling sequence, (ti )0:n is the knot array, (yi )0:n is the array of abscissa points, (ai )0:n+1 is the array of Ai coefficients, and (hi )0:n+1 is an array that contains hi = ti − ti−1. Only n, (ti ), and (yi ) are input values. They are available unchanged when the routine is finished. Arrays (ai ) and (hi ) are computed and available as output. q = 1 + δ 02 + δ 12 + · · · + δ n2 p=γ0δ0 +γ1δ1 +···+γnδn procedure BSpline2 Coef (n, (ti ), (yi ), (ai ), (hi )) integer i, n; real δ, γ, p, q real array (ai)0:n+1,(hi)0:n+1,(ti)0:n,(yi)0:n for i = 1 to n do hi ←ti −ti−1 end for h0 ← h1 hn+1 ← hn δ ← −1 γ ← 2y0 p ← δγ q←2 for i = 1 to n do r ← hi+1/hi δ ← −rδ γ ← −rγ + (r + 1)yi p←p+γδ q ← q + δ2 end for between t0 and tn . The result of Problem 9.3.26 is used. 9.3 B Splines: Interpolation and Approximation 413 a0 ←−p/q fori =1ton+1do ai ←[(hi−1 +hi)yi−1 −hiai−1]/hi−1 end for end procedure BSpline2 Coef Next we give a procedure function BSpline2 Eval for computing values of the quadratic splinegivenbyS(x)=􏰍n+1 A B2 (x).Itscallingsequencehassomeofthesamevariables i=0 i i−2 as in the preceding pseudocode. The input variable x is a single real number that should lie real function BSpline2 Eval(n, (ti ), (ai ), (hi ), x) integer i, n; real d, e, x; real array (ai )0:n+1, (hi )0:n+1, (ti )0:n fori =n−1to0 step−1do ifx−ti 􏶟0thenexitloop end for i←i+1 d ← [ai+1(x − ti−1) + ai (ti − x + hi+1)]/(hi + hi+1) e←[ai(x−ti−1 +hi−1)+ai−1(ti−1 −x+hi)]/(hi−1 +hi) BSpline2 Eval ← [d(x − ti−1) + e(ti − x)]/hi end function BSpline2 Eval Using the table of 20 points from Section 9.2, we can compare the resulting natural cubic spline curve with the quadratic spline produced by the procedures BSpline2 Coef and BSpline2 Eval. The first of these curves is shown in Figure 9.8, and the second is in Figure 9.16. The latter is reasonable but perhaps not as pleasing as the former. These curves show once again that cubic natural splines are simple and elegant functions for curve fitting. y 2 1.5 1 0.5 0x 12345678 􏶼0.5 􏶼1 FIGURE 9.16 􏶼1.5 Quadratic interpolating 􏶼2 spline 414 Chapter 9 Approximation by Spline Functions Schoenberg’s Process An efficient process due to Schoenberg [1967] can also be used to obtain B spline approx- imations to a given function. Its quadratic version is defined by 􏰃∞ 1 1. If f(x)=ax+b,thenS(x)= f(x). 2. If f (x) 􏶟 0 everywhere, then S(x) 􏶟 0 everywhere. 3. maxx |S(x)|􏶞 maxx |f(x)|. 4. If f is continuous on [a,b], if δ = maxi |ti+1 − ti|, and if δ < b − a, then for x in [a,b], Some of these properties are elementary; others are more abstruse. Property 1 is outlined in Problem 9.3.29. Property 2 is obvious because Bi2(x)􏶟 0 for all x. Property 3 follows easily from Equation (19) because if | f (x )| 􏶞 M , then f(τi)Bi2(x) where τi = 2(ti+1 +ti+2) (19) Here, of course, the knots are {t }∞ , and the points where f must be evaluated are S(x)= i =−∞ i i=−∞ Equation (19) is useful in producing a quadratic spline function that approximates f . midpoints between the knots. The salient properties of this process are as follows: |S(x)− f(x)|􏶞 3 max 2 a 􏶞 u 􏶞 v 􏶞 u+δ 􏶞 b |f(u)− f(v)| 5. ThegraphofSdoesnotcrossanylineintheplaneagreaternumberoftimesthandoes thegraphof f. 􏶨􏶨􏶨􏰃∞ |S(x)| 􏶞 􏶨􏶨 i=−∞ 􏶨􏶨􏶨 􏰃∞ f (τi )Bi2(x)􏶨􏶨 􏶞 | f (τi )∥Bi2(x) 􏶞 M 􏰃∞ i=−∞ Bi2(x) = M i=−∞ Properties 4 and 5 will be accepted without proof. Their significance, however, should not be overlooked. By Property 4, we can make the function S close to a continuous function f simply by making the mesh size δ small. This is because f (u)− f (v) can be made as small as we wish simply by imposing the inequality |u − v| 􏶞 δ (uniform continuity property). Property 5 can be interpreted as a shape-preserving attribute of the approximation process. In a crude interpretation, S should not exhibit more undulations than f . Pseudocode A pseudocode to obtain a spline approximation by means of Schoenberg’s process is devel- oped here. Suppose that f is defined on an interval [a, b] and that the spline approximation of Equation (19) is wanted on the same interval. We define nodes τi = a + ih, where h = (b − a)/n. Here, i can be any integer, but the nodes in [a,b] are only τ0,τ1,...,τn. To have τi = 1(ti+1 + ti+2), we define the knots ti = a + (i − 3)h. In Equation (19), the 22 only B splines B2 that are active on [a, b] are B2 , B2, . . . , B2 . Hence, for our purposes, (20) i Equation (19) becomes −10 n+1 S(x) = n+1 􏰃 f (τi )Bi2(x) i =−1 9.3 B Splines: Interpolation and Approximation 415 Thus, we require the values of f at τ−1, τ0, . . . , τn+1. Two of these nodes are outside the interval [a, b]; therefore, we furnish linearly extrapolated values in the code by defining f(τ−1) = 2f(τ0)− f(τ1) f(τn+1)=2f(τn)− f(τn−1) To use the formulas in Problem 9.3.26, we write n+3 􏰃 S(x)= D B2 (x) [D = f(τ )] ii−2 i i−2 i=1 A pseudocode to compute D1, D2, . . . , Dn+3 is given now. In the calling sequence for procedure Schoenberg Coef , f is an external function. After execution, the n + 3 desired coefficients are in the (di ) array. procedure Schoenberg Coef ( f, a, b, n, (di )) integer i; real a, b, h; real array (di )1:n+3 external function f h ← (b − a)/n fori =2ton+2do di ← f(a+(i−2)h) end for d1 ← 2d2 − d3 dn+3 ← 2dn+2 − dn+1 end procedure Schoenberg Coef After the coefficients Di have been obtained by the procedure just given, we can recover values of the spline S(x) in Equation (20). Here, we use the algorithm of Problem 9.3.26. Given an x, we first need to know where it is relative to the knots. To determine k such that tk−1 􏶞 x 􏶞 tk, we notice that k should be the largest integer such that tk−1 􏶞 x. This inequality is equivalent to the inequality k 􏶞 5 + (x − a)/ h, as is easily verified. This explains the 2 calculations of k in the pseudocode. The location of x is indicated in Figure 9.17. In the calling sequence for function Schoenberg Eval, a and b are the ends of the interval, and x is a point where the value of S(x) is desired. The procedure determines knots ti in such a way that the equally spaced points τi in the preceding procedure satisfy τi = 1 (ti +1 + ti +2 ). 2 FIGURE 9.17 Location of x tk􏶼1 x tk tk 􏶼 2 tk􏶽1 tk 􏶼 1 real function Schoenberg Eval(a, b, n, (di ), x) integer k: real c, h, p, w; real array (di )1:n+3 h ← (b − a)/n k ← integer[(x − a)/h + 5/2] p ← x − a − (k − 5/2)h c ← [dk+1 p + dk(2h − p)]/(2h) e ← [dk(p + h) + dk−1(h − p)]/(2h) Schoenberg Eval ← [cp + e(h − p)]/h end function Schoenberg Eval 416 Chapter 9 Approximation by Spline Functions Be ́zierCurves In computer-aided design, it is useful to have a procedure for producing a curve that goes through (or near to) some control points, or a curve that can be easily manipulated to give a desired shape. High-degree polynomial interpolation is generally not suitable for this sort of task, as one might guess from the negative remarks previously made about them. Experience shows that if one specifies a number of control points through which the polynomial must pass, the overall shape of the resulting curve may be severely disappointing! Polynomials can be used in a different way, however, leading to Be ́zier curves. Be ́zier curves use as a basis for the space 􏶜n (all polynomials of degree not exceeding n) a special set of polynomials that lend themselves to the task at hand. We standardize to the interval [0, 1] and fix a value of n. Next, we define basic polynomial functions 􏰔n􏰕 φni(x)= i xi(1−x)n−i (0􏶞i􏶞n) The polynomials φni are the constituents of the Bernstein polynomials. For a continuous function f defined on [0, 1], Bernstein, in 1912, proved that the sequence of polynomials 􏰃n 􏰔i􏰕 f n φni (x) (n 􏶟 1) y 1 0.8 􏷋70 0.6 0.4 0.2 􏷋71 􏷋75 pn(x) = i=0 converges uniformly to f , thus providing a very attractive proof of the Weierstrass Approx- imation Theorem. The graphs of a few polynomials φni are shown in Figure 9.18, where we used n = 7 and i = 0, 1, 5. The Bernstein basic polynomials are found in mathematical software systems such as Maple or Mathematica, for example. FIGURE 9.18 First few Bernstein basis polynomials ■ PROPERTIES 0x 0.2 0.4 0.6 0.8 1 Bernstein polynomials have two salient properties. For all x satisfying 0􏶞 x 􏶞 1, 1. φni(x)􏶟0 2. 􏰍n φni(x)=1 i=0 9.3 B Splines: Interpolation and Approximation 417 Any set of functions having these two properties is called a partition of unity on the interval [0, 1]. Notice that the second equation above is actually valid for all real x . The set {φn0,φn1,...,φnn} is a basis for the space 􏶜n. Consequently, every polynomial of degree at most n has a representation 􏰃n aiφni(x) i=0 If we want to create a polynomial that comes close to interpolating values (i/n, y ) for 􏰍n i 0 􏶞 i 􏶞 n, we can use i=0 yi φni to start and then, after examining the resulting curve, adjust the coefficients to change the shape of the curve. This is one procedure that can be used in computer-aided design. Changing the value of yi will change the curve principally in the vicinity of i/n because of the local nature of the basic polynomials φni . Another way in which these polynomials can be used is in creating curves that are not simply graphs of a function f . Here, we turn to a vector form of the procedure suggested above. If n + 1 vectors v0, v1, . . . , vn are prescribed, say, in R2 or R3, the expression 􏰃n i=0 makes sense, since the right-hand side is (for each t ) a linear combination of the vectors vi . As t runs over the interval [0,1], the vector u(t) describes a curve in the space where the vectorsvi aresituated.Thiscurveliesintheconvexhullofthevectorsvi,becauseu(t)isa convexlinearcombinationofthevi.Thisrequiresthetwopropertiesofφni mentionedabove. To illustrate this procedure, we have selected seven points in the plane and have drawn the closed curve generated by the above equation; that is, by the vector u(t). Figure 9.19 shows the resulting curve as well as the control points. In Figure 9.19, the control points y 5 4 3 2 1 u(t) u(t) = φni (t)vi (0 􏶞 t 􏶞 1) FIGURE 9.19 Curve using control points x 012345 418 Chapter 9 Approximation by Spline Functions are the vertices of the polygon, and the curve is the one that results in the manner de- scribed. Mathematical software systems such as Maple and Mathematica can be used to do this. A glance at Figure 9.18 will suggest to the reader that perhaps B splines can be used in the role of the Bernstein functions φni . Indeed, that is the case, and B splines have taken over in most programs for computer-aided design. Thus, to obtain a curve that comes close to a set of points (t , y ), we can set up a system of B splines (for example, cubic B splines) ii 􏰍n3 having knots ti . Then the linear combination i =0 yi Bi can be examined to see whether it has the desired shape. Here, of course, Bi3 denotes a cubic B spline whose support is the interval (ti , ti +4 ). Thevectorcaseisliketheonedescribedabove,exceptthatthefunctionsφni arereplaced by Bi3. Also, it is easier to take the knots as integers and let t run from 0 to n. The properties 1 and 2 of the φni displayed above are also shared by the B splines. Summary (1) The B spline of degree 0 is Bi0(x) = 􏰎 1 (ti 􏶞 x < ti+1) 0 (otherwise) Higher-degree B splines are defined recursively: 􏰔x−t􏰕 􏰔t −x􏰕 Bk(x) = i Bk−1(x)+ i+k+1 Bk−1(x) i+1 i ti+k −ti i where k = 1,2,... and i = 0,±1,±2,... . ti+k+1 −ti+1 (2) Some properties are An efficient method to evaluate a function of the form is to use Bk(x) = dx i A useful formula is Bk−1(x) − ti+k −ti i d􏰃∞ 􏰃∞ c Bk(x) = Bk−1(x) ti+k+1 −ti+1 i+1 B i k ( x ) = 0 B ik ( x ) > 0
x ∈/ [ t i , t i + k + 1 ) x ∈ ( t i , t i + k + 1 )
􏰃∞ i =−∞
􏰔x−t􏰕 Cj−1 =Cj i
C ik B ik ( x )
􏰔t −x􏰕
f ( x ) =
+Cj i+j
i it−ti−1t−t
i+j i i+j i
d 􏰔k􏰕 􏰔k􏰕
(3) The derivative of B splines is
d Bk−1(x) dxii ii
i =−∞ i =−∞

9.3 B Splines: Interpolation and Approximation 419 where di = k(ci − ci−1)/(ti+k − ti ). A basic result needed for integration is
􏶩x 􏰔t −t􏰕􏰃∞ Bk(s)ds = i+k+1 i Bk+1(x)
−∞i k+1j j=i
A resulting useful formula is
􏶩 x 􏰃∞ 􏰃∞ c Bk(s)ds =
−∞ i=−∞ i=−∞ whereei =1/(k+1)􏰍i cj(tj+k+1 −tj).
j =−∞
(4) To determine the coefficients in the expression
e Bk+1(x) ii ii
􏰃∞
i i−k
S(x)=
i =−∞
so that the resulting spline function interpolates a prescribed table, we use the condition
Aj(tj+1 −tj)+ Aj+1(tj −tj−1)= yj(tj+1 −tj−1) (0􏶞 j􏶞n) Thisisasystemofn+1linearequationsinn+2unknowns A0,A1,…,An+1 thatcanbe
solved recursively.
(5) Schoenberg’s process is an efficient process to obtain B spline approximations to a given function. For example, its quadratic version is defined by
􏰃∞
S(x) = f (τi )Bi2(x)
i =−∞
whereτ = 1(t +t )andtheknotsare{t }∞ .Thepointsτ where f mustbeevaluated
i 2i+1 i+2
are midpoints between the knots.
ii=−∞ i
AB2 (x)
(6) Be ́zier curves are used in computer-aided design for producing a curve that goes through (or near to) control points, or a curve that can be manipulated easily to give a desired shape. Be ́zier curves use Bernstein polynomials. For a continuous function f defined on [0, 1], the sequence of Bernstein polynomials
􏰃n 􏰔i􏰕
f n φni (x)
converges uniformly to f . The polynomials φni are 􏰔n􏰕
φni(x)= i xi(1−x)n−i Additional References
pn(x) =
(n 􏶟 1)
(0􏶞i􏶞n)
i=0
See Ahlberg et al. [1967], de Boor [1978], Farin [1990], MacLeod [1973], Schoenberg [1946, 1967], Schultz [1973], Schumaker [1981], Subbotin [1967], and Yamaguchi [1988].

420 Chapter 9
Approximation by Spline Functions
1. Show that the functions fn(x) = cosnx are generated by this recursive definition:
􏰎
a2. Whatfunctionsaregeneratedbythefollowingrecursivedefinition?
􏰎
a3. FindanexpressionforBi2(x)andverifythatitispiecewisequadratic.ShowthatBi2(x) is zero at every knot except
Bi2(ti+1) = ti+1 − ti and Bi2(ti+2) = ti+3 − ti+2 ti+2 − ti ti+3 − ti+1
4. VerifyEquation(5).
a5. Establish that 􏰍∞ f (t )B1 (x) is a first-degree spline that interpolates f at every
f0(x) = 1, f1(x) = cos x
fn+1(x) = 2 f1(x) fn(x) − fn−1(x) (n 􏶟 1)
f0(x) = 1, f1(x) = x
fn+1(x) = 2x fn(x) − fn−1(x) (n 􏶟 1)
i=−∞ i i−1
knot. What is the zero-degree spline that does so?
6. Showthatiftm 􏶞x 0)thatBik(tj)=0ifandonlyifj􏶞iorj􏶟i+k+1.
What is the maximum value of Bi2 and where does it Let the knots be the integers, and prove that
⎧⎪ ⎪ ⎪ ⎪
⎪ ⎪ ⎪⎨
occur?
0 (x<0) 1x3 (0􏶞x<1) 6 1(4−3x(x−2)2) (1􏶞x<2) 6 B 03 ( x ) = ⎪1(4+3(x−4)(x−2)2) ⎪ 6 0 curve passes through the first point, v0. Show that a linear B spline with integer knots can be written in matrix form as where 35. 36. 􏰖􏰗􏰖􏰗 ⎪ 1(4−x)3 (2􏶞x<3) (3􏶞x<4) ⎪⎩ 6 In the theory of Be ́zier curves, using the Bernstein basic polynomials, show that the S(x)=[x 1] −1 1 c1 =b10c0 +b11c1 2 0 c0 ⎧ ⎪⎨ b10 = x B01(x)=⎪⎩b11 =2−x 0 (0 􏶞 x < 1) (1􏶞x<2) (otherwise) (x􏶟4) 9.3 B Splines: Interpolation and Approximation 423 37. Show that the quadratic B spline with integer knots can be written in matrix form as ⎡⎤⎡⎤ 1 1 −2 1 c2 S(x)=2[x2 x 1]⎣−6 6 0⎦⎣c1 ⎦ = b20c0 + b21c1 + b22c2 where Hint: See Problem 9.3.23. 9 −3 0 c0 (0􏶞x<1) (1􏶞x<2) (2􏶞x<3) (otherwise) B02(x)= ⎪⎩ b22 0 38. Show that the cubic B spline with integer knots can be written as ⎡ ⎤⎡⎤ ⎧⎪ b20 ⎨ b21 −1 3 −3 S(x)=1[x3 x2 x 1]⎢ 12 −24 12 1 c3 0⎥⎢c2 ⎥ 0⎦⎣c1⎦ 0 c0 6 ⎣−48 60 −12 64 −44 4 where Hint: See Problem 9.3.34. ⎧ ⎪ b30 ⎪⎨ b31 B03(x)= ⎪ b32 ⎪⎩ b33 0 (0􏶞x<1) (1􏶞x<2) (2􏶞x<3) (3􏶞x<4) (otherwise) = b30c0 + b31c1 + b32c2 + b33c3 1. Using an automatic plotter, graph B0k for k = 0, 1, 2, 3, 4. Use integer knots ti = i over the interval [0, 5]. 2. Let ti = i (so the knots are the integer points on the real line). Print a table of 100 values of the function 3B1 + 6B1 − 4B1 + 2B1 on the interval [6, 14]. Using a plotter, 7 8 9 10 construct the graph of this function on the given interval. 3. (Continuation) Repeat for the function 3B2 + 6B2 − 4B2 + 2B2 . 7 8 9 10 4. AssumingthatS(x)=􏰍n ciBik(x),writeaproceduretoevaluateS′(x)ataspecified i=0 x. Input is n,k,x,t0,...,tn+k+1 and c0,c1,...,cn. 5. Write a procedure to evaluate 􏶧b S(x)dx, using the assumption that S(x) = 􏰍nk a i=0 ci Bi (x). Input will be n,k,a,b,c0,c1,...,cn,t0,...,tn+k+1. Computer Problems 9.3 424 Chapter 9 Approximation by Spline Functions 6. (March of the B splines) Produce graphs of several B splines of the same degree marching across the x-axis. Use an automatic plotter or a computer package with on-screen graphics capabilities, such as Matlab. a7. HistorianshaveestimatedthesizeoftheArmyofFlandersasfollows: Date Number Feb. 1578 27, 603 Sept. 1572 67, 259 Dec. 1573 62, 280 Mar. 1574 62, 350 Apr. 1588 63, 455 Jan. 1575 59, 250 Nov. 1591 May 1576 51, 457 8. RewriteproceduresBSpline2CoefandBSplineEvalsothatthearray(hi)isnotused. 9. Rewrite procedures BSpline2 Coef and BSpline2 Eval for the special case of equally spaced knots, simplifying the code where possible. 10. Write a procedure to produce a spline approximation to F (x ) = 􏶧 x f (t ) d t . Assume a that a 􏶞 x 􏶞 b. Begin by finding a quadratic spline interpolant to f at the n points ti =a+i(b−a)/n.Testyourprogramonthefollowing: Sept. 1580 45, 435 Oct. 1582 61, 162 Mar. 1607 41, 471 62, 164 Fit the table with a quadratic B spline, and use it to find the average size of the army during the period given. (The average is defined by an integral.) a. f(x)=sinx b. f(x)=ex c. f(x)=(x2+1)−1 (0􏶞x􏶞π) (0􏶞x􏶞4) (0􏶞x􏶞2) 11. Write a procedure to produce a spline function that approximates f ′(x) for a given f on a given interval [a,b]. Begin by finding a quadratic spline interpolant to f at n + 1 points evenly spaced in [a, b], including endpoints. Test your procedure on the functions suggested in the preceding computer problem. 12. Define f on [0, 6] to be a polygonal line that joins points (0, 0), (1, 2), (3, 3), (5, 3), and (6, 0). Determine spline approximations to f , using Schoenberg’s process and taking 7, 13, 19, 25, and 31 knots. 13. Write suitable code to calculate 􏰍∞ f (si )Bi2(x) with si = 1 (ti+1 + ti+2). Assume i=−∞ 2 that f is defined on [a,b] and that x will lie in [a,b]. Assume also that t1 < a < t2 and tn+1 < b < tn+2. (Make no assumption about the spacing of knots.) 14. Writeaproceduretocarryoutthisapproximationscheme: 􏰃∞ 1 f(τi)Bi3(x) τi = 3(ti+1 +ti+2 +ti+3) Assumethatfisdefinedon[a,b]andthatτi =a+ihfor0􏶞i􏶞n,whereh= (b − a)/n. 15. UsingamathematicalsoftwaresystemsuchasMatlabwithBsplineroutines,compute and plot the spline curve in Figure 9.16 based on the 20 data points from Section 9.2. Vary the degree of the B splines from 0, 1, 2, 3, through 4 and observe the resulting curves. S(x)= i =−∞ 9.3 B Splines: Interpolation and Approximation 425 16. Using B splines, write a program to perform a natural cubic spline interpolation at knotst0 εmax,thenhisreplacedbyh/2,providedthat|h/2|􏶟hmin.
5. If hmin 􏶞 |h| 􏶞 hmax, then the step is repeated by returning to step 1 with x(t) and the new h value.
The procedure for this adaptive scheme is RK45 Adaptive. In the parameter list of the pseudocode, f is the function f (t , x ) for the differential equation, t and x contain the initial values, h is the initial step size, tb is the final value for t, itmax is the maximum number of steps to be taken in going from a = ta to b = tb, εmin and εmax are lower and upper bounds on the allowable error estimate ε, hmin and hmax are bounds on the step size h, and iflag is an error flag that returns one of the following values:
iflag
0 1
Meaning
Successful march from ta to tb Maximum number of iterations reached
On return, t and x are the exit values, and h is the final step size value considered or used:
procedure RK45 Adaptive( f, t, x, h, tb, itmax, εmax, εmin, hmin, hmax, iflag) integer iflag, itmax, n; external function f
real ε, εmax, εmin, d, h, hmin, hmax, t, tb, x, xsave, tsave
realδ← 1 ×10−5 2

454 Chapter 10
Ordinary Differential Equations
output 0, h, t, x iflag ← 1 k←0
while k 􏶞 itmax
k←k+1
if |h| < hmin then h ← sign(h)hmin if |h| > hmax then h ← sign(h)hmax d ← |tb − t|
if d 􏶞 |h| then
iflag ← 0
if d 􏶞 δ · max{|tb|, |t|} then exit loop h ← sign(h)d
end if
xsave ← x
tsave ← t
call RK45( f, t, x, h, ε) output n,h,t,x,ε
if iflag = 0 then exit loop ifε<εmin thenh←2h if ε > εmax then
h ← h/2 x ← xsave
t ← tsave
k←k−1 end if
end while
end procedure RK45 Adaptive
In the pseudocode, notice that several conditions must be checked to determine the size of the final step, since floating-point arithmetic is involved and the step size varies.
As an illustration, the reader should repeat the computer example in the previous section using RK45 Adaptive, which allows variable step size, instead of RK4. Compare the accuracy of these two computed solutions.
An Industrial Example
A first-order differential equation that arose in the modeling of an industrial chemical process is as follows:
x′ =a+bsint+cx x(0)=0 (1)
in which a = 3, b = 5, and c = 0.2 are constants. This equation is amenable to the solution techniques of calculus, in particular the use of an integrating factor. However, the analytic solution is complicated, and a numerical solution may be preferable.
To solve this problem numerically using the adaptive Runge-Kutta formulas, one need only identify (and program) the function f that appears in the general description. In this problem, it is f (t, x) = 3 + 5 sin t + 0.2x. Here is a brief pseudocode for solving the

10.3 Stability and Adaptive Runge-Kutta and Multistep Methods 455 equation on the interval [0, 10] with particular values assigned to the parameters in the
routine RK45 Adaptive:
program Test RK45 Adaptive
integer iflag; real t , x , h , tb ; external function f
integer itmax ← 1000
real εmax ← 10−5, εmin ← 10−8, hmin ← 10−6, hmax ← 1.0 t←0.0; x←0.0; h←0.01; tb ←10.0
callRK45 Adaptive(f,t,x,h,tb,itmax,εmax,εmin,hmin,hmax,iflag) output itmax, iflag
end program Test RK45 Adaptive
real function f (t, x) real t, x
f ←3+5sin(t)+0.2x end function f
We obtain the approximation x(10) ≈ 135.917. The output from the code is a table of values that can be sent to a plotting routine. The resulting graph helps the user to visualize the solution curve.
Adams-Bashforth-Moulton Formulas
We now introduce a strategy in which numerical quadrature formulas are used to solve a single first-order ordinary differential equation. The model equation is
x′(t)= f(t,x(t))
and we suppose that the values of the unknown function have been computed at several points to the left of t, namely, t,t − h,t − 2h,…,t − (n − 1)h. We want to compute x(t + h). By the theorems of calculus, we can write
􏶩 t+h t
􏶩 t+h t
􏰃n j=1
wheretheabbreviation fj = f(t−(j−1)h,x(t−(j−1)h))hasbeenused.Inthelast line of the above equation, we have brought in a suitable numerical integration formula. The simplest case of such a formula will be for the interval [0, 1] and will use values of the integrand at points 0, −1, −2, . . . , 1 − n in the case of an Adams-Bashforth formula. Once we have such a basic rule, a change of variable will produce the rule for any other interval with any other uniform spacing.
x(t + h) = x(t) +
= x(t)+
≈x(t)+
x′(s) ds f(s,x(s))ds
cj fj

456 Chapter 10
Ordinary Differential Equations
Let us find a rule of the form 􏶩1
F(r)dr ≈c1F(0)+c2F(−1)+···+cnF(1−n) 0
There are n coefficients cj at our disposal. We know from interpolation theory that the formula can be made exact for all polynomials of degree n − 1. It suffices that we insist on integrating each function 1,r,r2,…,rn−1 exactly. Hence, we write down the appropriate equation:
􏶩1 􏰃n ri−1 dt =
cj(1− j)i−1 (1􏶞i􏶞n)
0
j=1
This is a system Au = b of n equations in n unknowns. The elements of the matrix A are
Aij =(1−j)i−1,andtheright-handsideisbi =1/i.
When this program is run, the output is the vector of coefficients 55 , − 59 , 37 , − 3 .
get the Adams-Moulton formulas, we start with a quadrature rule of the form
􏰉􏰈
24 24 24 8 Of course, higher-order formulas are obtained by changing the value of n in the code. To
􏶩1 􏰃n G(r)dr ≈
CjG(2− j)
A program similar to the one above yields the coefficients 􏰉 9 , 19 , − 5 , 1 􏰈. The distinction
0
j=1
24 24 24 24
between the two quadrature rules is that one involves the value of the integrand at 1 and the
other does not. 􏶧 t +h
How do we arrive at formulas for t g(s)ds from the work already done? Use the
change of variable from s to σ given by s = hσ − t. In these considerations, think of t as a constant. The new integral will be h􏶧1 g(hσ +t)dσ, which can be treated with either
0
of the two formulas already designed for the interval [0, 1]. For example,
􏶩 t+h h
F(r)dr ≈ 24 [55F(t)−59F(t −h)+37F(t −2h)−9F(t −3h)] t+h h
G(r)dr ≈ 24 [9G(t +h)+19G(t)−5G(t −h)+G(t −2h)] t
The method of undetermined coefficients used here to obtain the quadrature formulas does not, by itself, provide the error terms that we would like to have. An assessment of the error can be made from interpolation theory, because the methods considered here come from integrating an interpolating polynomial. Details can be found in more advanced books. You can experiment with some of the Adams-Bashforth-Moulton formulas in Computer Problems 10.3.2–10.3.4. These methods are taken up again in Section 11.3.
Stability Analysis
Let us now resume the discussion of errors that inevitably occur in the numerical solution of an initial-value problem
􏶩t
􏰎
x′ = f(t,x) x(a) = s
(2)

10.3 Stability and Adaptive Runge-Kutta and Multistep Methods 457
The exact solution is a function x(t). It depends on the initial value s, and to show this, we write x(t,s). The differential equation therefore gives rise to a family of solution curves, each corresponding to one value of the parameter s. For example, the differential equation
􏰎
x′ = x x(a) = s
gives rise to the family of solution curves x = se(t−a) that differ in their initial values x(a) = s. A few such curves are shown in Figure 10.4. The fact that the curves there diverge from one another as t increases has important numerical significance. Suppose, for instance, that initial value s is read into the computer with some roundoff error. Then even if all subsequent calculations are precise and no truncation errors occur, the computed solution will be wrong. An error made at the beginning has the effect of selecting the wrong curve from the family of all solution curves. Since these curves diverge from one another, any minute error made at the beginning is responsible for an eventual complete loss of accuracy. This phenomenon is not restricted to errors made in the first step, because each point in the numerical solution can be interpreted as the initial value for succeeding points.
x
FIGURE 10.4
Solution curves tox′ =x with x(a) = s
a􏶾t0 t1 t2 t3 t4 t5
x 􏶾 se(t􏶼a)
s5
s4
s3 Global error s2
s1
t
For an example in which this difficulty does not arise, consider
􏰎
x′ =−x x(a) = s
Its solutions are x = se−(t−a). As t increases, these curves come closer together, as in Figure 10.5. Thus, errors made in the numerical solution still result in selecting the wrong curve, but the effect is not as serious because the curves coalesce.
At a given step, the global error of an approximate solution to an ordinary differential equation contains both the local error at that step and the accumulative effect of all the local errors at all previous steps. For divergent solution curves, the local errors at each step are magnified over time, and the global error may be greater than the sum of all the local errors. In Figure 10.4 and Figure 10.5, the steps in the numerical solution are indicated by dots connected by dark lines. Also, the local errors are indicated by small vertical bars and the global error by a vertical bar at the right end of the curves.
For convergent solution curves, the local errors at each step are reduced over time, and the global error may be less than the sum of all the local errors. For the general differential

458
Chapter 10
Ordinary Differential Equations
FIGURE 10.5
Solution curves
′
to x = −x with
x(a) = s
whence
a􏶾t0 t1 t2 t3 t4 t5
Global error
t
s5 s4 s3 s2 s1
x 􏶾 se􏶼(t􏶼a)
x
Equation (2), how can the two modes of behavior just discussed be distinguished? It is simple. If fx > δ for some positive δ, the curves diverge. However, if fx < −δ, they con- verge. To see why, consider two nearby solution curves that correspond to initial values s and s + h. By Taylor series, we have ∂ 1 2 ∂2 x(t,s+h)=x(t,s)+h∂sx(t,s)+2h ∂s2x(t,s)+··· x(t,s+h)−x(t,s)≈h ∂ x(t,s) ∂s Thus, the divergence of the curves means that lim |x(t, s + h) − x(t, s)| = ∞ and can be written as t→∞ 􏶨􏶨􏶨 ∂ 􏶨􏶨􏶨 lim􏶨 x(t,s)􏶨=∞ t→∞ ∂s To calculate this partial derivative, start with the differential equation satisfied by x(t,s): ∂ x(t,s)= f(t,x(t,s)) ∂t and differentiate partially with respect to s: Hence, ∂ ∂x(t,s)= ∂ f(t,x(t,s)) ∂s ∂t ∂s ∂ ∂ x(t,s)= fx(t,x(t,s)) ∂ x(t,s)+ ft(t,x(t,s))∂t (3) ∂t ∂s ∂s ∂s But s and t are independent variables (a change in s produces no change in t), so ∂t/∂s = 0. If s is now fixed and if we put u(t) = (∂/∂s)x(t, s) and q(t) = fx (t, x(t, s)), then Equa- tion (3) becomes u′ = qu (4) This is a linear differential equation with solution u(t) = ceQ(t), where Q is the indefinite integral (antiderivative) of q. The condition limt→∞ |u(t)| = ∞ is met if limt→∞ Q(t) = ∞. 10.3 Stability and Adaptive Runge-Kutta and Multistep Methods 459 This situation, in turn, occurs if q(t) is positive and bounded away from zero because then 􏶩t 􏶩t Q(t) = q(θ) dθ > δ dθ = δ(t − a) → ∞
aa
as t → ∞ if fx = q > δ > 0.
To illustrate, consider the differential equation x ′ = t + tan x . The solution curves
diverge from one another as t → ∞ because fx (t, x) = sec2 x > 1. Summary
(1) The Runge-Kutta-Fehlberg method is
􏶗x(t) = x(t)+ 25 K1 + 1408K3 + 2197K4 − 1K5
216 2565 4104 5 x(t+h)=x(t)+ 16K1+ 6656K3+28561K4− 9K5+ 2K6
where
⎧⎪ ⎪ K 1 = h f ⎪
⎪K2=hf ⎪
t+4h,x+4K1
􏰔3 3 9􏰕
135 12825
( t , x )
􏰔1 1􏰕
􏰔 􏰔
56430 50
􏰕
􏰕
55
⎪⎨K3=hf ⎪K4=hf
t+8h,x+32K1+32K2
12 1932 7200 7296
⎪ ⎪K5=hf
⎪ ⎩K6=hf
t + 13h, x + 2197K1 − 2197K2 + 2197K3 439 3680 845
t+h,x+216K1−8K2+513K3−4104K4 􏰔 1 8 3544 1859
11 􏰕 t+2h,x−27K1+2K2−2565K3+4104K4−40K5
The quantity ε = |x(t + h) − 􏶗x| can be used in an adaptive step-size procedure. (2) A fourth-order multistep method is the Adams-Bashforth-Moulton method:
􏶗x(t+h)=x(t)+ h [55f(t,x(t))−59f(t−h,x(t−h))
24
+37f(t −2h,x(t −2h))−9f(t −3h,x(t −3h))]
x(t+h)=x(t)+ h [9f(t+h,􏶗x(t+h))+19f(t,x)t))
24
−5f(t −h,x(t −h))+ f(t −2h,x(t −2h))]
The value 􏶗x(t + h) is the predicted value, and x(t + h) is the corrected value. The trunca- tion errors for these two formulas are O(h5). Since the value of x(a) is given, the values for x(a + h), x(a + 2h), x(a + 3h), x(a + 4h) are computed by some single-step method such as the fourth-order Runge-Kutta method.

460 Chapter 10
Ordinary Differential Equations
Additional References
See Aiken [1985], Butcher [1987], Dekker and Verwer [1984], England [1969], Fehlberg [1969], Henrici [1962], Hundsdorfer [1985], Lambert [1973], Lapidus and Seinfeld [1971], Miranker [1981], Moulton [1930], Shampine and Gordon [1975], and Stetter [1973].
a1. Solvetheproblem
􏰎
x′ =−x x(0) = 1
by using the Trapezoid Rule, as discussed at the beginning of this chapter. Compare the true solution at t = 1 to the approximate solution obtained with n steps. Show, for example, that for n = 5, the error is 0.00123.
a2. Derive an implicit multistep formula based on Simpson’s rule, involving uniformly spaced points x(t − h), x(t), and x(t + h), for numerically solving the ordinary differ- ential equation x′ = f .
3. An alert student noticed that the coefficients in the Adams-Bashforth formula add up to 1. Why is that so?
a4. Deriveaformulaoftheform
x(t + h) = ax(t) + bx(t − h) + h[cx′(t + h) + dx′′(t) + ex′′′(t − h)]
that is accurate for polynomials of as high a degree as possible. Hint: Use polynomials 1, t, t2, and so on.
a5. Determinethecoefficientsofanimplicit,one-step,ordinarydifferentialequationmethod of the form
x(t + h) = ax(t) + bx′(t) + cx′(t + h)
so that it is exact for polynomials of as high a degree as possible. What is the order of
the error term?
6. The differential equation that is used to illustrate the adaptive Runge-Kutta program can be solved with an integrating factor. Do so.
7. EstablishEquation(4).
a8. The initial-value problem x′ = (1 + t2)x with x(0) = 1 is to be solved on the interval
[0, 9]. How sensitive is x (9) to perturbations in the initial value x (0)?
9. For each differential equation, determine regions in which the solution curves tend to
diverge from one another as t increases:
aa. x′ =sint+ex b. x′ =x+te−t ac. x′ =xt
d. x′ =x3(t2 +1) ae. x′ =cost−ex f. x′ =(1−x3)(1+t2)
Problems 10.3

10.3 Stability and Adaptive Runge-Kutta and Multistep Methods 461
a10. For the differential equation x′ = t(x3 − 6×2 + 15x), determine whether the solution
curves diverge from one another as t → ∞.
a11. Determine whether the solution curves of x′ = (1 + t2)−1x diverge from one another
as t → ∞.
1. Usemathematicalsoftwaretosolvesystemsoflinearequationswhosesolutionsare a. Adams-Bahforthcoefficients b. Adams-Moultoncoefficients
2. The second-order Adams-Bashforth-Moulton method is given by 􏶗x(t+h)=x(t)+h[3f(t,x(t))− f(t−h,x(t−h))]
2
x(t+h)=x(t)+h[f(t+h,􏶗x(t+h))+ f(t,x(t))] 2
Theapproximatesingle-steperrorisε≡K|x(t+h)−􏶗x(t+h)|,whereK = 1.Using 6
ε to monitor the convergence, write and test an adaptive procedure for solving an ODE of your choice using these formulas.
3. (Continuation) Carry out the instructions of the previous computer problem for the third-order Adams-Bashforth-Moulton method:
􏶗x(t+h)=x(t)+ h [23f(t,x(t))−16f(t−h,x(t−h)) 12
+ 5 f (t − 2h, x(t − 2h))] x(t+h)=x(t)+ h [5f(t+h,􏶗x(t+h))+8f(t,x(t))
12
− f(t−h,x(t−h))]
where K = 1 in the expression for the approximate single-step error. 10
4. (Predictor-corrector scheme) Using the fourth-order Adams-Bashforth-Moulton method, derive the predictor-corrector scheme given by the following equations:
􏶗x(t+h)=x(t)+ h [55f(t,x(t))−59f(t−h,x(t−h)) 24
+37f(t −2h,x(t −2h))−9f(t −3h,x(t −3h))] x(t+h)=x(t)+ h [9f(t+h,􏶗x(t+h))+19f(t,x(t))
24
−5f(t −h,x(t −h))+ f(t −2h,x(t −2h))]
Write and test a procedure for the Adams-Bashforth-Moulton method. Note: This is a multistep process because values of x at t, t − h, t − 2h, and t − 3h are used to deter- mine the predicted value 􏶗x(t + h), which, in turn, is used with values of x at t, t − h, and t − 2h to obtain the corrected value x(t + h). The error terms for these form- ulas are (251/720)h5 f (4)(ξ) and −(19/720)h5 f (4)(η), respectively. (See Section 9.3 for additional discussion of these methods.)
Computer Problems 10.3

462 Chapter 10
Ordinary Differential Equations
a5. Solve
⎧⎨x′ = 3x + 9t −13 ⎩t2
x(3) = 6
at x 􏰉 1 􏰈 using procedure RK45 Adaptive to obtain the desired solution to nine decimal
2
places. Compare with the true solution:
x = t3 − 9t2 + 13t 22
a6. (Continuation)Repeatthepreviousproblemforx􏰉−1􏰈. 2
7. It is known that the fourth-order Runge-Kutta method described in Equation (12) of Section 10.2 has a local truncation error that is O(h5). Devise and carry out a numerical experiment to test this. Suggestions: Take just one step in the numerical solution of a nontrivial differential equation whose solution is known beforehand. However, use a variety of values for h , such as 2−n , where 1 􏶞 n 􏶞 24. Test whether the ratio of errors to h5 remains bounded as h → 0. A multiple-precision calculation may be needed. Print the indicated ratios.
8. Computethenumericalsolutionof
􏰎
x′ =−x x(0) = 1
x n + 1 = x n − 1 + 2 h x n′ √
with x0 = 1 and x1 = −h +
for this problem? Carry out an analysis of the stability of this method. Hint: Consider fixedhandassumexn =λn.
a9. Tabulateandgraphthefunction[1−lnv(x)]v(x)on[0,e],wherev(x)isthesolution of the initial-value problem (dv/dx)[lnv(x)] = 2x,v(0) = 1. Check value: v(1) = e.
10. Determinethenumericalvalueof
􏶩 5 es
2π sds
4
in three ways: solving the integral, an ordinary differential equation, and using the exact
formula.
11. Compute and print a table of the function
􏶩φ􏶫12
1−4sin θdθ
using the midpoint method
1 + h2. Are there any difficulties in using this method
f(φ)=
0
by solving an appropriate initial-value problem. Cover the interval [0, 90◦ ] with steps
of 1◦ and use the Runge-Kutta method of order 4. Check values: Use f(30◦) = 0.51788 193 and f (90◦) = 1.46746 221. Note: This is an example of an elliptic integral of the second kind. It arises in finding an arc length on an ellipse and in many engineering problems.

10.3 Stability and Adaptive Runge-Kutta and Multistep Methods 463 a12. Bysolvinganappropriateinitial-valueproblem,makeatableofthefunction
􏶩∞ dt
f (x) =
on the interval [0,1]. Determine how well f is approximated by xe−1/x. Hint: Let
t =−lns.
a13. Bysolvinganappropriateinitial-valueproblem,makeatableofthefunction 2􏶩x2
e−t dt
on the interval 0 􏶞 x 􏶞 2. Determine how accurately f (x ) is approximated on this
1/x
tet
f (x) = √π
􏰉 2 3􏰈 2 −x2
interval by the function
g(x)=1− ay+by +cy √πe
0
where
􏰎
a = 0.30842 84 b = −0.08497 13
c = 0.66276 98 y = (1 + 0.47047x )−1
14. UsetheRunge-Kuttamethodtocompute􏶧1√1+s3ds. 0
a15. Writeandrunaprogramtoprintanaccuratetableofthesineintegral
r dr
The table should cover the interval 0 􏶞 x 􏶞 1 in steps of size 0.01. [Use sin(0)/0 = 1.
See Computer Problem 5.1.2] 16. Compute a table of the function
Shi(x) =
0
􏶩 x sinht
t dt
Si(x)=
􏶩 x sinr
0
by finding an initial-value problem that it satisfies and then solving the initial-value problem. Your table should be accurate to nearly machine precision. [Use sinh(0)/
0 = 1.]
17. Design and carry out a numerical experiment to verify that a slight perturbation in an initial-value problem can cause catastrophic errors in the numerical solution. Note: An initial-value problem is an ordinary differential equation with conditions specified only at the initial point. (Compare this with a boundary value problem as given in Chapter 12.)
18. RunexampleprogramsforsolvingtheindustrialexampleinEquation(1),comparethe solutions, and produce the plots.
19. AnotheradaptiveRunge-KuttamethodwasdevelopedbyEngland[1969].TheRunge- Kutta-England method is similar to the Runge-Kutta-Fehlberg method in that it com- bines a fourth-order Runge-Kutta formula and a companion fifth-order one. To reduce the number of function evaluations, the formulas are derived so that some of the same function evaluations are used in each pair of formulas. (A fourth-order Runge-Kutta

464 Chapter 10
Ordinary Differential Equations
formula requires at least four function evaluations, and a fifth-order one requires at least six.) The Runge-Kutta-England method uses the fourth-order Runge-Kutta methods in Computer Problem 10.2.14a and takes two half steps as follows:
􏰔1􏰕1
x t+2h =x(t)+6(K1+4K3+K4)
where
and
where
⎧⎪K1 = 1hf(t,x(t))
⎪2􏰉 􏰈
⎨K2=1hf t+1h,x(t)+1K1 2􏰉42􏰈
⎪K3=1hf t+1h,x(t)+1K1+1K2 ⎪⎩2􏰉4 44􏰈
K4=1hf t+1h,x(t)−K2+2K3 22
􏰔1􏰕1
x(t+h)=x t+2h +6(K5+4K7+K8)
⎧⎪ ⎪ K 5 = 1 h f 􏰉 t + 1 h , x 􏰉 t + 1 h 􏰈 􏰈
⎪ 2 􏰉 2 􏰉 2 􏰈 􏰈 ⎨K6=1hf t+3h,xt+1h +1K5
2􏰉4􏰉2􏰈2 􏰈 ⎪K7=1hf t+3h,xt+1h +1K5+1K6 ⎪⎩ 2 􏰉 4 􏰉 2􏰈 4 4􏰈
K8=1hf t+h,xt+1h −K6+2K7 22
With these two half steps, there are enough function evaluations so that only one more 1􏰔1
K9=2hf t+h,x(t)−12(K1+96K2−92K3+121K4 􏰕
− 144K5 − 6K6 + 12K7) is needed to obtain a fifth-order Runge-Kutta method:
􏶤x(t+h)=x(t)+ 1 (14K1 +64K3 +32K3 −8K5 +64K7 +15K8 −K9) 90
An adaptive procedure can be developed by using an error estimation based on the two values x(t + h) and 􏶤x(t + h). Program and test such a procedure. (See, for example, Shampine, Allen, and Pruess [1997].)
20. Investigatethenumericalsolutionoftheinitial-valueproblem 􏰎′√2
x=−1−x x(0) = 1
This problem is ill-conditioned, since x (t ) = cos t is a solution and x (t ) = 1 is also. For more information on this and other test problems, see Cash [2003] or www.ma.ic.ac .uk/∼jcash/.
21. (Student research project) Learn about algebraic differential equations.
22. Write software to implement the following pseudocodes and verify the numerical
results given in the text:
a. Test RK45 and RK45 b. Test RK45 Adaptive and RK45 Adaptive

11
Systems of Ordinary Differential Equations
A simple model to account for the way in which two different animal species sometimes interact is the predator-prey model. If u(t) is the number of individuals in the predator species and v(t) the number of individuals in the prey species, then under suitable simplifying assumptions and with appropriate constants a, b, c, and d,
⎧⎪ ⎪ ⎪ d u = a ( v + b ) u ⎨ dt
⎪ ⎪ ⎪⎩ d v = c ( u + d ) v dt
This is a pair of nonlinear ordinary differential equations (ODEs) that govern the populations of the two species (as functions of time t). In this chapter, numerical procedures are developed for solving such problems.
11.1 Methods for First-Order Systems
In Chapter 10, ordinary differential equations were considered in the simplest context; that is, we restricted our attention to a single differential equation of the first order with an accompanying auxiliary condition. Scientific and technological problems often lead to more complicated situations, however. The next degree of complication occurs with systems of several first-order equations.
Uncoupled and Coupled Systems
The sun and the nine planets form a system of particles moving under the jurisdiction of Newton’s law of gravitation. The position vectors of the planets constitute a system of 27 functions, and the Newtonian laws of motion can be written, then, as a system of 54 first-order ordinary differential equations. In principle, the past and future positions of the planets can be obtained by solving these equations numerically.
465

466 Chapter 11
Systems of Ordinary Differential Equations
Taking an example of more modest scope, we consider two equations with two auxiliary conditions. Let x and y be two functions of t subject to the system
􏰎
(1)
This is an example of an initial-value problem that involves a system of two first-order differential equations. Note that in the example given, it is not possible to solve either of the two differential equations by itself because the first equation governing x′ involves the unknown function y, and the second equation governing y′ involves the unknown function x. In this situation, we say that the two differential equations are coupled.
The reader is invited to verify that the analytic solution is
􏰎
y(t) = et sin(t) − t3 = sin(t)[cosh(t) + sinh(t)] − t3
Let us look at another example that is superficially similar to the first but is actually
with initial conditions
x′(t)=x(t)−y(t)+2t−t2 −t3 y′(t)=x(t)+y(t)−4t2 +t3
􏰎
x(0) = 1
y(0) = 0
simpler:
with initial conditions
􏰎
x′(t) = x(t) + 2t − t2 − t3 y′(t) = y(t) − 4t2 + t3
􏰎
x(0) = 1
y(0) = 0
x(t) = et cos(t) + t2 = cos(t)[cosh(t) + sinh(t)] + t2
These two equations are not coupled and can be solved separately as two unrelated initial- value problems (using, for instance, the methods of Chapter 10). Naturally, our concern here is with systems that are coupled, although methods that solve coupled systems also solve those that are not. The procedures discussed in Chapter 10 extend to systems whether coupled or uncoupled.
Taylor Series Method
We illustrate the Taylor series method for System (1) and begin by differentiating the
equations constituting it:
􏰎
x′ = x − y + 2t − t2 − t3
􏰎 y′ = x + y − 4t2 + t3
x′′ = x′ − y′ + 2 − 2t − 3t2
􏰎 y′′ = x′ + y′ − 8t + 3t2
x′′′ = x′′ − y′′ − 2 − 6t
􏰎 y′′′ = x′′ + y′′ − 8 + 6t
x(4) = x′′′ − y′′′ − 6 y(4) = x′′′ + y′′′ + 6
etc.
(2)

11.1 Methods for First-Order Systems 467 A program to proceed from x(t) to x(t + h) and from y(t) to y(t + h) is easily written by
using a few terms of the Taylor series:
′ h2 ′′ h3 ′′′ h4 (4)
x(t+h)=x+hx+2x+6x+24x +··· ′ h2 ′′ h3 ′′′ h4 (4)
y(t+h)=y+hy+2y+6y+24y +···
together with equations for the various derivatives. Here, x and y and all their derivatives are functions of t; that is, x = x(t), y = y(t), x′ = x′(t), y′′ = y′′(t), and so on.
A pseudocode program that generates and prints a numerical solution from 0 to 1 in 100 steps is as follows. Terms up to h4 have been used in the Taylor series.
program Taylor System1
integer k; real h, t, x, y, x′, y′, x′′, y′′, x′′′, y′′′, x(4), y(4) integernsteps←100; reala←0,b←1
x←1; y←0; t←a
output 0, t , x , y
h ← (b − a)/nsteps
fork =1tonstepsdo
x′ ←x−y+t(2−t(1+t)) y′ ←x+y+t2(−4+t)
x′′ ← x′ − y′ + 2 − t(2 + 3t) y′′ ← x′ + y′ + t(−8 + 3t) x′′′ ← x′′ − y′′ − 2 − 6t
y′′′ ← x′′ + y′′ − 8 + 6t
x(4) ← x′′′ − y′′′ − 6
y(4) ← x′′′ + y′′′ + 6
x ← x +h􏰠x′ + 1h􏰠x′′ + 1h􏰠x′′′ + 1h􏰠x(4)􏰡􏰡􏰡􏰡
􏰠 2 􏰠 3 􏰠 4 􏰠 􏰡􏰡􏰡􏰡 y←y+h y′+1h y′′+1h y′′′+1h y(4)
t←t+h
output k, t, x, y end for
end program Taylor System1
234
Vector Notation
Observe that System (1) can be written in vector notation as 􏰖x′􏰗 􏰖x−y+2t−t2−t3􏰗
with initial conditions
y′ = x+y−4t2+t3 (3) 􏰖􏰗􏰖􏰗
This is a special case of a more general problem that can be written as
􏰎′
X =F(t,X)
(4)
x(0) = 1 y(0) 0
X(a) = S, given

468 Chapter 11
Systems of Ordinary Differential Equations
where
􏰖 x 􏰗 􏰖 x′ 􏰗 X= y X′= y′
and F is the vector whose two components are given by the right-hand sides in Equation (1). Since F depends on t and X, we write F(t, X).
Systems of ODEs
We can continue this idea in order to handle a system of n first-order differential equations. First, we write them as
Then we let
⎡s ⎤ 1111
⎧⎪x1′ = f1(t,x1,x2,…,xn) ⎪⎨x2′ = f2(t,x1,x2,…,xn)
⎪ .
⎪⎩xn′ = fn(t,x1,x2,…,xn)
x1(a) = s1,x2(a) = s2,…, xn(a) = sn ⎡x ⎤ ⎡x′ ⎤ ⎡ f ⎤
all given
⎢ ⎢ x 2 ⎥ ⎥ ′ ⎢ ⎢ x 2′ ⎥ ⎥ ⎢ ⎢ f 2 ⎥ ⎥ ⎢ ⎢ s 2 ⎥ ⎥ X=⎢⎣.⎥⎦ X =⎢⎣.⎥⎦ F=⎢⎣.⎥⎦ S=⎢⎣.⎥⎦
xn xn′ fn sn
and we obtain Equation (4), which is an ordinary differential equation written in vector
notation.
Taylor Series Method: Vector Notation
The m-order Taylor series method would be written as
′ h2 ′′ hm (m)
X(t+h)=X+hX+2X +···+m!X (5)
where X = X(t), X′ = X′(t), X′′ = X′′(t), and so on.
A pseudocode for the Taylor series method of order 4 applied to the preceding problem
can be easily rewritten by a simple change of variables and the introduction of an array and an inner loop.
program Taylor System2
integer i,k; real h,t; real array (xi)1:n,(dij)1:n×1:4 integer n ← 2, nsteps ← 100
real a ← 0, b ← 1
t←0; (xi)←(1,0)
output 0, t, (xi )
h ← (b − a)/nsteps

iji
11.1 Methods for First-Order Systems 469
fork =1tonstepsdo
d11 ← x1 −x2 +t(2−t(1+t)) d21 ← x1 + x2 + t2(−4 + t)
d12 ← d11 −d21 +2−t(2+3t) d22 ←d11 +d21 +t(−8+3t) d13 ← d12 −d22 −2−6t
d23 ← d12 +d22 −8+6t
d14 ←d13 −d23 −6
d24 ←d13 +d23 +6
for i = 1 to n do􏰠 1 􏰠
1 􏰠 1 􏰡􏰡􏰡 xi ←xi +h di1+2h di2+3h di3+4h[di4]
end for
t←t+h
output k, t, (xi ) end for
end program Taylor System2
Here, a two-dimensional array is used instead of all the different derivative variables; that is, d ↔ x(j). In fact, this and other methods in this chapter become particularly easy to
program if the computer language supports vector operations.
Runge-Kutta Method
The Runge-Kutta methods of Chapter 10 also extend to systems of differential equations. The classical fourth-order Runge-Kutta method for System (4) uses these formulas:
where
X(t+h)=X+h(K1 +2K2 +2K3 +K4) (6) 6
⎧
⎪K1 = F(t,X)
⎪⎨K =F􏰉t+1h,x+1hk􏰈 2 􏰉 2 2 1􏰈
⎪K3=F t+1h,x+1hk2 ⎪⎩ 2 2
K4 =F(t+h,X+hK3)
Here, X=X(t), and all quantities are vectors with n components except variables t and h.
A procedure for carrying out the Runge-Kutta procedure is given next. It is assumed that the system to be solved is in the form of Equation (4) and that there are n equations in the system. The user furnishes the initial value of t, the initial value of X, the step size h, and the number of steps to be taken, nsteps. Furthermore, procedure XP System(n, t, (xi ), ( fi )) is needed, which evaluates the right-hand side of Equation (4) for a given value of array (xi ) and stores the result in array ( fi ). (The name XP System2 is chosen as an abbreviation of X′ for a system.)

470 Chapter 11 Systems of Ordinary Differential Equations
procedure RK4 System1(n, h, t, (xi ), nsteps) integer i, j, n; real h, t; real array (xi )1:n allocate real array (yi )1:n , (Ki, j )1:n×1:4 output 0, t, (xi )
for j =1tonstepsdo
call XP System(n, t, (xi ), (Ki,1)) for i = 1 to n do
yi ←xi +1hKi,1 2
end for
call XP System(n, t + h/2, (yi ), (Ki,2)) for i = 1 to n do
yi ←xi +1hKi,2 2
end for
call XP System(n, t + h/2, (yi ), (Ki,3)) for i = 1 to n do
yi ←xi +hKi,3 end for
call XP System(n, t + h, (y)i , (Ki,4)) for i = 1 to n do
xi ← xi + 1h[Ki,1 +2Ki,2 +2Ki,3 + Ki,4] 6
end for
t←t+h
output j,t,(xi) end for
deallocate array (yi ), (Ki, j ) end procedure RK4 System1
To illustrate the use of this procedure, we again use System (1) for our example. Of course, it must be rewritten in the form of Equation (4). A suitable main program and a procedure for computing the right-hand side of Equation (4) follow:
program Test RK4 System1 integer n ← 2, nsteps ← 100 real a ← 0, b ← 1
real h, t; real array (xi )1:n t←0
(xi) ← (1,0)
h ← (b − a)/nsteps
call RK4 System1(n, h, t, (xi ), nsteps) end program Test RK4 System1
procedure XP System(n, t, (xi ), ( fi )) real array (xi )1:n , ( fi )1:n
integer n

11.1 Methods for First-Order Systems 471
real t
f1 ← x1 − x2 + t(2 − t(1 + t)) f2 ← x1 + x2 − t2(4 − t)
end procedure XP System
A numerical experiment to compare the results of the Taylor series method and the Runge-Kutta method with the analytic solution of System (1) is suggested in Computer Problem 11.1.1. At the point t = 1.0, the results are as follows:
Taylor Series
x (1.0) ≈ 2.46869 40 y(1.0) ≈ 1.28735 46
Runge-Kutta
2.46869 42 1.28735 61
Analytic Solution
2.46869 39399 1.28735 52872
We can use mathematical software routines found in Matlab, Maple, or Mathematica to obtain the numerical solution of the system of ordinary differential equations (1). For t over the interval [0,1], we invoke an ODE procedure to march from t = 0 at which x(0) = 1 and y(0) = 0 to t = 1 at which x(1) = 2.468693912 and y(1) = 1.287355325.
To obtain the numerical solution of the ordinary differential equation defined for t over the interval [1, 1.5], invoke an ordinary differential equation solving procedure to march from t = 0 at which x(1) = 2 and y(1) = −2 to t = 1.5 at which x(1.5) ≈ 15.5028 and y(1.5) ≈ 6.15486.
Autonomous ODE
When we wrote the system of differential equations in vector form
X′ = F(t, X)
we assumed that the variable t was explicitly separated from the other variables and treated differently. It is not necessary to do this. Indeed, we can introduce a new variable x0 that is t in disguise and add a new differential equation x0′ = 1. A new initial condi- tion must also be provided, x0(a) = a. In this way, we increase the number of differ- ential equations from n to n + 1 and obtain a system written in the more elegant vector form
􏰎′
X = F(X)
X(a) = S, given
Consider the system of two equations given by Equation (1). We write it as a system
with three variables by letting
x0 =t, x1 =x, x2 =y
Thus, we have
⎡x0′⎤ ⎡1
⎤ ⎢⎣x1′ ⎥⎦=⎢⎣x1 −x2 +2×0 −x02 −x03⎥⎦
x2′ x1 +x2 −4×02 +x03
The auxiliary condition for the vector X is X (0) = [0, 1, 0]T .

472 Chapter 11
Systems of Ordinary Differential Equations
As a result of the preceding remarks, we sacrifice no generality in considering a system of n + 1 first-order differential equations written as
⎧⎪x0′ = f0(x0,x1,x2,…,xn) ⎪x1′ = f1(x0,x1,x2,…,xn) ⎪⎨x2′ = f2(x0,x1,x2,…,xn)
⎪ .
⎪⎩xn′ = fn(x0,x1,x2,…,xn)
x0(a) = s0,x1(a) = s1,x2(a) = s2,…, xn(a) = sn We can write this system in general vector notation as
where
all given
􏰎′
X = F(X)
(7)
X(a)=S, given
⎡x′ ⎤ ⎡ f ⎤
⎡x ⎤ 0000
⎢ ⎢ x 1 ⎥ ⎥ X = ⎢ ⎢ x 2 ⎥ ⎥
⎢ ⎢ x 1′ ⎥ ⎥ X ′ = ⎢ ⎢ x 2′ ⎥ ⎥
⎢ ⎢ f 1 ⎥ ⎥ F = ⎢ ⎢ f 2 ⎥ ⎥
⎡s ⎤ ⎢ ⎢ s 1 ⎥ ⎥
⎢⎣ . . . ⎥⎦
xn xn′ fn sn
⎢⎣ . . . ⎥⎦
A system of differential equations without the t variable explicitly present is said to be autonomous. The numerical methods that we discuss do not require that x0 = t or f0 = 1 ors0 =a.
For an autonomous system, the classical fourth-order Runge-Kutta method for System (6) uses these formulas:
⎢⎣ . . . ⎥⎦
S = ⎢ ⎢ s 2 ⎥ ⎥ ⎢⎣ . . . ⎥⎦
X(t+h)=X+h(K1 +2K2 +2K3 +K4) (8) 6
where
Here, X = X(t), and all quantities are vectors with n+1 components except the variables h. In the previous example, the procedure RK4 System1 would need to be modified by beginningthearrayswith0ratherthan1andomittingthevariablet.(WecallitRK4 System2 and leave it as Computer Problem 11.1.4.) Then the calling programs would be as follows:
⎧⎪ ⎪ K 1 = F ( X )
⎪⎨ K = F 􏰉 X + 1 h K 􏰈
2 􏰉 2 1􏰈 ⎪K3=F X+1hK2
⎩2
K4 =F(X+hK3)
program Test RK4 System2 real h, t; real array (xi )0:n integer n ← 2, nsteps ← 100 real a ← 0, b ← 1
(xi) ← (0,1,0)
h ← (b − a)/nsteps
call RK4 System2(n, h, (xi ), nsteps) end program Test RK4 System2

Summary
(1) A system of ordinary differential equations ⎧⎪x1′ = f1(t,x1,x2,…,xn)
⎪⎩xn′ = fn(t,x1,x2,…,xn)
x1(a) = s1,x2(a) = s2,…,xn(a) = sn,
all given
can be written in vector notation as
􏰎′
X =F(t,X)
X(a)=S, given where we define the following n component vectors
⎧⎪ ⎪ ⎪ ⎨
⎪ ⎪ ⎪⎩
X =[x1,x2,…,xn]T X′ = [x1′,x2′,…,xn′ ]T F = [ f1, f2,…, fn]T
X(a) = [x1(a),x2(a),…,xn(a)]T (2) The Taylor series method of order m is
′ h2 ′′ hm (m) X(t+h)=X+hX+2X +···+m!X
11.1 Methods for First-Order Systems 473
procedure XP System(n, (xi ), ( fi )) real array (xi )0:n , ( fi )0:n
integer n
f0 ← 1
f1 ← x1 −x2 +x0(2−x0(1+x0)) f 2 ← x 1 + x 2 − x 02 ( 4 − x 0 )
end procedure XP System
It is typical in ordinary differential equation solvers, such as those found in mathe- matical software libraries, for the user to interface with them by writing a subprogram in a nonautonomous format. In other words, the ordinary differential equation solver takes as input both the independent variable and the dependent variable and returns values for the right-hand side to the ordinary differential equation. Consequently, the nonautonomous programming convention may seem more natural to those who are using these software packages.
It is a useful exercise to find a physical application in your field of study or profession involving the solution of an ordinary differential equation. It is instructive to analyze and solve the physical problem by determining the appropriate numerical method and translating the problem into the format that is compatible with the available software.
⎪⎨x2′ = f2(t,x1,x2,…,xn) ⎪ .
where X = X(t), X′ = X′(t), X′′ = X′′(t), and so on.

474 Chapter 11
Systems of Ordinary Differential Equations
(3) The Runge-Kutta method of order 4 is X(t+h)=X+h(K1 +2K2 +2K3 +K4)
where
6
⎧⎪ ⎪ K 1 = F ( t , X )
⎪⎨ K = F 􏰉 t + 1 h , X + 1 h K 􏰈 2􏰉2 21􏰈
⎪K3=F t+1h,X+1hK2 ⎩22
K4 =F(t+h,X+hK3)
Here, X = X(t), and all quantities are vectors with n components except variables t and h.
(4) We can absorb the t variable into the vector by letting x0 = t and then writing the autonomous form for the system of ordinary differential equations in vector notation as
􏰎′
X = F(X)
X(a)=S, given where vectors are defined to have n + 1 components. Then
a1. Consider
􏰎􏰎
x′ = y x(0) = −1
y′ = x with y(0) = 0
⎧⎪ ⎪ ⎪ ⎨
⎪⎩
X =[x0,x1,x2,…,xn]T X′ = [x0′,x1′,x2′,…,xn′ ]T F = [1, f1, f2,…, fn]T
X(a) = [a,x1(a),x2(a),…,xn(a)]T
(5) The Runge-Kutta method of order 4 for the system of ordinary differential equations
in autonomous form is
X(t+h)=X+h(K1 +2K2 +2K3 +K4) 6
where
Here, X = X (t ), and all quantities F and K i are vectors with n + 1 components except the variables t and h.
⎧⎪ ⎪ K 1 = F ( X )
⎪⎨ K = F 􏰉 X + 1 h K 􏰈
2 􏰉 2 1􏰈 ⎪K3=F X+1hK2
⎩2
K4 =F(X+hK3)
Write down the equations, without derivatives, to be used in the Taylor series method of order 5.
Problems 11.1

the right-hand side:
11.1 Methods for First-Order Systems 475
a2. Howwouldyousolvethissystemofdifferentialequationsnumerically? ⎧
⎪⎨ x 1′ = x 12 + e t − t 2 ⎪⎩x2′ =x2−cost
x1(0) = 0 x2(1) = 0
a3. Howwouldyousolvetheinitial-valueproblem ⎧
⎪⎨x1′(t)=x1(t)et +sint−t2 ⎪⎩ x2′ (t) = [x2(t)]2 − et + x2(t)
x1(1) = 2 x2(1) = 4
if a computer program were available to solve an initial-value problem of the form
x′ = f (t, x) involving a single unknown function x = x(t)?
a4. Writeanequivalentsystemoffirst-orderdifferentialequationswithouttappearingon
⎧⎪⎨x′ =x2 +log(y)+t2
⎪⎩ y′ = ey − cos(x) + sin(tx) − (xy)7
x(0)=1 y(0)=3
a1. Solvethesystemofdifferentialequations(1)byusingtwodifferentmethodsgivenin this section and compare the results with the analytic solution.
a2. Solvetheinitial-valueproblem
⎧⎪⎨ x ′ = t + x 2 − y ⎪⎩ y ′ = t 2 − x + y 2
x(0)=3 y(0)=2
by means of the Taylor series method using h = 1/128 on the interval [0, 0.38]. Include terms involving three derivatives in x and y. How accurate are the computed function values?
3. WritetheRunge-Kuttaproceduretosolve ⎧⎪⎨ x 1 ′ = − 3 x 2
x′ =1x ⎪⎩ 2 3 1
x1(0) = 0 on the interval 0 􏶞 t 􏶞 4. Plot the solution.
a4. WriteprocedureRK4 System2andadriverprogramforsolvingtheordinarydifferential equation system given by Equation (2). Use h = −10−2, and print out the values of x0, x1, and x2, together with the true solution on the interval [−1,0]. Verify that the true solutionisx(t)=et +6+6t+4t2 +t3 andy(t)=et −t3 +t2 +2t+2.
x2(0) = 1
Computer Problems 11.1

476 Chapter 11
Systems of Ordinary Differential Equations
a5. UsingtheRunge-Kuttaprocedure,solvethefollowinginitial-valueproblemonthein- terval 0 􏶞 t 􏶞 2π. Plot the resulting curves (x1(t), x2(t)) and (x3(t), x4(t)). They should
be circles.
6. Solvetheproblem
⎧⎡x⎤
⎪⎢3 ⎥ ⎪⎨′⎢x4􏰉 􏰈⎥ X = ⎢⎣ − x 1 x 12 + x 2 2 − 3 / 2 ⎥⎦
⎪ −x 􏰉x2 + x2􏰈−3/2 ⎪⎩ 212
X(0) = [1,0,0,1]T
⎧⎪ ⎪ ⎪ x 0 ′ = 1
⎨x1′ =−x2+cosx0 ⎪⎩x2′= x1+sinx0
x0(1)=1 x1(1)=0 x2(1)=−1 Use the Runge-Kutta method and the interval −1 􏶞 t 􏶞 2.
a7. Writeandtestaprogram,usingtheTaylorseriesmethodoforder5,tosolvethesystem
⎧⎪⎨ x ′ = t x − y 2 + 3 t ⎪⎩ y ′ = x 2 − t y − t 2
x(5)=2 y(5)=3
on the interval [5,6] using h = 10−3. Print values of x and y at steps of 0.1.
8. Printatableofsintandcostontheinterval[0,π/2]bynumericallysolvingthesystem
⎧⎪⎨ x ′ = y ⎪⎩ y ′ = − x
x(0)=0 y(0)=1
9. WriteaprogramforusingtheTaylorseriesmethodoforder3tosolvethesystem
⎧⎪ ⎪ ⎪ x ′ = t x + y ′ − t 2 ⎨y′ =ty+3t
⎪ ⎪ ⎪⎩ z ′ = t z − y ′ + 6 t 3
x(0)=1 y(0)=2 z(0)=3 on the interval [0, 0.75] using h = 0.01.
10. Writeandtestashortprogramforsolvingthesystemofdifferentialequations
⎧⎪⎨ y ′ = x 3 − t 2 y − t 2 ⎪⎩ x ′ = t x 2 − y 4 + 3 t
y(2)=5 x(2)=3
over the interval [2, 5] with h = 0.25. Use the Taylor series method of order 4.
11. Recode and test procedure RK4 System2 using a computer language that supports vector operations.

11.2 Higher-Order Equations and Systems 477
12. Verifythenumericalresultsgiveninthetextforthesystemofdifferentialequations(1)
from programs Test RK4 System1 and RK4 System2.
13. (Continuation) Using mathematical software such as Matlab, Maple, or Mathematica containing symbolic manipulation capabilities to verify the analytic solution for the system of differential equations (1).
14. (Continuation)UsemathematicalsoftwareroutinessuchasarefoundinMatlab,Maple, or Mathematica to verify the numerical solutions given in the text. Plot the result- ing solution curve. Compare with the results from programs Test RK4 System1 and Test RK4 System2.
11.2 Higher-Order Equations and Systems
Consider the initial-value problem for ordinary differential equations of order higher than 1. A differential equation of order n is normally accompanied by n auxiliary conditions. This many initial conditions are needed to specify the solution of the differential equation pre- cisely (assuming certain smoothness conditions are present). Take, for example, a particular second-order initial-value problem
􏰎
Without the auxiliary conditions, the general analytic solution is
x(t) = 1t2 + 3 cos(2t) + c1t + c2 48
x′′(t) = −3 cos2(t) + 2 x(0)=0 x′(0)=0
(1)
where c1 and c2 are arbitrary constants. To select one specific solution, c1 and c2 must be fixed, and two initial conditions allow this to be done. In fact, x (0) = 0 yields c2 = − 3 ,
andx′(0)=0forcesc1 =0.
Higher-Order Differential Equations
8
In general, higher-order problems can be much more complicated than this simple example because System (1) has the special property that the function on the right-hand side of the differential equation does not involve x. The most general form of an ordinary differential equation with initial conditions that we shall consider is
􏰎
(2)
This can be solved numerically by turning it into a system of first-order differential equa- tions.Todoso,wedefinenewvariablesx1,x2,…,xn asfollows:
x(n) = f(t,x,x′,x′′,…,x(n−1))
x(a), x′(a), x′′(a), . . . , x(n−1)(a) all given
x1 =x x2 =x′ x3 =x′′ … xn−1 =x(n−2) xn =x(n−1)

478 Chapter 11
Systems of Ordinary Differential Equations
Consequently, the original initial-value problem (2) is equivalent to ⎧⎪ x1′=x2
⎪ x2′=x3 ⎨ .
⎪ ⎪ ⎪ ⎪ x n′ − 1 = x n
⎪⎩ xn′ = f(t,x1,x2,…,xn)
or, in vector notation,
where
and
x1(a),x2(a),…,xn(a) allgiven 􏰎′
X =F(t,X) X(a)=S, given
(3)
X = [x1,x2,…,xn]T X′ =[x1′,x2′,…,xn′]T
F = [x2,x3,x4,…,xn, f]T X(a) = [x1(a),x2(a),…,xn(a)]
Whenever a problem must be transformed by introducing new variables, it is recom- mended that a dictionary be provided to show the relationship between the new and the old variables. At the same time, this information, together with the differential equations and the initial values, can be displayed in a chart. Such systematic bookkeeping can be helpful in a complicated situation.
(4)
into a form suitable for solution by the Runge-Kutta procedure. A chart summarizing the transformed problem is as follows:
Old Variable New Variable Initial Value Differential Equation
x x1 3 x1′=x2
x ′ x 2 7 x 2′ = x 3
x′′ x3 13 x3′ =cosx1+sinx2−ex3 +t2
So the corresponding first-order system is
⎡⎤
x2
X′ =⎣ x3 ⎦
cosx1 +sinx2 −ex3 +t2
To illustrate, let us transform the initial-value problem 􏰎′′′ ′ x′′ 2
and X(0) = [3, 7, 13]T .
x =cosx+sinx −e +t x(0)=3 x′(0)=7 x′′(0)=13

ing chart:
Old Variable New Variable
xx1 x′ x2 y x3
y ′ y′′
Hence, we have
x 4 x5
Initial Value Differential Equation
2×1′=x2
−4 x2′ = x1 −x3 −9×2 +x43 +6×5 +2t −2 x3′=x4
7 x 4′ = x 5
6 x5′ =x5−x2+ex1 −t
⎡⎤
x2
⎢x1 −x3 −9×2 +x43 +6×5 +2t⎥
and X(1) = [2,−4,−2,7,6]T . Autonomous ODE Systems
11.2 Higher-Order Equations and Systems 479
Systems of Higher-Order Differential Equations
By systematically introducing new variables, we can transform a system of differential equations of various orders into a larger system of first-order equations. For instance, the system
⎧⎪⎨ x′′ = x − y − (3x′)2 + (y′)3 + 6y′′ + 2t
⎪⎩ y′′′ = y′′ − x′ + ex − t (5)
x(1)=2 x′(1)=−4 y(1)=−2 y′(1)=7 y′′(1)=6
can be solved by the Runge-Kutta procedure if we first transform it according to the follow-
X′ =⎢ x4 ⎥ ⎣ x5 ⎦
x5−x2+ex1 −t
We notice that t is present on the right-hand side of Equation (3) and that therefore the equations x0 = t and x0′ = 1 can be introduced to form an autonomous system of ordinary differential equations in vector notation. It is easy to show that a higher-order system of differential equations having the form in Equation (2) can be written in vector notation as
where
and
􏰎′
X = F(X)
X(a)=S, given
X = [x0,x1,x2,…,xn]T X′ = [x0′,x1′,x2′,…,xn′ ]T
F = [1,×2,x3,x4,…,xn, f]T X(a) = [a,x1(a),x2(a),…,xn(a)]

480 Chapter 11
Systems of Ordinary Differential Equations
As an example, the ordinary differential equation system in Equation (4) can be written
in autonomous form as
and X(1) = [1,2,−4,−2,7,6]T . Summary
(1) A single nth-order ordinary differential equation with initial values has the form 􏰎x(n) = f(t,x,x′,x′′,…,x(n−1))
x(a), x′(a), x′′(a), . . . , x(n−1)(a), all given It can be turned into a system of first-order equations of the form
⎡⎤
1
⎢ x2 ⎥
X′ =⎢x1 −x3 −9×2 +x43 +6×5 +2×0⎥ ⎢ x4 ⎥
⎣ x5 ⎦ x5 −x2 +ex1 −x0
where
⎧⎪ ⎪ ⎪ ⎨
⎪ ⎪ ⎪⎩
􏰎′
X =F(t,X)
X(a)=S, given X =[x1,x2,…,xn]T
X′ = [x1′,x2′,…,xn′ ]T
F =[x2,x3,x4,…,xn, f]T
X(a) = [x1(a),x2(a),…,xn(a)]T
(2) We can absorb the variable t into the vector notation by letting x0 = t and extending the vectors to length n + 1. Thus, a single nth-order ordinary differential equation can be
written as
where
⎧⎪ ⎪ ⎪ ⎨
⎪⎩
􏰎′
X = F(X)
X(a)=S, given X =[x0,x1,x2,…,xn]T
X′ = [x0′,x1′,x2′,…,xn′ ]T
F =[1,×2,x3,x4,…,xn, f]T
X(a) = [a,x1(a),x2(a),…,xn(a)]
a1. Turnthisdifferentialequationintoasystemoffirst-orderequationssuitableforapplying the Runge-Kutta method:
􏰘 x′′′ = 2x′ + log(x′′) + cos(x) x(0)=1 x′(0)=−3 x′′(0)=5
Problems 11.2

11.2 Higher-Order Equations and Systems 481 2. a. Assuming that a program is available for solving initial-value problems of the form
in Equation (3), how can it be used to solve the following differential equation?
􏰎
b. How would this problem be solved if the initial conditions were x(1) = 3, x′(1) = −7, and x′′′(1) = 0?
a3. Howwouldyousolvethisdifferentialequationproblemnumerically? ⎧⎪x′′ =x′ +x2 −sint
x′′′ = t + x + 2x′ + 3x′′
x(1)=3 x′(1)=−7 x′′(1)=4
⎨111
x′′=x −(x′)1/2+t
⎪⎩2 2 2
x1(0)=1 x2(1)=3 x1′(0)=0 x2′(1)=−2
a4. Converttoafirst-ordersystemtheorbitalequations
􏰎
with initial conditions
x(0) = 0.5 x′(0) = 0.75 y(0) = 0.25 y′(0) = 1.0
a5. Rewritethefollowingequationasasystemoffirst-orderdifferentialequationswithout t appearing on the right-hand side:
􏰎 􏰒x􏰓 x(4) = (x′′′)2 + cos(x′x′′) − sin(tx) + log t
x(0)=1 x′(0)=3 x′′(0)=4 x′′′(0)=5
a6. Expressthesystemofordinarydifferentialequations
⎧⎪ ⎪ ⎪ d 2 z − 2 t d z = 2 t e x z ⎪ dt2 dt
⎪ ⎪⎨ d 2 x − 2 x z d x = 3 x 2 y t 2 ⎪ dt2 dt
⎪ d2 y − ey dy = 4xt2z ⎪ ⎪ ⎪⎩ d t 2 d t
z(1)=x′′(1)=y′(1)=2 z′(1)=x(1)=y(1)=3 as a system of first-order ordinary differential equations.
7. Determineasystemoffirst-orderequationsequivalenttoeachofthefollowing:
x′′ + x(x2 + y2)−3/2 = 0 y′′ + y(x2 + y2)−3/2 = 0
aa. x′′′+x′′sinx+tx′+x=0
c.
a8. Consider
􏰎
b. x(4)+x′′cosx′+txx′ =0
􏰘 x ′′ = x ′ − x x(0)=0 x′(0)=1
x′′ =3×2 −7y2 +sint+cos(x′y′) y′′′ =y+x2 −cost−sin(xy′′)
Determine the associated first-order system and its auxiliary initial conditions.

482 Chapter 11
Systems of Ordinary Differential Equations
a9. Theproblem
⎧⎪ x′′(t) = x + y − 2x′ + 3y′ + log t ⎨y′′(t)=2x−3y+5x′ +ty′ −sint ⎪⎩ x(0) = 1 x′(0) = 2
y(0)=3 y′(0)=4
is to be put into the form of an autonomous system of five first-order equations. Give
the resulting system and the appropriate initial values.
10. Write procedure XP System for use with the fourth-order Runge-Kutta routine RK4 System1 for the following differential equation:
x alone:
􏰘x′ =−x+axy y′ =3y−xy
􏰎 ′′′ x′′ ′′′ ′ 10 x =10e −x sin(xx)−(xt)
x(2) = 6.5 x′(2) = 4.1 x′′(2) = 3.2
11. Ifwearegoingtosolvetheinitial-valueproblem
􏰎
x(1) = x′(1) = x′′(1) = 1
using Runge-Kutta formulas, how should the problem be transformed?
12. Convert this problem involving differential equations into an autonomous system of first-order equations (with initial values):
⎧⎪⎨3x′ +tanx′′ −x2 =√t2 +1+y2 +(y′)2 ⎪⎩−3y′+coty′′+y2 =t2+(x+1)1/2+4x′
x(1)=2 x′(1)=−2 y(1)=7 y′(1)=3
13. Follow the instructions in the preceding problem on this example:
⎧⎪txyz+x′y′/t =tx2 +x/y′′ +z ⎨t2x/z + y′z′t = y2 − (z′′)2x + x′y′ ⎪⎩tyz−x′z′y′ =z2 −zx′′ −(yz)′
x(3)=1 y(3)=2 z(3)=4 x′(3)=5 y′(3)=6 z′(3)=7
14. Turnthispairofdifferentialequationsintoasecondorderdifferentialequationinvolving
x ′′′ = x ′ − t x ′′ + x + ln t
1. UseRK4System1tosolveeachofthefollowingfor0􏶞t􏶞1.Useh=2−k withk=5,
6, and 7, and compare results.
􏰎′′ 2t 21/2 ⎪⎨′′ 2 t
⎧⎪x′′ =x2 −y+et x = 2(e − x ) y = x − y − e
a. x(0)=0 x′(0)=1 b. ⎪⎩x(0)=0 x′(0)=0 y(0) = 1 y′(0) = −2
Computer Problems 11.2

3. Solve
􏰎
x′′ + x′ + x2 − 2t = 0
4. Solve
on the interval [−2, 0].
􏰎
x′′ = 2x′ − 5x
x(0) = 0 x′(0) = 0.4
11.3 Adams-Bashforth-Moulton Methods 483
2. Solve the Airy differential equation ⎧⎪⎨x′′ =tx
⎪⎩ x (0) = 0.35502 80538 87817 x′(0) = −0.25881 94037 92807
on the interval [0, 4.5] using the Runge-Kutta method. Check value: The value x (4.5) = 0.00033 02503 is correct.
x(0) = 0 x′(0) = 0.1
on [0, 3] by any convenient method. If a plotter is available, graph the solution.
5. Write computer programs based on the pseudocode in the text to find the numerical solution of these ordinary differential equation systems:
a. (1) b. (4) c. (5)
6. (Continuation) Use mathematical software such as Matlab, Maple, or Mathematica with symbolic manipulation capabilities to find their analytical solutions.
7. (Continuation)UsemathematicalsoftwareroutinessuchasarefoundinMatlab,Maple, or Mathematica to verify the numerical solutions for these ordinary differential equation systems. Plot the resulting solution curves.
11.3 Adams-Bashforth-Moulton Methods A Predictor-Corrector Scheme
The procedures explained so far have solved the initial-value problem
􏰘X′ = F(X) X(a)=S, given
(1)
by means of single-step numerical methods. In other words, if the solution X(t) is known at a particular point t, then X(t + h) can be computed with no knowledge of the solution at points earlier than t. The Runge-Kutta and Taylor series methods compute X(t + h) in terms of X(t) and various values of F.
Moreefficientmethodscanbedevisedifseveralvalues X(t), X(t−h), X(t−2h),… are used in computing X(t + h). Such methods are called multistep methods. They have the obvious drawback that at the beginning of the numerical solution, no prior values of X are available. So it is usual to start a numerical solution with a single-step method, such as the Runge-Kutta procedure, and transfer to a multistep procedure for efficiency as soon as enough starting values have been computed.

484 Chapter 11
Systems of Ordinary Differential Equations
An example of a multistep formula is known as the Adams-Bashforth method (see Section 10.3 and the related problem). It is
X􏶗(t + h) = X(t) + h {55F[X(t)] − 59F[X(t − h)] + 37F[X(t − 2h)] 24
−9F[X(t − 3h)]} (2)
Here, X􏶗(t + h) is the predicted value of X(t + h) computed by using Formula (2). If the solution X has been computed at the four points t, t − h, t − 2h, and t − 3h, then Formula (2) can be used to compute X􏶗 (t + h). If this is done systematically, then only one evaluation of F is required for each step. This represents a considerable savings over the fourth-order Runge-Kutta procedure; the latter requires four evaluations of F per step. (Of course, a consideration of truncation error and stability might permit a larger step size in the Runge-Kutta method and make it much more competitive.)
In practice, Formula (2) is never used by itself. Instead, it is used as a predictor, and then another formula is used as a corrector. The corrector that is usually used with Formula (2) is the Adams-Moulton formula:
X(t + h) = X(t) + h {9F[X􏶗(t + h)] + 19F[X(t)] − 5F[X(t − h)] 24
+ F[X(t − 2h)]} (3)
Thus, Equation (2) predicts a tentative value of X(t +h), and Equation (3) computes this X value more accurately. The combination of the two formulas results in a predictor-corrector scheme.
With initial values of X specified at a, three steps of a Runge-Kutta method can be performed to determine enough X values that the Adams-Bashforth-Moulton procedure can begin. The fourth-order Adams-Bashforth and Adams-Moulton formulas, started with the fourth-order Runge-Kutta method, are referred to as the Adams-Moulton method. Predictor and corrector formulas of the same order are used so that only one application of the corrector formula is needed. Some suggest iterating the corrector formula, but experience has demonstrated that the best overall approach is only one application per step.
Pseudocode
Storage of the approximate solution at previous steps in the Adams-Moulton method is usually handled either by storing in an array of dimension larger than the total number of steps to be taken or by physically shifting data after each step (discarding the oldest data and storing the newest in their place). If an adaptive process is used, the total number of steps to be taken cannot be determined beforehand. Physical shifting of data can be eliminated by cycling the indices of a storage array of fixed dimension. For the Adams-Moulton method, the xi data for X(t) are stored in a two-dimensional array with entries zim in locations m = 1,2,3,4,5,1,2,… for t = a,a + h,a + 2h,a + 3h,a + 4h,a + 5h,a + 6h,…, respectively. The sketch in Figure 11.1 shows the first several t values with corresponding m values and abbreviations for the formulas used.
An error analysis can be conducted after each step of the Adams-Moulton method. If
x(p) is the numerical approximation of the ith equation in System (1) at t + h obtained by i

FIGURE 11.1
Starting values for applications of RK and AB/AM methods
m:1 2 3 4 5 1 2
11.3 Adams-Bashforth-Moulton Methods 485
a a 􏶽 h RK
a 􏶽 2h RK
a 􏶽 3h RK
a 􏶽 4h AB / AM
a 􏶽 5h AB / AM
a 􏶽 6h AB / AM
predictorFormula(2)andxi isthatfromcorrectorFormula(3)att+h,thenitcanbeshown that the single-step error for the i th component at t + h is given approximately by
19 􏶨􏶨xi − x(p)􏶨􏶨 εi= i
est = max |εi | 1􏶞i􏶞n
So we compute
270 |xi |
in the Adams-Moulton procedure AM System to obtain an estimate of the maximum single- step error at t + h.
A control procedure is needed that calls the Runge-Kutta procedure three times and then calls the Adams-Moulton predictor-corrector scheme to compute the remaining steps. Such a procedure for doing nsteps steps with a fixed step size h follows:
procedure AMRK(n, h, (xi ), nsteps)
integer i, k, m, n; real est, h; real array (xi )0:n allocate real array ( fi j )0:n×0:4, (zi j )0:n×0:4
m←0
output h
output 0, (xi )
for i = 0 to n do
zim ←xi end for
fork =1to3do
callRK System(m,n,h,(zij),(fij)) output k,(zim)
end for
fork =4tonstepsdo
callAM System(m,n,h,est,(zij),(fij),) output k,(zim)
output est
end for
for i = 0 to n do
xi ←zim end for
deallocate array ( f, z) end procedure AMRK

486 Chapter 11 Systems of Ordinary Differential Equations
The Adams-Moulton method for a system and the computation of the single-step error are accomplished in the following pseudocode:
procedureAM System(m,n,h,est,(zij),(fij))
integer i, j, k, m, mp1; real d, dmax, est, h
real array (zi j )0:n×0:4, ( fi j )0:n×0:4 allocate real array (si )0:n , ( yi )0:n realarray(ai)1:4 ←(55,−59,37,−9) realarray(bi)1:4 ←(9,19,−5,1) mp1 ← (1 + m) mod 5
callXP System(n,(zim),(fim)) for i = 0 to n do
si ←0 end for
fork =1to4do
j ← (m − k + 6) mod 5
for i = 0 to n do
si ←si +ak fij
end for end for
for i = 0 to n do
yi ←zim +hsi/24
end for
callXP System(n,(yi),(fi,mp1)) for i = 0 to n do
si ←0 end for
fork =1to4do
j ←(mp1−k+6)mod5
for i = 0 to n do
si ←si +bk fij
end for end for
for i = 0 to n do
zi,mp1 ←zim +hsi/24
end for
m ← mp1
dmax ← 0
for i = 0 to n do
d←|zim −yi|/|zim| if d > dmax then
dmax ← d j←i
end if end for
est ← 19dmax/270 deallocate array (s, y) end procedure AM System

11.3 Adams-Bashforth-Moulton Methods 487
Here, the function evaluations are stored cyclically in fim for use by Formulas (2) and (3).
Various optimization techniques are possible in this pseudocode. For example, the program-
mer may wish to move the computation of 1 h outside of the loops. 24
A companion Runge-Kutta procedure is needed, which is a modification of procedure RK4 System2 from Section 11.1:
procedureRK System(m,n,h,(zij),(fij))
integer i, m, mp1, n; real h; real array (zi j )0:n×0:4, ( fi j )0:n×0:4 allocate real array (gi j )0:n×0:3, (yi )0:n
mp1 ← (1 + m) mod 5
callXP System(n,(zim),(fim))
for i = 0 to n do
yi ←zim +1hfim 2
end for
callXP System(n,(yi),(gi,1)) for i = 0 to n do
yi ←zim +1hgi,1 2
end for
callXP System(n,(yi),(gi,2)) for i = 0 to n do
yi ←zim +hgi,2 end for
callXP System(n,(yi),(gi,3)) for i = 0 to n do
zi,mp1 ← zim + h[ fim + 2gi,1 + 2gi,2 + gi,3]/6 end for
m ← mp1
deallocate array (gi j ), (yi ) end procedure RK System
As before, the programmer may wish to move 1 h out of the loop. 6
To use the Adams-Moulton pseudocode, we supply the procedure XP System that defines the system of ordinary differential equations and write a driver program with a call to procedure AMRK. The complete program then consists of the following five parts: the main program and procedures XP System, AMRK, RK System, and AM System.
As an illustration, the pseudocode for the last example in Section 11.2 (p. 479) is as follows:
program Test AMRK
real h; real array (xi )0:n integer n ← 5, nsteps ← 100 real a ← 0, b ← 1
(xi) ← (1,2,−4,−2,7,6)
h ← (b − a)/nsteps
call AMRK(n, h, (xi ), nsteps) end program Test AMRK

488 Chapter 11
Systems of Ordinary Differential Equations
procedure XP System(n, (xi ), ( fi )) integer n; real array (xi )0:n , ( fi )0:n
f0 ← 1
f1 ← x2
f2 ← x1 −x3 −9×2 +x43 +6×5 +2×0 f3 ← x4
f4 ← x5
f5 ← x5 − x2 + ex1 − x0
end procedure XP System
Here, we have programmed this procedure for an autonomous system of ordinary differential equations.
An Adaptive Scheme
Since an estimate of the error is available from the Adams-Moulton method, it is natural to replace procedure AMRK with one that employs an adaptive scheme—that is, one that changes the step size. A procedure similar to the one used in Section 10.3 is outlined here. The Runge-Kutta method is used to compute the first three steps, and then the Adams- Moulton method is used. If the error test determines that halving or doubling of the step size is necessary in the first step using the Adams-Moulton method, then the step size is halved or doubled, and the whole process starts again with the initial values—so at least one step of the Adams-Moulton method must take place. If during this process the error test indicates that halving is required at some point within the interval [a, b], then the step size is halved. A retreat is made back to a previously computed value, and after three Runge-Kutta steps have been computed, the process continues, using the Adams-Moulton method again but with the new step size. In other words, the point at which the error was too large should be computed by the Adams-Moulton method, not the Runge-Kutta method. Doubling the step size is handled in an analogous manner. Doubling the step size requires only saving an appropriate number of previous values; however, one can simplify this process (whether halving or doubling the step size) by always backing up two steps with the old step size and then using this as the beginning point of a new initial-value problem with the new step size. Other, more complicated procedures can be designed and can be the subject of numerical experimentation. (See Computer Problem 11.3.3.)
An Engineering Example
In chemical engineering, a complicated production activity may involve several reactors connected with inflow and outflow pipes. The concentration of a certain chemical in the i th reactorisanunknownquantity,xi.Eachxi isafunctionoftime.Iftherearenreactors,the whole process is governed by a system of n differential equations of the form
X′ = AX + V X(0)=S, given
where X is the vector containing the unknown quantities xi , A is an n × n matrix, and V is a constant vector. The entries in A depend on the flow rates permitted between different reactors of the system.

11.3 Adams-Bashforth-Moulton Methods 489
There are several approaches to solving this problem. One is to diagonalize the matrix A by finding a nonsingular matrix P for which is P −1 A P is diagonal and then using the matrix exponential function to solve the system in an analytic form. This is a task that mathematical software can handle. On the other hand, we can simply turn the problem over to an ODE solver and get the numerical solution. One piece of information that is always wanted in such a problem is a description of the steady state of the system. That means the values of all variables at t = ∞. Each function xi should be a linear combination of exponential functions of the form t 􏶙→ eλt , in which λ < 0. Here is a simple example that can illustrate all of this: ⎡x′⎤ ⎡ ⎤⎡ ⎤ ⎡ ⎤ ⎢1⎥ −8/3−4/3 1 x1 12 ⎣x2′ ⎦=⎣−17/3 −4/3 1⎦⎣x2⎦+⎣29⎦ (4) x3′ −35/3 14/3 −2 x3 48 Using mathematical software such as Matlab, Maple, or Mathematica, we can obtain a closed-form solution: x(t) = 1e−3t(6 − 50et + 10e2t + 34e3t) 6 y(t) = 1e−3t(12 − 125et + 40e2t + 73e3t) 6 z(t) = 1e−3t(14 − 200et + 70e2t + 116e3t) 6 For a system of ordinary differential equations with a large number of variables, it may be more convenient to represent them in a computer program with an array such as x(i,t) rather than by separate variables names. To see the numerical value of the analytic solution at a single point, say, t = 2.5, we obtain x(2.5) ≈ 5.74788, y(2.5) ≈ 12.5746, z(2.5) ≈ 20.0677. Also, we can produce a graphing of the analytic solution to the problem. Finally, the programs presented in this section can be used to generate a numerical solution on a prescribed interval with a prescribed number of points. Some Remarks about Stiff Equations In many applications of differential equations there are several functions to be tracked together as functions of time. A system of ordinary differential equations may be used to model the physical phenomena. In such a situation, it can happen that different solution functions (or different components of a single solution) have quite disparate behavior that makes the selection of the step size in the numerical solution problematic. For example, one component of a function may require a small step in the numerical solution because it is varying rapidly, whereas another component may vary slowly and not require a small step size for its computation. Such a system is said to be stiff. Figure 11.2 illustrates a slowly varying solution surrounded by other solutions with rapidly decaying transients. An example will illustrate this possibility. Consider a system of two differential equa- tions with initial conditions: 􏰎 x′ =−20x−19y x(0)=2 y′ =−19x−20y y(0)=0 (5) 490 Chapter 11 Systems of Ordinary Differential Equations x FIGURE 11.2 Solution curves for a stiff ode t The solution is easily seen to be x(t)=e−39t +e−t y(t)=e−39t −e−t The component e−39t quickly becomes negligible as t increases, starting at 0. The solution is then approximately given by x(t) = −y(t) = e−t, and this function is smooth and decreasing to 0. It would seem that in almost any numerical solution, a large step size could be used. However, let us examine the simplest of numerical procedures: Euler’s method. It generates the solution by using the following equations: xn+1 =xn +h(−20xn −19yn) x0 =2 yn+1 =yn +h(−19xn −20yn) y0 =0 These difference equations can be solved in closed form, and we have xn =(1−39h)n +(1−h)n yn =(1−39h)n −(1−h)n For the numerical solution to converge to 0 (and thus imitate the actual solution), it is necessary that h < 2 . If we were solving only the differential equation x′ = −x to get the 39 solution x(t) = e−t , the step size could be as large as h = 2 to get the correct behavior as t increased. (See Problem 11.3.2.) To see that numerical success (in the sense of being able to use a reasonable step size) depends on the method used, let us consider the implicit Euler method. For a single differential equation, this employs the formula xn+1 =xn +hf(tn+1,xn+1) Since xn+1 appears on both sides of this equation, the equation must be solved for xn+1. In the example being considered, the Euler equations are xn+1 = xn + h(−20xn+1 − 19yn+1) yn+1 = yn + h(−19xn+1 − 20yn+1) This pair of equations has the form Xn+1 = Xn + AXn+1, where A is the 2×2 matrix in the previous pair of equations and Xn is the vector having components xn and yn. This matrix equation can be written (I − A)Xn+1 = Xn or Xn+1 = (I − A)−1 Xn. A consequence is that the explicit solution is Xn = (I − A)−n X0. At this point, it is necessary to appeal to a result concerning such iterative processes. For Xn to converge to 0 for any choice of initial vector X0, it is necessary and sufficient that all eigenvalues of (I − A)−1 be less than one in 11.3 Adams-Bashforth-Moulton Methods 491 modulus (see Kincaid and Cheney [2002]). Equivalently, the eigenvalues of I − A should be greater than 1 in modulus. An easy calculation shows that for positive h this condition is met, without further hypotheses. Thus, the implicit Euler method can be used with any reasonable step size on this problem. In the literature on stiff equations, much more infor- mation can be found, and there are books that address this topic thoroughly. Some essential references are Dekker and Verwer [1984], Gear [1971], Miranker [1981], and Shampine and Gordon [1975]. In general, stiff ordinary differential equations are rather difficult to solve. This is com- pounded by the fact that in most cases, one does not know beforehand whether an ordinary differential equation that one is trying to solve numerically is stiff. Software packages usu- ally have ordinary differential equation solvers specifically designed to handle stiff ordinary differential equations. Some of these procedures may vary both the step size and the order of the method. In such algorithms, the Jacobian matrix ∂ F /∂ X y may play a role. Solving an associated linear system involving the Jacobian matrix is critical to the reliability and efficiency of the code. The Jacobian matrix may be sparse, an indication that the function F does not depend on some of the variables in the problem. For readers who are interested in the history of numerical analysis, we recommend the book by Goldstine [1977]. The textbook on differential equations by Moulton [1930] gives some insight into the numerical methods used prior to the advent of high-speed computing machines. He also (page 224) gives some of the history, going back to Newton! The calculation of orbits in celestial mechanics has always been a stimulus for the invention of numerical methods; so also have been the needs of ballistic science. Moulton mentions that the retardation of a projectile by air resistance is a very complicated function of velocity that necessitates numerical solution of the otherwise simple equations of ballistics. Summary (1) For the autonomous form for a system of ordinary differential equations in vector notation 􏰘X′ = F(X) X(a) = S, given the Adams-Bashforth-Moulton method of fourth order is X􏶗(t + h) = X(t) + h 􏶮55F[X(t)] − 59F[X(t − h)] + 37F[X(t − 2h)] 24 􏶯 − 9F[X(t − 3h)] X(t + h) = X(t) + h 􏶮9F[X􏶗(t + h)] + 19F[X(t)] − 5F[X(t − h)] 24 􏶯 + F[X(t − 2h)] Here, X􏶗 (t + h) is the predictor, and X(t + h) is the corrector. The Adams-Bashforth- Moulton method needs five evaluations of F per step. With the initial vector X(a) given, the values for X(a + h), X(a + 2h), X(a + 3h) are computed by the Runge-Kutta method of fourth order. Then the Adams-Bashforth-Moulton method can be used repeatedly. The predicted value X􏶗 is computed from the four X values at t, t − h, t − 2h, and t − 3h, and then the corrected value X(t + h) can be computed by using the predictor value X􏶗(t + h) and previously evaluated values of F at t, t − h, and t − 2h. 492 Chapter 11 Systems of Ordinary Differential Equations Additional References See Aiken [1985], Ascher and Petzold [1998], Boyce and DiPrima [2003], Butcher [1987], Carrier and Pearson [1991], Chicone [2006], Collatz [1966], Dekker and Verwer [1984], Edwards and Penny [2004], England [1969], Enright [2006], Fehlberg [1969], Gear [1971], Golub and Ortega [1992], Henrici [1962], Hull et al. [1972], Hundsdorfer [1985], Lambert [1973, 1991], Lapidus and Seinfeld [1971], Miranker [1981], Moulton [1930], and Shampine and Gordon [1975]. Problems 11.3 a 1. Find the general solution of this system by turning it into a first-order system of four equations: 􏰎 x′′ =αy y′′ =βx 2. Verify the assertions made about the step size h in the discussion of stiff equations. 1. Test the procedure AMRK on the system given in Computer Problem 11.2.2. 2. The single-step error is closely controlled by using fourth-order formulas; however, the roundoff error in performing the computations in Equations (3) and (4) can be large. It is logical to carry these out in what is known as partial double-precision Computer Problems 11.3 arithmetic. The function F would be evaluated in single precision at the desired points X(t + ih), but the linear combination 􏰍 ci F(X(t + ih)) would be accumulated in i double precision. Also, the addition of X(t) to this result is done in double precision. Recode the Adams-Moulton method so that partial double-precision arithmetic is used. Compare this code with that in the text for a system with a known solution. How do they compare with regard to roundoff error at each step? 3. WriteandtestanadaptiveprocesssimilartoRK45 AdaptiveinSection10.3withcalling sequence procedure AMRK Adaptive(n,h,ta,tb,(xi), itmax,εmin, εmax, hmin, hmax, iflag) This routine should carry out the adaptive procedure outlined in this section and be used in place of the AMRK procedure. 4. Solve the predator-prey problem in the example at the beginning of this chapter with a = −10−2, b = −1 ×102, c = 10−2 and d = −102 and with initial values u(0) = 80, 4 v(0) = 30. Plot u (the prey) and v (the predator) as functions of time t. 5. Solve and plot the numerical solution of the system of ordinary differential equa- tions given by Equation (4) using mathematical software such as Matlab, Maple, or Mathematica. 11.3 Adams-Bashforth-Moulton Methods 493 6. (Continuation)RepeatforEquation(5)usingaroutinespecificallydesignedtohandle stiff ordinary differential equations. 7. Solvethefollowingtestproblemsandplottheirsolutioncurves. a. This problem corresponds to a recently discovered stable orbit that arises in the restricted three-body problem in which the orbits are co-planar. The two spatial coordinates of the j th body are x1 j and x2 j for j = 1, 2, 3. Each of the six coordinates satisfies a second-order differential equation: ′′ 􏰃3 􏰉 􏰈3 mk xik−xij /djk whered2 =􏰍2 (x −x )2fork,j=1,2,3.Assumethatthebodieshaveequal xij= jk i=1 ij ik k=1 k̸=j mass, say, m1 = m2 = m3 = 1, and with the appropriate starting conditions, they will follow the same figure-eight orbit as a periodic steady-state solution. When the system is rewritten as a first order system, the dimension of the problem is 12, and the initial conditions at t = 0 are given by ⎧x ⎪ 11 ⎪x ⎪ 21 = −0.97000436 = 0.24308753 = 0.0 = 0.0 x′ = 0.466203685 11 x′ = 0.43236573 21 x′ = −0.93240737 12 x′ = −0.86473146 22 x′ = 0.466203685 13 x′ = 0.43236573 23 ⎨x ⎪x 12 ⎪ 22 ⎪x ⎪⎩ 13 = 0.97000436 = −0.24308753 x Solve the problem for t ∈ [0, 20]. 23 b. TheLorenzproblemiswellknown,anditarisesinthestudyofdynamicalsystems: ⎧⎪x1′ =10(x2−x1) ⎨x2′ =x1(28−x3)−x2 ⎪ ⎪ ⎪ x 3′ = x 1 x 2 − 8 x 3 ⎩3 x1(0) = 15, x2(0) = 15, x3(0) = 36 Solve the problem for t ∈ [0, 20]. It is known to have solutions that are potentially poorly conditioned. For additional details on these problems, see Enright [2006]. 8. Write a computer program based on pseudocode Test AMRK to find the numerical solution to the ordinary differential equation systems, and compare the results with that by using a built-in routine such as can be found in Matlab, Maple, or Mathematica. Plot the resulting solution curves. 9. (Tacoma Narrows Bridge project) In 1940, the third longest suspension bridge in the world collapsed in a high wind. The following system of differential equations is a mathematical model that attempts to explain how twisting oscillations can be magnified 494 Chapter 11 Systems of Ordinary Differential Equations and cause such a calamity: 􏰎 ′′ ′ 􏰠 a(y−lsinθ) a(y+lsinθ) 􏰡 y =−yd−[K/(ma)] e −1+e −1 +0.2Wsinωt θ′′ = −θy′d +(3cosθ/l)[K/(ma)]􏰠ea(y−lsinθ) −ea(y+lsinθ)􏰡 The last term in the y equation is the forcing term for the wind W , which adds a strictly vertical oscillation to the bridge. Here, the roadway has width 2l hanging between two suspended cables, y is the current distance from the center of the roadway as it hangs below its equilibrium point, and θ is the angle the roadway makes with the horizontal. Also, Newton’s Law F = ma is used and Hooke’s constant K . Explore how ODE solvers are used to generate numerical trajectories for various parameter settings. Illustrate different types of phenomena that are available in this model. For additional details, see McKenna and Tuama [2001] and Sauer [2006]. 12 Smoothing of Data and the Method of Least Squares Surface tension S in a liquid is known to be a linear function of temperature T . For a particular liquid, measurements have been made of the surface tension at certain temperatures. The results were as follows: T 0 10 20 30 40 80 90 95 S 68.0 67.1 66.4 65.6 64.6 61.8 61.0 60.0 How can the most probable values of the constants in the equation S = aT + b be determined? Methods for solving such problems are developed in this chapter. 12.1 Method of Least Squares Linear Least Squares In experimental, social, and behavioral sciences, an experiment or survey often produces a mass of data. To interpret the data, the investigator may resort to graphical methods. For instance, an experiment in physics might produce a numerical table of the form x x0 x1 ··· xm y y0 y1 ··· ym (1) and from it, m + 1 points on a graph could be plotted. Suppose that the resulting graph looks like Figure 12.1. A reasonable tentative conclusion is that the underlying function is linear and that the failure of the points to fall precisely on a straight line is due to experimental error. If one proceeds on this assumption—or if theoretical reasons exist for believing that the function is indeed linear—the next step is to determine the correct function. Assuming that y = ax + b what are the coefficients a and b? Thinking geometrically, we ask: What line most nearly passes through the eight points plotted? 495 496 Chapter 12 Smoothing of Data and the Method of Least Squares y7 y y5 y6 4 y2 y3 y0 y1 FIGURE 12.1 Experimental data x0 x1x2x3 x4 x5 x6x7 x To answer this question, suppose that a guess is made about the correct values of a and b. This is equivalent to deciding on a specific line to represent the data. In general, the data points will not fall on the line y = ax + b. If by chance the kth datum falls on the line, then axk +b−yk =0 If it does not, then there is a discrepancy or error of magnitude | axk + b − yk | The total absolute error for all m + 1 points is therefore 􏰃m | axk + b − yk | k=0 This is a function of a and b, and it would be reasonable to choose a and b so that the function assumes its minimum value. This problem is an example of l1 approximation and can be solved by the techniques of linear programming, a subject dealt with in Chapter 17. (The methods of calculus do not work on this function because it is not generally differentiable.) In practice, it is common to minimize a different error function of a and b: 􏰃m k=0 This function is suitable because of statistical considerations. Explicitly, if the errors follow a normal probability distribution, then the minimization of φ produces a best estimate of a and b. This is called an l2 approximation. Another advantage is that the methods of calculus can be used on Equation (2). The l1 and l2 approximations are related to specific cases of the lp norm defined by 􏶬 1 / p ∥x∥p = forthevectorx =[x1,x2,...,xn]T. (1􏶞p<∞) φ(a,b)= (axk +b−yk)2 (2) 􏰎 􏰃n i=1 |xi|p Let us try to make φ(a, b) a minimum. By calculus, the conditions ∂φ = 0 ∂φ = 0 ∂a ∂b 12.1 Method of Least Squares 497 (partial derivatives of φ with respect to a and b, respectively) are necessary at the minimum. Taking derivatives in Equation (2), we obtain notation, we set ⎧⎪ ⎪ 􏰃m ⎪⎨ 2(axk+b−yk)xk=0 k=0 ⎪ 􏰃m 2(axk +b−yk)=0 This is a pair of simultaneous linear equations in the unknowns a and b. They are called ⎩ the normal equations and can be written as ⎧􏶠􏰃m 􏶡 􏶠􏰃m 􏶡 􏰃m k=0 ⎪⎨ xk2 a+ xk b= ykxk 􏶠k=0 􏶡 k=0 k=0 (3) ⎪ ⎪ ⎪ 􏰃m 􏰃m ⎪⎩ x k a + ( m + 1 ) b = y k k=0 k=0 Here, of course, 􏰍mk=0 1 = m + 1, which is the number of data points. To simplify the 􏰃n 􏰃n 􏰃n 􏰃n p= xk q= yk r= xkyk k=0 s= xk2 k=0 The system of Equations (3) is now k=0 k=0 􏰖 􏰗􏰖􏰗􏰖􏰗 spa=r pm+1bq We solve this pair of equations by Gaussian elimination and obtain the following algorithm. Alternatively, since this is a 2 × 2 linear system, we can use Cramer’s Rule∗ to solve it. The determinant of the coefficient matrix is 􏰖􏰗 s p = (m + 1)s − p2 p m+1 􏰗 = 1 [(m + 1)r − pq] b=dDet p q =d[sq−pr] We can write this as an algorithm: Linear Least Squares d = Det Moreover, we obtain 􏰖 a = 1 Det 1􏰖sr􏰗1 r p dqm+1d ■ ALGORITHM1 The coefficients in the least-squares line y = ax + b through the set of m + 1 data points (xk, yk) for k = 0,1,2,...,m are computed (in order) as follows: 1. p=􏰍mk=0xk 2. q = 􏰍mk=0 yk ∗Cramer’s Rule is given in Appendix D. 498 Chapter 12 Smoothing of Data and the Method of Least Squares 3. r = 􏰍mk=0 xk yk 4. s = 􏰍mk=0 xk2 5. d=(m+1)s−p2 6. a=[(m+1)r−pq]/d 7. b=[sq−pr]/d EXAMPLE 1 Solution The preceding analysis illustrates the least-squares procedure in the simple linear case. As a concrete example, find the linear least-squares solution for the following table of values: x 4 7 11 13 17 y20267 Plot the original data points and the line using a finer set of grid points. The equations in Algorithm 1 leads to this system of two equations: Another form of this result is where Linear Example 􏰙 􏶠􏰃m 􏶡 􏶠􏰃m 􏶡􏶠􏰃m 􏶡􏰚 a=1 (m+1) xkyk − d k=0 xk yk k=0 k=0 􏰙􏶠􏰃m􏶡􏶠􏰃m􏶡􏶠􏰃m􏶡􏶠􏰃m􏶡􏰚 (4) b=1xk2 yk−xk xkyk d k=0 k=0 k=0 k=0 􏶠􏰃m 􏶡 􏶠􏰃m 􏶡2 d=(m+1) xk2 − xk k=0 k=0 􏰎 whose solution is a = 0.4864 and b = −1.6589. By Equation (3), we obtain the value φ(a,b) = 10.7810. Figure 12.2 is a plot of the given data and the linear least squares straight line. y 644a + 52b = 227 52a+ 5b=17 FIGURE 12.2 Linear least squares 10 8 6 4 2 0 –2 0 2 4 6 8 10 12 14 16 18 20 x ■ We can use mathematical software such as Matlab, Maple, or Mathematica to fit a lin- ear least-squares polynomial to the data and verify the value of φ. (See Computer Prob- lem 12.1.5.) To understand what is going on here, we want to determine the equation of a line of the form y = ax + b that fits the data best in the least-squares sense. With four data points (xi,yi),wehavefourequationsyi =axi +bfori=1,2,3,4thatcanbewrittenas where Ax = y ⎡⎤⎡⎤ ⎢x1 1⎥􏰖 􏰗 ⎢y1⎥ ⎢x2 1⎥ a =⎢y2⎥ ⎣x3 1⎦b ⎣y3⎦ x41 y4 In general, we want to solve a linear system Ax = b where A is an m × n matrix and m > n. The solution coincides with the solution of the
normal equations
AT Ax = AT b This corresponds to minimizing ||Ax − b||2.
Nonpolynomial Example
The method of least squares is not restricted to linear (first-degree) polynomials or to any specific functional form. Suppose, for instance, that we want to fit a table of values (xk , yk ), where k = 0,1,…,m, by a function of the form
y = a ln x + b cos x + cex
in the least-squares sense. The unknowns in this problem are the three coefficients a, b, and c. We consider the function
equations:
⎧⎪ ⎪ 􏰃m
􏰃m k=0
􏰃m k=0
(lnxk)exk (cosxk)exk (exk )2
􏰃m
􏰃m k=0
(alnxk +bcosxk +cexk −yk)2
φ(a,b,c)=
and set ∂φ/∂a = 0, ∂φ/∂b = 0, and ∂φ/∂c = 0. This results in the following three normal
(lnxk)2
⎨􏰃m 􏰃m 􏰃m 􏰃m
⎪a
⎪ k=0
+ b (lnxk)(cosxk) + b
(lnxk)(cosxk) + c
= = =
yk lnxk yk cosxk yk exk
⎪a
⎪ k=0
k=0 􏰃m
(cosxk)2 (cos xk )exk
+ c + c
k=0 􏰃m
k=0 􏰃m
⎪ ⎪ ⎪ 􏰃m ⎪⎩a
(ln xk )exk
+ b
k=0
k=0
k=0
k=0
12.1 Method of Least Squares 499
k=0

500
Chapter 12
Smoothing of Data and the Method of Least Squares
EXAMPLE 2
Solution
Fitafunctionoftheformy=alnx+bcosx+cex tothefollowingtablevalues:
x 0.24 0.65 0.95 1.24 1.73 2.01 2.23 2.52 2.77 2.99
y 0.23 −0.26 −1.10 −0.45 0.27 0.10 −0.29 0.24 0.56 1.00 Using the table and the equations above, we obtain the 3 × 3 system
⎧
⎪⎨ 6.79410a − 5.34749b + 63.25889c = 1.61627
⎪⎩ −5.34749a + 5.10842b − 49.00859c = −2.38271 63.25889a − 49.00859b + 1002.50650c = 26.77277
It has the solution a = −1.04103, b = −1.26132, and c = 0.03073. So the curve y = −1.04103 ln x − 1.26132 cos x + 0.03073ex
has the required form and fits the table in the least-squares sense. The value of φ(a, b, c) is 0.92557. Figure 12.3 is a plot of the given data and the nonpolynomial least squares curve.
FIGURE 12.3
Nonpolynomial least squares
y
1 0.5 0 􏶼0.5 􏶼1
􏶼1.5 x 0 0.5 1 1.5 2 2.5 3
■
We can use mathematical software such as Matlab, Maple, or Mathematica to verify these results and to plot the solution curve. (See Computer Problem 12.1.6.)
Basis Functions {g0, g1, . . . , gn}
The principle of least squares, illustrated in these two simple cases, can be extended to general linear families of functions without involving any new ideas. Suppose that the data in Equation (1) are thought to conform to a relationship such as
􏰃n
y= cjgj(x) (5)
j=0
inwhichthefunctionsg0,g1,…,gn (calledbasisfunctions)areknownandheldfixed.The coefficientsc0,c1,…,cn aretobedeterminedaccordingtotheprincipleofleastsquares.

In other words, we define the expression
φ(c0,c1,…,cn) = cjgj(xk)− yk
􏰚2
k=0
When set equal to zero, the resulting equations can be rearranged as
􏰃m 􏰙􏰃n k=0 j=0
(6)
and select the coefficients to make it as small as possible. Of course, the expression φ(c0,c1,…,cn) is the sum of the squares of the errors associated with each entry (xk, yk) in the given table.
Proceeding as before, we write down as necessary conditions for the minimum the n equations
∂φ =0 (0􏶞i􏶞n) ∂ci
These partial derivatives are obtained from Equation (7). Indeed,
􏰃n􏰙􏰃m 􏰚 􏰃m
gi(xk)gj(xk) cj = ykgi(xk) (0􏶞i􏶞n) (7)
j=0 k=0 k=0
These are the normal equations in this situation and serve to determine the best values of the parameters c0 , c1 , . . . , cn . The normal equations are linear in ci ; thus, in principle, they can be solved by the method of Gaussian elimination (see Chapter 7).
In practice, the normal equations may be difficult to solve if care is not taken in choosing the basis functions g ,g ,…,g . First, the set {g ,g ,…,g } should be linearly inde-
01n 􏰍01n pendent.Thismeansthatnolinearcombination n cigi canbethezerofunction(except
i=0
in the trivial case when c0 = c1 = ··· = cn = 0). Second, the functions g0,g1,…,gn
should be appropriate to the problem at hand. Finally, one should choose a set of basis functions that is well conditioned for numerical work. We elaborate on this aspect of the problem in the next section.
Summary
(1) We wish to find a line y = ax + b that most nearly passes through the m + 1 pairs of points (xi, yi) for 0􏶞 i 􏶞 m. An example of l1 approximation is to choose a and b so that the total absolute error for all these points is minimized:
􏰃m
| axk + b − yk |
k=0
This can be solved by the techniques of linear programming.
(2) An l2 approximation will minimize a different error function of a and b:
􏰃m k=0
12.1 Method of Least Squares 501
∂φ 􏰃m 􏰙􏰃n 􏰚
∂c = i
2 cj gj(xk) − yk j=0
gi(xk) (0􏶞 i 􏶞 n)
φ(a,b)=
(axk +b−yk)2

502 Chapter 12
Smoothing of Data and the Method of Least Squares
The minimization of φ produces a best estimate of a and b in the least-squares sense. One solves the normal equations
⎧􏶠􏰃m 􏶡 􏶠􏰃m 􏶡 􏰃m
⎪⎨ xk2 a+ xk b= ykxk
􏶠k=0 􏶡 k=0 k=0
k=0 k=0
(3) In a more general case, the data points conform to a relationship such as
􏰃n
y= cjgj(x)
j=0
in which the basis functions g0 , g1 , . . . , gn are known and held fixed. The coefficients c0,c1,…,cn aretobedeterminedaccordingtotheprincipleofleastsquares.Thenormal equations in this situation are
􏰃n􏰙􏰃m 􏰚 􏰃m
gi(xk)gj(xk) cj = ykgi(xk) (0􏶞i􏶞n)
j=0 k=0 k=0
and can be solved, in principle, by the method of Gaussian elimination to determine the
best values of the parameters c0,c1,…,cn.
a1. Usingthemethodofleastsquares,findtheconstantfunctionthatbestfitsthefollowing data:
x −1 2 3
545 4 3 12
a2. Determinetheconstantfunctioncthatisproducedbytheleast-squarestheoryapplied to the Table on p. 495. Does the resulting formula involve the points xk in any way? Apply your general formula to the preceding problem.
a3. Find an equation of the form y = aex2 + bx3 that best fits the points (−1,0), (0,1), and (1, 2) in the least-squares sense.
4. SupposethatthexpointsinTable(1)aresituatedsymmetricallyabout0onthex-axis. In this case, there is an especially simple formula for the line that best fits the points. Find it.
a5. Findtheequationofaparabolaofformy=ax2+bthatbestrepresentsthefollowing data. Use the method of least squares.
x −1 0 1 y 3.1 0.9 2.9
6. Suppose that Table (1) is known to conform to a function like y = x2 − x + c. What value of c is obtained by the least-squares theory?
⎪ ⎪ ⎪ 􏰃m
􏰃m ⎪⎩ x k a + ( m + 1 ) b = y k
y
Problems 12.1

a 7. 8.
9.
10.
12.1 Method of Least Squares 503 Suppose that Table (1) is thought to be represented by a function y = c log x . If so,
what value for c emerges from the least-squares theory? Show that Equation (4) is the solution of Equation (3).
(Continuation) How do we know that divisor d is not zero? In fact, show that d is positive for m 􏶟 1. Hint: Show that
m k−1 􏰃􏰃
(xk − xl )2
by induction on m. The Cauchy-Schwarz inequality can also be used to prove that
d > 0.
(Continuation) Show that a and b can also be computed as follows:
d =
k=0 l=0
1 􏰃m
􏶤x = m + 1
􏰃m
c =
Hint: Show that d = (m + 1)c.
Howdoweknowthatthecoefficientsc0,c1,…,cn thatsatisfythenormalEquations(7) do not lead to a maximum in the function defined by Equation (6)?
If Table (1) is thought to conform to a relationship y = log(cx), what is the value of c obtained by the method of least squares?
Whatstraightlinebestfitsthefollowingdata x1234
y0112
in the least-squares sense?
Inanalyticgeometry,welearnthatthedistancefromapoint(x0,y0)toalinerepresented by the equation ax + by = c is (ax0 + by0 − c)(a2 + b2)−1/2. Determine a straight line that fits a table of data points (xi , yi ), for 0 􏶞 i 􏶞 m, in such a way that the sum of the squares of the distances from the points to the line is minimized.
Show that if a straight line is fitted to a table (xi , yi ) by the method of least squares, then the line will pass through the point (x∗, y∗), where x∗ and y∗ are the averages of the xi ’s and yi ’s, respectively.
The viscosity V of a liquid is known to vary with temperature according to a quadratic law V = a + bT + cT2. Find the best values of a, b, and c for the following table:
T1234567 V 2.31 2.01 1.80 1.66 1.55 1.47 1.41
x k
1 􏰃m
􏶤y = m + 1 y k
k=0
a = c (xk − 􏶤x)(yk − 􏶤y) b = 􏶤y − a􏶤x
k=0 (xk − 􏶤x )2
1 􏰃m k=0
a11. a12. a13.
14.
15.
a 16.
k=0

504 Chapter 12 Smoothing of Data and the Method of Least Squares
17. Anexperimentinvolvestwoindependentvariablesxandyandonedependentvariable z. How can a function z = a + bx + cy be fitted to the table of points (xk , yk , zk )? Give the normal equations.
a18. Findthebestfunction(intheleast-squaressense)thatfitsthefollowingdatapointsand isoftheform f(x)=asinπx+bcosπx:
x −1 −1 0 1 1 22
y −1 0 1 2 1
a19. Find the quadratic polynomial that best fits the following data in the sense of least
squares:
x −2 −1 0 1 2
y21112
a20. Whatlinebestrepresentsthefollowingdataintheleast-squaressense?
x012
y 5 −6 7
a21. Whatconstantcmakestheexpression
as small as possible?
􏰃m
[ f (xk ) − cexk ]2
k=0
22. Show that the formula for the best line to fit data (k, yk) at the integers k for 1􏶞 k 􏶞 n is
where
k=1 k=1
23. Establish the normal equations and verify the results in Example 1.
24. Avectorvisassertedtobetheleast-squaressolutionofaninconsistentsystemAx=b. How can we test v without going through the entire least-squares procedure?
25. Find the normal equations for the following data points:
x 1.0 2.0 2.5 3.0
y 3.7 4.1 4.3 5.0
Determine the straight line that best fits the data in the least-squares sense. Plot the data point and the least-squares line.
26. For the case n = 4, show directly that by forming the normal equations from the data points (xi , yi ), we obtain the results in Theorem 1.
y = ax + b
􏰙􏰃n 􏰃n􏰚
6 2 kyk − (n + 1) yk n(n2 − 1)
a =
b=n(n−1) (2n+1) yk−3 kyk
􏰙 k=1 k=1 􏰚 2 􏰃n 􏰃n

12.2 Orthogonal Systems and Chebyshev Polynomials 505
1. Write a procedure that sets up the normal Equations (7). Using that procedure and other routines, such as Gauss and Solve from Chapter 7, verify the solution given for the problem involving ln x , cos x , and ex in the subsection entitled “Nonpolynomial Example.”
2. Write a procedure that fits a straight line to Table (1). Use this procedure to find the constants in the equation S = aT + b for the table in the example that begins this chapter. Also, verify the results obtained for the problem in the subsection entitled “Linear Example.”
3. Writeandtestaprogramthattakesm+1pointsintheplane(xi,yi),where0􏶞i􏶞m, with x0 < x1 < ··· < xm, and computes the best linear fit by the method of least squares. Then the program should create a plot of the points and the best line determined by the least-squares method. 4. The Internal Revenue Service (IRS) publishes the following table of values having to do with minimal distributions of pension plans: x12345678 y 29.9 29.0 28.1 27.1 26.2 25.3 24.4 23.6 9 10 11 12 13 14 15 16 22.7 21.8 21.0 20.1 19.3 18.5 17.7 16.9 What simple function represents the data? Use Equation (5), and plot the data and the results using either plotting software such as gnuplot or some mathematics software system such as Maple, Matlab, or Mathematica. 5. UsingmathematicalsoftwaresuchasMatlab,Maple,orMathematica,fitalinearleast- squares polynomial to the data in Example 1. Then plot the original data and the polynomial using a fine set of grid points. 6. (Continuation) Verify the results in Example 2 and plot the curve. 12.2 Orthogonal Systems and Chebyshev Polynomials Orthonormal Basis Functions {g0, g1, . . . , gn} Once the functions g0 , g1 , . . . gn of Equation (5) in Section 12.1 have been chosen, the least-squares problem can be interpreted as follows: The set of all functions g that can be expressedaslinearcombinationsofg0,g1,...,gn isavectorspaceG.(Familiaritywith vector spaces is not essential to understanding the discussion here.) In symbols, we have 􏰎 􏰃n􏶬 G= g:thereexistc0,c1,...,cn suchthatg(x)= cjgj(x) j=0 Computer Problems 12.1 506 Chapter 12 Smoothing of Data and the Method of Least Squares The function that is being sought in the least-squares problem is thus an element of the vectorspaceG.Sincethefunctionsg0,g1,...,gn formabasisforG,thesetisnotlinearly dependent. However, a given vector space has many different bases, and they can differ drastically in their numerical properties. Let us turn our attention away from the given basis {g0 , g1 , . . . , gn } to the vector space G generated by that basis. Without changing G, we ask: What basis for G should be chosen for numerical work? In the present problem, the principal numerical task is to solve the normal equations—that is, Equation (7) in Section 12.1: 􏰃n􏰙􏰃m 􏰚 􏰃m gi(xk)gj(xk) cj = ykgi(xk) (0􏶞i􏶞n) (1) j=0 k=0 k=0 The nature of this system obviously depends on the basis {g0,g1,...,gn}. We want these equations to be easily solved or to be capable of being accurately solved. The ideal situation occurs when the coefficient matrix in Equation (1) is the identity matrix. This happens if the basis {g0,g1,...,gn} has the property of orthonormality: 􏰃m 􏰘1i=j gi(xk)gj(xk)=δij= 0 i≠ j k=0 In the presence of this property, Equation (1) simplifies dramatically to 􏰃m cj = ykgj(xk) (0􏶞 j􏶞n) k=0 which is no longer a system of equations to be solved but rather an explicit formula for the coefficients c j . Under rather general conditions, the space G has a basis that is orthonormal in the sense just described. A procedure known as the Gram-Schmidt process can be used to obtain such a basis. There are some situations in which the effort of obtaining an orthonormal basis is justified, but simpler procedures often suffice. We describe one such procedure now. Remember that our goal is to make Equation (1) well disposed for numerical solution. We want to avoid any matrix of coefficients that involves the difficulties encountered in connection with the Hilbert matrix (see Computer Problem 7.2.4). This objective can be met if the basis for the space G is well chosen. We now consider the space G that consists of all polynomials of degree 􏶞 n, which is an important example of the least-squares theory. It may seem natural to use the following n + 1 functions as a basis for G: g0(x) = 1 g1(x) = x g2(x) = x2 ... gn(x) = xn Using this basis, we write a typical element of the space G in the form 􏰃n g(x)= j=0 cjgj(x)= 􏰃n j=0 cjxj =c0 +c1x+c2x2 +···+cnxn This basis, however natural, is almost always a poor choice for numerical work. For many purposes, the Chebyshev polynomials (suitably defined for the interval involved) do form a good basis. Figure12.4givesanindicationofwhythemonomialsxj donotformagoodbasisfor numerical work: These functions are too much alike! If a function g is given and we wish y 1 0.5 T5 T4 x x2 x3 x4 12.2 Orthogonal Systems and Chebyshev Polynomials 507 T1 0x 0.2 0.4 x5 0.6 0.8 1 􏶼0.5 􏶼1 T2 T3 FIGURE 12.4 Polynomials xk and Chebyshev polynomials Tk to express it as a linear combination of the monomials, g(x) = 􏰍nj=0 cjxj, it is difficult to determine the coefficients cj precisely. Figure 12.4 also shows a few of the Chebyshev polynomials; they are quite different from one another. For simplicity, assume that the points in our least-squares problem have the property −1=x0 n) (6) Statistical theory tells us that if the trend of the table is truly a polynomial of degree N (but
σn2 =m−n infected by noise), then
i=0
■ PROPERTIES
σ2 >σ2 >···>σ2 =σ2 =σ2 =···=σ2
0 1 N N+1 N+2 m−1
This fact suggests the following strategy for dealing with the case in which N is not known: Compute σ02 , σ12 , . . . in succession. As long as these are decreasing significantly, continue the calculation. When an integer N is reached for which σN2 ≈ σN2 +1 ≈ σN2 +2 ≈ · · · , stop and declare pN to be the polynomial sought.
If σ02 , σ12 , . . . are to be computed directly from the definition in Equation (6), then each of the polynomials p0 , p1 , . . . will have to be determined. The procedure described below can avoid the determination of all but the one desired polynomial.
In the remainder of the discussion, the abscissas xi are to be held fixed. These points are assumed to be distinct, although the theory can be extended to include cases in which some points repeat. If f and g are two functions whose domains include the points {x0, x1, . . . , xm }, then the following notation is used:
􏰃m i=0
This quantity is called the inner product of f and g. Much of our discussion does not depend on the exact form of the inner product but only on certain of its properties. An inner product ⟨· , ·⟩ has the following properties:
Defining Properties of an Inner Product
The reader should verify that the inner product defined in Equation (7) has the properties listed.
A set of functions is now said to be orthogonal if ⟨ f, g⟩ = 0 for any two different
functions f and g in that set. An orthogonal set of polynomials can be generated recursively
by the following formulas:
⎧
⎪⎨ q 0 ( x ) = 1
⎪⎩ q1(x)=x−α0
qn+1(x) = xqn(x) − αnqn(x) − βnqn−1(x) (n 􏶟 1)
⟨ f, g⟩ =
f (xi )g(xi ) (7)
1. ⟨f,g⟩=⟨g,f⟩
2. ⟨f, f⟩>0unless f(xi)=0foralli 3. ⟨af,g⟩=a⟨f,g⟩wherea∈R
4. ⟨f,g+h⟩=⟨f,g⟩+⟨f,h⟩

where
⎧⎪⎨αn = ⟨xqn,qn⟩ ⟨qn,qn⟩
⎪⎩βn = ⟨xqn,qn−1⟩ ⟨qn−1, qn−1⟩
12.2 Orthogonal Systems and Chebyshev Polynomials 513
In these formulas, a slight abuse of notation occurs where “xqn” is used to denote the function whose value at x is xqn(x).
To understand how this definition leads to an orthogonal system, let’s examine a few cases. First,
⟨q1,q0⟩=⟨x−α0,q0⟩=⟨xq0 −α0q0,q0⟩=⟨xq0,q0⟩−α0⟨q0,q0⟩=0
Notice that several properties of an inner product listed previously have been used here.
Also, the definition of α0 was used. Another of the first few cases is this: ⟨q2, q1⟩ = ⟨xq1 − α1q1 − β1q0, q1⟩
= ⟨xq1, q1⟩ − α1⟨q1, q1⟩ − β1⟨q0, q1⟩ = 0
Here, the definition of α1 has been used, as well as the fact (established above) that ⟨q1 , q0 ⟩ = 0. The next step in a formal proof is to verify that ⟨q2,q0⟩ = 0. Then an inductive proof completes the argument.
One part of this proof consists in showing that the coefficients αn and βn are well defined. This means that the denominators ⟨q , q ⟩ are not zero. To verify that this is the
􏰍m n n
case,supposethat⟨qn,qn⟩=0.Then i=0[qn(xi)]2 =0,andconsequently,qn(xi)=0for
eachvalueofi.Thismeansthatthepolynomialqn hasm+1roots,x0,x1,…,xm.Since the degree n is less than m, we conclude that qn is the zero polynomial. However, this is not possible because obviously
q0(x) = 1
q1(x) = x − α0
q2(x) = x2 + (lower-order terms)
and so on. Observe that this argument requires n < m. The system of orthogonal polynomials {q0, q1, . . . , qm−1} generated by the above algo- rithm is a basis for the vector space 􏶥 clearfromthealgorithmthateachqn startswiththehighesttermxn.Ifitisdesiredtoexpress agivenpolynomial pofdegreen(n􏶞m−1)asalinearcombinationofq0,q1,...,qn,this can be done as follows: Set m−1 of all polynomials of degree at most m − 1. It is 􏰃n i=0 On the right-hand side, only one summand contains x n . It is the term an qn . On the left-hand side, there is also a term in xn. One chooses an so that anxn on the right is equal to the corresponding term in p. Now write n−1 􏰃 p = ai qi (8) p−anqn = aiqi i=0 514 Chapter 12 Smoothing of Data and the Method of Least Squares On both sides of this equation, there are polynomials of degree at most n − 1 (because of the choice of an). Hence, we can now choose an−1 in the way we chose an; that is, choose an−1 so that the terms in xn−1 are the same on both sides. By continuing in this way, we discover take the inner product of both sides of Equation (8) with q j . The result is ⟨p,qj⟩ = aj⟨qj,qj⟩ the unique values that the coefficients a must have. This establishes that {q , q , . . . , q } is 􏶥 i01n n, for n = 0,1,...,m − 1. a basis for Another way of determining the coefficients ai (once we know that they exist!) is to 􏰃n 􏰙􏰃m 􏰚 􏰃m qi(xk)qj(xk) cj = 􏰃n i=0 ⟨p,qj⟩= Since the set q0,q1,...,qn is orthogonal, ⟨qi,qj⟩ = 0 for each i different from j. Hence, we obtain Thisgivesaj asaquotientoftwoinnerproducts. Now we return to the least-squares problem. Let F be a function that we wish to fit by a polynomial pn of degree n. We shall find the polynomial that minimizes the expression 􏰃m [F(xi)− pn(xi)]2 i=0 The solution is given by the formulas pn = It is especially noteworthy that ci does not depend on n. This implies that the various polynomials p , p , . . . that we are seeking can all be obtained by simply truncating one 01 series—namely, m−1 ciqi.Toprovethat pn,asgiveninEquation(9),solvesourproblem, i=0 we return to the normal equations, Equation (1). The basic functions now being used are q0, q1, . . . , qn . Thus, the normal equations are ai⟨qi,qj⟩ (0􏶞 j􏶞n) 􏰃n ⟨ F , q i ⟩ 􏰍 i=0 ciqi ci = ⟨qi,qi⟩ (9) ykqi(xk) ⟨qi,qj⟩cj =⟨F,qi⟩ whereFissomefunctionsuchthatF(xk)=yk for0􏶞k􏶞m.Next,applytheorthogonality property ⟨qi,qj⟩ = 0 when i ≠ j. The result is ⟨qi,qi⟩ci = ⟨F,qi⟩ (0􏶞 i 􏶞 n) (10) Now we return to the variance numbers σ02,σ12,... and show how they can be easily computed. First, an important observation: The set {q0 , q1 , . . . , qn , F − pn } is orthogonal! j=0 k=0 Using the inner product notation, we get k=0 􏰃n j=0 (0􏶞i􏶞n) (0􏶞 i 􏶞 n) 12.2 Orthogonal Systems and Chebyshev Polynomials 515 The only new fact here is that ⟨F − pn,qi⟩ = 0 for 0􏶞 i 􏶞 n. To check this, write = ⟨F,qi⟩− ⟨F − pn,qi⟩ = ⟨F,qi⟩−⟨pn,qi⟩ j=0 In this computation, we used Equations (9) and (10). Since pn is a linear combination of q0,q1,...,qn, it follows easily that ⟨F − pn, pn⟩ = 0 Now recall that the variance σn2 was defined by σ n2 = m − n ρ n = The quantities ρn can be written in another way: ρn =⟨F−pn,F−pn⟩ = ⟨F − pn, F⟩ = ⟨F, F⟩ − ⟨F, pn⟩ 􏰃n ci ⟨F, qi ⟩ =⟨F,F⟩−􏰃n ⟨F,qi⟩2 i=0 ⟨qi,qi⟩ Thus, the numbers ρ0 , ρ1 , . . . can be generated recursively by the algorithm = ⟨F, F⟩ − i=0 􏶷􏰃n 􏶸 cjqj,qi 􏰃n j=0 cj⟨qj,qi⟩ = ⟨F, qi ⟩ − ci ⟨qi , qi ⟩ = 0 = ⟨F,qi⟩− ρn 􏰃m [ y i − p n ( x i ) ] 2 i=0 ⎧⎪⎨ρ0 =⟨F,F⟩−⟨F,q0⟩2 ⟨q0, q0⟩ ⎪ ⟨F,qn⟩2 ⎩ρn =ρn−1−⟨qn,qn⟩ (n􏶟1) Summary (1) We use Chebyshev polynomials {T j } as an orthogonal basis that can be generated recur- sively by T j (x ) = 2x T j −1 (x ) − T j −2 (x ) ( j 􏶟 2) 516 Chapter 12 Smoothing of Data and the Method of Least Squares with T0(x) = 1 and T1(x) = x. The coefficient matrix A = (aij)0:n×0:n and the right-hand side b = (bi )0:n of the normal equations are k=0 A linear combination of Chebyshev polynomials can be evaluated recursively: aij = bi = 􏰃m k=0 􏰃m Ti(zk)Tj(zk) ykTi(zk) (0􏶞i, j􏶞n) (0􏶞i􏶞n) ⎧ ⎪⎨ w n + 2 = w n + 1 = 0 ⎪⎩ w j = c j + 2 x w j + 1 − w j + 2 g(x) = w0 − xw1 g(x) = 􏰃n j=0 cj Tj (x) ( j = n , n − 1 , . . . , 0 ) (2) We discuss smoothing of data by polynomial regression. 1. Let g0,g1,...,gn be a set of functions such that 􏰍mk=0 gi(xk)gj(xk) = 0 if i ≠ j. What linear combination of these functions best fits the data of Table (1) in Section 12.1? a2. Considerpolynomialsg0,g1,...,gn definedbyg0(x) = 1,g1(x) = x−1,andgj(x) = 3xgj−1(x) + 2gj−2(x). Develop an efficient algorithm for computing values of the function f(x)=􏰍nj=0cjgj(x). a3. Showthatcosnθ=2cosθcos(n−1)θ−cos(n−2)θ.Hint:Usethefamiliaridentity cos(A ∓ B) = cos A cos B ± sin A sin B. 4. (Continuation) Show that if fn(x) = cos(n arccos x), then f0(x) = 1, f1(x) = x, and fn(x)=2xfn−1(x)− fn−2(x). a5. (Continuation)ShowthatanalternatedefinitionofChebyshevpolynomialsisTn(x)= cos(n arccos x ) for −1 􏶞 x 􏶞 1. a6. (Continuation) Give a one-line proof that Tn(Tm(x)) = Tnm(x). a7. (Continuation)Showthat|Tn(x)|􏶞1forxintheinterval[−1,1]. a8. Defineg(x)=T 􏰉1x+1􏰈.Whatrecursiverelationdothesefunctionssatisfy? kk22 9. Show that T0, T2, T4, . . . are even and that T1, T3, . . . are odd functions. Recall that an even function satisfies the equation f (x) = f (−x); an odd function satisfies the equation f (x) = − f (−x). Problems 12.2 12.2 Orthogonal Systems and Chebyshev Polynomials 517 a10. Count the number of operations involved in the algorithm used to compute g(x) = 􏰍 nj = 0 c j T j ( x ) . 11. Showthatthealgorithmforcomputingg(x)=􏰍nj=0cjTj(x)canbemodifiedtoread ⎧ ⎪⎨ w n − 1 = c n − 1 + 2 x c n ⎪⎩ wk=ck+2xwk+1−wk−2 (n−2􏶟k􏶟1) g(x) = c0 + xw1 − w2 thus making wn+2, wn+1, and w0 unnecessary. a12. (Continuation)Counttheoperationsforthealgorithmintheprecedingproblem. a13. Determine T6(x) as a polynomial in x. 14. Verifythefourpropertiesofaninnerproductthatwerelistedinthetext,usingDefini- tion (7). 15. Verifytheseformulas: 1 􏰃m ⟨qn,qn⟩ ρn−1−ρn p0(x)=m+1 16. Completetheproofthatthealgorithmforgeneratingtheorthogonalsystemofpolyno- a 17. mials works. There is a function f of the form f(x)=αx12 +βx13 for which f (0.1) = 6×10−13 and f (0.9) = 3×10−2. What is it? Are α and β sensitive to perturbations in the two given values of f (x)? i=0 yi βn =⟨qn−1,qn−1⟩ cn = ⟨F,qn⟩ 18. (Multiple choice) Let x1 = [2,2,1]T , x2 = [1,1,5]T , and x3 = [−3,2,1]T . If the Gram-Schmidt process is applied to this ordered set of vectors to produce an orthonormal set {u1, u2, u3}, what is u1? a. 􏰠2, 2, 1􏰡T b. [2,2,1]T c. 􏰠2, 2, 1􏰡T 333 555 e. Noneofthese. 19. (Multiple choice, continuation) What is u2? d. [1,0,0]T 1T1TT a. √27[1,1,5] b. √18[−1,−1,4] c. [2,2,1] d. [1,1,−4]T e. None of these. 1. Carry out an experiment in data smoothing as follows: Start with a polynomial of modest degree, say, 7. Compute 100 values of this polynomial at random points in the interval [−1, 1]. Perturb these values by adding random numbers chosen from a small interval, say, 􏰠− 1 , 1 􏰡. Try to recover the polynomial from these perturbed values by 88 using the method of least squares. Computer Problems 12.2 518 Chapter 12 Smoothing of Data and the Method of Least Squares 2. WriterealfunctionCheb(n,x)forevaluatingTn(x).Usetherecursiveformulasatisfied by Chebyshev polynomials. Do not use a subscripted variable. Test the program on these 15 cases: n = 0,1,3,6,12 and x = 0,−1,0.5. 3. Write real function Cheb(n,x,(yi)) to calculate T0(x),T1(x),...,Tn(x), and store these numbers in the array (yi ). Use your routine, together with suitable plotting rou- tines,toobtaingraphsofT0,T1,T2,...,T8 on[−1,1]. 4. Write real function F(n,(ci),x) for evaluating f(x) = 􏰍n cjTj(x). Test your 􏰍∞k j=02 routine by means of the formula k=0 t Tk(x) = (1 − tx)/(1 − 2tx + t ), valid for |t| < 1. If |t| 􏶞 1 , then only a few terms of the series are needed to give full machine 2 precision. Add terms in ascending order of magnitude. 12.3 k=0 Interpret the results in terms of the least-squares polynomial-fitting problem. 8. Programthealgorithmforfindingσ02,σ12,...inthepolynomialregressionproblem. 9. Programthecompletepolynomialregressionalgorithm.Theoutputshouldbeαn,βn, σn2,andcn for0􏶞n􏶞 N,where N isdeterminedbytheconditionσN2−1 >σN2 ≈σN2+1.
10. Using orthogonal polynomials, find the quadratic polynomial that fits the following
data in the sense of least squares:
a. x −1 −1 0 1 1 b. x −2 −1 0 1 2 22
y −1 0 1 2 1 y 2 1 1 1 2
Other Examples of the Least-Squares Principle
The principle of least squares is also used in other situations. In one of these, we attempt to solve an inconsistent system of linear equations of the form
􏰃n
akjxj =bk (0􏶞k􏶞m) (1)
5. Obtain a graph of Tn for some reasonable value of n by means of the following idea: Generate 100 equally spaced angles θi in the interval [0, π ]. Define xi cos θi and yi = Tn (xi ) = cos(n arccos xi ) = cos nθi . Send the points (xi , yi ) to a suitable plotting routine.
6. Writesuitablecodetocarryouttheprocedureoutlinedinthetextforfittingatablewith a linear combination of Chebyshev polynomials. Test it in the manner of Computer Problem 12.2.1, first by using an unperturbed polynomial. Find out experimentally how large n can be in this process before roundoff errors become serious.
a7. Define xk = cos[(2k − 1)π/(2m)]. Select modest values of n and m > 2n. Compute and print the matrix A whose elements are
􏰃m
aij =
Ti(xk)Tj(xk) (0􏶞i, j􏶞n)
j=0

begins by factoring
Ax = b A=QR
j=0 k=0
k=0
12.3 Other Examples of the Least-Squares Principle 519
in which m > n. Here, there are m + 1 equations but only n + 1 unknowns. If a given n+1-tuple(x0,x1,…,xn)issubstitutedontheleft,thediscrepancybetweenthetwosides of the kth equation is termed the kth residual. Ideally, of course, all residuals should be zero.Ifitisnotpossibletoselect(x0,x1,…,xn)soastomakeallresidualszero,System(1) is said to be inconsistent or incompatible. In this case, an alternative is to minimize the sum of the squares of the residuals. So we are led to minimize the expression
􏰃m 􏶠􏰃n 􏶡2 φ(x0,x1,…,xn)= akjxj −bk
k=0 j=0
(2)
by making an appropriate choice of (x0,x1,…,xn). Proceeding as before, we take partial derivatives with respect to xi and set them equal to zero, thereby arriving at the normal equations
􏰃n􏶠􏰃m 􏶡􏰃m
akiakj xj = bkaki (0􏶞i􏶞n) (3)
This is a linear system of just n + 1 equations involving unknowns x0, x1, . . ., xn . It can be shown that this system is consistent, provided that the column vectors in the original coef- ficient array are linearly independent. System (3) can be solved, for instance, by Gaussian elimination. The solution of System (3) is then a best approximate solution of Equation (1) in the least-squares sense.
Special methods have been devised for the problem just discussed. Generally, they gain in precision over the simple approach outlined above. One such algorithm for solving System (1),
wherematrixQis(m+1)×(n+1)satisfyingQT Q=IandmatrixRis(n+1)×(n+1) satisfying ri i > 0 and ri j = 0 for j < i . Then the least-squares solution is obtained by an algorithm called the modified Gram-Schmidt process. A more elaborate (and more versatile) algorithm depends on the singular value decomposition of the matrix A. This is a factoring, A = U􏷏VT , in which UT U = Im+1, VTV = In+1,and􏷏isan(m+1)×(n+1)diagonalmatrixthathasnonnegativeentries. For these more reliable procedures, the reader is referred to material at the end of this section and to Stewart [1973] and Lawson and Hanson [1995]. Use of a Weight Function w (x) Another important example of the principle of least squares occurs in fitting or approximat- ing functions on intervals rather than discrete sets. For example, a given function f defined on an interval [a, b] may have to be approximated by a function such as 􏰃n j=0 g(x) = cj gj (x) 520 Chapter 12 Smoothing of Data and the Method of Least Squares It is natural, then, to attempt to minimize the expression 􏶩b a by choosing coefficients appropriately. In some applications, it is desirable to force functions g and f into better agreement in certain parts of the interval. For this purpose, we can modify Equation (4) by including a positive weight function w(x), which can, of course, be w(x) ≡ 1 if all parts of the interval are to be treated the same. The result is 􏶩b j=0 􏶩1 − 1 Ti(x)Tj(x)(1−x2)−1/2dx= ⎪⎨0 i≠j π i= j>0
φ(c0,c1,…,cn)=
[g(x)− f(x)]2dx (4)
φ(c0,c1,…,cn)=
[g(x)− f(x)]2w(x)dx
a Theminimumofφisagainsoughtbydifferentiatingwithrespecttoeachci andsettingthe
partial derivatives equal to zero. The result is a system of normal equations:
􏰃n􏰖􏶩b 􏰗 􏶩b
gi(x)gj(x)w(x)dx cj = f(x)gi(x)w(x)dx (0􏶞i􏶞n) (5)
aa
Thisisasystemofn+1linearequationsinn+1unknownsc0,c1,…,cn andcanbesolved by Gaussian elimination. Earlier remarks about choosing a good basis apply here also. The idealsituationistohavefunctionsg0,g1,…,gn thathavetheorthogonalityproperty:
􏶩b
gi(x)gj(x)w(x)dx = 0 (i ≠ j) (6)
a
Many such orthogonal systems have been developed over the years. For example,
Chebyshev polynomials form one such system, namely, ⎧
The weight function (1 − x 2 )−1/2 assigns heavy weight to the ends of the interval [−1, 1]. If a sequence of nonzero functions g0 , g1 , . . . , gn is orthogonal according to Equa- tion (6), then the sequence λ0 g0 , λ1 g1 , . . . , λn gn is orthonormal for appropriate positive
real numbers λ j , namely,
Nonlinear Example
􏰘􏶩 b a
􏶪−1/2
λj =
[gj(x)]2w(x)dx
As another example of the least-squares principle, here is a nonlinear problem. Suppose that a table of points (xk , yk ) is to be fitted by a function of the form
y = ecx
Proceeding as before leads to the problem of minimizing the function
􏰃m k=0
φ(c)=
(ecxk −yk)2
⎪ ⎪⎩ 2
π i=j=0

12.3 Other Examples of the Least-Squares Principle 521 The minimum occurs for a value of c such that
is easy and leads to
φ(c)=
(cxk −zk)2
zk =lnyk
∂φ 􏰃m
2(ecxk −yk)ecxkxk
This equation is nonlinear in c. One could contemplate solving it by Newton’s method or the secant method. On the other hand, the problem of minimizing φ(c) could be attacked directly. Since there can be multiple roots in the normal equation and local minima in φ itself, a direct minimization of φ would be safer. This type of difficulty is typical of nonlinear least-squares problems. Consequently, other methods of curve fitting are often preferred if the unknown parameters do not occur linearly in the problem.
Alternatively, this particular example can be linearized by a change of variables z = ln y and by considering
0=∂c=
k=0
z = cx The problem of minimizing the function
􏰃m k=0
􏰃m 􏶰􏰃m c = z k x k
k=0
This value of c is not the solution of the original problem but may be satisfactory in some
applications.
Linear and Nonlinear Example
The final example contains elements of linear and nonlinear theory. Suppose that an (xk , yk ) table is given with m + 1 entries and that a functional relationship such as
y = a sin(bx)
is suspected. Can the least-squares principle be used to obtain the appropriate values of the parameters a and b?
Notice that parameter b enters this function in a nonlinear way, creating some difficulty, as will be seen. According to the principle of least squares, the parameters should be chosen such that the expression
􏰃m
[a sin(bxk ) − yk ]2
k=0
has a minimum value. The minimum value is sought by differentiating this expression with
respect to a and b and setting these partial derivatives equal to zero. The results are
⎧⎪ ⎪ 􏰃m
⎪⎨
k=0 ⎪ ⎪ ⎪ 􏰃m
⎪⎩
k=0
2[a sin(bxk ) − yk ] sin(bxk ) = 0 2[a sin(bxk ) − yk ]axk cos(bxk ) = 0
x k2 k=0

522 Chapter 12
Smoothing of Data and the Method of Least Squares
If b were known, a could be obtained from either equation. The correct value of b is the one for which these corresponding two a values are identical. So each of the preceding equations should be solved for a, and the results set equal to each other. This process leads to the equation
􏰃m k=0
􏰃m k=0
yk sin bxk (sinbxk)2
􏰃m = k=0
⎢ ⎢
⎢
UTAV=D= ⎢ ⎢
⎢ ⎢⎣
σ2
⎥ ⎥
⎥
⎥ ⎥
0 ⎥ ⎥⎦
􏰃m k=0
xk yk cos bxk
xk sinbxk cosbxk
which can now be solved for parameter b, using, for example, the bisection method or the secant method. Then either side of this equation can be evaluated as the value of a.
Additional Details on SVD
The singular value decomposition (SVD) of a matrix is a factorization that can reveal important properties of the matrix that otherwise could escape detection. For example, from the SVD decomposition of a square matrix one could be alerted to the near-singularity of the matrix. Or from the SVD factorization of a nonsquare matrix an unexpected loss of rank could be revealed. Since the SVD factorization of a matrix yields a complete orthogonal decomposition, it provides a technique for computing the least squares solution of a system of equations and at the same time producing the norm of the error vector.
Suppose that a given m × n matrix has the factorization A = U DV T
where U = [u1,u2,…,um] is an m × m orthogonal matrix, V = [v1,v2, …,vn] is an n × n orthogonal matrix, and the m × n diagonal matrix D contains the singular values of A on its diagonal, listed in decreasing order. The singular values of a matrix A are the positive square roots of the eigenvalues of AT A. These are denoted by σ1 􏶟 σ2 􏶟 ··· 􏶟 σr 􏶟 0. In detail, we have
⎡σ⎤ 1
where UT U = Im and VT V = In. (In the above matrix, blank space corresponds to zero entries.) Moreover, we have Avi = σi ui and σi = || Avi ||2 where vi is column i in V and
…
σ
r
0
…
m×n

■ THEOREM1
EXAMPLE 1
Find the least-squares solution of this nonsquare system
⎡⎤⎡⎤⎡⎤
11×1 ⎣0 1⎦⎣y⎦=⎣−1⎦ 10z1
12.3 Other Examples of the Least-Squares Principle 523
ui is column i in U. Since U is orthogonal, we obtain
􏶨􏶨􏶨􏶨Ax − b􏶨􏶨􏶨􏶨2 = 􏶨􏶨􏶨􏶨UT (Ax − b)􏶨􏶨􏶨􏶨2 = 􏶨􏶨􏶨􏶨UT Ax − UT b􏶨􏶨􏶨􏶨2
=􏶨􏶨􏶨􏶨UT A(VVT)x−UTb􏶨􏶨􏶨􏶨2
=􏶨􏶨􏶨􏶨(UT AV)(VTx)−UTb􏶨􏶨􏶨􏶨2
= 􏶨􏶨􏶨􏶨DVT x − UT b􏶨􏶨􏶨􏶨2 = 􏶨􏶨􏶨􏶨Dy − c􏶨􏶨􏶨􏶨2
􏰃r =
􏰃m i=r+1
ci2
c =uTbandx=Vy,ify =σ−1c for1􏶞i􏶞rthentheleast-squaressolutionis
and
(σi yi − ci )2 + wherey=VTxandc=UTb.Here,yisdefinedbyyi =ci/σj andxbyx=Vy.Since
i=1 ii iii
􏰃n 􏰃r 􏰃r 􏰉 􏰈 x = yv= σ−1cv= σ−1 uTbv
LS ii iii iii i=1 i=1 i=1
􏶨􏶨 􏶨􏶨􏰃m 􏰃m􏰉􏰈 􏶨􏶨AxLS −b􏶨􏶨2 = ci2 = uiTb 2
i=r+1 i=r+1
which is the smallest of all two-norm minimizers. For additional, details see Golub and Van Loan [1996].
In conclusion, we obtain the following theorem.
SVD LEAST SQUARES THEOREM
Let Abeanm×nmatrixofrankr.LettheSVDfactorizationbe A=UDVT. The least-squares solution of the system Ax = b is x = 􏰍n (σ−1c )v , where
LS i=1 i i i
ci = uiT b. If there exist many least-squares solutions to the given system, then the
one of least 2-norm is x as described above.
using the singular value decomposition:
⎡⎤⎡1√ 1√⎤⎡√⎤
1 1 ⎢3√6 √0 3√3⎥ 3 0 􏰙1√2 1√2􏰚 ⎣0 1⎦=⎢1 6 1 2 −1 3⎥⎣ 0 1⎦ 2√ 2√
⎣6√ 2√ 3√⎦ 12−12 1016−12−13002 2
Solution
√
623
We have r = rank( A) = 2 and the singular values σ1 = 3 and σ2 = 1. This leads to
􏰖 􏰗⎡1⎤
c1=u1Tb= 1√6 1√6 1√6 ⎣−1⎦=1√6 36613

524 Chapter 12
Smoothing of Data and the Method of Least Squares
and
􏰖 􏰗⎡1⎤ c2=u2Tb= 0 −1√2 1√2 ⎣−1⎦=√2
221
􏰉 􏰈 􏰉 􏰈 1􏰔1√􏰕􏰙1√2􏰚√􏰙1√2􏰚
and
x=σ−1cv+σ−1cv=√ 6 2√+2 2√
LS 111 222 33 12 −12 􏰙1􏰚􏰖1􏰗􏰙4􏰚 2 2
=3+1=3 1 −2
33
This solution is the same as that from the normal equations. ■
Using the Singular Value Decomposition
This material requires the theory of the singular value decomposition discussed in Sec- tion 8.3.
An important application of the singular value decomposition is in the matrix least- squares problem, to which we now return. For any system of linear equations Ax = b, we want to define a unique minimal solution. This is described as follows. Let A be m × n, and define
ρ=inf{||Ax−b||2 :x∈Rn}
The minimal solution of our system is taken to be the point of smallest norm in the set {x: || Ax − b||2 = ρ}. If the system is consistent, then ρ = 0, and we are simply asking for the point of least norm among all solutions. If the system is inconsistent, we want Ax to be ascloseaspossibletob;thatis,||Ax−b||2 =ρ.Iftherearemanysuchpoints,wechoose the one closest to the origin.
The minimal solution is produced by using the pseudo-inverse of A, and this object, in turn, can be computed from the singular value decomposition of A as discussed in Section 8.3. First, consider a diagonal m × n matrix of the following form, where the σ j are positive numbers:
⎡σ⎤ 1
⎢ ⎢
⎢ D= ⎢
⎢ ⎢ ⎢⎣
σ2
⎥ ⎥
⎥
⎥ ⎥
0 ⎥ ⎥⎦
…
σ
r
0
…
m×n

MINIMAL SOLUTION THEOREM
Consider a system of linear equations Ax = b, in which A is an m × n matrix. The minimal solution of the system is A+ b.
■ THEOREM2
Proof
12.3 Other Examples of the Least-Squares Principle 525 Its pseudo-inverse D+ is defined to be of the same form, except that it is to be n × m and it
has 1/σ j on its diagonal. For example, 􏰖􏰗⎡10⎤
500+⎣51⎦ D=020 D=02
00
If A is any m × n matrix and if UDV T is one of its singular value decompositions, we
define the pseudo-inverse of A to be
A+ = VD+UT
We do not stop to prove that the pseudo-inverse of A is unique if we impose the order σ1 􏶟 σ2 􏶟 · · ·.
Use the notation established above, and let x be any point in Rn . Define y = V T x and c = UT b. Using the properties of V and U, we obtain
= inf || D V T x − U T b||2 x
= inf || D y − c||2 y
Exploiting the special nature of D, we have
ρ = inf||Ax − b||2 x
= inf ||U DV T x − b||2 x
= inf ||U T (U D V T x − b)||2 x
􏶨􏶨 􏶨􏶨 􏰃r 􏶨􏶨Dy−c􏶨􏶨2 =
i=1
(σiyi −ci)2 +
􏰃m i=r+1
ci2
Tominimizethislastexpression,wedefineyi =ci/σi for1􏶞i􏶞r.Theothercomponents canremainunspecified.Buttogetthe yofleastnorm,wemustsetyi =0forr+1􏶞i􏶞m. This construction is carried out by the pseudo-inverse D+, so y = D+c. Hence, we obtain
x=Vy=VD+c=VD+UTb= A+b
Let us express the minimal solution in another form, taking advantage of the zero compo- nents in the vector y. Since yi = 0 for i > r, we require only the first r components of y. Thesearegivenbyyi =ci/σi.Nowitisevidentthatonlythefirstrcomponentsofcare needed. Since c = UT b, ci is the inner product of row i in UT with the vector b. That is the same as the inner product of the ith column of U with b. Thus,
yi =uiTb/σi 1􏶞i􏶞r

526 Chapter 12
Smoothing of Data and the Method of Least Squares
■ THEOREM3
The minimal solution, which we may denote by x∗, is then
􏰃r i=1
An example of this procedure can be carried out in mathematical software such as Matlab, Maple or Mathematica. We can generate a system of 20 equations with three unknowns by a random process. This technique is often used in testing software, especially in benchmarking studies, in which a large number of examples is run with careful timing. The software has a provision for entering random matrices. When executed, the computer program first exhibits the random input. The three singular values of matrix A are displayed. Then the diagonal 20 × 3 matrix D is displayed. A check on the numerical work is made by computing U DV T , which should equal A. Then the pseudo-inverse of D+ is computed. Next, the pseudo-inverse A+ is computed. The minimal solution, x = A+b, is computed, as well as the residual vector, r = A+ b = b. Then the orthogonality condition AT r = 0 is checked. This program is therefore carrying out all the steps described above for obtaining the minimal solution of a system of equations. Another example will be given below to show what happens in the case of a loss in rank. (See Computer Problem 12.3.10.)
In problems of this type, the user must examine the singular values and decide whether any are small enough to warrant being set equal to zero. The necessity of this step becomes clear when we look at the definition of D+. The reciprocals of the singular values are the principal constituents of this matrix. Any very small singular value that is not set equal to zero will therefore have a disruptive effect on the subsequent calculations. A rule of thumb that has been recommended is to drop any singular value whose magnitude is less than σ1 times the inherent accuracy of the coefficient matrix. Thus, if the data are accurate to three decimal places and if σ1 = 5, then any σi less than 0.005 should be set equal to zero.
An example of a small matrix having a near-deficiency in rank is given next. In the Maple program, certain singular values are set equal to zero if they fail to meet the relative size criterion mentioned in the previous paragraph. Also, we have added, as a check on the calculations, a verification of the following four Penrose properties for a pseudo-matrix.
x∗ =Vy=
yivi ■
PENROSE PROPERTIES OF THE PSEUDO-INVERSE
The pseudo-inverse A+ for the matrix A has these four properties:
A = A A+ A A+ = A+ A A+ AA+ = (AA+)T A+ A = (A+ A)T
We can use mathematical software such as Matlab, Maple, or Mathematica for finding the pseudo-inverse of a matrix that has a deficiency in rank. For example, consider this
5 × 3 matrix:
⎡⎤
−85 −55 −115 ⎢ −35 97 −167 ⎥
A = ⎢ 79 56 102 ⎥ (7) ⎣63 57 69⎦
45 −8 97.5

when we minimize
12.3 Other Examples of the Least-Squares Principle 527
A tolerance value is set so that in the evaluation of singular values any value whose magnitude is less than the tolerance is treated as zero. We can verify the Penrose properties for this matrix. (See Computer Problem 12.3.11.)
Summary
(1) We attempt to solve an inconsistent system
􏰃n j=0
(0􏶞k􏶞m)
in which there are m + 1 equations but only n + 1 unknowns with m > n. We minimize the
akjxj =bk
sum of the squares of the residuals and are led to minimize the expression
􏰃m 􏶠􏰃n 􏶡2 φ(x0,x1,…,xn)= akjxj −bk
k=0 j=0
We solve the (n + 1) × (n + 1) system of normal equations
􏰃n􏶠􏰃m 􏶡􏰃m
akiakj xj = bkaki (0􏶞i􏶞n)
j=0 k=0
by Gaussian elimination, and the solution is a best approximate solution of the original
system in the least-squares sense.
Additional References
See Acton [1959], Bjo ̈rck [1996], Branham [1990], Cheney [1982, 2001], Forsythe [1957], van Huffel and Vandewalle [1991], Lawson and Hanson [1995], Rice [1971], Rice and White [1964], Rivlin [1990], Spa ̈th [1992], and Whittaker and Robinson [1944].
1. Analyzetheleast-squaresproblemoffittingdatabyafunctionoftheformy=xc. a2. ShowthattheHilbertmatrix(ComputerProblem7.2.4)arisesinthenormalequations
j=0
a3. Find a function of the form y = ecx that best fits this table:
x01
y11 2
k=0
􏶩 1 􏰙 􏰃n 􏰚 2 cjxj−f(x) dx
0
Problems 12.3

528 Chapter 12
Smoothing of Data and the Method of Least Squares
a4.
5. a 6.
7.
a8.
(Continuation)Repeattheprecedingproblemforthefollowingtable: x01
yab
(Continuation)Repeattheprecedingproblemunderthesuppositionthatbisnegative.
Show that the normal equation for the problem of fitting y = ecx to points (1, −12) and (2, 7.5) has two real roots: c = ln 2 and c = 0. Which value is correct for the fitting problem?
Consider the inconsistent System (1). Suppose that each equation has associated with it a positive number wi indicating its relative importance or reliability. How should Equations (2) and (3) be modified to reflect this?
Determinethebestapproximatesolutionoftheinconsistentsystemoflinearequations
⎧
⎪⎨2x+3y= 1
⎪⎩ x − 4 y = − 9 2x−y =−1
10. 11.
12. 13.
a14.
a15.
a 16.
a17. 18.
squares to the function sin x on the interval [0, π/2]. ab. Do the same for ex on [0, 1].
Analyze the problem of fitting a function y = (c − x)−1 to a table of m + 1 points. Showthatthenormalequationsfortheleast-squaressolutionofAx=bcanbewritten
(AT A)x = AT b.
Derive the normal equations given by System (5).
A table of values (xk,yk), where k = 0,1,…,m, is obtained from an experiment. When plotted on semilogarithmic graph paper, the points lie nearly on a straight line, implying that y ≈ eax+b. Suggest a simple procedure for obtaining parameters a and b.
in the least-squares sense.
9.aa. Find the constant c for which cx is the best approximation in the sense of least
In fitting a table of values to a function of the form a + bx−1 + cx−2, we try to make eachpointlieonthecurve.Thisleadstoa+bx−1 +cx−2 = y for0􏶞k􏶞m.An
kkk
equivalent equation is axk2 + bxk + c = yk xk2 for 0 􏶞 k 􏶞 m. Are the least-squares
problems for these systems of equations equivalent?
A table of points (xk,yk) is plotted and appears to lie on a hyperbola of the form y = (a + bx)−1. How can the linear theory of least squares be used to obtain good estimates of a and b?
Consider f (x ) = e2x over [0, π ]. We wish to approximate the function by a trigono- metric polynomial of the form p(x) = a + b cos(x) + c sin(x). Determine the linear system to be solved for determining the least squares fit of p to f .
Find the constant c that makes the expression 􏶧 1(ex − cx)2 dx a minimum. 0
Showthatineveryleast-squaresmatrixproblem,thenormalequationshaveasymmetric coefficient matrix.

12.3 Other Examples of the Least-Squares Principle 529 19. Verify that the following steps produce the least-squares solution of Ax = b.
a. Factor A = Q R, where Q and R have the properties described in the text. b. Define y = QT b. c. Solve the lower triangular system Rx = y.
a20. What value of c should be used if a table of experimental data (xi, yi) for 0􏶞 i 􏶞 m is to be represented by the formula y = c sin x ? An explicit usable formula for c is required. Use the principle of least squares.
21. Refer to the formulas leading to the minimal solution of the system Ax = b. Prove thatthe y-vectorisgivenbytheformulayi =σ−2bT Av for1􏶞i􏶞r.
ii
22. Prove that the pseudo-inverse satisfies the four Penrose equations.
23. UsethefourPenrosepropertiestofindthepseudo-inverseofthematrix[a,0]T,where a > 0. Prove that the pseudo-inverse is a discontinuous function of a.
24. Usethetechniquesuggestedintheprecedingproblemtofindthepseudo-inverseofthe m × n matrix consisting solely of 1’s.
25. UsethePenroseequationstofindthepseudo-inverseofany1×nmatrixandanym×1 matrix.
26. (Multiplechoice)LetA=PDQ,whereAisanm×nmatrix,Pisanm×munitary matrix, D is an m × n diagonal matrix, and Q is an n × n unitary matrix. Which equation can be deduced from those hypotheses?
a. A∗=P∗D∗Q∗ b. A−1=Q∗D−1P∗ c. D=PAQ d. A∗A= Q∗D∗DQ e. Noneofthese.
27. (Multiplechoice,continuation)Assumethehypothesesoftheprecedingproblem.Use the notation + to indicate a pseudo-inverse. Which equation is correct?
28.
(Multiple choice) Let D be an m × n diagonal matrix with diagonal elements p1, p2, . . . , pr , 0, 0, . . . , 0. Here all the numbers pi , for 1 􏶞 i 􏶞 r , are positive. Which assertion is not valid?
a. D+ isthem×ndiagonalmatrixwithdiagonalelements(1/p1,1/p2,…,1/pr,0, 0, …, 0)
b. D+ isthen×m diagonalmatrixwithdiagonalelements(1/p1,1/p2,…,1/pr,0, 0, …, 0)
29.
(Multiple choice) Consider an inconsistent system of equations Ax = b. Let U be a unitary matrix and let E = U∗ A. Let v, w, and z be vectors such that Uv = Eb, Uw = E∗b, E y = U∗b, and Ex = Ub. A vector that solves the least-squares problem for the original system Ax = b is:
a. v b. w c. y d. z e. None of these.
a. A+=PD+Q b. A∗=Q∗D−1P∗ c. A+=Q∗D+P∗
d. A−1=Q∗D+P∗
e. None of these.
c.(D+)∗=(D∗)+ d. D++=D
e. None of these.

530 Chapter 12
Smoothing of Data and the Method of Least Squares
a1. Usingthemethodsuggestedinthetext,fitthedatainthetable
x 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
y 0.6 1.1 1.6 1.8 2.0 1.9 1.7 1.3
by a function y = a sin bx.
2. (Prony’s method, n = 1) To fit a table of the form
x12···m
y y1 y2 ··· ym
bythefunctiony=abx,wecanproceedasfollows:Ifyisactuallyabx,thenyk =abk and yk+1 = byk for k = 1,2,…,m − 1. So we determine b by solving this system of equations using the least-squares method. Having found b, we find a by solving the equations yk = abk in the least-squares sense. Write a program to carry out this procedure, and test it on an artificial example.
3. (Continuation) Modify the procedure of the preceding computer problem to handle any case of equally spaced points.
4. Aquickwayoffittingafunctionoftheform f(x)≈ a+bx
1+cx
is to apply the least-squares method to the problem (1 + cx) f (x) ≈ a + bx. Use this
technique to fit the world population data given here:
Year Population (billions)
1000 0.340 1650 0.545 1800 0.907 1900 1.61 1950 2.51 1960 3.15 1970 3.65 1980 4.20 1990 5.30
Determine when the world population will become infinite!
5. (Studentresearchproject)Explorethequestionofwhethertheleast-squaresmethod should be used to predict. For example, study the variances in the preceding problem to determine whether a polynomial of any degree would be satisfactory.
6. Writeaprocedurethattakesasinputan(m+1)×(n+1)matrix Aandanm+1 vector b and returns the least-squares solution of the system Ax = b.
7. WriteaMapleprogramtofindtheminimalsolutionofanysystemofequations,Ax=b.
Computer Problems 12.3

12.3 Other Examples of the Least-Squares Principle 531
8. (Continuation) Write a Matlab program for the task in the preceding problem.
9. Investigate some of the newer methods for solving inconsistent linear equations
Ax = b, when the criterion is to make Ax close to b in one of the other useful norms,
namely, the maximum norm ||x||∞ = max1 􏶞 i 􏶞 n |xi | or the l1 norm ||x||1 = 􏰍n |xi |. i=1
Use some of the available software.
10. UsingmathematicalsoftwaresuchasMatlab,Maple,orMathematica,generateasys- tem of twenty equations with three unknowns by a random number generator. Form the pseudo-inverse matrix and verify the properties in Theorem 2.
11. (Continuation.) Repeat using Matrix (7).
12. WriteacomputerprogramforcarryingouttheleastsquarescurvefitusingChebyshev polynomials. Test the code on a suitable data set and plot the results.

13
Monte Carlo Methods and Simulation
532
FIGURE 13.1
Traffic flow
A highway engineer wishes to simulate the flow of traffic for a proposed design of a major freeway intersection. The information that is obtained will then be used to determine the capacity of storage lanes (in which cars must slow down to yield the right of way). The intersection has the form shown in Figure 13.1, and various flows (cars per minute) are postulated at the points where arrows are drawn. By writing and running a simulation program, the engineer can study the effect of different speed limits, determine which flows lead to saturation (bottlenecks), and so on. Some techniques for constructing such programs are developed in this chapter.
13.1 Random Numbers
This chapter differs from most of the others in its point of view. Instead of addressing clear- cut mathematical problems, it attempts to develop methods for simulating complicated proc- esses or phenomena. If the computer can be made to imitate an experiment or a process, then by repeating the computer simulation with different data, we can draw statistical conclu- sions. In such an approach, the conclusions may lack a high degree of mathematical precision but still be sufficiently accurate to enable us to understand the process being simulated.
Particular emphasis is given to problems in which the computer simulation involves an element of chance. The whimsical name of Monte Carlo methods was applied some years

ago by Stanislaw M. Ulam (1909–1984) to this way of imitating reality by a computer. Since chance or randomness is part of the method, we begin with the elusive concept of random numbers.
Consider a sequence of real numbers x1, x2, . . . all lying in the unit interval (0, 1). Expressed informally, the sequence is random if the numbers seem to be distributed hap- hazardly throughout the interval and if there seems to be no pattern in the progression x1, x2, . . . For example, if all the numbers in decimal form begin with the digit 3, then the numbers are clustered in the subinterval 0.3 􏶞 x < 0.4 and are not randomly distributed in (0, 1). If the numbers are monotonically increasing, they are not random. If each xi is obtained from its predecessor by a simple continuous function, say, xi = f (xi−1), then the sequence is not random (although it might appear to be so). A precise definition of randomness is quite difficult to formulate, and the interested reader may wish to consult an article by Chaitlin [1975], in which randomness is related to the complexity of computer algorithms! Thus, it seems best, at least in introductory material, to accept intuitively the notion of a random sequence of numbers in an interval and to accept certain algorithms for generating sequences that are more or less random. A recommended reference is the book of Niederreiter [1992]. Random-Number Algorithms and Generators Most computer systems have random-number generators, which are procedures that produce either a single random number or an entire array of random numbers with each call. In this chapter, we call such a procedure Random. The reader can use a random-number generator available on his or her own computing system, one available within the computer language being used, or one of the generators described below. For example, random- number generators are contained in mathematical software systems such as Matlab, Maple, and Mathematica as well as many computer programming languages. These random-number procedures return one or an array of uniformly distributed pseudo-random numbers in the unit interval (0, 1) depending on whether the argument is a scalar variable or an array. A random seed procedure restarts or queries the pseudo-random-number generator. The random number generator can produce hundreds of thousands of pseudo-random numbers before repeating itself, at least theoretically. For the problems in this chapter, one should select a routine to provide random numbers uniformly distributed in the interval (0, 1). A sequence of numbers is uniformly distributed in the interval (0, 1) if no subset of the interval contains more than its share of the numbers. In particular, the probability that an element x drawn from the sequence falls within the subinterval [a, a + h] should be h and hence independent of the number a. Similarly, if pi =(xi,yi)arerandompointsintheplaneuniformlydistributedinsomerectangle,then the number of these points that fall inside a small square of area k should depend only on k and not on where the square is located inside the rectangle. Random numbers produced by a computer code cannot be truly random because the manner in which they are produced is completely deterministic; that is, no element of chance is actually present. But the sequences that are produced by these routines appear to be random, and they do pass certain tests for randomness. Some authors prefer to emphasize this point by calling such computer-generated sequences pseudo-random numbers. If the reader wishes to program a random-number generator, the following one should be satisfactory on a machine that has 32-bit word length. This algorithm generates n random 13.1 Random Numbers 533 534 Chapter 13 Monte Carlo Methods and Simulation numbersx1,x2,...,xn uniformlydistributedintheopeninterval(0,1)bymeansofthe following recursive algorithm: Here, all li ’s are integers in the range 1 < li < 231 − 1. The initial integer l0 is called the seed for the sequence and is selected as any integer between 1 and the Mersenne prime number 231 − 1 = 21474 83647. For information on portable random-number generators, the reader should consult the article by Schrage [1979]. A fast normal random-number generator can be written in only a few lines of code as presented in Leva [1992]. It is based on the ratio of uniform deviates method of Kinderman and Monahan [1977]. An external function procedure to generate a new array of pseudo-random numbers per call could be based on the following pseudocode: integer array (li )0:n ; real array (xi )1:n l0 ←anyintegersuchthat1 43x? Write a program to test this on 1000 random numbers.
Writeaprogramtogenerateandprint1000pointsuniformlyandrandomlydistributed inthecircle(x−3)2 +(y+1)2􏶞9.
Generate 1000 random numbers xi according to a uniform distribution in the interval (0, 1). Define a function f on (0, 1) as follows: f (t ) is the number of random numbers x1, x2, . . . , x1000 less than t. Compute f (t)/1000 for 200 points t uniformly distributed in (0, 1). What do you expect f (t)/1000 to be? Is this expectation borne out by the experiment? If a plotter is available, plot f (t )/1000.
Let ni (1 􏶞 i 􏶞 1000) be a sequence of integers that satisfies 0 􏶞 ni 􏶞 9. Write a program to test the given sequence for periodicity. (The sequence is periodic if there is an integer k such that ni = ni+k for all i.)
Generateinthecomputer1000randomnumbersintheinterval(0,1).Printandexamine them for evidence of nonrandom behavior.
Generate1000randomnumbersxi (1􏶞i􏶞1000)onyourcomputer.Letni denotethe eighth decimal digit in xi . Count how many 0’s, 1’s, . . . , 9’s there are among the 1000 numbers ni . How many of each would you expect? This code can be written with nine statements.
(Continuation) Using a random-number generator, generate 1000 random numbers, and count how many times the digit i occurs in the jth decimal place. Print a table of these values—that is, frequency of digit versus decimal place. By examining the table, determine which decimal place seems to produce the best uniform distribution of random digits. Hint: Use the routine from Computer Problem 1.1.7 to compute the arithmetic mean, variance, and standard deviations of the table entries.
Using random integers, write a short program to simulate five people matching coin flips. Print the percentage of match-ups (five of a kind) after 125 flips.
Write a program to generate 1600 random points uniformly distributed in the sphere defined by x2 + y2 + z2 􏶞 1. Count the number of random points in the first octant.
Write a program to simulate 1000 simultaneous flips of three coins. Print the number of times that two of the three coins come up heads.
Compute1000triplesofrandomnumbersdrawnfromauniformdistribution.Foreach triple (x, y, z), compute the leading significant digit of the product xyz. (The leading significant digit is one of 1, 2, . . . , 9.) Determine the frequencies with which the digits 1 through 9 occur among the 1000 cases. Try to account for the fact that these digits do not occur with the same frequency. (For example, 1 occurs approximately 7 times more often than 9.) If you are intrigued by this, you may wish to consult the articles by Flehinger [1966], Raimi [1969], and Turner [1982].
Runtheexampleprogramsinthissectionandseewhethersimilarresultsareobtained on your computer system.
13.1 Random Numbers 543

544
Chapter 13
Monte Carlo Methods and Simulation
20. Write a program to generate and plot 1000 pseudo-random points with the following exponential distribution inside the figure below: x = − ln(1 − r )/λ for r ∈ [0, 1) and λ = 1/30.
x
2
z
3–2 0
y
1
13.2
21. Improve the program Coarse Check by using ten or a hundred buckets instead of two.
22. (Student research project) Investigate some of the latest developments on random- number generators and explore parallel random number generators. Random numbers are often needed for distributions other than the uniform distribution, so this has a statistical aspect.
Estimation of Areas and Volumes by Monte Carlo Techniques
Numerical Integration
Now we turn to applications, the first being the approximation of a definite integral by the Monte Carlo method. If we select the first n elements x1, x2, . . . , xn from a random sequence in the interval (0, 1), then
􏶩 1 1 􏰃n f(x)dx ≈ n f(xi)
0 i=1
Here, the integral is approximated by the average of n numbers f(x1), f(x2),…, f(xn). When this is actually carried out, the error is of order 1/√n, which is not at all competitive with good algorithms, such as the Romberg method. However, in higher dimensions, the Monte Carlo method can be quite attractive. For example,
􏶩 1 􏶩 1 􏶩 1 1 􏰃n f(x,y,z)dxdydz≈ n f(xi,yi,zi)
000 i=1
where (xi , yi , zi ) is a random sequence of n points in the unit cube 0 􏶞 x 􏶞 1, 0 􏶞 y 􏶞 1, and 0 􏶞 z 􏶞 1. To obtain random points in the cube, we assume that we have a random sequence

13.2 Estimation of Areas and Volumes by Monte Carlo Techniques 545
in (0, 1) denoted by ξ1, ξ2, ξ3, ξ4, ξ5, ξ6, . . . To get our first random point p1 in the cube, just let p1 = (ξ1, ξ2, ξ3). The second is, of course, p2 = (ξ4, ξ5, ξ6), and so on.
If the interval (in a one-dimensional integral) is not of length 1 but, say, is the gen- eral case (a,b), then the average of f over n random points in (a,b) is not simply an approximation for the integral but rather for
1􏶩b
b − a
which agrees with our intention that the function f (x) = 1 have an average of 1. Similarly, in higher dimensions, the average of f over a region is obtained by integrating and dividing by the area, volume, or measure of that region. For instance,
1􏶩3􏶩1􏶩2
f (x ) d x
􏶩a2􏶩b2􏶩c2 a1 b1 c1
f(x,y)dxdy=
􏶩a2􏰘􏶩b2􏰖􏶩c2 􏰗 􏶪
8
f (x, y, z) dx dy dz
1 −1 0
a
is the average of f over the parallelepiped described by the following three inequalities: 0 􏶞 x 􏶞 2, −1 􏶞 y 􏶞 1, 1 􏶞 z 􏶞 3.
To keep the limits of integration straight, recall that
and
0
􏶩 b 􏶩 d
ac ac
􏶩 b􏰖􏶩 d 􏰗 f(x,y)dx dy
f(x,y,z)dxdydz=
So if (xi , yi ) denote random points with appropriate uniform distribution, the following
examples illustrate Monte Carlo techniques: 􏶩 5
􏶩 5 􏶩 6
21 i=1
In each case, the random points should be uniformly distributed in the regions involved. In general, we have
􏶩
Here, we are using the fact that the average of a function on a set is equal to the integral of the function over the set divided by the measure of the set.
Example and Pseudocode
Let us consider the problem of obtaining the numerical value of the integral
􏶩􏶩􏶣 􏶩􏶩
sin ln(x + y + 1) dx dy = f (x, y) dx dy 􏶝􏶝
a1 b1 c1
5 􏰃n f(x)dx ≈ n f(xi)
f(x,y)dxdy≈ n f(xi,yi)
i=1 15 􏰃n
f(x,y,z)dx dy dz
f ≈ (measure of A) × (average of f over n random points in A) A

546
Chapter 13
Monte Carlo Methods and Simulation
z
Surface f
1
Disk 􏷀
y
FIGURE 13.6
Sketch of surface f(x,y) above disk 􏶝
1
x
over the disk in xy-space, defined by the inequality
􏰎 􏰔 􏰕2 􏰔 􏰕2 􏶬
􏶝= (x,y): x−1 + y−1 􏶞1 224
A sketch of this domain, with a surface above it, is shown in Figure 13.6. We proceed by gen- erating random points in the square and discarding those that do not lie in the disk. We take n=5000pointsinthedisk.Ifthepointsarepi =(xi,yi),thentheintegralisestimatedtobe 􏶩􏶩 􏰔 􏰕
􏶝
f(x,y)dxdy ≈(areaofdisk􏶝)× averageheightof f over n random points
􏰙􏰃n 􏰚 =􏰉πr2􏰈 1 f(pi)
n i=1
π 􏰃n
= 4n
i=1
f(pi)
The pseudocode for this example follows. Intermediate estimates of the integral are printed when n is a multiple of 1000. This gives us some idea of how the correct value is being approached by our averaging process.
program Double Integral integer i, j: real sum, vol, x, y; integer n ← 5000, iprt ← 1000; call Random((ri j ))
j←0; sum←0 for i = 1 to n do
real array (ri j )1:n×1:2 external function f
x = ri,1; y = ri,2
if(x−1/2)2 +(y−1/2)2􏶞1/4then
j←j+1 sum←sum+ f(x,y) if mod( j, iprt) = 0 then
vol ← (π/4)sum/real( j) output j,vol

13.2 Estimation of Areas and Volumes by Monte Carlo Techniques 547
end if end if
end for
vol ← (π/4)sum/real( j) output j,vol
end program Double Integral
real function f (x, y)
real x, y􏰉√ 􏰈
f ←sin ln(x+y+1) end function
We obtain an approximate value of 0.57 for the integral. Computing Volumes
The volume of a complicated region in 3-space can be computed by a Monte Carlo technique. Taking a simple case, let us determine the volume of the region whose points satisfy the inequalities
⎧
⎪⎨ 0 􏶞 x 􏶞 1 0 􏶞 y 􏶞 1 0 􏶞 z 􏶞 1
⎪⎩x2 +siny􏶞z x − z + ey 􏶞 1
The first line defines a cube whose volume is 1. The region defined by all the given in- equalities is therefore a subset of this cube. If we generate n random points in the cube and determine that m of them satisfy the last two inequalities, then the volume of the desired region is approximately m/n. Here is a pseudocode that carries out this procedure:
program Volume Region
integer i, m; real array (ri j )1:n×1:3; real vol, x, y, z integer n ← 5000, iprt ← 1000
call Random((ri j ))
for i = 1 to n do
x ← ri,1
y ← ri,2
z ← ri,3
if x2 + sin y 􏶞 z, x − z + ey 􏶞 1 then m ← m + 1 if mod(i, iprt) = 0 then
vol ← real(m)/real(i)
output i, vol end if
end for
end program Volume Region

548
Chapter 13
Monte Carlo Methods and Simulation
Observe that intermediate estimates are printed out when we reach 1000, 2000, . . . , 5000 points. An approximate value of 0.14 is determined for the volume of the region.
Ice Cream Cone Example
Consider the problem of finding the volume above the cone z2 = x2 + y2 and inside the sphere x2 + y2 + (z − 1)2 = 1 as shown in Figure 13.7. The volume is contained in the box bounded by −1 􏶞 x 􏶞 1, −1 􏶞 y 􏶞 1, and 0 􏶞 z 􏶞 2, which has volume 8. Thus, we want to generate random points inside this box and multiply by 8 the ratio of those inside the desired volume to the total number generated. A pseudocode for doing this follows:
program Cone
integer i, m; real vol, x, y, z; real array (ri j )1:n×1:3 integer n ← 5000, iprt ← 1000; m ← 0
call Random((ri j ))
for i = 1 to n do
x ← 2ri,1 − 1; y ← 2ri,2 − 1; z ← 2ri,3
ifx2 +y2􏶞z2,x2 +y2􏶞z(2−z)thenm←m+1 if mod(i, iprt) = 0 then
vol ← 8 real(m)/real(i)
output i, vol end if
end for
end program Cone
The volume of the cone is approximately 3.3.
z
FIGURE 13.7
Ice cream cone region
􏶼1 0 1
x
y

13.2 Estimation of Areas and Volumes by Monte Carlo Techniques 549
Summary
(1) We discuss the approximating of integrals by the Monte Carlo method to estimate areas and volumes. We use
􏶩 1 0
1 􏰃n
f(x)dx ≈ n
1 􏰃n
f(xi) f(xi,yi,zi)
􏶩 1 􏶩 1 􏶩 1
000 i=1
f(x,y,z)dxdydz ≈ n
where {xi } is a sequence of random numbers in the unit interval and (xi , yi , zi ) is a random
sequence of n points in the unit cube. (2) In general, we have
i=1
􏶩
f ≈ (measure of A) × (average of f over n random points in A) A
Problems 13.2
a 1.
1. a 2.
3. a4.
It is proposed to calculate π by using the Monte Carlo method. A circle of radius 1 is inside a square of side 2. We count how many of m random points in the square happen to lie in the circle. Assume that the error is 1/√m. How many points must be taken to obtain π with three accurate figures (i.e., 3.142)?
Runthecodesgiveninthissectiononyourcomputersystemandverifythattheyproduce reasonable answers.
Write and test a program to evaluate the integral 􏶧 1 ex d x by the Monte Carlo method, 0
using n = 25, 50, 100, 200, 400, 800, 16000, and 32000. Observe that 32,000 random numbers are needed and that the work in each case can be used in the next case. Print the exact answer. Plot the results using a logarithmic scale to show the rate of growth.
Writeaprogramtoverifynumericallythatπ =􏶧2(4−x2)1/2 dx.UsetheMonteCarlo 0
method and 2500 random numbers. UsetheMonteCarlomethodtoapproximatetheintegral
􏶩1􏶩1􏶩1
(x2 +y2 +z2)dxdydz
−1 −1 −1 Compare with the correct answer.
Computer Problems 13.2

550 Chapter 13
Monte Carlo Methods and Simulation
a5. Writeaprogramtoestimate
􏶩 2􏶩 6 􏶩 1
(yx2 +zlogy+ex)dxdydz 0 3 −1
6. UsingtheMonteCarlotechnique,writeapseudocodetoapproximatetheintegral 􏶩􏶩􏶩
(ex sinylogz)dxdydz 􏶝
where 􏶝 is the circular cylinder that has height 3 and circular base x2 + y2 􏶞 4.
a7. Estimate the area under the curve y = e−(x+1)2 and inside the triangle that has vertices
(1, 0), (0, 1), and (−1, 0) by writing and testing a short program.
8. UsingtheMonteCarloapproach,findtheareaoftheirregularfiguredefinedby ⎧
⎪⎨ 1 􏶞 x 􏶞 3 − 1 􏶞 y 􏶞 4 ⎪⎩x3 +y3􏶞29
y􏶟ex −2
a9. UsetheMonteCarlomethodtoestimatethevolumeofthesolidwhosepoints(x,y,z)
satisfy
⎧
⎪⎨ 0 􏶞 x 􏶞 y 1 􏶞 y 􏶞 2 − 1 􏶞 z 􏶞 3
⎪⎩ e x 􏶞 y
(sin z)y 􏶟 0
a10. Using a Monte Carlo technique, estimate the area of the region determined by the inequalities 0 􏶞 x 􏶞 1, 10 􏶞 y 􏶞 13, y 􏶟 12 cos x, and y 􏶟 10 + x3. Print intermediate answers.
11. Use the Monte Carlo method to approximate the following integrals. 􏶩1􏶩1􏶩1
a.
b.
(x2 −y2 −z2)dxdydz (x2 −y2 +xy−3)dxdy
􏶩−1􏶩−1 −1 45
􏶩1 􏶩2√ 3 y
􏶩 􏶩√ 􏶩
1 y y+z
c.
12. Thevalueoftheintegral
(x2y + xy2)dx dy
􏶩 π/4􏶩 2cosφ 􏶩 2π
d. xy dx dy dz 0 y2 0
ρ2 sin φ dθ dρ dφ
000
using spherical coordinates is the volume above the cone z2 = x2 + y2 and inside the sphere x2 + y2 + (z − 1)2 = 1. Use the Monte Carlo method to approximate this integral and compare the results with that from the example in the text.
13. Let R denote the region in the xy-plane defined by the inequalities 􏰎1 􏶞3x􏶞9−y
3
√x 􏶞 y 􏶞 3
2 1+y

13.2 Estimation of Areas and Volumes by Monte Carlo Techniques 551
Estimate the integral
􏶩􏶩
i
􏰉√􏰈 00
(ex +cosxy)dxdy R
a14. UsingaMonteCarlotechnique,estimatetheareaoftheregiondefinedbytheinequali- ties4x2 +9y2􏶞36andy􏶞arctan(x+1).
15. Writeaprogramtoestimatetheareaoftheregiondefinedbytheinequalities
􏰎
x2 + y2 􏶞 4 |y| 􏶞 ex
16. Anintegralcanbeestimatedbytheformula
􏶩 1 1 􏰃n f(x)dx ≈ n f(xi)
numerical integration scheme. Test whether the estimates converge at the rate 1/n or
17. Consider the ellipsoid
x2 y2 z2
4 + 16 + 4 = 1
a. Writeaprogramtogenerateandstore5000randompointsuniformlydistributedin the first octant of this ellipsoid.
0 i=1
even if the x ’s are not random numbers; in fact, some nonrandom sequences may
be better. Use the sequence xi = fractional part of i 2 and test the corresponding
1/√n by using some simple examples, such as 􏶧 1 ex dx and 􏶧 1(1 + x2)−1 dx.
ab. Writeaprogramtoestimatethevolumeofthisellipsoidinthefirstoctant.
18. A Monte Carlo method for estimating 􏶧 b f (x) dx if f (x) 􏶟 0 is as follows: Let
a
c􏶟 maxa􏶞x􏶞b f(x).Thengeneratenrandompoints(x,y)intherectanglea􏶞x􏶞b,
0􏶞y􏶞c.Countthenumberkoftheserandompoints(x,y)thatsatisfyy􏶞 f(x).Then 􏶧b f(x)dx≈kc(b−a)/n.Verifythisandtestthemethodon􏶧2x2dx,􏶧1(2×2−
a􏶧110 x+1)dx,and 0(x2+sin2x)dx.
19. (Continuation) Use the method of Computer Problem 13.2.18 to estimate the value ofπ = 4􏶧1√1−x2dx.Generaterandompointsin0􏶞x􏶞1,0􏶞y􏶞1.Usen =
0√ 1000, 2000, . . . , 10000 and try to determine whether the error is behaving like 1/ n.
20. (Continuation)ModifythemethodoutlinedinComputerProblem13.2.19tohandlethe case when f takes positive and negative values on [a, b]. Test the method on 􏶧 1 x3 dx.
−1
21. Another Monte Carlo method for evaluating 􏶧 b f (x ) d x is as follows: Generate an odd
a
number of random numbers in (a, b). Reorder these points so that a < x1 < x2 < · · · < xn < b. Now compute f(x1)(x2 −a)+ f(x3)(x4 −x2)+ f(x5)(x6 −x4)+···+ f(xn)(b−xn−1) Test this method on 􏶩1􏶩1􏶩1 (1+x2)−1 dx (1−x2)−1/2 dx x−1 sinxdx 000 552 Chapter 13 Monte Carlo Methods and Simulation 22. What is the expected value of the volume of a tetrahedron formed by four points chosen randomly inside the tetrahedron whose vertices are (0, 0, 0), (0, 1, 0), (0, 0, 1), and (1, 0, 0)? (The precise answer is unknown!) 23. Write a program to compute the area under the curve y = sin x and above the curve y = ln(x + 2). Use the Monte Carlo method, and print intermediate results. 24. Estimatetheintegral by the Monte Carlo method. 􏶩5.9􏶠 sinx+x2􏶡 e dx lnx 25. Test the random-number generator that is available to you in the following manner: Begin by creating a list of N random numbers rk , uniformly distributed in the interval [0,1]. Create a list of random integers nk by extracting the integer part of 10rk for 1􏶞k􏶞 N.Computetheelementsina10×10matrix(mij),wheremij isthenumberof times i is followed by j in the list (nk ). Compare these numbers to the values predicted byelementaryprobabilitytheory.Ifpossible,displaythevaluesofmij graphically. 26. (Student research project) Investigate some of the latest developments on Monte Carlo methods for multivariable integration. 13.3 Simulation We next illustrate the idea of simulation. We consider a physical situation in which an element of chance is present and try to imitate the situation on the computer. Statistical conclusions can be drawn if the experiment is performed many times. Applications include the simulation of servers, clients, and queues as might occur in businesses such as banks or grocery stores. Loaded Die Problem In simulation problems, we must often produce random variables with a prescribed distri- bution. Suppose, for example, that we want to simulate the throw of a loaded die and that the probabilities of various outcomes have been determined as shown: Outcome 1 2 3 4 5 6 Probability 0.2 0.14 0.22 0.16 0.17 0.11 If the random variable x is uniformly distributed in the interval (0, 1), then by breaking this interval into six subintervals of lengths given by the table, we can simulate the throw of this loaded die. For example, we agree that if x is in (0, 0.2), the die shows 1; if x is in [0.2, 0.34), the die shows 2, and so on. A pseudocode to count the outcome of 5000 throws of 3.2 13.3 Simulation 553 this die and compute the probability might be written as follows: program Loaded Die integer i, j; real array (yi )1:6, (mi )1:6, (ri )1:n real n ← 5000 (yi)6 ←(0.2,0.34,0.56,0.72,0.89,1.0) (mi)6 ←(0.0,0.0,0.0,0.0,0.0,0.0) call Random((ri )) for i = 1 to n do for j = 1 to 6 do ifri 0,thensomechoiceofhexists for which the matrix is diagonally dominant.
6. Establish the properties claimed for the function g in Equation (10).
7. Showthatforthesimpleproblem
􏰎
the tridiagonal system to be solved can be written as
⎧⎪⎨ ( 2 − h 2 ) x 1 ⎪⎩−xi−1 +(2−h2)xi
happens:
􏰎
−xn−2 + (2 − h2)xn−1
x′′ =−x
x(a) = α x(b) = β
− x 2 = α −xi+1 =0 = β
(2􏶞i􏶞n−2)
a8. Write down the system of equations Ax = b that results from using the usual second-
order central difference approximation to solve
􏰎
a9. Letubeasolutionoftheinitial-valueproblem
􏰎
How do we solve the following two-point boundary-value problem by utilizing u?
􏰎
10. How would you solve the problem
􏰎
where a, b, A, B, and C are given real numbers? (Assume that A and B are not both zero.) a11. Usetheshootingmethodonthistwo-pointboundary-valueproblem,andexplainwhat
x′′ =(1+t)x
x(0) = 0 x(1) = 1
u′′ =etu+t2u′
u(1) = 0 u′(1) = 1
x′′ =etx+t2x′
x(1) = 0 x(2) = 7
x′ = f(t,x)
Ax(a) + Bx(b) = C
x′′ =−x
x(0) = 3 x(π) = 7
This problem is to be solved analytically, not by computer or calculator.
14.2 A Discretization Method 579

580 Chapter 14
Boundary-Value Problems for Ordinary Differential Equations
1. Explainthemainstepsinsettingupaprogramtosolvethistwo-pointboundaryvalue problem by the finite-difference method.
􏰎
Show any preliminary work that must be done before programming. Exploit the linearity of the differential equation. Program and compare the results when different values of n are used, say, n = 10, 100, and 1000.
2. Solvethefollowingtwo-pointboundaryvalueproblemnumerically.Forcomparisons, the exact solutions are given.
⎧⎨ x′′ = (1 − t)x + 1
aa. ⎩ (1+t)2
x(0) = 1 x(1) = 0.5
􏰎1􏰠􏰡 ab. x′′ = 3 (2−t)e2x +(1+t)−1
x(0) = 0 x(1) = −log2
3. Solvetheboundary-valueproblem
􏰎
by discretization. Compare with the exact solution, which is x (t ) = t + 2 sin t .
4. Repeat Computer Problem 14.1.2, using a discretization method.
5. Writeacomputerprogramtoimplement
a. program BVP1. b. program BVP2.
6. (Continuation)Usingbuilt-inroutinesinmathematicalsoftwaresystemssuchasMatlab, Maple, or Mathematical, solve and plot the solution curve for the boundary-value prob- lem associated with
a. program BVP1. b. program BVP2.
7. Investigate the computation of numerical solutions to the following challenging test
x′′ =xsint+x′cost−et x(0) = 0 x(1) = 1
problems, which are nonlinear:
a.
􏰎􏰎
x′′ = ex εx′′ + (x′)2 = 1 x(0) = 0, x(1) = 0 b. x(0) = 0, x(1) = 1
Vary ε = 10−1, 10−2, 10−3, . . . . Compare to the true solution x(t) = 1 + ε ln cosh((x − 0.745)/ε)
which has a corner at t = 0.745.
x′′ =−x+tx′−2tcost+t x(0) = 0 x(π) = π
Computer Problems 14.2

c. Troesch’sproblem: d. Bratu’s problem:
􏰎
􏰎
x′′ =μsinh(μx) x(0) = 0,x(1) = 1
usingμ=50. using λ = 3.55.
e.
λ = λ∗, and no solutions when λ > λ∗.
εx′′ + tx′ = 0 using ε = 10−8. x(−1) = 0, x(1) = 2
􏰎
x′′ + λex = 0
x(0) = 0,x(1) = 0
If we let λ = 3.51383 . . . , there are two solutions when λ < λ∗, one solution when 14.2 A Discretization Method 581 √√ Compare to the true solution x(t) = 1 + erf(t/ Cash [2003] uses these and other test problems in his research. For more information on them, see www.ma.ic.ac.uk/∼jcash/ 8. (Buckingofacircularringproject)Amodelforacircularringwithcompressibility c under hydrostatic pressure p from all directions is given by the following boundary- value problem involving a system of seven differential equations: 􏰒 􏰓 y′ =−1−cy +(c+1)y , y (0)= π, y π =0 1571212 2ε)/erf(1/ 2ε). y2′ =[1+c(y5−y7)]cosy1, y3′ =[1+c(y5 −y7)]siny1, y4′ =1+c(y5 −y7), y5′ = y6[−1−cy5 +(c+1)y7], y3(0)=0 y4(0)=0 􏰒π􏰓 y2 2 =0 􏰒π􏰓 y6′ =y5y7−[1+c(y5−y7)](y5+p), y6(0)=0, y6 2 =0 y7′ =[1+c(y5−y7)]y6 Various simplifications are useful in the study of the buckling or collapse of the circular ring such as by considering only a quarter-circle by symmetry (sketch (a) below). As the pressure increases, the radius of the circle decreases, and a bifurcation or a change of state can occur (sketch (b) below). The shooting method together with more advanced numerical methods can be used to solve this problem. Explore some of them. See Huddleston [2000] and Sauer [2006] for additional details. s 􏶾 􏷅/2 (y2, y3) y1 y4 p s􏶾0 l 􏶼1 1 􏶼1 (b) p p (a) 15 Partial Differential Equations In the theory of elasticity, it is shown that the stress in a cylindrical beam under torsion can be derived from a function u(x, y) that satisfies the Poisson equation ∂2u + ∂2u + 2 = 0 ∂x2 ∂y2 In the case of a beam whose cross section is the square defined by |x| 􏶞 1, | y | 􏶞 1, the function u must satisfy Poisson’s equation inside the square and must be zero at each point on the perimeter of the square. By using the methods of this chapter, we can construct a table of approximate values of u(x, y). 15.1 Parabolic Problems Many physical phenomena can be modeled mathematically by differential equations. When the function that is being studied involves two or more independent variables, the differential equation is usually a partial differential equation. Since functions of several variables are intrinsically more complicated than those of one variable, partial differential equations can lead to some of the most challenging of numerical problems. In fact, their numerical solution is one type of scientific calculation in which the resources of the fastest and most expensive computing systems easily become taxed. We shall see later why this is so. Some Partial Differential Equations from Applied Problems Some important partial differential equations and the physical phenomena that they govern are listed here: • Thewaveequationinthreespatialvariables(x,y,z)andtimetis ∂2u = ∂2u + ∂2u + ∂2u ∂t2 ∂x2 ∂y2 ∂z2 The function u represents the displacement at time t of a particle whose position at rest is (x, y, z). With appropriate boundary conditions, this equation governs vibrations of a three-dimensional elastic body. 582 • Theheatequationis ∂u = ∂2u + ∂2u + ∂2u ∂t ∂x2 ∂y2 ∂z2 15.1 Parabolic Problems 583 The function u represents the temperature at time t in a physical body at the point that has coordinates (x, y, z). • Laplace’sequationis ∂2u + ∂2u + ∂2u = 0 ∂x2 ∂y2 ∂z2 It governs the steady-state distribution of heat in a body or the steady-state distribution of electrical charge in a body. Laplace’s equation also governs gravitational, electric, and magnetic potentials and velocity potentials in irrotational flows of incompressible fluids. The form of Laplace’s equation given above applies to rectangular coordinates. In cylindrical and spherical coordinates, it takes these respective forms: ∂2u + 1 ∂u + 1 ∂2u + ∂2u = 0 ∂r2 r ∂r r2 ∂φ2 ∂z2 1∂2 1 ∂􏰔 ∂u􏰕 1 ∂2u r∂r2(ru)+r2sinθ∂θ sinθ∂θ +r2sin2θ∂φ2 =0 • Thebiharmonicequationis ∂4u+2 ∂4u +∂4u=0 ∂x4 ∂x2 ∂y2 ∂y4 It occurs in the study of elastic stress, and from its solution the shearing and normal stresses can be derived for an elastic body. • TheNavier-Stokesequationsare ∂u +u∂u +v∂u + ∂p = ∂2u + ∂2u ∂t ∂x ∂y ∂x ∂x2 ∂y2 ∂v +u∂v +v∂v + ∂p = ∂2v + ∂2v ∂t ∂x ∂y ∂y ∂x2 ∂y2 Here, u and v are components of the velocity vector in a fluid flow. The function p is the pressure, and the fluid is assumed to be incompressible but viscous. In three dimensions, the following operators are useful in writing many standard partial differential equations ∇=∂+∂+∂ ∂x ∂y ∂z 2 ∂2 ∂2 ∂2 ∇ = ∂x2 + ∂y2 + ∂z2 (Laplacian operator) 584 Chapter 15 Partial Differential Equations For example, we have Heat equation Diffusion equation Wave equation Laplace equation Poisson equation Helmholtz equation 1 ∂u = ∇2u k ∂t ∂u = ∇(d∇u) + ρ ∂t 1 ∂2u = ∇2u ν2 ∂t2 ∇2u = 0 ∇2u = −4πρ ∇2u = −k2u The diffusion equation with diffusion constant d has the same structure as the heat equation because heat transfer is a diffusion process. Some authors use alternate notation such as 􏶚u = curl(grad(u)) = ∇2u. Additional examples from quantum mechanics, electromagnetism, hydrodynamics, elasticity, and so on could also be given, but the five partial differential equations shown already exhibit a great diversity. The Navier-Stokes equation, in particular, illustrates a very complicated problem: a pair of nonlinear, simultaneous partial differential equations. To specify a unique solution to a partial differential equation, additional conditions must be imposed on the solution function. Typically, these conditions occur in the form of bound- ary values that are prescribed on all or part of the perimeter of the region in which the solution is sought. The nature of the boundary and the boundary values are usually the determining factors in setting up an appropriate numerical scheme for obtaining the approximate solution. Matlab includes a PDE Toolbox for partial differential equations. It contains many commands for such tasks as describing the domain of an equation, generating meshes, computing numerical solutions, and plotting. Within Matlab, the command pdetool in- vokes a graphical user interface (GUI) that is a self-contained graphical environment for solving partial differential equations. One draws the domain and indicates the boundary, fills in menus with the problem and boundary specifications, and selects buttons to solve the problem and plot the results. Although this interface may provide a convenient working environment, there are situations in which command-line functions are needed for addi- tional flexibility. A suite of demonstrations and help files is useful in finding one’s way. For example, this software can handle PDEs of the following types ParabolicPDE HyperbolicPDE Elliptic PDE b∂u−∇·(c∇u)+au= f ∂t b∂2u−∇·(c∇u)+au= f ∂t2 −∇ · (c∇u) + au = f for x and y on the two-dimensional domain 􏶝 for the problem. On the boundaries of the domain, the following boundary conditions can be handled: Dirichlet Generalized Neumann Mixed hu = r n⃗ · (c∇u) + qu = g combination of Dirichlet/Neumann Here, n⃗ = du/dν is the outward unit length normal derivative. While the PDE can be entered via a dialog box, both the boundary conditions and the PDE coefficients a, c, d can be entered in a variety of ways. One can construct the geometry of the domain by drawing solid objects (circle, polygon, rectangle, and ellipse) that may be overlapped, moved, and rotated. Heat Equation Model Problem In this section, we consider a model problem of modest scope to introduce some of the essential ideas. For technical reasons, the problem is said to be of the parabolic type. In it we have the heat equation in one spatial variable accompanied by boundary conditions appropriate to a certain physical phenomenon: ⎧⎪ ⎪ ∂ 2 ∂ ⎪⎨ ∂ x 2 u ( x , t ) = ∂ t u ( x , t ) ⎪ ⎪ ⎪⎩ u(0,t) = u(1,t) = 0 u(x,0) = sinπx (1) 15.1 Parabolic Problems 585 These equations govern the temperature u(x, t) in a thin rod of length 1 when the ends are held at temperature 0, under the assumption that the initial temperature in the rod is given by the function sinπx (see Figure 15.1). In the xt-plane, the region in which the solution is sought is described by inequalities 0 􏶞 x 􏶞 1 and t 􏶟 0. On the boundary of this region (shaded in Figure 15.2), the values of u have been prescribed. Rod Ice Ice FIGURE 15.1 Heated rod 01 x FIGURE 15.2 Heat equation: xt-plane t 01 Finite-Difference Method x A principal approach to the numerical solution of such a problem is the finite-difference method. It proceeds by replacing the derivatives in the equation by finite differences. Two 586 Chapter 15 Partial Differential Equations formulas from Section 4.3 are useful in this context: f′(x)≈ 1[f(x+h)− f(x)] h f′′(x)≈ 1[f(x+h)−2f(x)+ f(x−h)] h2 If the formulas are used in the differential Equation (1), with possibly different step lengths h and k, the result is 1 [u(x +h,t)−2u(x,t)+u(x −h,t)] = 1[u(x,t +k)−u(x,t)] (2) This equation is now interpreted as a means of advancing the solution step by step in the t variable. That is, if u(x,t) is known for 0􏶞 x 􏶞 1 and 0􏶞 t 􏶞 t0, then Equation (2) allows us to evaluate the solution for t = t0 + k. Equation (2) can be rewritten in the form u(x,t +k)=σu(x +h,t)+(1−2σ)u(x,t)+σu(x −h,t) (3) where σ=k h2 A sketch showing the location of the four points involved in this equation is given in Figure 15.3. Since the solution is known on the boundary of the region, it is possible to compute an approximate solution inside the region by systematically using Equation (3). It is, of course, an approximate solution because Equation (2) is only a finite-difference analog of Equation (1). (x, t 􏶽 k) (x 􏶼 h, t) (x, t) (x 􏶽 h, t) To obtain an approximate solution on a computer, we select values for h and k and use Equation (3). An analysis of this procedure, which is outside the scope of this text, shows that for stability of the computation, the coefficient 1−2σ in Equation (3) should be nonnegative. (If this condition is not met, errors made at one step will probably be magnified at subsequent steps, ultimately spoiling the solution.) The reader is referred to Kincaid and Cheney [2002] or Forsythe and Wasow [1960] for a discussion of stability. Using this algorithm, we can continue the solution indefinitely in the t-variable by computations involving only prior values of t. This is an example of a marching problem or marching method. FIGURE 15.3 Heat equation: Explicit stencil h2 k Pseudocode for Explicit Method For utmost simplicity, we select h = 0.1 and k = 0.005. Coefficient σ is now 0.5. This choicemakesthecoefficient1−2σ equaltozero.Ourpseudocodefirstprintsu(ih,0)for 0 􏶞 i 􏶞 10 because they are known boundary values. Then it computes and prints u(ih, k) for 0 􏶞 i 􏶞 10 using Equation (3) and boundary values u(0, t) = u(1, t) = 0. This procedure is continued until t reaches the value 0.1. The single subscripted arrays (ui ) and (vi ) are used to store the values of the approximate solution at t and t + k , respectively. Since the analytic solution of the problem is u(x,t) = e−π2t sinπx (see Problem 15.1.3), the error can be printed out at each step. The procedure described is an example of an explicit method. The approximate values ofu(x,t+k)arecalculatedexplicitlyintermsofu(x,t).Notonlyisthissituationatypical, but even in this problem the procedure is rather slow because considerations of stability force us to select k􏶞 1h2 2 Since h must be rather small to represent the derivative accurately by the finite difference formula, the corresponding k must be extremely small. Values such as h = 0.1 and k = 0.005 are representative, as are h = 0.01 and k = 0.00005. With such small values of k, an inordinate amount of computation is necessary to make much progress in the t variable. 15.1 Parabolic Problems 587 program Parabolic1 integer i, j ; real array (ui )0:n , (vi )0:n integer n ← 10, m ← 20 real h ← 0.1, k ← 0.005 realu0 ←0,v0 ←0,un ←0,vn ←0 fori =1ton−1do ui ← sin(πih) end for output (ui ) for j = 1 to m do fori =1ton−1do vi ← (ui−1 + ui+1)/2 end for output (vi ) t ← jk fori =1ton−1do ui ← e−π2t sin(πih) − vi end for output (ui ) fori =1ton−1do ui ←vi end for end for end program Parabolic1 588 Chapter 15 Partial Differential Equations Crank-Nicolson Method An alternative procedure of the implicit type goes by the name of its inventors, John Crank and Phyllis Nicolson, and is based on a simple variant of Equation (2): 1 [u(x +h,t)−2u(x,t)+u(x −h,t)] = 1[u(x,t)−u(x,t −k)] (4) h2 k If a numerical solution at grid points x = ih, t = jk has been obtained up to a certain level in the t variable, Equation (4) governs the values of u on the next t level. Therefore, Equation (4) should be rewritten as in which −u(x − h, t) + ru(x, t) − u(x + h, t) = su(x, t − k) (5) r = 2 + s and s = h2 k FIGURE 15.4 Crank-Nicolson method: Implicit stencil The locations of the four points in this equation are shown in Figure 15.4. (x 􏶼 h, t) (x, t) (x 􏶽 h, t) (x, t 􏶼 k) On the t level, u is unknown, but on the (t − k) level, u is known. So we can introduce unknownsui =u(ih,t)andknownquantitiesbi =su(ih,t−k)andwriteEquation(5)in matrix form: ⎡ r −1 ⎤⎡u⎤⎡b ⎤ 11 ⎢−1 r −1 ⎥⎢u2 ⎥ ⎢b2 ⎥ ⎢ −1 r −1 ⎥⎢u3 ⎥ ⎢b3 ⎥ ⎥=⎢ . ⎥ (6) ⎢ ... ... ... ⎥⎢ . ⎢⎣ −1r−1 ⎥⎦ ⎢⎣ u ⎥⎦ ⎢⎣ b ⎥⎦ n−2 bn−1 n−2 −1 r un−1 The simplifying assumption that u(0,t) = u(1,t) = 0 has been used here. Also, h = 1/n. The system of equations is tridiagonal and diagonally dominant because |r| = 2 + h2/k > 2. Hence, it can be solved by the efficient method of Section 7.3.
An elementary argument shows that this method is stable. We shall see that if the initial values u(x, 0) lie in an interval [α, β], then values subsequently calculated by using Equation (5) will also lie in [α, β ], thereby ruling out any unstable growth. Since the solution is built up line by line in a uniform way, we need only verify that the values on the first computed line, u(x, k), lie in [α, β]. Let j be the index of the largest ui that occurs on this line t = k. Then
−uj−1 +ruj −uj+1 =bj

Since uj is the largest of the u’s, uj−1 􏶞 uj and uj+1 􏶞 uj. Thus, ruj =bj +uj−1+uj+1􏶞bj +2uj
Sincer=2+sandbj =su(jh,0),thepreviousinequalityleadsatonceto uj 􏶞u(jh,0)􏶞β
Sinceuj isthelargestoftheui,wehave
ui􏶞β foralli
Similarly,
thus establishing our assertion.
ui􏶟α foralli
Pseudocode for the Crank-Nicolson Method
A pseudocode to carry out the Crank-Nicolson method on the model program is given next. In it, h = 0.1, k = h2/2, and the solution is continued until t = 0.1. The value of r is 4 and s = 2. It is easier to compute and print only the values of u at interior points on each horizontal line. At boundary points, we have u(0, t) = u(1, t) = 0. The program calls procedure Tri from Section 7.3.
15.1 Parabolic Problems 589
program Parabolic2
integer i, j; real array (ci )1:n−1, (di )1:n−1, (ui )1:n−1, (vi )1:n−1 integer n ← 10, m ← 20
real h ← 0.1, k ← 0.005
real r, s, t
s ← h2/k
r←2+s
fori =1ton−1do
di ←r
ci ←−1
ui ← sin(πih)
end for
output (ui )
for j = 1 to m do
fori =1ton−1do di ←r
vi ←sui end for

590 Chapter 15
Partial Differential Equations
call Tri(n − 1, (ci ), (di ), (ci ), (vi ), (vi )) output (vi )
t ← jk
fori =1ton−1do
ui ← e−π2t sin(πih) − vi end for
output (ui )
fori =1ton−1do
ui ←vi end for
end for
end program Parabolic2
We used the same values for h and k in the pseudocode for two methods (explicit and Crank- Nicolson), so a fair comparison can be made of the outputs. Because the Crank-Nicolson method is stable, a much larger k could have been used.
Alternative Version of the Crank-Nicolson Method
Another version of the Crank-Nicolson method is obtained as follows: The central differ-
ences at 􏰉x, t − 1 k􏰈 in Equation (4) produce 2
1􏰖􏰔 1􏰕􏰔1􏰕􏰔 1􏰕􏰗 h2 u x+h,t−2k −2u x,t−2k +u x−h,t−2k
= 1[u(x,t)−u(x,t −k)] k
Since the u values are known only at integer multiples of k, terms such as u􏰉x, t − 1 k􏰈 are 2
replaced by the average of u values at adjacent grid points; that is, 􏰔1􏰕1
u x,t−2k ≈2[u(x,t)+u(x,t−k)] 1 [u(x +h,t)−2u(x,t)+u(x −h,t)+u(x +h,t −k)
So we have 2h2
−2u(x,t−k)+u(x−h,t−k)]= 1[u(x,t)−u(x,t−k)] k
The computational form of this equation is
−u(x −h,t)+2(1+s)u(x,t)−u(x +h,t)
where
= u(x − h, t − k) + 2(s − 1)u(x, t − k) + u(x + h, t − k) (7)
h2 1 s=k≡σ

FIGURE 15.5
Crank-Nicolson method: Alternative stencil
Stability
(x 􏶼 h, t)
(x 􏶼 h, t 􏶼 k)
(x, t)
(x, t 􏶼 k)
(x 􏶽 h, t)
(x 􏶽 h, t 􏶼 k)
15.1 Parabolic Problems 591 The six points in this equation are shown in Figure 15.5. This leads to a tridiagonal system
ofform(6)withr =2(1+s)and
bi =u((i−1)h,t−k)+2(s−1)u(ih,t−k)+u((i+1)h,t−k)
At the heart of the explicit method is Equation (3), which shows how the values of u for t + k depend on the values of u at the previous time step, t . If we introduce the values of u onthemeshbywritinguij =u(ih,jk),thenwecanassembleallthevaluesforonet-level into a vector v( j ) as follows:
v(j) = [u0j,u1j,u2j,…,unj]T Equation (3) can now be written in the form
ui,j+1 =σui+1,j +(1−2σ)uij +σui−1,j This equation shows how v( j+1) is obtained from v( j). It is simply
v(j+1) = Av(j) where A is the matrix whose elements are
⎤
Our equations tell us that
v(j) = Av(j−1) = A2v(j−2) = A3v(j−3) = ··· = Ajv(0)
From physical considerations, the temperature in the bar should approach zero. After all, the heat is being lost through the ends of the rod, which are being kept at temperature 0. Hence, A j v(0) should converge to 0 as j → ∞.
At this juncture, we need a theorem in linear algebra that asserts (for any matrix A) that Ajv → 0 for all vectors v if and only if all eigenvalues of A satisfy |λi| < 1. The eigenvalues of the matrix A in the present analysis are known to be λi =1−2σ(1−cosθi) θi = iπ n+1 ⎡1−2σ σ ⎢ σ 1−2σ σ ⎢σ1−2σσ ⎥ ⎥ ⎢ ... ... ⎢⎣ σ 1−2σ σ ⎥⎦ ... ⎥ σ 1−2σ 592 Chapter 15 Partial Differential Equations FIGURE 15.6 Heat equation: (a) Solution surface; (b) Contour plot 0 0 0.2 0.4 0.6 0.8 1 In our problem, we therefore must have −1 < 1 − 2σ (1 − cos θi ) < 1 This leads to 0 < σ 􏶞 1 , because θi can be arbitrarily close to π . This in turn leads to 2 the step-size condition k 􏶞 1 h2. 2 Mathematical software systems such as Matlab, Maple, or Mathematica contain rou- tines that solve partial differential equations. For example in Maple and Mathematica, we can invoke commands to verify the general analytical solution. (See Problem 15.1.3.) In Matlab, there is a sample program to numerically solve our model heat equation. In Fig- ure 15.6, we solve the heat equation, generate a three-dimensional plot of its solution surface, and produce a two-dimensional contour plot, which is displayed in color for indicating the various contours. 1 0.75 0.5 0.25 0 0 0.5 0.4 0.3 0.2 0.1 0.2 0.5 0.4 0.3 0.2 0.4 0.6 0.1 10 0.8 The PDE Toolbox within Matlab produces solutions to partial differential equations using the finite-element formulation of the scalar PDE problem. (See Section 15.3 for addi- tional discussion of the finite-element method.) This software library contains a graphical user interface with graphical tools for describing domains, generating triangular meshes on them, discretizing the PDEs on the mesh, building systems of equations, obtaining numerical approximations for their solution, and visualizing the results. In particular, Matlab has the function parabolic for solving parabolic PDEs. As is found in the Matlab documentation, one can solve the two-dimensional heat equation ∂u =∇2u ∂t on the square −1 􏶞 x , y 􏶞 1. There are Dirichlet boundary conditions u = 0 and discon- tinuous initial conditions u(0) = 1 in the circle x2 + y2 < 2 and u(0) = 0 otherwise. A 15.1 Parabolic Problems 593 5 Matlab demonstration continues with a movie of the solution curves. Summary (1) We consider a model problem involving the following parabolic partial differential equation ∂2 ∂ ∂x2u(x,t)= ∂tu(x,t) Using finite differences with step size h in the x -direction and k in the t -direction, we obtain 1 [u(x +h,t)−2u(x,t)+u(x −h,t)] = 1[u(x,t +k)−u(x,t)] h2 k The computational form is u(x,t +k)=σu(x +h,t)+(1−2σ)u(x,t)+σu(x −h,t) where σ = k/h2. An alternative approach is the Crank-Nicolson method based on other finite differences for the right-hand side: 1 [u(x +h,t)−2u(x,t)+u(x −h,t)] = 1[u(x,t)−u(x,t −k)] h2 k Its computational form is −u(x − h, t) + ru(x, t) − u(x + h, t) = su(x, t − k) where r = 2 + s and s = h2/k. Yet another variant of the Crank-Nicolson method is based on these finite differences: 1􏰖􏰔 1􏰕􏰔1􏰕􏰔 1􏰕􏰗 h2 u x+h,t−2k −2u x,t−2k +u x−h,t−2k = 1[u(x,t)−u(x,t −k)] Then by using k 􏰔1􏰕1 u x,t−2k ≈2[u(x,t)+u(x,t−k)] 594 Chapter 15 Partial Differential Equations the computational form is −u(x −h,t) + 2(1+s)u(x,t)−u(x +h,t) = u(x −h,t −k)+2(s −1)u(x,t −k)+u(x +h,t −k) where s = h2/k. This results in a tridiagonal system of equations to be solved. 1. Asecond-orderlineardifferentialequationwithtwovariableshastheform A∂2u+B ∂2u +C∂2u+···=0 Here, A, B, and C are functions of x and y, and the terms not written are of lower order. The equation is said to be elliptic, parabolic, or hyperbolic at a point (x, y), depending on whether B2 − 4AC is negative, zero, or positive, respectively. Classify each of these equations in this manner: aa. uxx+uyy+ux+sinxuy−u=x2+y2 b. uxx −uyy +2ux +2uy +exu=x−y a2. 3. h. xuxx +yuxy +uyy =0 Derivethetwo-dimensionalformofLaplace’sequationinpolarcoordinates. a4. a5. a 6. ac. uxx =uy +u−ux +y e. 3uxx +uxy +uyy =exy g. uxx +2uxy +uyy =0 d. uxy =u−ux −uy af. exuxx +cosyuxy −uyy =0 ∂x2 ∂x ∂y ∂y2 Showthatthefunction u(x,t) = is a solution of the heat conduction problem uxx = ut and satisfies the boundary condition 􏰃N n=1 Refertothemodelproblemsolvednumericallyinthissectionandshowthatifthereis no roundoff, the approximate solution values obtained by using Equation (3) lie in the interval [0, 1]. (Assume 1 􏶟 2k/h2.) Find a solution of Equation (3) that has the form u(x,t) = at sinπx, where a is a constant. In using Equation (5), how must the linear System (6) be modified for u (0, t ) = c0 and u(1, t) = cn with c0 ≠ 0, cn ≠ 0? When using Equation (7)? u(0,t) = u(1,t) = 0 u(x,0) = cn sinnπx for all N 􏶟 1 􏰃N n=1 cne−(nπ)2t sinnπx Problems 15.1 a7. DescribeindetailhowEquation(1)withboundaryconditionsu(0,t)=q(t),u(1,t)= g(t), and u(x, 0) = f (x) can be solved numerically by using System (6). Here q, g, and f are known functions. a8. What finite difference equation should be a suitable replacement for the equation ∂2u/∂x2 = ∂u/∂t + ∂u/∂x in numerical work? a9. Consider the partial differential equation ∂u/∂x + ∂u/∂t = 0 with u = u(x,t) in the region [0, 1] × [0, ∞], subject to the boundary conditions u(0, t) = 0 and u(x, 0) specified. For fixed t, we discretize only the first term using (ui+1 − ui−1)/(2h) for i = 1,2,...,n − 1 and (un − un−1)/h, where h = 1/n. Here, ui = u(xi,t) and xi = ih with fixed t. In this way, the original problem can be considered a first-order initial-value problem y = [u , u , . . . , u ]T Determine the n × n matrix A. 15.1 Parabolic Problems 595 where dy+1Ay=0 dx 2h d y = 􏰠u′ , u′ , . . . , u′ 􏰡T u′ = ∂ui 12 n dx12 n i∂t 10. RefertothediscussionofthestabilityoftheCrank-Nicolsonprocedure,andestablish the inequality ui 􏶟 α. 11. What happens to System (6) when k = h2? 12. (Multiple choice) In solving the heat equation uxx = ut on the domain t􏶟1 and 0 􏶞 x 􏶞 1, one can use the explicit method. Suppose the approximate solution on one horizontal line is a vector V j . Then the whole process turns out to be described by Vj+1 = AVj where A is a tridiagonal matrix, having 1−2σ on its diagonal and σ in the superdiagonal and subdiagonal positions. Here σ = k/h2, where k is the time step and h is the x-step. For stability in the numerical solution, what should we require? a. σ = 1 b. Alleigenvaluesof Asatisfy|λ|<1. c. k􏶟h2/2 2 d. h=0.01andk=5×10−3 e. Noneofthese. 13. (Continuation) The fully implicit method for solving the heat conduction problem requires at each step the solution of the equation AVj−1 = Vj Here, A is not the same as in the preceding problem, but is similar: It has 1 + 2σ on the diagonal and −σ on the subdiagonal and superdiagonal. What do we know about the eigenvalues of this matrix, A? Hint: This question concerns eigenvalues of A, not A−1. a. They are all negative. b. They are all in the open interval (0, 1). c. They are greater than 1. d. They are in the interval (−1, 0). e. Noneofthese. 596 Chapter 15 Partial Differential Equations 15.2 1. Solve the same heat conduction problem as in the text except use h = 2−4, k = 2−10, and u(x, 0) = x(1 − x). Carry out the solution until t = 0.0125. 2. Modify the Crank-Nicolson code in the text so that it uses the alternative scheme (7). Compare the two programs on the same problems with the same spacing. 3. Recodeandtestthepseudocodeinthissectionusingacomputerlanguagethatsupports vector operations. 4. RuntheCrank-Nicolsoncodewithdifferentchoicesofhandk,inparticular,lettingk be much larger. Try k = h, for example. 5. Trytotakeadvantageofanyspecialcommandsorproceduresinmathematicalsoftware such as in Matlab, Maple, or Mathematica to solve the numerical example (1). 6. (Continuation) Use the symbolic manipulation capabilities in Maple or Mathematica to verify the general analytical solution of (1). Hint: See Problem 15.1.3. Hyperbolic Problems Wave Equation Model Problem The wave equation with one space variable ∂2u = ∂2u (1) ∂t2 ∂x2 governs the vibration of a string (transverse vibration in a plane) or the vibration in a rod (longitudinal vibration). It is an example of a second-order linear differential equation of the hyperbolic type. If Equation (1) is used to model the vibrating string, then u(x, t) represents the deflection at time t of a point on the string whose coordinate is x when the string is at rest. To pose a definite model problem, we suppose that the points on the string have coordinates x in the interval 0 􏶞 x 􏶞 1 (see Figure 15.7). Let us suppose that at time t = 0, the deflections satisfy equations u(x, 0) = f (x) and ut (x, 0) = 0. Assume also that the ends of the string remain fixed. Then u(0, t) = u(1, t) = 0. A fully defined boundary-value FIGURE 15.7 x Vibrating string 0 x 1 u u(x, t) Computer Problems 15.1 problem, then, is ⎧⎪ ⎪ ⎪ u t t − u x x = 0 ⎨ u(x,0)=f(x) ⎪⎩ ut(x,0)=0 u(0,t) = u(1,t) = 0 (2) The region in the xt-plane where a solution is sought is the semi-infinite strip defined by inequalities 0 􏶞 x 􏶞 1 and t 􏶟 0. As in the heat conduction problem of Section 15.1, the values of the unknown function are prescribed on the boundary of the region shown (see Figure 15.8). 15.2 Hyperbolic Problems 597 FIGURE 15.8 Wave equation: xt-plane t 01 x Analytic Solution The model problem in (2) is so simple that it can be immediately solved. Indeed, the solution is u(x,t)= 1[f(x+t)+ f(x−t)] (3) 2 provided that f possesses two derivatives and has been extended to the whole real line by defining f(−x)=−f(x) and f(x+2)= f(x) To verify that Equation (3) is a solution, we compute derivatives using the chain rule: ux = 1[f′(x +t)+ f′(x −t)] ut = 1[f′(x +t)− f′(x −t)] 22 uxx = 1[f′′(x +t)+ f′′(x −t)] utt = 1[f′′(x +t)+ f′′(x −t)] 22 Obviously, Also, Furthermore, we have utt =uxx u(x,0)= f(x) ut(x,0)= 1[f′(x)− f′(x)]=0 2 598 Chapter 15 Partial Differential Equations FIGURE 15.9 Wave equation: f stencil (x 􏶼 t, 0) Numerical Solution (x, t) (x, 0) (x 􏶽 t, 0) x In checking endpoint conditions, we use the formulas by which f was extended: u(0,t)= 1[f(t)+ f(−t)]=0 2 u(1,t)= 1[f(1+t)+ f(1−t)] 2 = 1[f(1+t)− f(t −1)] 2 = 1[f(1+t)− f(t−1+2)]=0 2 The extension of f from its original domain to the entire real line makes it an odd periodic function of period 2. Odd means that f (x) = − f (−x) and the periodicity is expressed by f(x+2)= f(x) for all x. To compute u(x, t), we need to know f at only two points on the x-axis, x + t and x − t, as in Figure 15.9. The model problem is used next to illustrate again the principle of numerical solution. Choosing step sizes h and k for x and t , respectively, and using the familiar approximations for derivatives, we have from Equation (1) 1 [u(x +h,t)−2u(x,t)+u(x −h,t)] h2 = 1 [u(x,t +k)−2u(x,t)+u(x,t −k)] k2 which can be rearranged as u(x, t + k) = ρu(x + h, t) + 2(1 − ρ)u(x, t) + ρu(x − h, t) − u(x, t − k) (4) Here, ρ = k2 h2 15.2 Hyperbolic Problems 599 FIGURE 15.10 Wave equation: Explicit stencil Figure 15.10 shows the point (x, t + k) and the nearby points that enter into Equation (4). (x, t 􏶽 k) (x 􏶼 h, t) (x, t) (x 􏶽 h, t) (x, t 􏶼 k) The boundary conditions in Problem (2) can be written as ⎧ ⎪⎨ u(x,0)= f(x) 1[u(x,k)−u(x,0)] = 0 (5) ⎪ ⎪ ⎪⎩ k u(0,t) = u(1,t) = 0 The problem defined by Equations (4) and (5) can be solved by beginning at the line t = 0, where u is known, and then progressing one line at a time with t = k, t = 2k, t = 3k,.... Note that because of (5), our approximate solution satisfies u(x,k)=u(x,0)= f(x) (6) The use of the O(k) approximation for ut leads to low accuracy in the computed solution to Problem (2). Suppose that there is a row of grid points (x, −k). Letting t = 0 in Equation (4), we have u(x, k) = ρu(x + h, 0) + 2(1 − ρ)u(x, 0) + ρu(x − h, 0) − u(x, −k) Now the central difference approximation 1 [u(x,k)−u(x,−k)]=0 2k for ut(x,0) = 0 can be used to eliminate the fictitious grid point (x, −k). So instead of Equation (6), we set u(x,k)= 1ρ[f(x+h)+ f(x−h)]+(1−ρ)f(x) (7) 2 becauseu(x,0)= f(x).Valuesofu(x,nk),n􏶟2,cannowbecomputedfromEquation(4). 600 Chapter 15 Partial Differential Equations Pseudocode A pseudocode to carry out this numerical process is given next. For simplicity, three one- dimensional arrays (ui ), (vi ), and (wi ) are used: (ui ) represents the solution being computed on the new t line; (vi ) and (wi ) represent solutions on the preceding two t lines. program Hyperbolic integer i, j; real t,x,ρ; real array (ui)0:n,(vi)0:n,(wi)0:n integer n ← 10, m ← 20 real h ← 0.1, k ← 0.05 u0 ←0;v0 ←0;w0 ←0;un ←0;vn ←0;wn ←0 ρ ← (k/h)2 fori =1ton−1do x ← ih wi ← f(x) vi ← 1ρ[f(x −h)+ f(x +h)]+(1−ρ)f(x) 2 end for for j = 2 to m do fori =1ton−1do ui ←ρ(vi+1 +vi−1)+2(1−ρ)vi −wi end for output j,(ui) fori =1ton−1do wi ←vi vi ←ui t ← jk x ← ih ui ←TrueSolution(x,t)−vi end for output j,(ui) end for end program Hyperbolic real function f (x) real x f ←sin(πx) end function f real function True Solution(x,t) real t, x True Solution ← sin(πx)cos(πt) end function True Solution This pseudocode requires accompanying functions to compute values of f (x) and the true solution. We chose f (x ) = sin(π x ) in our example. It is assumed that the x interval is [0, 1], but when h or n is changed, the interval can be [0, b]; that is, nh = b. The numerical solution is printed on the t lines that correspond to 1k, 2k, . . . , mk. More advanced treatments show that the ratios ρ = k2 h2 must not exceed 1 if the solution of the finite difference equations is to converge to a solution of the differential problem as k → 0 and h → 0. Furthermore, if ρ > 1, roundoff errors that occur at one stage of the computation would probably be magnified at later stages and thereby ruin the numerical solution.
In Matlab, the PDE Toolbox has a function for producing the solution of hyperbolic problems using the finite element formulation of the scalar PDE problem. An example found in the Matlab documentation finds the numerical solution of the two-dimensional wave propagation problem
∂2u = ∇2u ∂t2
on the square −1 􏶞 x , y 􏶞 1 with Dirichlet boundary conditions on the left and right bound-
aries, u = 0 for x = ±1, and zero values of the normal derivatives on the top and bottom
boundaries. Further, there are Neumann boundary conditions ∂u/∂ν = 0 for y = ±1. The
initial conditions u(0) = arctan􏰉cos􏰉π x􏰈􏰈 and du(0)/dt = 3sin(πx)exp􏰉sin􏰉π y􏰈􏰈 are 22
chosen to avoid putting too much energy into the higher vibration modes.
Advection Equation
We focus on the advection equation
∂u =−c∂u
∂t ∂x
Here, u = u(x,t) and c = c(x,t) in which one can consider x as space and t as time. The advection equation is a hyperbolic partial differential equation that governs the motion of a conserved scalar as it is advected by a known velocity field. For example, the advection equation applies to the transport of dissolved salt in water. Even in one space dimension and constant velocity, the system remains difficult to solve. Since the advection equation is difficult to solve numerically, interest typically centers on discontinuous shock solutions, which are notoriously hard for numerical schemes to handle.
Using the forward difference approximation in time and the central-difference approx- imations in space, we have
15.2 Hyperbolic Problems 601
This gives
1[u(x,t+k)−u(x,t)]=−c 1 [u(x+h,t)−u(x−h,t)] k 2h
u(x, t + k) = u(x, t) − 1σ [u(x + h, t) − u(x − h, t)] 2
where σ = (k/h)c(x,t). All numerical solutions grow in magnitude for all time steps k. For all σ > 0, this scheme is unstable by Fourier stability analysis.

602 Chapter 15
Partial Differential Equations
Lax Method
In the central-difference scheme above, replace the first term on the right-hand side, u(x, t), by 1 [u(x,t−k)+u(x,t+k)].Thenweobtain
2
This is the Lax method, and this simple change makes the method conditionally stable. Upwind Method
Another way of obtaining a stable method is by using a one-sided approximation to ux in the advection equation as long as the side is taken in the upwind direction. If c > 0, the transport is to the right. This can be interpreted as the wind of speed c blowing the solution from left to right. So the upwind direction is to the left for c > 0 and to the right for c < 0. Thus, the upwind difference approximation is u(x,t +k) = 1 [u(x,t −k)+u(x,t +k)]− 1σ [u(x +h,t)−u(x −h,t)] 22 = 1(1+σ)u(x −h,t)+ 1(1+σ)u(x,t −k) 22 ux(x,t) ≈ Then the upwind scheme for the advection equation is 􏰎 u(x, t + k) = u(x, t) − σ Lax-Wendroff Method −c [u(x, t) − u(x − h, t)] /h −c [u(x + h, t) − u(x, t)] /h (c > 0) (c < 0) 􏰎 −c [u(x, t) − u(x − h, t)] /h −c [u(x + h, t) − u(x, t)] /h (c > 0) (c < 0) The Lax-Wendroff scheme is second-order in space and time. The following is one of several possible forms of this method. We start with a Taylor series expansion over one time step: u(x,t + k) = u(x,t) + kut(x,t) + 1k2utt(x,t) + O(k3) 2 Now use the advection equation to replace time derivatives on the right-hand side by space derivatives: ut = −cux utt =(−cux)t =−ctux −c(ux)t =−ctux −c(ut)x =−ctux +c(cux)t Here, we have let c = c(x, t) and have not assumed c is a constant. Substituting for ut and uxx givesus u(x,t+k)=u(x,t)−cku +1k2􏰠−cu +c(cu)􏰡+O(k3) x2txxx where everything on the right-hand side is evaluated at (x , t ). If we approximate the space derivative with second-order differences, we will have a second-order scheme in space and time: u(x,t+k)≈u(x,t)−ck 1 [u(x+h,t)−u(x−h,t)] 2h 12􏰛 1 􏰜 +2k −ct2h[u(x+h,t)−u(x−h,t)]+c(cux)x The difficulty with this scheme arises when c depends on space and we must evaluate the last term in the expression above. In the case in which c is a constant, we obtain c(cux)x =c2uxx ≈ 1 [u(x +h,t)−2u(x,t)+u(x −h,t)] 2h The Lax-Wendroff scheme becomes u(x,t+k)=u(x,t)− 1σ[u(x+h,t)−u(x−h,t)] 2 + 1cσ2 [u(x +h,t)−2u(x,t)+u(x −h,t)] 2 where σ = c(k/h). As does the Lax method, this method has numerical dissipation (lose of amplitude); however, it is relatively weak. Summary (1) We consider a model problem involving the following hyperbolic partial differential equation: ∂2u = ∂2u ∂t2 ∂x2 15.2 Hyperbolic Problems 603 Using finite differences, we approximate it by 1 [u(x +h,t)−2u(x,t)+u(x −h,t)] h2 = 1 [u(x,t +k)−2u(x,t)+u(x,t −k)] k2 The computational form is u(x, t + k) = ρu(x + h, t) + 2(1 − ρ)u(x, t) + ρu(x − h, t) − u(x, t − k) whereρ=k2/h2 <1.Att=0,weuse u(x,k)= 1ρ[f(x+h)+ f(x−h)]+(1−ρ)f(x) 2 604 Chapter 15 Partial Differential Equations Problems 15.2 a1. Whatisthesolutionoftheboundary-valueproblem utt =uxx u(x,0)=x(1−x) ut(x,0)=0 u(0,t)=u(1,t)=0 atthepointwherex =0.3andt =4? a2. Show that the function u(x,t) = f(x + at) + g(x − at) satisfies the wave equa- tionutt =a2uxx. a3. (Continuation) Using the idea in the preceding problem, solve this boundary-value problem: utt = uxx u(x,0) = F(x) ut(x,0) = G(x) u(0,t) = u(1,t) = 0 4. Show that the boundary-value problem utt =uxx u(x,0)=2f(x) ut(x,0)=2g(x) has the solution u(x,t)= f(x+t)+ f(x−t)+G(x+t)−G(x−t) where G is an antiderivative (i.e., indefinite integral) of g. Here, we assume that −∞ < x <∞andt􏶟0. 5. (Continuation)Solvetheprecedingproblemonafinitexinterval,forexample,0􏶞x􏶞1, adding boundary condition u(0,t) = u(1,t) = 0. In this case, f and g are defined only for 0 􏶞 x 􏶞 1. Computer Problems 15.2 a 1. 2. (Continuation) Write a program to compute the solution of u(x, t) at any given point Given f (x ) defined on [0, 1], write and test a function for calculating the extended f thatobeystheequations f(−x)=−f(x)and f(x+2)= f(x). (x , t ) for the boundary-value problem of Equation (2). 3. Compare the accuracy of the computed solution, using first Equation (6) and then Equation (7), in the computer program in the text. 4. Usetheprograminthetexttosolveboundary-valueProblem(2)with 1􏰔1􏶨􏶨1􏶨􏶨􏰕 1 1 f(x)= −􏶨􏶨x− 􏶨􏶨 h= k= 42 2 16 32 5. Modifythecodeinthetexttosolveboundary-valueProblem(2)whenut(x,0)=g(x). Hint: Equations (5) and (7) will be slightly different (a fact that affects only the initial loop in the program). 15.3 Elliptic Problems 605 6. (Continuation)Usetheprogramthatyouwrotefortheprecedingcomputerproblemto solve the following boundary-value problem: ⎧⎪ ⎪ ⎪ u t t = u x x ( 0 􏶞 x 􏶞 1 , t 􏶟 0 ) ⎪⎨ u(x,0)=sinπx ⎪ut(x,0)= 1sin2πx ⎪⎩ 4 u(0,t) = u(1,t) = 0 7. Modifythecodeinthetexttosolvethefollowingboundary-valueproblem: ⎧⎪ ⎪ ⎪ utt =uxx (−1􏶞x􏶞1,t􏶟0) ⎨ u(x,0)=|x|−1 ⎪ ⎪ ⎪⎩ u t ( x , 0 ) = 0 u(−1,t) = u(1,t) = 0 8. Modifythecodeinthetexttoavoidstorageofthe(vi)and(ui)arrays. 9. Simplifythecodeinthetextforthespecialcaseinwhichρ=1.Comparethenumerical solution at the same grid points for a problem in which ρ = 1 and ρ ≠ 1. 10. Use mathematical software such as in Matlab, Maple, or Mathematica to solve the wave Equation (2) and plot both the solution surface and the contour plot. 11. Use the symbolic manipulation capabilities in Maple or Mathematica to verify that Equation (3) is the general analytical solution of the wave equation. 15.3 Elliptic Problems One of the most important partial differential equations in mathematical physics and engi- neering is Laplace’s equation, which has the following form in two variables: ∇2u ≡ ∂2u + ∂2u = 0 ∂x2 ∂y2 Closely related to it is Poisson’s equation: ∇2u = g(x, y) These are examples of elliptic equations. (Refer to Problem 17.1.1 for the classification of equations.) The boundary conditions associated with elliptic equations generally differ from those for parabolic and hyperbolic equations. A model problem is considered here to illustrate the numerical procedures that are often used. Helmholtz Equation Model Problem Suppose that a function u = u(x, y) of two variables is the solution to a certain physical problem. This function is unknown but has some properties that, theoretically, determine it 606 Chapter 15 Partial Differential Equations uniquely. We assume that on a given region R in the xy-plane, 􏰎 ∇2u + f u = g u(x, y) known on the boundary of R (1) Here, f = f (x, y) and g = g(x, y) are given continuous functions defined in R. The boundary values could be given by a third function u(x, y) = q(x, y) on the perimeter of R. When f is a constant, this partial differential equation is called the Helmholtz equation. It arises in looking for oscillatory solutions of the wave equations. Finite-Difference Method As before, we find an approximate solution of such a problem by the finite-difference method. The first step is to select approximate formulas for the derivatives in our problem. In the present situation, we use the standard formula f′′(x)≈ 1[f(x+h)−2f(x)+ f(x−h)] (2) h2 derived in Section 4.3. If it is used on a function of two variables, we obtain the five-point formula approximation to Laplace’s equation: ∇2u≈ 1[u(x+h,y)+u(x−h,y)+u(x,y+h)+u(x,y−h)−4u(x,y)] (3) h2 This formula involves the five points displayed in Figure 15.11. The local error inherent in the five-point formula is h2 􏰖∂4u ∂4u 􏰗 −12 ∂x4(ξ,y)+∂y4(x,η) (4) and for this reason, Formula (3) is said to provide an approximation of order O(h2). In other words, if grids are used with smaller and smaller spacing, h → 0, then the error that is committed in replacing ∇2u by its finite-difference approximation goes to zero as rapidly as does h2. Equation (3) is called the five-point formula because it involves values of u at (x, y) and at the four nearest grid points. (x 􏶼 h, y) (x, y 􏶽 h) (x, y) (x, y 􏶼 h) (x 􏶽 h, y) FIGURE 15.11 Laplace’s equation: Five-point stencil It should be emphasized that when the differential equation in (1) is replaced by the finite-difference analog, we have changed the problem. Even if the analogous finite- difference problem is solved with complete precision, the solution is that of a problem that only simulates the original one. This simulation of one problem by another becomes better and better as h is made to decrease to zero, but the computing cost will inevitably increase. We should also note that other representations of the derivatives can be used. For example, the nine-point formula is ∇2u≈ 1 [4u(x+h,y)+4u(x−h,y)+4u(x,y+h)+4u(x,y−h) 6h2 + u(x + h, y + h) + u(x − h, y + h) + u(x + h, y − h) + u(x − h, y − h) − 20u(x, y)] (5) This formula is of order O(h2). In the special case that u is a harmonic function (which means it is a solution of Laplace’s equation), the nine-point formula is of order O(h6). For additional details, see Forsythe and Wasow [1960, pp. 194–195]. Hence, it is an extremely accurate approximation in using finite-difference methods and solving the Poisson equation ∇2u = g, with g a harmonic function. For more general problems, the nine-point Formula (5) has the same order error term as the five-point Formula (3) [namely, O(h2)] and would not be an improvement over it. If the mesh spacing is not regular (say, h1, h2, h3, and h4 are the left, bottom, right, and top spacing, respectively), then it is not difficult to show that at (x, y) the irregular five-point formula is ∇2u ≈ which is only of order h when h1 = αih for 0 < αi < 1. This formula is usually used near boundary points, as in Figure 15.12. If the mesh is small, however, the boundary points can be moved over slightly to avoid the use of (6). This perturbation of the region R (in most 15.3 Elliptic Problems 607 1 [h1u(x +h3,y)+h3u(x −h1,y)] 1h1h3(h1 +h3) 2 + 1 1h2h4(h2 +h4) 2􏰔11􏰕 [h2u(x, y + h4) + h4u(x, y − h2)] −2hh+hh u(x,y) (6) 13 24 h1 h4 h2 h3 FIGURE 15.12 Boundary points: Irregular mesh spacing h h 608 Chapter 15 Partial Differential Equations cases for small h) produces an error no greater than that introduced by using the irregular scheme (6). Returning to the model Problem (1), we cover the region R by mesh points xi =ih yj =jh (i,j􏶟0) (7) At this time, it is convenient to introduce an abbreviated notation: uij =u(xi,yi) fij = f(xi,yi) gij =g(xi,yj) (8) With it, the five-point formula takes on a simple form at the point (xi , y j ): (∇2u)ij ≈ 1 (ui+1, j +ui−1, j +ui, j+1 +ui, j−1 −4uij) (9) h2 If this approximation is made in the differential Equation (1), the result is (the reader should verify it) −u +􏰉4−h2 f 􏰈u =−h2g (10) i, j−1 ij ij ij −u The coefficients of this equation can be illustrated by a five-point star in which each point −u −u i+1, j i−1, j i, j+1 corresponds to the coefficient of u in the grid (see Figure 15.13). 􏶼1 FIGURE 15.13 Helmholtz equations: Five-point star 􏶼1 4􏶼hfij 􏶼1 􏶼1 FIGURE 15.14 Uniform grid spacing To be specific, we assume that the region R is a unit square and that the grid has spacing h = 1 (see Figure 15.14). We obtain a single linear equation of the form (10) for each of 4 5 4 3 2 1 12345 15.3 Elliptic Problems 609 the nine interior grid points. These nine equations are as follows: ⎧⎪−u21 − u01 − u12 − u10 + (4 − h2 f11)u11 = −h2g11 ⎪−u31 − u11 − u22 − u20 + (4 − h2 f21)u21 = −h2g21 ⎪−u41 − u21 − u32 − u30 + (4 − h2 f31)u31 = −h2g31 ⎪⎨−u22 −u02 −u13 −u11 +(4−h2 f12)u12 =−h2g12 ⎪−u32 −u12 −u23 −u21 +(4−h2 f22)u22 = −h2g22 ⎪−u42 − u22 − u33 − u31 + (4 − h2 f32)u32 = −h2g32 ⎪−u23 − u03 − u14 − u12 + (4 − h2 f13)u13 = −h2g13 ⎪⎩−u33 − u13 − u24 − u22 + (4 − h2 f23)u23 = −h2g23 −u43 −u23 −u34 −u32 +(4−h2 f33)u33 = −h2g33 This system of equations could be solved through Gaussian elimination, but let us examine them more closely. There are 45 coefficients. Since u is known at the boundary points, we move these 12 terms to the right-hand side, leaving only 33 nonzero entries out of 81 in our 9 × 9 system. The standard Gaussian elimination causes a great deal of fill-in, in the forward elimination phase—that is, zero entries are replaced by nonzero values. So we seek a method that retains the sparse structure of this system. To illustrate how sparse this system of equations is, we write it in matrix notation: Au = b Suppose that we order the unknowns from left to right and bottom to top: u = [u11, u21, u31, u12, u22, u32, u13, u23, u33]T This is called the natural ordering. Now the coefficient matrix is (11) (12) ⎡4−h2f11 −1 0 −1 0 0 ⎢−14−h2f21−1 0−1 0 ⎢0 −14−h2f31 0 0 −1 0 0 0 0 0 0 0⎤ 0⎥ ⎢ −1 0 A=⎢ 0 −1 ⎢0 0 ⎢⎣0 0 0 0 0 0 and the right-hand side is 04−h2f12−1 0 0⎥ 0⎥ 0⎥ 0 −1 −1 0 0−1 0 0 00 4−h2 f22 −1 −1 0 −1 4−h2f23 −1 + u43 Notice that if f (x, y) < 0, then A is a diagonally dominant matrix. −1 0 0 −1 −14−h2f32 0 0 0 04−h2f13−1 ⎡−h2g +u +u ⎤ 11 ⎢−h2g21 ⎢−h2g31 ⎢−h2g12 b= ⎢−h2g22 ⎢−h2g32 ⎢−h2g13 ⎣−h2g23 −h2g33 10 +u20 +u30 +u02 01 ⎥ 0 −1 0 −1 4−h2 f33 +u42 +u14 +u24 + u34 ⎥ ⎥ +u03⎥ ⎦ +u41⎥ ⎥ −1 ⎥ 0⎥⎦ 610 Chapter 15 Partial Differential Equations Gauss-Seidel Iterative Method Since the equations are similar in form, iterative methods are often used to solve such sparse systems. Solving for the diagonal unknown, we have from Equation (10) the Gauss-Seidel method or iteration given by 􏰒 (k) (k+1) (k) (k+1) 2 􏰓 ui+1, j +ui−1, j +ui, j+1 +ui, j−1 −h gij If we have approximate values of the unknowns at each grid point, this equation can be used to generate new values. We call u(k) the current values of the unknowns at iteration k and u(k+1) the value in the next iteration. Moreover, the new values are used in this equation as soon as they become available. The Gauss-Seidel method and other iterative methods are discussed in Section 8.2. The pseudocode for this method on a rectangle is as follows: (k+1) 1 uij = 4−h2 f ij procedure Seidel(ax,ay,nx,ny,h,itmax,(uij)) integer i, j,k,nx,ny,itmax realax,ay,x,y; realarray(uij)0:nx,0:ny fork =1toitmaxdo for j = 1 to ny − 1 do y ← ay + jh for i = 1 to nx − 1 do x←ax +ih v ← ui+1, j +ui−1, j +ui, j+1 +ui, j−1 uij ←(v−h2g(x,y))/(4−h2 f(x,y)) end for end for end for end procedure Seidel In using this procedure, one must decide on the number of iterative steps to be computed, itmax. The coordinates of the lower left-hand corner of the rectangle, (ax , ay ), and the step size h are specified. The number of x grid points is n x , and the number of y grid points is n y . Numerical Example and Pseudocode Let us illustrate this procedure on the boundary-value problem ⎧⎨∇2u − 1 u = 0 inside R (unit square) 55 pseudocode for the Gauss-Seidel procedure, starting with u = 1 and taking 20 iterations, is given next. Notice that only 81 words of storage are needed for the array in solving the 49 × 49 linear system iteratively. Here, h = 1 . 8 ⎩ 25 where q = cosh􏰉1 x􏰈 + cosh􏰉1 y􏰈. This problem has the known solution u = q. A driver u=q ontheboundaryofR (13) For this model problem, the accompanying functions are given next: 15.3 Elliptic Problems 611 program Elliptic integer i, j; real h, x, y; real array (ui j )0:nx ,0:ny integernx ←8,ny ←8,itmax←20 realax ←0,bx ←1,ay ←0,by ←1 h←(bx −ax)/nx for j = 0 to ny do y ← ay + jh u0j ←Bndy(ax,y) unx,j ←Bndy(bx,y) end for for i = 0 to nx do x←ax +ih ui0 ←Bndy(x,ay) ui,ny ←Bndy(x,by) end for for j = 1 to ny − 1 do y ← ay + jh for i = 1 to nx − 1 x←ax +ih uij ←Ustart(x,y) end for end for output 0,Norm((uij),nx,ny) call Seidel(ax,ay,nx,ny,h,itmax,(uij)) output itmax,Norm((uij),nx,ny) for j = 0 to ny do y ← ay + jh for i = 0 to nx do x←ax +ih uij ←|TrueSolution(x,y)−uij| end for end for output itmax,Norm((uij),nx,ny) end program Elliptic real function f (x, y) real x, y f ← −0.04 end function f real function Bndy(x, y) real x, y Bndy ← True Solution(x, y) end function Bndy real function g(x, y) real x, y g ← 0 end function g real function Ustart(x, y) real x, y Ustart ← 1 end function Ustart 612 Chapter 15 Partial Differential Equations real function True Solution(x, y) real x, y True Solution ← cosh(0.2x) + cosh(0.2y) end function True Solution real function Norm((uij),nx,ny) real array (ui j )0:nx ,0:ny t←0 for i = 1 to nx − 1 do for j = 1 to ny − 1 do t ← t + u2 ij Norm← t end function Norm end for end for √ After 75 iterations, the computed values at the 49 interior grid points are as follows: 2.0000 2.0003 2.0013 2.0028 2.0050 2.0078 2.0113 2.0154 2.0201 2.0003 2.0006 2.0016 2.0031 2.0053 2.0081 2.0116 2.0157 2.0204 2.0013 2.0016 2.0025 2.0041 2.0062 2.0091 2.0125 2.0166 2.0213 2.0028 2.0031 2.0041 2.0056 2.0078 2.0106 2.0141 2.0182 2.0229 2.0050 2.0053 2.0062 2.0078 2.0100 2.0128 2.0163 2.0204 2.0251 2.0078 2.0081 2.0091 2.0106 2.0128 2.0156 2.0191 2.0232 2.0279 2.0113 2.0116 2.0125 2.0141 2.0163 2.0191 2.0225 2.0266 2.0313 2.0154 2.0157 2.0166 2.0182 2.0204 2.0232 2.0266 2.0307 2.0354 2.0201 2.0204 2.0213 2.0229 2.0251 2.0279 2.0313 2.0354 2.0401 The Euclidean norm ||u||2 = 􏰍nx −1 􏰍ny −1 u2 of the difference between the computed 2 i=1 j=1 ij values and the known solution of the boundary-value problem (13) is approximately 0.47 × 10−4. This example is a good illustration of the fact that the numerical problem being solved is the system of linear Equations (11), which is a discrete approximation to the continuous- boundary-value Problem (13). When comparing the true solution of (13) with the computed solution of the system, remember the discretization error involved in making the approxi- mation. This error is O(h2). With h as large as h = 1 , most of the errors in the computed 8 solution are due to the discretization error! To obtain a better agreement between the dis- crete and continuous problems, select a much smaller mesh size. Of course, the resulting linear system will have a coefficient matrix that is extremely large and quite sparse. Iterative methods are ideal for solving such systems that arise from partial differential equations. For additional information, see the references listed at the end of this section. For a range of engineering and science applications, Matlab has a PDE Toolbox for the numerical solution of partial differential equations. It can accommodate two space variables and one time variable. After discretizing the equation over an unstructured mesh, it applies finite elements to solve it and offers a provision for visualizing the results. The first example is Poisson’s equation ∇2u = −1 in the unit circle with u = 0 on the boundary. A comparison of the finite-element solution is made with the exact solution. Finite-Element Methods The finite-element method has become one of the major strategies for solving partial dif- ferential equations. It provides an alternative to the finite-difference methods discussed up to now in this chapter. As an illustration, we develop a version of the finite-element method for Poisson’s equation ∇2u≡uxx +uyy =r where r is a constant function. The partial differential equation holds over a specified region R in a two-dimensional plane. Solving Poisson’s equation is equivalent to minimizing the expression 􏶩􏶩􏰖1􏰉2 2􏰈 􏰗 2 ux+uy +ru dxdy J(u)= This means that if the function u minimizes the expression above, then u obeys Poisson’s equation. Suppose the region is subdivided into triangles using approximations as necessary. The function u is approximated by a function φ that is a composite of plane triangular elements, each defined over a triangular piece of R. Then consider the substitute problem of minimizing 􏰃 􏰉 (e)􏰈 Je φ e where each term in the summation is evaluated over its own base triangle T as described below. (By accepting this theory on faith, you should be able to grasp the general idea of the finite-element method.) Assume that a base triangle has vertices (xi , yi ), (x j , y j ), and (xk , yk ). The solution surface above the triangle is approximated by a plane triangular element denoted φ(e)(x, y), where the superscript indicated this element. Let zi , z j , and zk be the distances up to the plane at the triangle corners called nodes. Let L(e) be one at node i and zero at nodes j and i k. Similarly, let L(e) be one at node j and zero at nodes i and k, and let L(e) be one at node k jk and zero at nodes i and j. As is shown in Figure 15.15, the area of the base triangle, denoted 􏶚e, is given by ⎡⎤ 1 1 xi yi 􏶚e = 2Det⎣1 xj yj ⎦ 1 xk yk =xjyk +xiyj +xkyi −xjyi −xiyk −xkyj 15.3 Elliptic Problems 613 R 614 Chapter 15 Partial Differential Equations FIGURE 15.15 Base triangle (xi, yi) (xk, yk) Consequently, we obtain 1 1xy ⎡⎤ L(e) = 􏶚−1Det⎣1 x y ⎦ i2ejj 1 xk yk (xj, yj) = 1􏶚−1[(x y − x y ) + (y − y )x + (x − x )y] 2ejkkjjkkj 1 −1􏰒(e) (e) (e) 􏰓 ≡ 2 􏶚e ai + bi x + ci y We have defined the coefficients a(e), b(e), and c(e). Similarly, we find iii and Finally, we obtain 1 1xy L(e) = 􏶚−1Det⎣1 x y ⎦ j2ekk 1 xi yi ⎡⎤ = 1􏶚−1[(x y − x y ) + (y − y )x + (x − x )y] 2ekiikkiik 1 −1􏰒(e) (e) (e) 􏰓 ≡2􏶚e aj +bj x+cj y ⎡⎤ =1􏶚−1[(xy −xy)+(y−y)x+(x −x)y] 2eijjiijji 1 −1􏰒 (e) (e) (e) 􏰓 ≡2􏶚e ak+bkx+cky φ(e) = L(e)z + L(e)z + L(e)z iijjkk 1 1xy L(e) = 􏶚−1Det⎣1 x y ⎦ k2eii 1 xj yj Wehave 􏶩􏶩􏰖􏰒 􏰓 􏰗 J 􏰉φ(e)􏰈 = 1 􏰉φ(e)􏰈2 +􏰉φ(e)􏰈2 +rφ(e) dx dy ≡ F(z ,z ,z ) e2xyijk T To solve the minimization problem, we set the appropriate derivatives to zero, which requires derivatives of the components. Notice that and (e) 1 −1􏰒 (e) (e) (e) 􏰓 φx =2􏶚e bi zi+bj zj+bk zk (e) 1 −1􏰒(e) (e) (e) 􏰓 φy =2􏶚e ci zi+cj zj+ck zk We carry out the differentiations ∂F/∂zi = = 􏶩􏶩􏰉(e) (e) (e) (e) (e)􏰈 φx φxzi +φy φyzi +rφzi dxdy 􏶩􏶩T􏰔 1 1 φ(e) 􏶚−1b(e) + φ(e) 􏰕 􏶚−1c(e) + r L(e) x2ei y2ei i dx dy 􏰓 (e) (e) =4􏶚e bi + ci zi+ bi bj +ci cj zj 15.3 Elliptic Problems 615 T􏰖􏰔􏰒 􏰓 􏰒 􏰓􏰕 􏰒 1 −1 (e) 2 (e) 2 (e) (e) 􏰒 (e) (e) (e) (e)􏰓 􏰗 1 +bibk+cick zk+r3􏶚e Here, the integrations are straightforward by elementary calculus. Moreover, it can be shown that where􏶚e istheareaofeachtriangleT.Similarresultsareobtainedfor∂F/∂zj and∂F/∂zk. Consequently, we set ⎡∂F/∂z ⎤ ⎡0⎤ i ⎢⎣∂F/∂zj ⎥⎦=⎣0⎦ 0 b(e)b(e) + c(e)c(e) b(e)b(e) + c(e)c(e) ⎤ ⎡ z ⎤ ⎡ 1 ⎤ 􏶩􏶩􏶩􏶩􏶩􏶩1 L(e)dxdy= L(e)dxdy= L(e)dxdy= 􏶚 ijk3e TTT ∂F/∂zk ⎢iiijijikik⎥1 and we obtain ⎡ 􏰉b(e)􏰈2 + 􏰉c(e)􏰈2 ⎢ b ( e ) b ( e ) + c ( e ) c ( e ) 􏰉 b ( e ) 􏰈 2 + 􏰉 c ( e ) 􏰈 2 b ( e ) b ( e ) + c ( e ) c ( e ) ⎥ ⎢⎣ z 2 ⎥⎦ = − 4 r 􏶚 2 ⎢⎣ 1 ⎥⎦ ⎣ijijjjjkjk⎦3e b(e)b(e) + c(e)c(e) b(e)b(e) + c(e)c(e) 􏰉b(e)􏰈2 + 􏰉c(e)􏰈2 z3 ikikjkjkk k 1 EXAMPLE 1 This matrix equation contains all the ingredients we need to assemble the partial derivatives. In a particular application, we need to do the proper assembling. For each element φ(e), the ac- tive nodes i, j, and k are those that contribute nonzero values. These contributions are re- corded for derivatives relative to the corresponding variables among the zi , z j , zk , and so on. Apply the finite-element method to solve Poisson’s equation u x x + u y y = 4 over the unit square with the triangularizations shown in Figure 15.16 and using boundary values corre- sponding to the exact solution u(x, y) = x2 + y2. y 4 23x FIGURE 15.16 1 (e 􏶾 2) (e 􏶾 1) Triangularization 616 Chapter 15 Solution Partial Differential Equations By symmetry, we need to consider only the bottom right-hand part of the square, which has been split into two triangles. The input ingredients are nodes 1 to 4, where the coordinates (x,y) are as follows: node 1: 􏰉1,1􏰈, node 2: (0, 0), node 3: (1, 0), and node 4: (1, 1). 22 The elements are two triangles with node numbers indicated: e = 1: 1, 2, 3 and e = 2: 1, 3, 4. The astute reader will notice that the z coordinates need to be determined only for node 1, since they are boundary values for nodes 2, 3, 4! However, we will ignore this fact for the moment to illustrate the assembly process in the finite-element method. Notice that the areas of the triangular elements are 􏶚1 = 􏶚2 = 1 and r = 4. First, we compute the 4 a(e), b(e), c(e) coefficients from this basic information. In the following table, each column corresponds to a node (i, j, k): e=1 e=2 a(e) 0 1 0 1 0−1 22 b(e) 0−1 1 −1 1 1 22 22 c(e) 1−1−1 0−1 1 22 22 One can verify that the columns do produce the desired L(e), L(e), and L(e) functions. For ijk example, the first column gives L(1) = 1􏶚−1[0+0 · x +1 · y] = 2y. At node 1, this gives i21 the value of 1, while at nodes 2 and 3, it gives the value 0. Similarly, the other columns produce the desired results. Next, we obtain the matrix equation for element e = 1: ⎡ 1 1⎤⎡⎤⎡1⎤ ⎢1−2 −2⎥z1 −3 ⎢ − 1 1 0 ⎥ ⎣ z 2 ⎦ = ⎢⎣ − 1 ⎥⎦ ⎣22⎦3 −1 0 1 z3 −1 223 and the matrix equation for element e = 2: ⎡ 1 1⎤⎡⎤⎡1⎤ ⎢1−2 −2⎥z1 −3 ⎢ − 1 1 0 ⎥ ⎣ z 3 ⎦ = ⎢⎣ − 1 ⎥⎦ ⎣22⎦3 −1 0−1 z4 −1 223 Then we assemble the two matrices, obtaining ⎡ 1 ⎢ 2 −2 1⎤⎡ ⎤ ⎡−2⎤ −1 −2⎥ z1 ⎢ 3⎥ ⎢−1 1 ⎢2 2 0⎥⎢z ⎥ ⎢−1 ⎥ ⎥⎢2⎥=⎢3⎥ 0 ⎥⎦ ⎣ z 3 ⎦ ⎢⎣ − 2 ⎥⎦ 0 1 Now that we have illustrated the process of assembling the elements, we can quickly find the solution using the fact that z2 = 0, z3 = 1, and z4 = 2, since they are boundary values. Using these values in the last matrix equation above, we immediately find that z1 = 2 . This 3 ⎢⎣ − 1 0 −100−1z 3 2 24−1 3 is a rough approximation, since the true value is 1 . Remember that u(x, y) = x2 + y2 is 2 the exact solution. ■ We can obtain more accurate approximations by adding more elements and writing a com- puter program to handle the computations. (See Computer Problem 15.3.15.) For additional details, see Scheid [1990] and Sauer [2006]. More on Finite Elements At first, we take a very general approach to this topic, supposing that we have a linear transformation A and want to solve the equation Au = b for u, when b is given. This obviously includes the case when A is an m × n matrix and b is a vector of m components. But there are many complicated problems that fit this same mold. For example, A can be a linear differential operator, and we may wish to solve a two-point boundary-value problem involving it, such as 􏰎 Here, A operates on functions and is defined by the equation Au = u′′ + 2u. Another example of great importance is the model problem Equation (1). In this case, A would be the Laplacian differential operator. This problem is discussed in Chapter 17 as well. The basic strategy of the finite-element method for solving the equation Au = b is to selectbasicfunctionsv1,v2,...,vn andtrytosolvetheequationwithalinearcombination of these basic functions. Since A is assumed to be a linear transformation, we obtain u′′(t) + 2u(t) = t2 (0 􏶞 t 􏶞 1) u(0) = u(1) = 0 􏰃n Au=A j=1 cjvj = 􏰃n j=1 􏰉􏰈 cj Avj =b Now the unknowns in the problem are the coefficients c j . Typically, the equation just dis- played is inconsistent because b is not in the linear span of the set of functions { Av1, Av2, . . . , Avn}. In this case, one must compromise and accept an approximate solution to the set of equations. Many different tactics can be used to arrive at an approximate solution to the problem. For example, a least-squares approach can be used if the linear space involved has aninnerproduct,⟨·,·⟩.Thecoefficientscj wouldthenbechosensothattheorthogonality condition was fulfilled; that is, 􏰃n cj Avj − b ⊥ This leads to the normal equations Span{v1,v2,...,vn} j=1 􏰃n ⟨Avj,vi⟩cj =⟨b,vi⟩ (1􏶞i􏶞n) j=1 These equations for the unknown coefficients cj are also known (in this context) as the Galerkin equations. They form a system of n linear equations in n unknowns. We shall illustrate this process with a two-point boundary-value problem involving a second-order ordinary differential equation: 􏰎 The finite element method usually uses local functions as the basic functions in the previous discussion. This means that each basic function should be zero except on a short interval. B splines have this property and are therefore often used in the finite-element method. In the u′′(t) + g(t)u(t) = f (t) u(0) = a u(1) = b 15.3 Elliptic Problems 617 618 Chapter 15 Partial Differential Equations present problem, we shall want to use B splines having two continuous derivatives because the operator A will be defined by Au = u′′ + gu Hence,cubicsplineswouldsuggestthemselves.Defineknotsti =ih,wherehisachosen step size. (Its reciprocal should be an integer in this example.) Let B3j be the cubic B splines corresponding to the given knots. This is an infinite list of B splines, as discussed in Chapter 9. All but a finite number are zero on the interval [0, 1]. The ones that are not identically zero on the interval [0, 1] can be relabeled as v1, v2, ..., vn. These are our test functions. Proceeding as before, we arrive at a set of n linear equations in n unknowns. The details require one to find the functions Av j by using the B spline formulas in Chapter 9. This is tedious and not very instructive. Similar considerations can be applied to Laplace’s equation on a given domain. To illustrate, we take the domain to be a square of side 2, where 0 􏶞 x , y 􏶞 2. On the boundary of the square, we require u(x, y) = sin(xy). Such a problem is called a Dirichlet problem. For base functions, we use functions vj that already satisfy the homogeneous part of the problem. That is, we want each vj to satisfy Laplace’s equation inside the square domain. Functions that satisfy Laplace’s equation are said to be harmonic. We can exploit the fact that the real and imaginary parts of an analytic function are harmonic. Thus, if we set z = x + iy and compute zk, we will be able to extract harmonic functions that are polynomials. Here are a few harmonic polynomials, v j for 0 􏶞 j 􏶞 6: z = 1 z = x + iy z2 =(x+iy)2 z3 =(x+iy)3 v0(x, y) = 1 v1(x, y) = x v3(x,y)=x2 −y2 v5(x,y)=x3 −3xy2 v2(x, y) = y v4(x,y)=2xy v6(x,y)=3x2y−y3 Using these seven functions, we form u = 􏰍6j=0 cjvj. This satisfies Laplace’s equation, and we can concentrate on making u close to the specified boundary value x3 − y2 on the perimeter of the square. There are many ways to proceed, and we choose first to use a method called collocation. In this process, we select a number of points on the boundary and writedownanequationateachpointthatsaysthevalueof􏰍6j=0cjvj equalstheprescribed value. If the number of points equals the number of basic functions, we have the classical collocation method. Here, we took eight points, whereas there are only seven functions and seven coefficients. Hence, we ask for a least-squares solution. We took the so-called collocation points to be (0, 2), (1, 2), (2, 2), (2, 1), (2, 0), (1, 0), (0, 0) and (0, 1). This led to the following system of eight equations: ⎡ ⎤⎡⎤⎡⎤ 2 −4 0 0 2 −3 4 −11 2 0 8 −16 1 3 4 2 0 4 0 8 0 1 0 1 0 0 0 0 1 −1 0 0 −1 c7 0 The least-squares solution is c = [0.3219, −0.8585, −0.8585, 0, 1.1931, 0.2146, −0.2146]T 1 0 ⎢1 1 ⎢1 2 ⎢1 2 ⎢1 2 ⎢1 1 ⎣1 0 1 0 −8c0 0 −2⎥⎢c1⎥ ⎢sin(2)⎥ 16⎥⎢c2⎥ ⎢sin(4)⎥ 11⎥⎢c3 ⎥ = ⎢sin(2)⎥ 0⎥⎢c4⎥ ⎢ 0 ⎥ a c-vector having components 0⎥⎢c5⎥ ⎢ 0⎦⎣c6⎦ ⎣ 0 ⎥ 0 ⎦ The residual function is 􏰍6j=0 cjvj − b, where bi(x, y) = sin(xy). Its absolute value is 0.3219 at each of the eight collocation points. To improve the accuracy, one must employ more basic functions and more collocation points. Another technique that is often used in the finite-element method is the replacement of a differential equation by an optimization problem. This can be illustrated by a two-point boundary-value problem such a􏰎s (hu′)′ − gu = f u(a) = α u(b) = β Here, u is the unknown function, while h, g, and f are prescribed functions, all defined on the interval [a,b]. This problem is called a Sturm-Liouville problem. There is an accompanying functional, defined by 􏶩b􏰠 􏰡 (u′)2h + u2g + 2u f dx a The functional and the two-point boundary-value problem are related by several theorems. One of these states roughly that if we find the function u that minimizes the functional 􏶛(u) subject to the side conditions u(a) = α and u(b) = β, then we will have the solution of the boundary-value problem. It is possible to exploit the fact that 􏶛(u) is defined as long as u has a derivative, whereas in the differential equation, we require a function possessing two derivatives. In fact, for the functional, we require only that u be piecewise differentiable, a property that spline functions of degrees 0 and 1 possess. These ideas extend to functions of two or more variables and allow one to use spline functions of low degree in two or more variables to approximate the solution to a differential equation. These are the principal features of the finite element method. For the mathematical theory of finite-element methods, see the books by Brenner and Scott [2002], Strang [2006], and others. Summary (1) We study a model problem involving the following elliptic partial differential equation ∇2u + f u = g over a region, with the value of u given on the boundary. The first term involves the Laplace operator ∇2, which is ∇2u ≡ ∂2u + ∂2u ∂x2 ∂y2 By placing a grid over the region with uniform spacing h in both directions the Laplacian term can be approximated by using the five-point finite differences ∇2u≈ 1[u(x+h,y)+u(x−h,y)+u(x,y+h)+u(x,y−h)−4u(x,y)] h2 At each interior grid point, we write ui j = u(xi , yj ) = u(ih, jh), and we obtain the following equation for our model problem: 􏶛(u) = 15.3 Elliptic Problems 619 −u −u i+1, j i−1, j −u i, j+1 −u +􏰉4−h2 f 􏰈u =−h2g i, j−1 ij ij ij Usually, the resulting linear system of equations is large and sparse, and iterative methods can be used to solve it. For example, the Gauss-Seidel iterative method for our linear 620 Chapter 15 Partial Differential Equations system is (k+1) 1 uij = 4−h2 f ij 􏰒 (k) (k+1) (k) (k+1) 2 􏰓 ui+1, j +ui−1, j +ui, j+1 +ui, j−1 −h gij The grid points can be ordered in different ways, such as the natural ordering or the red-black ordering, which affects the rate of convergence of the iterative procedures. (2) The distinguishing feature of the finite-element method is that we solve an equation Ax = b approximately by setting u = 􏰍n cjvj, where v1,v2,...,vn are chosen by the j=1 􏰍n user. The unknown coefficients c j are computed so that j =1 c j Av j is as close as possible to b. Typically, in partial differential equations, the functions v j will be multidimensional spline functions. Additional References For additional study and reading, see Ames [1992], Evans [2000], Forsythe and Wasow [1960], Gockenbach [2002], Mattheij, Rienstra and Boonkkamp [2005], Ortega and Voigt [1985], Rice and Boisvert [1984], Smith [1965], Street [1973], Varga [1962, 2002], Wachspress [1966], Young [1971], and Young and Gregory [1972]. 1. Establishtheformulafortheerrorinthe a. five-pointformula,Equation(3). b. nine-point formula, Equation (5). 2. Establishtheirregularfive-pointFormula(6)anditserrorterm. 3. WritethematricesthatoccurinEquation(11)whentheunknownsareorderedaccording to the vector u = [u11, u31, u22, u13, u33, u21, u12, u32, u23]T . This is known as red- black or checkerboard ordering. 4. a. Verify Equation (10). b. VerifythatthesolutionofEquation(13)isasgiveninthetext. a5. Considertheproblemofsolvingthepartialdifferentialequation 20uxx−30uyy+ 5 ux+1uy=69 x+y y in a region R with u prescribed on the boundary. Derive a five-point finite difference equation of order O(h2) that corresponds to this equation at some interior point (xi , yj ). a6. Solvethisboundary-valueproblemtoestimateu􏰉1,1􏰈andu􏰉0,1􏰈: 􏰎 ∇2u = 0 (x, y) ∈ R u = x (x, y) ∈ ∂ R 222 Problems 15.3 15.3 Elliptic Problems 621 The region R with boundary ∂ R is shown in the figure (the arc is circular). Use h = 1 . 2 Note: This problem (and many others in this text) can be stated in physical terms also. For example, in this problem, we are finding the steady-state temperature in a beam of cross section R if the surface of the beam is held at temperature u(x, y) = x. y 1 􏶼12 0 12 1 a7. Considertheboundary-valueproblem 􏰎 for the region in the unit square with h = 1 in the figure below. Here, ∂ R is the 􏶮232􏶯 boundaryofR,∂R2 = (x,y)∈∂R:3􏶞x<1, 3􏶞y<1 ,and∂R1 =∂R−∂R2. At the mesh points, determine the system of linear equations that yields an approximate value for u(x, y). Write the system in the form Au = b. x ∇2u = 9(x2 + y2) (x, y) ∈ R u = x − y (x, y) ∈ ∂ R1 y 1 2 3 1 3 12 33 x 1 8. Determine the linear system to be solved if the nine-point Formula (5) is used as the approximation in the problem of Equation (1). Notice the pattern in the coefficient matrix with both the five-point and nine-point formulas when unknowns in each row are grouped together. (Draw dotted lines through A to form 3 × 3 submatrices.) 9. In Equation (11), show that A is diagonally dominant when f (x, y) 􏶞 0. 10. Whatisthelinearsystemifanalternativenine-pointformula ∇2u≈ 1 [16u(x+h,y)+16u(x−h,y)+16u(x,y+h) 12h2 +16u(x, y − h) − u(x + 2h, y) − u(x − 2h, y) −u(x, y + 2h) − u(x, y − 2h) − 60u(x, y)] 622 Chapter 15 Partial Differential Equations is used? What are the advantages and disadvantages of using it? Hint: It has accuracy O(h4). 11. (Multiple choice) What is Laplace’s equation in three variables? a. u−x+uy +uz =0 b. uxx +uyy =0 c. uxx +uyy +yzz =0 d. uxx +uyy =yut e. Noneofthese. 12. (Multiplechoice)Whichoftheseisnotaharmonicfunctionof(x,y)? a. x2−y2 b.2xy c.x3y−xy3 d. x3 − xy3 e. None of these. 13. (Multiplechoice)InsolvingtheDirichletproblemontheunitsquare,where0 0). If F(b′) 􏶟 F(b′ + δ),
then x∗ ∈ [b′, b]. Taking the midpoint of this interval, we obtain 􏶤x = 1 (b′ + b) as our 2
estimate of x∗ and find that |􏶤x − x∗| 􏶞 1 (b − a). On the other hand, if F(b′) < F(b′ + δ), 6 then x∗ ∈ [a′,b′ + δ]. Again we take the midpoint, 􏶤x = 1(a′ + b′ + δ), and find that 2 |􏶤x − x∗| 􏶞 1 (b − a) + 1 δ. So if we ignore the small quantity δ/2, our accuracy is 1 (b − a) 626 in using three evaluations of F. By continuing the search pattern outlined, we find an estimate 􏶤x of x∗ with only n evaluations of F and with an error not exceeding 1􏰔b − a􏰕 2 λn where λn is the (n + 1)st member of the Fibonacci sequence: 􏰎 (1) (2) λ1=1, λ2=1 λk =λk−1 +λk−2 (k􏶟3) For example, elements λ1 through λ8 are 1, 1, 2, 3, 5, 8, 13, and 21. In the Fibonacci search algorithm, we initially determine the number of steps N for a desired accuracy ε > δ by selecting N to be the subscript of the smallest Fibonacci number greater than 1 (b − a)/ε. We define a sequence of intervals, starting with the given
2
interval [a,b] of length l = b − a, and, for k = N, N − 1,…,3, use these formulas

630
Chapter 16
Minimization of Functions
for updating:
Atthestepk =2,weset
22 􏰎a = a′ if F(a′)􏶟 F(b′)
􏰔λ 􏰕 􏶚 = k−2
λk a′ =a+􏶚
b = b′ if F(a′) < F(b′) a′ = 1(a+b)−2δ b′ = 1(a+b)+2δ b′ =b−􏶚 􏰎a = a′ if F(a′)􏶟 F(b′) (b − a) (3) b = b′ if F(a′) < F(b′) and we have the final interval [a, b], from which we compute 􏶤x = 1 (a + b). This algorithm FIGURE 16.7 Fibonacci search algorithm: Verify using a typical situation l 􏷄 l′ a a′ b′ b To verify the algorithm, consider the situation shown in Figure 16.7. Since λk = 2 requires only one function evaluation per step after the initial step. λk−1 + λk−2, we have 􏰔λ 􏰕 􏰔λ 􏰕 l′ =l−􏶚=l− k−2 l= k−1 l (4) λk λk and the length of the interval of uncertainty has been reduced by the factor (λk−1/λk). The next step yields 􏰔λ 􏰕 􏶚′ = k−3 l′ (5) λk −1 and 􏶚′ is actually the distance between a′ and b′. Therefore, one of the preceding points at which the function was evaluated is at one end or the other of [a, b]; that is, 􏰔λ−2λ 􏰕 k−2 l 􏰔λ −λ 􏰕 􏰔λ 􏰕 b′ − a′ = l = 2􏶚 = k = k−1 􏰔λ 􏰕 λk k−2 l= k−3 l λk = k−3 l′=􏶚′ λk −1 λk by Equations (2), (4), and (5). It is clear by Equation (4) that after N − 1 function evaluations, the next-to-last interval has length (1/λN) times the length of the initial interval [a,b]. So the final interval is (b − a)(1/λN ) wide, and the maximum error (1) is established. The final step is similar to that outlined, and F is evaluated at a point 2δ away from the midpoint of the next-to-last interval. Finally, set 􏶤x = 1 (b + a) from the last interval [a, b]. 2 One disadvantage of the Fibonacci search is that the algorithm is rather complicated. Also, the desired precision must be given in advance, and the number of steps to be computed for this precision must be determined before beginning the computation. Thus, the initial evaluation points for the function F depend on N , the number of steps. Golden Section Search Algorithm A similar algorithm that is free of these drawbacks is described next. It has been termed the golden section search because it depends on a ratio ρ known to the early Greeks as the golden section ratio: 1􏰒 √􏰓 ρ=2 1+ 5 ≈1.6180339887 16.1 One-Variable Case 631 The mathematical history of this number can be found in Roger [1998], and ρ satisfies the equation ρ2 = ρ + 1, which has roots 1􏰉1 + √5􏰈 ≈ 1.61803... and 1􏰉1 − √5􏰈 ≈ 22 −0.61803. . . . In each step of this iterative algorithm, an interval [a, b] is available from the previous work. It is an interval that is known to contain the minimum point x∗, and our objective is to replace it by a smaller interval that is also known to contain x∗. In each step, two values of F are needed: 􏰎 x = a + r(b − a) u = F(x) y = a + r2(b − a) v = F(y) where r = 1/ρ and r2 + r = 1, which has roots 1􏰉−1 + √5􏰈 ≈ 0.61803... and 1􏰉√􏰈 2 2 −1 − 5 ≈ −1.61803.... There are two cases to consider: Either u > v or u 􏶞 v. Let us take the first. Figure 16.8 depicts this situation. Since F is assumed continuous and unimodal, the minimum of F must be in the interval [a, x]. This interval is the input interval at the beginning of the next step. Observe now that within the interval [a, x], one evaluation of F is already available, namely, at y. Also note that
a + r(x − a) = y
u
v
(6)
FIGURE 16.8
Golden section search
r(b 􏶼 a)
r(b 􏶼 a)
algorithm: u>v a
y x* x
b

632
Chapter 16
Minimization of Functions
because x −a = r(b−a). In the next step, therefore, y will play the role of x, and we shall need the value of F at the point a + r2(x − a).
So what must be done in this step is to carry out the following replacements in order:
b←x
x←y
u←v
y ← a + r2(b − a) v ← F(y)
The other case is similar. If u 􏶞 v, the picture might be as in Figure 16.9. In this case, the minimum point must lie in [y, b]. Within this interval, one value of F is available, namely, at x. Observe that
y + r2(b − y) = x
(See Problem 16.1.9.) Thus, x should now be given the role of y, and the value of F is to
be computed at y + r(b − y). The following ordered replacements accomplish this:
a←y
y←x
v←u
x ← a + r(b − a) u ← F(x)
Problems 16.1.10 and 16.1.11 hint at a shortcoming of this procedure: It is quite slow. Slowness in this context refers to the large number of function evaluations that are needed to achieve reasonable precision. It can be surmised that this slowness is attributable to the extreme generality of the algorithm. No advantage has been taken of any smoothness that the function F may possess.
If [a, b] is the starting interval in the search for a minimum of F, then at the beginning,
with one evaluation of F , we can be sure only that the minimum point, x ∗ , is in an interval of
width b − a. In the golden section search, the corresponding lengths in successive steps are
r(b − a) for two evaluations of F, r2(b − a) for three evaluations of F, and so on. After n
steps, the minimum point has been pinned down to an interval of length rn−1(b − a). How
does this compare with the Fibonacci search algorithm using n evaluations? The correspond-
ing width of interval, at the last step of this algorithm, is λ−1(b − a). Now, the Fibonacci n
algorithm should be better, because it is designed to do as well as possible with a prescribed
v
u
FIGURE 16.9
Golden section search
r(b 􏶼 a)
r(b 􏶼 a)
algorithm: u 􏶞 v a
y x x*
b

16.1 One-Variable Case 633 number of steps. So we expect the ratio rn−1/λ−1 to be greater than 1. But it approaches
n
1.17 as n → ∞. (See Problem 16.1.8.) Thus, one may conclude that the extra complexity
of the Fibonacci algorithm, together with the disadvantage of having the algorithm itself depend on the number of evaluations permitted, mitigates against its use in general.
In the golden section search algorithm, how is the correct ratio r determined? Remember that when we pass from one interval to the next in the algorithm, one of the points x or y is to be retained in the next step. Here, we present first a sketch of the first interval in which we let x = a + r(b − a) and y = b + r(a − b). It is followed by a sketch of the next interval.
ayxb
azyx􏶾b
In this new interval, the same ratios should hold, so we have y = a + r(x − a). Since x − a = r(b − a), we can write y = a + r[r(b − a)]. Setting the two formulas for y equal to each other gives us
whence
Dividing by (a − b) gives
a + r2(b − a) = b + r(a − b)
a − b + r2(b − a) = r(a − b)
r2 + r − 1 = 0
The roots of this quadratic equation are as given previously.
Quadratic Interpolation Algorithm
Suppose that F is represented by a Taylor series in the vicinity of the point x∗. Then F(x)= F(x∗)+(x −x∗)F′(x∗)+ 1(x −x∗)2F′′(x∗)+···
2
Since x∗ is a minimum point of F, we have F′(x∗) = 0. Thus,
F(x) ≈ F(x∗) + 1(x − x∗)2 F′′(x∗) 2
This tells us that, in the neighborhood of x∗, F(x) is approximated by a quadratic function whose minimum is also at x ∗ . Since we do not know x ∗ and do not want to involve derivatives in our algorithms, a natural stratagem is to interpolate F by a quadratic polynomial. Any three values (xi , F (xi )), i = 1, 2, 3, can be used for this purpose. The minimum point of the resulting quadratic function may be a better approximation to x∗ than is x1, x2, or x3. Writing an algorithm that carries out this idea iteratively is not trivial, and many unpleasant cases must be handled. What should be done if the quadratic interpolant has a maximum instead of a minimum, for example? There is also the possibility that F′′(x∗) = 0, in which case higher-order terms of the Taylor series determine the nature of F near x∗.
Here is the outline of an algorithm for this procedure. At the beginning, we have a function F whose minimum is sought. Two starting points x and y are given, as well as two

634 Chapter 16
Minimization of Functions
control numbers δ and ε. Computing begins by evaluating the two numbers
Now let
In either case, the number
􏰎
z=
2x − y 2y − x
if u < v if u 􏶟 v 􏰎 u = F(x) v = F(y) w = F(z) is to be computed. At this stage, we have three points x, y, and z together with corresponding function values u, v, and w. In the main iteration step of the algorithm, one of these points and its accompanying function value are replaced by a new point and new function value. The process is repeated until a success or failure is reached. In the main calculation, a quadratic polynomial q is determined to interpolate F at the three current points x, y, and z. The formulas are discussed below. Next, the point t where q′(t) = 0 is determined. Under ideal circumstances, t is a minimum point of q and an approximate minimum point of F. So one of the x, y, or z should be replaced by t. We are interested in examining q′′(t) to determine the shape of the curve q near t. For the complete description of this algorithm, the formulas for t and q′′(t) must be given. They are obtained as follows: ⎧⎪ ⎪ ⎪ a = v − u ⎪ y−x ⎪ b = w − v ⎪⎨ z−y ⎪ c = b − a ⎪ z−x ⎪ 1􏰛 a􏰜 ⎪t=2 x+y−c ⎪⎩q′′(t) = 2c Their derivation is outlined in Problem 16.1.12. The solution case occurs if q′′(t) > 0 and max{|t − x|,|t − y|,|t − z|} < ε The condition q′′(t) > 0 indicates, of course, that q′ is increasing in the vicinity of t, so t is indeed a minimum point of q. The second condition indicates that this estimate, t, of the minimum point of F is within distance ε of each of the three points x, y, and z. In this case, t is accepted as a solution.
The usual case occurs if
q′′(t)>0 and δ􏶟 max{|t−x|,|t−y|,|t−z|}􏶟ε
These inequalities indicate that t is a minimum point of q but not near enough to the three initial points to be accepted as a solution. Also, t is not farther than δ units from each of x, y, and z and can thus be accepted as a reasonable new point. The old point that has the greatest function value is now replaced by t and its function value by F(t).

FIGURE 16.10
Taylor series algorithm: First bad case
t z􏶽􏷎sign(t􏶼z) z y x The second bad case occurs if
q′′(t) < 0 thus indicating that t is a maximum point of q. In this case, identify the greatest and the least among u, v, and w. Suppose, for example, that u 􏶟 v 􏶟 w. Then replace x by z + δ sign(z − x). An example is shown in Figure 16.11. The first bad case occurs if q′′(t) > 0 and max{|t − x|,|t − y|,|t − z|} > δ
Here, t is a minimum point of q but is so remote that there is some danger in using it as a new point. We identify one of the original three points that is farthest from t, for example, x, and also we identify the point closest to t, say z. Then we replace x by z + δ sign(t − z) and u by F(x). Figure 16.10 shows this case. The curve is the graph of q.
q
􏷈
16.1 One-Variable Case 635
u
v
v
u
􏷈
z􏶽􏷎sign(z􏶼x) z
q
FIGURE 16.11
Taylor series algorithm: Second bad case
Summary
y x t
We consider the problem of finding the local minimum of a unimodal function of a one-variable. Algorithms discussed are Fibonacci search, golden section search, and quadratic interpolation.
a1. For the function F(x1, x2, x3) = x12 + 3×2 + 2×32 − 4×1 − 6×2 + 8×3, find the uncon- strained minimum point. Then find the constrained minimum over the set K defined by inequalities x1 􏶞 0, x2 􏶞 0, and x3 􏶞 0. Next, solve the same problem when K is defined byx1􏶞2,×2􏶞0,andx3􏶞 −2.
Problems 16.1

636 Chapter 16
Minimization of Functions
a2. 3.
a4. 5.
6.
a 7.
8.
a9. a10.
a 11. 12.
ForthefunctionF(x,y)=13×2+13y2−10xy−18x−18y,findtheunconstrained minimum. Hint: Try substituting x = u +v and y = u −v.
If F is unimodal and continuous on the interval [a, b], how many local maxima may F have on [a, b]?
FortheFibonaccisearchalgorithm,writeexpressionsfor􏶤xinthetwocasesn=2,3. Carry out four steps of the Fibonacci search algorithm using ε = 1 to determine the
following:
aa. MinimumofF(x)=x2−6x+2on[0,10]
b. MinimumofF(x)=2×3 −9×2 +12x+2on[0,3] c. MaximumofF(x)=2×3−9×2+12xon[0,2]
Let F be a continuous unimodal function defined on the interval [a, b]. Suppose that thevaluesofFareknownatnpoints,namely,a=t1 F(b′);b=b′,a′ = 2
1(a+a′),andb′ =aifF(a′)u:
⎧⎪ ⎪ ⎪ ⎪ b ← b ′
⎪b′←a′
⎪ ⎪ ⎪⎨ v ← u
􏰔λ 􏰕
⎪􏶚← k−2 (b−a) ⎪ λk
⎪a′←a+􏶚 ⎩u←F(a′)
􏰔λ 􏰕
􏶚 = N−2 (b − a)
λN a′ =a+􏶚
b′ =b−􏶚 u = F(a′) v = F(b′)
Then loop on k from N − 1 downward to 3, updating as follows:
5. (Berman algorithm) Suppose that F is unimodal on [a, b]. Then if x1 and x2 are any twopointssuchthata􏶞x1 F(x2) F(x1) = F(x2) F(x1) < F(x2) implies implies implies x∗ ∈ (x1, b] x∗ ∈ [x1, x2] x∗ ∈ [a, x2) So by evaluating F at x1 and x2 and comparing function values, we are able to reduce the size of the interval that is known to contain x∗. The simplest approach is to start atthemidpointx0 =1(a+b)andifFis,say,decreasingforx>x0,wetestF 2
at x0 +ih, i = 1,2,…,q, with h = (b−a)/2q, until we find a point x1 from which F begins to increase again (or until we reach b). Then we repeat this procedure starting at x1 and using a smaller step length h/q. Here, q is the maximal number of evaluations at each step, say, 4. Write a subroutine to perform the Berman algorithm and test it for evaluating the approximate minimization of one-dimensional functions. Note: The total number of evaluations of F needed for executing this algorithm up to some iterative step k depends on the location of x∗. If, for example, x∗ = b, then clearly, we need q evaluations at each iteration and hence kq evaluations. This number will decrease the closer x∗ is to x0, and it can be shown that with q = 4, the expected number of evaluations is three per step. It is interesting to compare the efficiency of the Berman algorithm (q = 4) with that of the Fibonacci search algorithm. The expected number of evaluations per step is three, and the uncertainty interval decreases by a factor 4−1/3 ≈ 0.63 per evaluation. In comparison, the Fibonacci search algorithm has

16.2 Multivariate Case 639 a reduction factor of 1 􏰉1 + √5 􏰈 ≈ 0.62. Of course, the factor 0.63 in the Berman
2
algorithm represents only an average and can be considerably lower but also as high
as 4−1/4 ≈ 0.87.
6. SelectaroutinefromyourprogramlibraryorfromapackagesuchasMatlab,Maple,or Mathematica for finding the minimum point of a function of one variable. Experiment with the function F(x) = x4 + sin(23x) to determine whether this routine encounters any difficulties in finding a global minimum point. Use starting values both near to and far from the global minimum point. (See Figure 16.2.)
7. (Student project) The Greek mathematician Euclid of Alexandria (325–265 B.C.E.) wrote a collection of 13 books on mathematics and geometry. In book six, Proposition 30 shows how to divide a line into its mean and extreme mean, which is finding the golden section point on a line. This states that the ratio of the smaller part of a line segment to the larger part is the same as the ratio of the larger part to the whole line segment. For a line segment of length 1, denote the larger part by r and the smaller part by 1 − r as shown here:
Hence, we have the ratios
r 1􏶼r
01
1−r=r r1
and we obtain the quadratic equation
r2 =1−r
This equation has two roots, one positive and one negative. The reciprocal of the
positive root is the golden ratio 1 􏰉1 + √5􏰈, which was of interest to Pythagoras 2
(580–500 B.C.E.). It was also used in the construction of the Great Pyramid of Gizah. Mathematical software systems such as Matlab, Maple, or Mathematica contain the golden ratio constant. In fact, the default width-to-height ratio for the plot function is the golden ratio. Investigate the golden section ratio and its use in scientific computing.
8. Using a mathematical software system such as Matlab, Maple, or Mathematica, write computer program to reproduce
a. Figure16.1.
b. Figure16.2.Also,findtheglobalminimumofthefunctionaswellasseverallocal
minimum points near the origin.
16.2 Multivariate Case
Now we consider a real-valued function of n real variables F: Rn → R. As before, a point x∗ is sought such that
F(x∗)􏶞F(x) forallx∈Rn

640
Chapter 16
Minimization of Functions
Some of the theory of multivariate functions must be developed to understand the rather sophisticated minimization algorithms in current use.
Taylor Series for F : Gradient Vector and Hessian Matrix
If the function F possesses partial derivatives of certain low orders (which is usually assumed in the development of these algorithms), then at any given point x, a gradient vector G(x) = (Gi )n is defined with components
Gi =Gi(x)=∂F(x) (1􏶞i􏶞n) (1) ∂xi
and a Hessian matrix H(x) = (Hi j )n×n is defined with components
Hij =Hij(x)=∂2F(x) (1􏶞i, j􏶞n) (2)
∂xi ∂xj
We interpret G(x) as an n-component vector and H(x) as an n × n matrix, both depending on x.
Using the gradient and Hessian, we can write the first few terms of the Taylor series for F as
􏰃n F(x+h)=F(x)+
i=1
Equation (3) can also be written in an elegant matrix-vector form:
F(x + h) = F(x) + G(x)T h + 1 hT H(x)h + · · · (4) 2
Here, x is the fixed point of expansion in Rn , and h is the variable in Rn with components h1,h2,…,hn. The three dots indicate higher-order terms in h that are not needed in this discussion.
A result in calculus states that the order in which partial derivatives are taken is imma- terial if all partial derivatives that occur are continuous. In the special case of the Hessian matrix, if the second partial derivatives of F are all continuous, then H is a symmetric matrix; that is, H = HT because
Hij= ∂2F = ∂2F =Hji ∂xi ∂xj ∂xj ∂xi
To illustrate Formula (4), let us compute the first three terms in the Taylor series for the function
F(x1, x2) = cos(πx1) + sin(πx2) + ex1x2 taking (1, 1) as the point of expansion.
1 􏰃n 􏰃n
Gi(x)hi +2 hiHij(x)hj +··· (3)
i=1 j=1
EXAMPLE 1

Solution
Partial derivatives are
∂F = −π sin(πx1) + x2ex1x2
16.2 Multivariate Case 641
∂x1
∂2F =−π2cos(πx1)+x2ex1x2
∂F = π cos(πx2) + x1ex1x2 ∂x2
∂2F =(x1x2+1)ex1x2 ∂ x 2 ∂ x 1
∂ x 12 ∂2F
∂2F =−π2sin(πx2)+x12ex1x2 ∂x2
=(x1x2+1)ex1x2
Note the equality of cross derivatives; that is, ∂2 F/∂x1 ∂x2 = ∂2 F/∂x2 ∂x1. At the particular
∂x1 ∂x2
point x = [1, 1]T , we have
F(x) = −1 + e, So by Equation (4),
G(x) =
􏰙 e 􏰚 −π + e
, H(x) = 􏰖􏰗
􏰙π2+e 2e􏰚
2e e
F(1+h1,1+h2)=−1+e+[e,−π+e] h1 h2
1 􏰖π2+e 2e􏰗􏰖h􏰗
+ [h1,h2] 1 +···
+ 1􏰠(π2 +e)h2 +(2e)h h +(2e)h h +eh2􏰡+··· ■ 2112212
In mathematical software systems such Maple or Mathematica, we can verify these calculations using built-in routines for the gradient and Hessian. Also, we can obtain two terms in the Taylor series in two variables expanded about the point (1, 1) and then carry out a change of variables to obtain similar results as above.
Alternative Form of Taylor Series
Another form of the Taylor series is useful. First let z be the point of expansion, and then let h = x − z. Now from Equation (4),
F(x)= F(z)+G(z)T(x−z)+ 1(x−z)T H(z)(x−z)+··· (5) 2
We illustrate with two special types of functions. First, the linear function has the form
2 2eeh2 F(1+h1,1+h2)=−1+e+eh1 +(−π+e)h2
or equivalently, by Equation (3),
􏰃n i=1
bixi =c+bTx
for appropriate coefficients c,b1,b2,…,bn. Clearly, the gradient and Hessian are Gi(z) =
F(x)=c+ bi andHij(z)=0,soEquation(5)yields
F(x)= F(z)+
􏰃n i=1
bi(xi −zi)= F(z)+bT(x−z)

642 Chapter 16
Minimization of Functions
Second, consider a general quadratic function. For simplicity, we take only two vari- ables. The form of the function is
F(x1, x2) = c + (b1x1 + b2x2) + 1􏰉a11x12 + 2a12x1x2 + a22x2􏰈 (6) 2
which can be interpreted as the Taylor series for F when the point of expansion is (0, 0). To verify this assertion, the partial derivatives must be computed and evaluated at (0, 0):
∂F=b+a x+a x ∂F=b+a x+a x ∂x 1 111 122 ∂x 2 222 121
12 ∂2F=a ∂2F =a
∂x2 11 ∂x ∂x 12 112
∂2F =a ∂2F=a ∂x ∂x 12 ∂x2 22
212 Letting z = [0, 0]T , we obtain from Equation (5)
􏰖x􏰗1 􏰖aa􏰗􏰖x􏰗 F(x) = c + [b1, b2] 1 + [x1, x2] 11 12 1
written as
F(x)=c+bTx+1xT Ax (7) 2
where c is a scalar, b a vector, and A a matrix. Equation (7) holds for a general quadratic function of n variables, with b an n-component vector and A an n × n matrix.
Returning to Equation (3), we now write out the complicated double sum in complete detail to assist in understanding it:
⎧􏰍n xHx⎫ ⎪ 􏰍j=1 1 1j j⎪
⎪+ nj=1xHx⎪ 􏰃n􏰃n ⎨22jj⎬
x2 2 a12 a22 x2
This is the matrix form of the original quadratic function of two variables. It can also be
xTHx=
i=1 j=1
xiHijxj =⎪+··· ⎪
⎪+ ··· ⎪
⎩+􏰍n xHx⎭ j=1 n nj j
⎧⎫ ⎪ x1H11x1 + x1H12x2 + ··· + x1H1nxn ⎪
⎪+xH x +xH x ⎪⎨22112222
=⎪+··· ⎪+ ···
⎪⎩+xH x +xH x nn11nn22
+···+xH x ⎪ 2 2 n n ⎪ ⎪⎬
+··· ⎪
+··· ⎪
+···+xH x⎪⎭ n nn n
Thus, xT Hx can be interpreted as the sum of all n2 terms in a square array of which the (i, j ) element is xi Hi j x j .

Steepest Descent Procedure
A crucial property of the gradient vector G(x) is that it points in the direction of the most
rapid increase in the function F, which is the direction of steepest ascent. Conversely,
−G(x) points in the direction of the steepest descent. This fact is so important that it is
worth a few words of justification. Suppose that h is a unit vector, 􏰍n h2 = 1. The rate i=1 i
of change of F (at x) in the direction h is defined naturally by d 􏶨􏶨􏶨
that
F(x + th) = F(x) + tG(x)T h + 1t2hT H(x)h + ··· (8) 2
Differentiation with respect to t leads to
d F(x+th)=G(x)Th+thTH(x)h+··· (9)
dt
By letting t = 0 here, we see that the rate of change of F in the direction h is nothing else
than
G(x)T h
Now we ask: For what unit vector h is the rate of change a maximum? The simplest path
to the answer is to invoke the powerful Cauchy-Schwarz inequality:
dt F(x + th)􏶨t=0
This rate of change can be evaluated by using Equation (4). From that equation, it follows
16.2 Multivariate Case 643
􏰃n i=1
􏶠􏰃n i=1
􏶡1/2 􏶠􏰃n 􏶡1/2 ui2 vi2
uivi 􏶞
where equality holds only if one of the vectors u or v is a nonnegative multiple of the other.
Applying this to
On the basis of the foregoing discussion, a minimization procedure called best-step steepest descent can be described. At any given point x, the gradient vector G(x) is calculated. Then a one-dimensional minimization problem is solved by determining the value t∗ for which the function
φ(t) = F(x + tG(x))
is a minimum. Then we replace x by x + t∗ G(x) and begin anew.
The general method of steepest descent takes a step of any size in the direction of the
negative gradient. It is not usually competitive with other methods, but it has the advantage of simplicity. One way of speeding it up is described in Computer Problem 16.2.2.
􏰃n i=1
G(x)T h =
and remembering that 􏰍n h2 = 1, we conclude that the maximum occurs when h is a
i=1 i
positive multiple of G(x), that is, when h points in the direction of G.
Gi (x)hi
i=1
(10)

644 Chapter 16
Minimization of Functions
Contour Diagrams
In understanding how these methods work on functions of two variables, it is often helpful to draw contour diagrams. A contour of a function F is a set of the form
{x : F(x) = c}
where c is a given constant. For example, the contours of function
F(x) = 25×12 + x2
are ellipses, as shown in Figure 16.12. Contours are also called level sets by some authors. At any point on a contour, the gradient of F is perpendicular to the curve. So, in general, the path of steepest descent may look like Figure 16.13.
y
6.00
FIGURE 16.12
Contours of F(x) = 25×12+x2
􏶼6.00
More Advanced Algorithms
􏶼2.00
Ellipse c􏶾25×2 􏶽y2
x
2.00
To explain more advanced algorithms, we consider a general real-valued function F of n variables. Suppose that we have obtained the first three terms in the Taylor series of F in the vicinity of a point z. How can they be used to guess the minimum point of F? Obviously,

16.2
Multivariate Case 645
F(x) 􏶾 F(x1) F(x) 􏶾 F(x2) F(x) 􏶾 F(x3) F(x) 􏶾 F(x4) F(x) 􏶾 F(x5)
x3
x2
x5
x4
x1
FIGURE 16.13
Path of steepest descent
we could ignore all terms beyond the quadratic terms and find the minimum of the resulting quadratic function:
F(x + z) = F(z) + G(z)T x + 1 xT H(z)x + · · · (11) 2
Here, z is fixed and x is the variable. To find the minimum of this quadratic function of x, we must compute the first partial derivatives and set them equal to zero. Denoting this quadratic function by Q and simplifying the notation slightly, we have
Q(x)=F(z)+ from which it follows that
xiHijxj (12)
􏰃n i=1
1 􏰃n 􏰃n
Gixi +2 ∂Q 􏰃n
(1􏶞k􏶞n) (13) (See Problem 16.2.13.) The point x that is sought is thus a solution of the system of n
∂x =Gk+ Hkjxj k j=1
􏰃n
Hkjxj =−Gk (1􏶞k􏶞n)
j=1
H(z)x = −G(z) (14)
The preceding analysis suggests the following iterative procedure for locating a mini- mum point of a function F: Start with a point z that is a current estimate of the minimum point. Compute the gradient and Hessian of F at the point z. They can be denoted by G and H, respectively. Of course, G is an n-component vector of numbers and H is an n × n matrix of numbers. Then solve the matrix equation
i=1 j=1
equations
or, equivalently,
Hx = −G

646 Chapter 16
Minimization of Functions
obtaining an n-component vector x. Replace z by z + x and return to the beginning of the procedure.
Minimum, Maximum, and Saddle Points
There are many reasons for expecting trouble from the iterative procedure just outlined. One especially noisome aspect is that we can expect to find a point only where the first partial derivatives of F vanish; it need not be a minimum point. It is what we call a stationary point. Such points can be classified into three types: minimum point, maximum point, and saddle point. They can be illustrated by simple quadratic surfaces familiar from analytic geometry:
• Minimum of F(x, y) = x2 + y2 at (0, 0)
• Maximum of F(x, y) = 1 − x2 − y2 at (0, 0) • Saddle point of F(x, y) = x2 − y2 at (0, 0)
(See Figure 16.14(a).) (See Figure 16.14(b).) (See Figure 16.14(c).)
FIGURE 16.14
Simple quadratic surfaces
(c) Saddle point
(a) Minimum point
(b) Maximum point

QUADRATIC FUNCTION THEOREM
If the matrix H has the property that xT Hx > 0 for every nonzero vector x, then the quadratic function Q has a minimum point.
■ THEOREM1
Positive Definite Matrix
If z is a stationary point of F, then
G(z) = 0
Moreover, a criterion ensuring that Q, as defined in Equation (12), has a minimum point is
as follows:
(See Problem 16.2.15.) A matrix that has this property is said to be positive definite. Notice that this theorem involves only second-degree terms in the quadratic function Q.
As examples of quadratic functions that do not have minima, consider the following: −x12 −x2 +13×1 +6×2 +12 x12 −x2 +3×1 +5×2 +7
x12 −2x1x2 +x1 +2×2 +3 2×1 +4×2 +6
Inthefirsttwoexamples,letx1 =0andx2 →∞.Inthethird,letx1 =x2 →∞.Inthe last, let x1 = 0 and x2 → −∞. In each case, the function values approach −∞, and no global minimum can exist.
Quasi-Newton Methods
Algorithms that converge faster than steepest descent in general and that are currently recommended for minimization are of a type called quasi-Newton. The principal example is an algorithm introduced in 1959 by Davidon, called the variable metric algorithm. Subsequently, important modifications and improvements were made by others, such as R. Fletcher, M. J. D. Powell, C. G. Broyden, P. E. Gill, and W. Murray. These algorithms proceed iteratively, assuming in each step that a local quadratic approximation is known for the function F whose minimum is sought. The minimum of this quadratic function either provides the new point directly or is used to determine a line along which a one-dimensional search can be carried out. In implementation of the algorithm, the gradient can be either provided in the form of a procedure or computed numerically by finite differences. The Hessian H is not computed, but an estimate of its LU factorization is kept up to date as the process continues.
Nelder-Mead Algorithm
ForminimizingafunctionF:Rn →R,anothermethodcalledtheNelder-Meadalgorithm is available. It is a method of direct search and proceeds without involving any derivatives of the function F and without any line searches.
Before beginning the calculations, the user assigns values to three parameters: α, β,
and γ . The default values are 1, 1 , and 1, respectively. In each step of the algorithm, a set 2
16.2 Multivariate Case 647

648 Chapter 16
Minimization of Functions
ofn+1pointsinRn isgiven:{x0,x1,…,xn}.ThissetisingeneralpositioninRn.This means that the set of n points xi − x0, with 1 􏶞 i 􏶞 n, is linearly independent. A consequence ofthisassumptionisthattheconvexhulloftheoriginalset{x0,x1,…,xn}isann-simplex. For example, a 2-simplex is a triangle in R2, and a 3-simplex is a tetrahedron in R3. To make the description of the algorithm as simple as possible, we assume that the points have been relabeled (if necessary) so that F(x0)􏶟 F(x1)􏶟 ··· 􏶟 F(xn). Since we are trying to minimize the function F, the point x0 is the worst of the current set, because it produces the highest value of F.
We compute the point
1 􏰃n u=n xi
i=1
This is the centroid of the face of the current simplex opposite the worst vertex, x0. Next, we compute a reflected point v = (1 + α)u − αx0.
If F(v) is less than F(xn), then this is a favorable situation, and one is tempted to replace x0 by v and begin anew. However, we first compute an expanded reflected point w = (1 + γ)v − γu and test to see whether F(w) is less than F(xn). If so, we replace x0 by w and begin anew. Otherwise, we replace x0 by v as originally suggested and begin with the new simplex.
Assume now that F(v) is not less than F(xn). If F(v)􏶞 F(x1), then replace x0 by v
and begin again. Having disposed of all cases when F(v)􏶞 F(x1), we now consider two
further cases. First, if F(v)􏶞 F(x0), then define w = u + β(v − u). If F(v) > F(x0),
compute w = u + β(x0 − u). With w now defined, test whether F(w) < F(x0). If this is true, replace x0 by w and begin anew. However, if F(w)􏶟 F(x0), shrink the simplex by usingxi ← 1(xi +xn)for0􏶞i􏶞n−1.Thenbeginanew. 2 The algorithm needs a stopping test in each major step. One such test is whether the relative flatness is small. That is the quantity F(x0) − F(xn) |F(x0)| + |F(xn)| Other tests to make sure progress is being made can be added. In programming the algorithm, one keeps the number of evaluations of f to a minimum. In fact, only three indices are needed: the indices of the greatest F(xi), the next greatest, and the least. In addition to the original paper of Nelder and Mead [1965], one can consult Dennis and Woods [1987], Dixon [1974], and Torczon [1997]. Different authors give slightly different versions of the algorithm. We have followed the original description by Nelder and Mead. Method of Simulated Annealing This method has been proposed and found to be effective for the minimization of difficult functions, especially if they have many purely local minimum points. It involves no deriva- tives or line searches; indeed, it has found great success in minimizing discrete functions, such as arise in the traveling salesman problem. Suppose we are given a real-valued function of n real variables; that is, F: Rn → R. We must be able to compute the values F(x) for any x in Rn. It is desired to locate a global minimum point of F, which is a point x∗ such that F(x∗)􏶞F(x) for all x in Rn. In other words, F(x∗) is equal to infx∈Rn F(x). The algorithm generates a sequenceofpointsx1,x2,x3,...,andonehopesthatminj􏶞k F(xj)convergestoinfF(x) as k → ∞. It suffices to describe the computation that leads to xk+1, assuming that xk has been computed.Webeginbygeneratingamodestnumberofrandompointsu1,u2,...,um in a large neighborhood of xk. For each of these points, the value of F must be computed. The next point, xk+1, in our sequence is chosen to be one of the points u1, u2, . . . , um . This choice is made as follows. Select an index j such that F(uj) = min{F(u1), F(u2),..., F(um)} If F(uj) < F(xk), then set xk+1 = uj. In the other case, for each i, we assign a probability pi to ui by the formula pi =eα[F(xk)−F(ui)] (1􏶞i􏶞m) Here, α is a positive parameter chosen by the user of the code. We normalize the probabilities by dividing each by their sum. That is, we compute and then carry out a replacement 􏰃m S= pi i=1 pi ← pi/S Finally, a random choice is made among the points u1, u2, . . . , um , taking account of the probabilities pi that have been assigned to them. This randomly chosen ui becomes xk+1. The simplest way to make this random choice is to employ a random number generator to get a random point ξ in the interval (0, 1). Select i to be the first integer such that ξ 􏶞 p1 + p2 + · · · + pi Thus,ifξ 􏶞 p1,leti=1(andxn+1 = u1).If p1 < ξ 􏶞 p1+p2,thenleti = 2(andxn+1 = u2), and so on. The formula for the probabilities pi is taken from the theory of thermodynamics. The interested reader can consult the original articles by Metropolis et al. [1953] or Otten and van Ginneken [1989]. Presumably, other functions can serve in this role as well. What is the purpose of the complicated choice for xk+1? Because of the possibility of encountering local minima, the algorithm must occasionally choose a point that is uphill from the current point. Then there is a chance that subsequent points might begin to move toward a different local minimum. An element of randomness is introduced to make this possible. With minor modifications, the algorithm can be used for functions f : X → R, where X is any set. For example, in the traveling salesman problem, X will be the set of all permutations of a set of integers {1, 2, 3, . . . , N }. All that is required is a procedure for generating random permutations and, of course, a code for evaluating the function f . Computer programs for this algorithm can be found on the Internet such as at the web- sites http://www.netlib.gov and http://www.ingber.com. A collection of papers on this subject, emphasizing parallel computation, is Azencott [1992]. 16.2 Multivariate Case 649 650 Chapter 16 Minimization of Functions Summary (1) In a typical minimization problem, we seek a point x∗ such that F(x∗)􏶞F(x) forallx∈Rn where F is a real-valued multivariate function. (2) A gradient vector G(x) has components Gi =Gi(x)=∂F(x) ∂xi and a Hessian matrix H(x) has components Hij =Hij(x)=∂2F(x) (1􏶞i􏶞n) (3) The Taylor series for F is F(x + h) = F(x) + G(x)T h + 1 hT H(x)h + · · · 2 Here, x is the fixed point of expansion in Rn and h is the variable in Rn with components h1,h2,...,hn. The three dots indicate higher-order terms in h that are not needed in this discussion. (4) An alternative form of the Taylor series is F(x)= F(z)+G(z)T(x−z)+ 1(x−z)T H(z)(x−z)+··· 2 For example, a linear function F(x) = c + bT x has the Taylor series (1􏶞i, j􏶞n) It is a symmetric matrix if the second-order derivatives are continuous. ∂xi ∂xj A quadratic function is F(x) = F(z) + bT (x − z) F(x)=c+bTx+1xT Ax 2 (5) An iterative procedure for locating a minimum point of a function F is to start with a point z that is a current estimate of the minimum point, compute the gradient G and Hessian H of F at the point z, and solve the matrix equation Hx = −G for x. Then replace z by z + x and repeat. (6) If the matrix H has the property that xT Hx > 0 for every nonzero vector x, then the quadratic function Q has a unique minimum point.
(7) Algorithms that are discussed are steepest descent, Nelder-Mead, and simulated annealing.

Additional References
For more reading on the subject of optimization, see books and papers by Azencott [1992], Baldick [2006], Beale [1988], Cvijovic and Kilnowski [1995], Dennis and Schnabel [1983, 1996], Dennis and Woods [1987], Dixon [1974], Fletcher [1976], Floudas and Pardalos [1992], Gill, Murray and Wright [1981], Herz-Fischer [1998], Horst, Pardalos, and Thoai [2000], Kelley [2003], Kirkpatrick et al. [1983], Lootsam [1972], More ́ and Wright [1993], Nelder and Mead [1965], Nocedal and Wright [2006], Otten and van Ginneken [1989], Rheinboldt [1998], Roos, Terlaky, and Vial [1997], Torczon [1997], and To ̈rn and Zilinskas [1989].
1. DeterminewhetherthesefunctionshaveminimumvaluesinR2:
a2. a 3. 4. 5.
aa. x12 −x1x2 +x2 +3×1 +6×2 −4 ab. x12 −3x1x2 +x2 +7×1 +3×2 +5 c. 2×12 −3x1x2 +x2 +4×1 −x2 +6
d. ax12 −2bx1x2 +cx2 +dx1 +ex2 + f
Hint: Use the method of completing the square.
Locate the minimum point of 3×2 − 2xy + y2 + 3x − 46 + 7 by finding the gradient and Hessian and solving the appropriate linear equations.
Using (0, 0) as the point of expansion, write the first three terms of the Taylor series forF(x,y)=ex cosy−yln(x+1).
Using (1, 1) as the point of expansion, write the first three terms of the Taylor series for F(x, y) = 2×2 − 4xy + 7y2 − 3x + 5y.
TheTaylorseriesexpansionaboutzerocanbewrittenas
F(x) = F(0) + G(0)T x + 1 xT H(0)x + · · · 2
Show that the Taylor series about z can be written in a similar form by using matrix- vector notation; that is,
F(x)= F(z)+G(z)TX + 1XTH(z)X +··· 2
a6.
Show that the gradient of F(x, y) is perpendicular to the contour. Hint: Interpret the equation F(x, y) = c as defining y as a function of x. Then by the chain rule,
∂F + ∂F dy = 0 ∂x ∂y dx
where 􏰖x􏰗 􏰙 G(z)􏰚 X = z , G(z)= −G(z) ,
􏰙 H(z) −H(z)􏰚 H(z)= −H(z) H(z)
16.2 Multivariate Case 651
From it obtain the slope of the tangent to the contour.
Problems 16.2

652 Chapter 16
Minimization of Functions
7.
8.
9.
a 10.
11.
a12.
13. 14. 15.
Considerthefunction
F(x1,x2,x3)=3ex1x2 −x3cosx1+x2lnx3
a. DeterminethegradientvectorandHessianmatrix.
ab. DerivethefirstthreetermsoftheTaylorseriesexpansionabout(0,1,1).
c. What linear system should be solved for a reasonable guess as to the minimum point for F? What is the value of F at this point?
It is asserted that the Hessian of an unknown function F at a certain point is 􏰖􏰗
32 14
What conclusion can be drawn about F?
Whatarethegradientsofthefollowingfunctionsatthepointsindicated? aa. F(x,y)=x2y−2x+yat(1,0)
ab. F(x,y,z)=xy+yz2+x2zat(1,2,1)
Consider F(x, y, z) = y2z2(1 + sin2 x) + (y + 1)2(z + 3)2. We want to find the mini- mum of the function. The program to be used requires the gradient of the function. What formulas must we program for the gradient?
Let F be a function of two variables whose gradient at (0, 0) is [−5, 1]T and whose Hessian is
􏰖􏰗
6 −1 −1 2
Make a reasonable guess as to the minimum point of F. Explain.
Write the function F(x1, x2) = 3×12 + 6x1x2 − 2×2 + 5×1 + 3×2 + 7 in the form of Equation (7) with appropriate A, b, and c. Show in matrix form the linear equations that must be solved in order to find a point where the first partial derivatives of F vanish. Finally, solve these equations to locate this point numerically.
VerifyEquation(13).IndifferentiatingthedoublesuminEquation(12),firstwriteall terms that contain xk. Then differentiate and use the symmetry of the matrix H.
ConsiderthequadraticfunctionQinEquation(12).ShowthatifHispositivedefinite, then the stationary point is a minimum point.
(General quadratic function) Generalize Equation (6) to n variables. Show that a general quadratic function Q(x) of n variables can be written in the matrix-vector form of Equation (7), where A is an n × n symmetric matrix, b a vector of length n, and c a scalar. Establish that the gradient and Hessian are
G(x)= Ax+b and H(x)= A
respectively.

16.2 Multivariate Case 653 16. Let A be an n × n symmetric matrix and define an upper triangular matrix U = (ui j )
by putting
⎧
⎪⎨ a i j i = j
u i j = ⎪⎩ 2 a i j i < j 0 i>j
Show that xT Ux = xT Ax for all vectors x.
17. Show that the general quadratic function Q(x) of n variables can be written
Q(x) = c + bT x + 1 xT U x 2
where U is an upper triangular matrix. Can this simplify the work of finding the station- ary point of Q?
18. Show that the gradient and Hessian satisfy the equation H(z)(x − z) = G(x) − G(z)
for a general quadratic function of n variables.
19. UsingTaylorseries,showthatageneralquadraticfunctionofnvariablescanbewritten
in block form
where
Q(x)= 1XTAX +BTX +c 2
􏰖x􏰗 􏰙A−A􏰚 􏰙b􏰚 X= z , A= −A A , B= −b
Here z is the point of expansion.
20. (Least-squares problem) Consider the function
F(x) = (b − Ax)T (b − Ax) + αxT x
where A is a real m ×n matrix, b is a real column vector of order m, and α is a positive
real number. We want the minimum point of F for given A, b, and α. Show that F(x + h) − F(x) = (Ah)T (Ah) + αhT h 􏶟 0
for h a vector of order n, provided that
(AT A+αI)x= ATb
This means that any solution of this linear system minimizes F(x); hence, this is the normal equation.
21. (Multiple choice) What is the gradient of the function f (x) = 3×12 − sin(x1x2) at the point (3, 0)?
a. (6, −3) b. (3, −1) c. (18, 0) d. (18, −3) e. None of these.

654 Chapter 16
Minimization of Functions
22. (Multiple choice, continuation) The directional derivative of the function f at the point x in the direction u is given by the expression
d f(x+tu)|t=0 dt
In this description, u should be a unit vector. What is the numerical value of the directional derivative where f (x) is the function defined in the preceding problem,
√ x = (1,π/2), and u = (1,1)/ 2.
√
a. 6/ 2 b.6 c.18
d.3 e.Noneofthese.
23. (Multiple choice, continuation) If f is a real-valued function of n variables, the Hessian H=(Hij)isgivenbyHij =∂2f/∂xi∂xj,alltermsbeingevaluatedataspecificpoint x. What is the entry H22 in this matrix in the case of f as given in the previous problem and x = (1, π/2)?
a. 6 b. 6/√2 c. 1 d. π2/2 e. Noneofthese.
24. (Multiple choice) Let f be a real-valued function of n real variables. Let x and u be given as numerical vectors, and u ≠ 0. Then the expression f (x + t u) defines a functionoft.Supposethattheminimumof f(x+tu)occursatt = 0.Whatconclusion can be drawn?
a. The gradient of f at x, denoted by G(x), is 0.
b. u is perpendicular to the gradient of f at x.
c. u = G(x), where G(x) denotes the gradient of f at x. d. G(x)isperpendiculartox. e.Noneofthese.
25. (Multiple choice) If f is a (real-valued) quadratic function of n real variables, we can writeitintheform fx)=c−bTx+1xT Ax.Thegradientof f isthen:
2
a. Ax b.b−Ax c.Ax−b d.1Ax−b e.Noneofthese. 2
1. SelectaroutinefromyourprogramlibraryorfromapackagesuchasMatlab,Maple,or Mathematica for minimizing a function of many variables without the need to program derivatives. Test it on one or more of the following well-known functions. The ordering of our variables is (x, y, z, w).
a. Rosenbrock’s: 100(y − x2)2 + (1 − x)2. Start at (−1.2, 1.0).
b. Powell’s: (x +10y)2 +5(z −w)2 +(y −2z)4 +10(x −w)4. Start at (3,−1,0,1).
c. Powell’s:x2 +2y2 +3z2 +4w2 +(x+y+z+w)4.Startat(1,−1,1,1).
d. Fletcher and Powell’s: 100(z − 10φ)2 + 􏰉􏶣x2 + y2 − 1􏰈2 + z2 in which φ is an angle determined from (x, y) by
cos2πφ = x sin2πφ = y 􏶣 and􏶣
x2 + y2 x2 + y2 where −π/2 < 2πφ 􏶞 3π/2. Start at (1, 1, 1). Computer Problems 16.2 16.2 Multivariate Case 655 e. Wood’s:100(x2 −y)2 +(1−x)2 +90(z2 −w)2 +(1−z)2 +10(y−1)2 + (w − 1)2 + 19.8(y − 1)(w − 1). Start at (−3, −1, −3, −1). 2. (Acceleratedsteepestdescent)Thisversionofsteepestdescentissuperiortothebasic one. A sequence of points x1, x2, . . . is generated as follows: Point x1 is specified as the starting point. Then x2 is obtained by one step of steepest descent from x1. In the general step, if x1, x2, . . . , xm have been obtained, we find a point z by steepest descent from xm. Then xm+1 is taken as the minimum point on the line xm−1 + t(z − zm−1). Program and test this algorithm on one of the examples in Computer Problem 16.2.1. 3. Using a routine in your program library or in Matlab, Maple, or Mathematica, a. solve the minimization problem that begins this chapter. b. plotandsolvefortheminimumpoint,themaximumpoint,andthesaddlepointof these functions, respectively: x2 + y2, 1 − x2 − y2, x2 − y2. c. plot and numerically experiment with these functions that do not have minima: −x2 −y2 +13x+6y+12,x2 −y2 +3x+5y+7,x2 −2xy+x+2y+3, 2x + 4y + 6. 4. WewanttofindtheminimumofF(x,y,z)=z2cosx+x2y2+x2ez usingacomputer program that requires procedures for the gradient of F together with F. Write the necessary procedures. Find the minimum using a preprogrammed code that uses the gradient. 5. Assumethat procedure Xmin( f, (gradi ), n, (xi), (gi j )) is available to compute the minimum value of a function of two variables. Suppose that this routine requires not only the function but also its gradient. If we are going to use this routine with the function F(x, y) = ex cos2(xy), what procedure will be needed? Write the appropriate code. Find the minimum using a preprogrammed code that uses the gradient. 6. ProgramandtesttheNelder-Meadalgorithm. 7. Program and test the Simulated Annealing algorithm. 8. (Studentresearchproject)Exploreoneofthenewermethodsforminimizationsuch as generic algorithms, methods of simulated annealing, or the Nelder-Mead algorithm. Use some of the software that is available for them. 9. Usebuilt-inroutinesinmathematicalsoftwaresystemssuchasMapleorMathematica to verify the calculations in Example 1. Hint: In Maple, use grad and Hessian, and in Mathematica, use Series. For example, obtain two terms in the Taylor series in two variables expanded about the point (1, 1), and then carry out a change of variables. 10. (Molecular conformation: Protein folding project) Forces that govern folding of amino acids into proteins are due to bonds between individual atoms and to weaker interactions between unbound atoms such as electrostatic and Van der Waals forces. The Van der Waals forces are modeled by the Lennard-Jones potential U(r)= 1 − 2 r12 r6 656 Chapter 16 Minimization of Functions where r is the distance between atoms. y 􏶼1 123 x In the figure, the energy minimum is −1 and it is achieved at r = 1. Explore this subject and the numerical methods used. One approach is to predict the conformation of the proteins in finding the minimum potential energy of the total configuration of amino acids. For a cluster of atoms with positions (x1, y1, z1) to (xn, yn, zn), the objective function to be minimized is U=􏰃1−2 r12 r6 i 1 ⎪3×2 −8×3 +x4 =5 ⎪⎩x1,x2,x3,x4 􏶟0
⎪⎩ constraints:
yT A 􏶟 cT y􏶟0
y􏶞x−2

668 Chapter 17
Linear Programming
Here, nonunique and unbounded “solutions” may be obtained.
aa.c=[2,−4]T 􏰖 1􏰗T
b. c= 2, 2 ac. c=[3,2]T
d.c=[2,−3]T e. c = [−4, 11]T
af. c=[−3,4]T g. c = [2, 1]T ah. c=[3,1]T
􏰖􏰗
A= −3 −5 49
b = [−15, 36] b = [30, 12]T
b=[6,36]T
b=[0,5]T
b = [12, 44]T
b = [6, −20]T
b=[0,−2]T
􏰖6 5􏰗
A= 41 􏰖􏰗
A= −3 2 􏰖−4 9􏰗
A= −1 1 􏰖01􏰗
A= −3 4 􏰖−4 11􏰗
A= 23 −4 −5
􏰖􏰗
A= 11 􏰖1 2􏰗
A= 24 53
b = [21, 18]T
a14. Solvethefollowinglinearprogrammingproblembyhand,usingagraphforhelp:
⎧⎪maximize: 4x+4y+z
⎪ ⎪⎨
⎪ constraints: ⎪
⎧⎪ 3x+2y+ z= 12 ⎪⎨ 7x+7y+2z 􏶞 144 ⎪ 7x + 5y + 2z 􏶞 80 ⎪11x+7y+3z􏶞 132 ⎪⎩ x 􏶟 0 y 􏶟 0
⎩
Hint: Use the equation to eliminate z from all other expressions. Solve the resulting two-dimensional problem.
15. Putthislinearprogrammingproblemintosecondprimalform.Youmaywanttomake changes of variables. If so, include a dictionary relating new and old variables.
⎧⎪ minimize:
⎪⎨
ε1 + ε2 + ε3 ⎧⎪|3x+4y+6| 􏶞 ε1 ⎨|2x−8y−4| 􏶞 ε2
⎪ ⎪constraints:⎪|−x−3y+5|􏶞 ε
⎪⎩ ⎪⎩ 3
ε1>0 ε2>0 ε3>0 x>0 y>0
Solve the resulting problem.

16.
17.1 Standard Forms and Duality 669 Consider the following linear programming problem:
a 17.
18.
⎪⎩ constraints:
In the special case in which all data are positive, show that the dual problem has the
same extreme value as the original problem.
Suppose that a linear programming problem in first primal form has the property that cTx is not bounded on the feasible set. What conclusion can be drawn about the dual problem?
(Multiplechoice)Whichoftheseproblemsisformulatedinthefirstprimalformfora linear programming problem?
a. maximize cTx subject to Ax 􏶞 b
b. minimizecTxsubjecttoAx􏶞b,x􏶟0
c. maximizecTxsubjecttoAx=b,x􏶟0
d. maximize cTx subject to Ax 􏶞 b, x 􏶟 0 e. None of these.
Awesternshopwishestopurchase300feltand200strawcowboyhats.Bidshavebeen received from three wholesalers. Texas Hatters has agreed to supply not more than 200 hats, Lone Star Hatters not more than 250, and Lariat Ranch Wear not more than 150. The owner of the shop has estimated that his profit per hat sold from Texas Hatters would be $3/felt and $4/straw, from Lone Star Hatters $3.80/felt and $3.50/straw, and from Lariat Ranch Wear $4/felt and $3.60/straw. Set up a linear programming problem to maximize the owner’s profits. Solve by using a program from your software library.
The ABC Drug Company makes two types of liquid painkiller that have brand names Relieve (R) and Ease (E) and contain different mixtures of three basic drugs, A, B, and C, produced by the company. Each bottle of R requires 7 unit of drug A, 1 unit
of drug B, and 3 unit of drug C. Each bottle of E requires 4 unit of drug A, 5 unit of 492
drug A, 7 units of drug B, and 9 units of C. Moreover, Food and Drug Administration regulations stipulate that the number of bottles of R manufactured cannot exceed twice the number of bottles of E. The profit margin for each bottle of E and R is $7 and $3, respectively. Set up the linear programming problem in first primal form to determine the number of bottles of the two painkillers that the company should produce each day so as to maximize their profits. Solve by using available software.
Supposethattheuniversitystudentgovernmentwishestocharterplanestotransportat least 750 students to the bowl game. Two airlines, α and β, agree to supply aircraft for
a1.
2.
⎧
⎪⎨ maximize:
c1 x1 + c2 x2
􏰎
a1x1 + a2x2 􏶞 b x1􏶟0 x2􏶟0
92
drug B, and 1 unit of drug C. The company is able to produce each day only 5 units of 4
a3.
Computer Problems 17.1

670
Chapter 17
Linear Programming
17.2
the trip. Airline α has five aircraft available carrying 75 passengers each, and airline β has three aircraft available carrying 250 passengers each. The cost per aircraft is $900 and $3250 for the trip from airlines α and β, respectively. The student government wants to charter at most six aircraft. How many of each type should be chartered to minimize the cost of the airlift? How much should the student government charge to make 50c/ profit per student? Solve by the graphical method, and verify by using a routine from your program library.
4. (Continuation)Reworktheprecedingcomputerprobleminthefollowingtwopossibly different ways:
a. Thenumberofstudentsgoingontheairliftismaximized. b. Thecostperstudentisminimized.
a5. (Dietproblem)Auniversitydininghallwishestoprovideatleast5unitsofvitaminC and 3 units of vitamin E per serving. Three foods are available containing these vitamins. Food f1 contains 2.5 and 1.25 units per ounce of vitamins C and E, respectively, whereas food f2 contains just the opposite amounts. The third food f3 contains an equal amount of each vitamin at 1 unit per ounce. Food f1 costs 25c/ per ounce, food f2 costs 56c/ per ounce, and food f3 costs 10c/ per ounce. The dietitian wishes to provide the meal at a minimum cost per serving that satisfies the minimum vitamin requirements. Set up this linear programming problem in second primal form. Solve with the aid of a code from your computer program library.
6. Use built-in routines in mathematical software systems such as Matlab, Maple, or Mathematica to solve linear programming problem with equation number below in first primal form, in second primal form, and in dual form:
a. (2) b. (3) c. (4) d. (5) e. (6)
Simplex Method
The principal algorithm that is used in solving linear programming problems is the simplex method. Here, enough of the background of this method is described that the reader can use available computer programs that incorporate it.
Consider a linear programming problem in second primal form:
⎧
⎪⎨ maximize: cTx
􏰎
Ax = b x􏶟0
It is assumed that c and x are n-component vectors, b is an m-component vector, and A is an m × n matrix. Also, it is assumed that b􏶟 0 and that A contains an m × m identity
⎪⎩ constraints:

matrix in its last m columns. As before, we define the set of feasible points as K = {x ∈ Rn: Ax = b, x 􏶟 0}
The points of K are exactly the points that are competing to maximize cTx. Vertices in K and Linearly Independent Columns of A
The set K is a polyhedral set in Rn , and the algorithm to be described proceeds from vertex
to vertex in K, always increasing the value of cTx as it goes from one to another. Let us
give a precise definition of vertex. A point x in K is called a vertex if it is impossible to
expressitasx=1(u+v),withbothuandvinKandu≠ v.Inotherwords,xisnotthe 2
midpoint of any line segment whose endpoints lie in K .
We denote by a(1), a(2), . . . , a(n) the column vectors constituting the matrix A. The
following theorem relates the columns of A to the vertices of K :
everyindexi thatisnotinthesetI(x),wehavexi =0,ui 􏶟0,vi 􏶟0,andxi = 1(ui +vi). 2
17.2 Simplex Method 671
THEOREM ON VERTICES AND COLUMN VECTORS
Let x ∈ K and define I(x) = {i: xi > 0}. Then the following are equivalent:
1. x isavertexof K.
2. The set {a(i): i ∈ I(x)} is linearly independent.
■ THEOREM1
Proof
IfStatement1isfalse,thenwecanwritex=1(u+v),withu∈K,v∈K,andu≠ v.For 2
This forces ui and vi to be zero. Thus, all the nonzero components of u and v correspond to indices i in I(x). Since u and v belong to K,
and
Hence, we obtain
􏰃n 􏰃
b = Au = ui a(i) = ui a(i)
i=1 i∈I(x) 􏰃n 􏰃
b = Av = vi a(i) = vi a(i) i=1 i∈I(x)
􏰃 (ui −vi)a(i) =0 i∈I(x)
showing the linear dependence of the set {a(i): i ∈ I(x)}. Thus, Statement 2 is false. Con- sequently, Statement 2 implies Statement 1.
For the converse, assume that Statement 2 is false. From the linear dependence of
column vectors a(i) for i ∈ I(x), we have
􏰃
i∈I(x)
􏰃 (i) yi a
i∈I(x)
=0
with
|yi|≠ 0

672 Chapter 17
Linear Programming
for appropriate coefficients yi. For each i ∈/ I(x), let yi = 0. Form the vector y with components yi for i = 1,2,…,n. Then, for any λ, we see that because x ∈ K,
􏰃n 􏰃n 􏰃
A(x ±λy) = (xi ±λyi)a(i) = i=1
i=1
xi a(i) ±λ yi a(i) = Ax = b i∈I(x)
Now select the real number λ positive but so small that x +λy􏶟0 and x −λy􏶟0. [To
see that it is possible, consider separately the components for i ∈ I(x) and i ∈/ I(x).] The
resulting vectors, u = x + λy and v = x − λy, belong to K. They differ, and obviously,
x = 1 (u + v). Thus, x is not a vertex of K; that is, Statement 1 is false. So Statement 1 2
implies Statement 2. ■
Given a linear programming problem, there are three possibilities:
1. There are no feasible points; that is, the set K is empty.
2. K is not empty, and cTx is not bounded on K.
3. K is not empty, and cTx is bounded on K.
It is true (but not obvious) that in the third case, there is a point x in K such that cTx 􏶟 cT y for all y in K. We have assumed that our problem is in the second primal form so that possibility 1 cannot occur. Indeed, A contains an m × m identity matrix and so has the form
⎡a11 a12 ··· a1k 1 0 ··· 0⎤
⎢ a21 a22 · · · a2k 0 1 · · · 0 ⎥ A=⎢ . . .. . . . .. .⎥
⎣……..⎦ am1 am2 ··· amk 0 0 ··· 1
where k = n − m. Consequently, we can construct a feasible point x easily by setting x1 =x2 =···=xk =0andxk+1 =b1,xk+2 =b2,andsoon.Itisthenclearthat Ax=b. The inequality x 􏶟 0 follows from our initial assumption that b 􏶟 0.
Simplex Method
Next we present a brief outline of the simplex method for solving linear programming prob- lems. It involves a sequence of exchanges so that the trial solution proceeds systematically from one vertex to another in K. This procedure is stopped when the value of cTx is no longer increased as a result of the exchange.
The following is an outline of the simplex algorithm.

■ ALGORITHM1
Simplex
A few remarks on this algorithm are in order. In the beginning, select the indices k1,k2,…,km such that a(k1),a(k2),…,a(km) form an m × m identity matrix. At step 5, where we say that x is a solution, we mean that the vector v = (vi ) given by vki = xi for 1􏶞 i 􏶞 n and vi = 0 for i ∈/ {k1,k2,…,km} is the solution. A convenient choice for the tolerance ε that occurs in steps 5 and 7 might be 10−6.
In any reasonable implementation of the simplex method, advantage must be taken of the fact that succeeding occurrences of step 1 are very similar. In fact, only one column of B changes at a time. Similar remarks hold for steps 3 and 6.
We do not recommend that the reader attempt to program the simplex algorithm. Efficient codes, refined over many years of experience, are usually available in software libraries. Many of them can provide solutions to a given problem and to its dual with very little additional computing. Sometimes this feature can be exploited to decrease the execution time of a problem. To see why, consider a linear programming problem in first primal form:
⎧⎪ maximize: cTx ⎨􏰎
As usual, we assume that x is an n vector and that A is an m × n matrix. When the simplex algorithm is applied to this problem, it performs an iterative process on an m × m matrix denoted by B in the preceding description. If the number of inequality constraints m is very large relative to n, then the dual problem may be easier to solve, since the B matrices for it will be of dimension n × n. Indeed, the dual problem is
⎧⎪ minimize: bT y ⎨􏰎
(D) ⎪⎩constraints: ATy􏶟c y􏶟0
(P) ⎪⎩constraints:
x􏶟0
Ax 􏶞 b
17.2 Simplex Method 673
Select a small positive value for ε. In each step, we have a set of m indices {k1,k2,…,km}.
1. Put columns a(k1), a(k2), . . . , a(km ) into B, and solve Bx = b.
2. If xi > 0 for 1 􏶞 i 􏶞 m, continue. Otherwise, exit because the algorithm has
failed.
3. Sete=[ck1,ck2,…,ckm]T,andsolveBTy=e.
4. Choose any s in {1,2,…,n} but not in {k1,k2,…,km} for which cs − yTa(s) is greatest.
5. If cs − yTa(s) < ε, exit because x is the solution. 6. Solve Bz = a(s). 7. If zi 􏶞 ε for 1 􏶞 i 􏶞 m, then exit because the objective function is unbounded on K. 8. Among the ratios xi/zi that have zi > 0 for 1􏶞 i 􏶞 m, let xr/zr be the smallest. In case of a tie, let r be the first occurrence.
9. Replace kr by s, and go to step 1.

674 Chapter 17
Linear Programming
and the number of inequality constraints here is n. An example of this technique appears in the next section.
Summary
(1) For the second primal form, the set of feasible points is K = {x ∈ Rn: Ax = b, x 􏶟 0}
which are the points of K competing to maximize cTx.
(2) For a linear programming problem, there are these possibilities: There are no feasible points, that is, the set K is empty; K is not empty, and cTx is not bounded on K; K is not empty, and cTx is bounded on K.
(3) Denote by a(1), a(2), . . . , a(n) the column vectors constituting the matrix A. Let x ∈ K anddefineI(x)={i:xi >0}.ThenxisavertexofKifandonlyiftheset{a(i):i∈I(x)} is linearly independent.
(4) The simplex method involves a sequence of exchanges so that the trial solution proceeds systematically from one vertex to another in the set of feasible points K . This procedure is stopped when the value of cTx is no longer increased as a result of exchanges.
a1.
a 2. 3.
4. a5.
a 6. 7.
Showthatthelinearprogrammingproblem
􏰎
can be put into first primal form by increasing the number of variables by just one. Hint: Replace xj by yj − y0.
Show that the set K can have only a finite number of vertices.
Suppose that u and v are solution points for a linear programming problem and that
x = 1 (u + v). Show that x is also a solution. 2
Usingthesimplexmethodasdescribed,solvethenumericalexampleinthetext.
Using standard manipulations, put the dual problem (D) into first and second primal forms.
Show how a code for solving a linear programming problem in first primal form can be used to solve a system of n linear equations in n variables.
Using standard techniques, put the dual problem (D) into first primal form (P); then take the dual of it. What is the result?
maximize: cT x constraints: Ax 􏶞 b
Problems 17.2

⎧⎪ minimize: ⎪ ⎪ ⎪ ⎪
8×1 +6×2 +6×3 +9×4
⎪ ⎪⎨
+ x 4 􏶟 2
+ x4 􏶟 4 x3 + x4 􏶟 1 +x3 􏶟 1
a. ⎪ ⎪
⎧⎪ minimize: ⎪⎨
ab. ⎪ ⎪⎩constraints:
⎧⎪ maximize: ⎪⎨
ac. ⎪ ⎪constraints:
⎪⎩
⎩
⎪⎩x1􏶟0 x2􏶟0 x3􏶟0 x4􏶟0 10×1 −5×2 −4×3 +7×4 +x5
constraints:
⎪
⎪ x1
17.3 Approximate Solution of Inconsistent Linear Systems 675
1. Select a linear programming code from your computing center library and use it to solve these problems:
⎧⎪ x 1 + 2 x 2 ⎪ ⎪ ⎪ ⎪ ⎪⎨ 3×1 + x2
⎧⎪4×1−3×2− x3+4×4+ x5=1 ⎨
⎪⎩−x1 + 2×2 + 2×3 + x4 + 3×5 = 4 x1􏶟0 x2􏶟0 x3􏶟0 x4􏶟0 x5􏶟0
2×1 + 4×2 + 3×3
⎧⎪4×1 +2×2 +3×3 􏶞 15
⎨3×1+2×2+x3􏶞 7
⎪ x + x + 2x 􏶞 6 ⎪⎩1 2 3
x1􏶟0 x2􏶟0 x3􏶟0
2. (Student research project) Investigate recent developments in computational linear
programming algorithms, especially by interior-point methods.
17.3 Approximate Solution of Inconsistent Linear Systems
Linear programming can be used for the approximate solution of systems of linear equations that are inconsistent. An m × n system of equations
􏰃n
aijxj =bi (1􏶞i􏶞m)
j=1
is said to be inconsistent if there is no vector x = [x1,x2,…,xn]T that simultaneously
satisfies all m equations in the system. For instance, the system ⎧
⎪⎨ 2 x 1 + 3 x 2 = 4
⎪⎩ x1 − x2 = 2 (1)
x1 +2×2 =7
is inconsistent, as can be seen by attempting to carry out the Gaussian elimination process.
Computer Problems 17.2

676 Chapter 17
Linear Programming
l1 Problem
Since no vector x can solve an inconsistent system of equations, the residuals
􏰃n j=1
ri =
cannot be made to vanish simultaneously. Hence, 􏰍m |r | > 0. Now it is natural to ask for
⎪⎨
n+1 ⎪ 􏰃
(1􏶞i􏶞m) (1􏶞i􏶞m)
⎪ ⎪ ⎪
⎪ aijyj −εi 􏶞bi ⎪⎨j=1
(3)
⎪ ⎪ ⎩
n+1 ⎪ 􏰃
εi
⎧⎪ 􏰃n
aijxj −bi (1􏶞i􏶞m) 􏰍m i=1 i
an x vector that renders the expression i =1 |ri | as small as possible. This problem is called thel problemforthissystemofequations.Othercriteria,leadingtodifferentapproximate
1 􏰍m2 solutions, might be to minimize r or max
which special algorithms have been designed (see Barrodale and Roberts [1974]). However, if one of these special programs is not available or if the problem is small in scope, linear programming can be used.
􏰍m 2i=1 i the problem of minimizing r .
|r |. Chapter 12 discusses in detail 1􏶞i􏶞m i
􏰍ni=1 i
The minimization of i =1 |ri | by appropriate choice of the x vector is a problem for
A simple, direct restatement of the problem is
⎧⎪
⎪ minimize:
⎪ ⎪⎨
⎪
⎪ constraints: ⎪
⎪⎩
􏰃m i=1
⎪ ⎪⎨ aijxj−bi􏶞εi (1􏶞i􏶞m) j=1
⎪ 􏰃n
⎩− aijxj+bi􏶞εi (1􏶞i􏶞m)
j=1
(2)
If a linear programming code is at hand in which the variables are not required to be nonnegative, then it can be used on Problem (2). If the variables must be nonnegative, the following technique can be applied. Introduce a variable yn+1, and write xj = yj − yn+1. Then define ai,n+1 = − 􏰍nj =1 ai j . This step creates an additional column in the matrix A. Now consider the linear programming problem
⎧⎪
􏰃m
− εi
⎧i=1
⎪ ⎪ ⎪
maximize:
constraints:
⎪−
⎪ j=1
aijyj −εi 􏶞 −bi ⎩ y􏶟0 ε􏶟0
which is in first primal form with m + n + 1 variables and 2m inequality constraints.

that
17.3 Approximate Solution of Inconsistent Linear Systems 677 It is not hard to verify that Problem (3) is equivalent to Problem (2). The main point is
n+1 n 􏰃􏰃
aijyj = aij(xj +yn+1)+ai,n+1yn+1 j=1 j=1
⎧⎪
⎪ maximize:
􏰃m 􏰃m
ui − vi
⎪⎨
⎪ ⎪⎨􏰃
i = 1
aijyj −ui +vi =bi
n+1 ⎪ ⎪⎩ ⎪⎩ j = 1
(1􏶞i􏶞m)
⎪constraints:
Using the preceding formulas, we have
ri =
=
aijxj −bi =
aijyj −yn+1
aij(yj −yn+1)−bi aij −bi
= =
aijxj +yn+1 aij +yn+1 j=1
−
aij
􏶠 􏶡 􏰃n 􏰃n 􏰃n
j=1 􏰃n
j=1
ai j x j
Another technique can be used to replace the 2m inequality constraints in Problem (3)
j=1
by a set of m equality constraints. We write
εi =|ri|=ui +vi
whereui =ri andvi =0ifri􏶟0butvi =−ri andui =0ifri <0.Theresultinglinear programming problem is − ⎧ i = 1 u􏶟0 v􏶟0 y􏶟0 􏰃n j=1 􏰃n j=1 n+1 􏰃 = Fromit,weconcludethatr +v =u 􏶟0.Nowv andu shouldbeassmallaspossible, aijyj −bi =ui −vi 􏰃n j=1 􏰃n j=1 j=1 consistentwiththisrestriction,becauseweareattemptingtominimize i=1(ui +vi).Soif r 􏶟0,wetakev 􏶟0andu =r ,whereasifr <0,wetakev =−r andu =0.Ineither i iii􏰍mi iii􏰍m case, |ri | = ui + vi . Thus, minimizing i=1(ui + vi ) is the same as minimizing i=1 |ri |. The example of the inconsistent linear system given by (1) could be solved in the l1 sense by solving the linear programming problem ⎧⎪minimize: u1 +v1 +u2 +v2 +u3 +v3 ⎪⎨ ⎧⎪2y1 +3y2 −5y3 −u1 +v1 =4 ⎪ ⎨ y1− y2 −u2+v2=2 (4) ⎪constraints:⎪ y +2y −3y −u +v =7 ⎪⎩ ⎪⎩1 2 3 3 3 i i i i i 􏰍m y1, y2, y3 􏶟 0 u1,u2,u3 􏶟 0 v1,v2,v3 􏶟 0 678 Chapter 17 Linear Programming The solution is ⎧⎪ minimize: ⎪ ε ⎧⎪ 􏰃n ⎪⎨ aijxj −ε 􏶞bi ⎨ (1􏶞i􏶞m) (1􏶞i􏶞m) u1 = 0 v1 = 0 y1 = 2 u2 = 0 v2 = 0 y2 = 0 u3 = 0 v3 = 5 y3 = 0 From it, we recover the l1 solution of System (1) in the form x1 = y1 − y3 = 2 r1 = u1 − v1 = 0 x2 = y2 − y3 = 0 r2 = u2 − v2 = 0 r3 = u3 − v3 = −5 We can use mathematical software systems such as Matlab, Maple, or Mathematica to solve this linear programming problem. For example, we obtain u1 = v1 = u2 = v2 = u3 = y2 = y3 = 0, v3 = 5, and y1 = 2, with 5 as the value of the objective function. For another system, we need to set the equality constraints. We obtain the solution corresponding to y1 = y2 = y3 =684.2887,u1 =u2 =u3 =v1 =v2 =0,andv3 =5with5asthevalue of the objective function. The x vector is x1 = 2 and x2 = 3.1494 × 10−11. This solution is slightly different from the one previously obtained, owing to roundoff errors, but the minimum value for the objective function is the same and all the constraints are satisfied. l∞ Problem Consider again a system of m linear equations in n unknowns: 􏰃n aijxj =bi (1􏶞i􏶞m) j=1 If the system is inconsistent, we know that the residuals ri = 􏰍nj =1 ai j x j − bi cannot all be zero for any x vector. So the quantity ε = max1 􏶞 i 􏶞 m |ri | is positive. The problem of making ε a minimum is called the l∞ problem for the system of equations. An equivalent linear programming problem is j=1 ⎪ 􏰃n ⎪⎩− j=1 If a linear programming code is available in which the variables need not be greater than or equal to zero, then it can be used to solve the l∞ problem as formulated above. If the variables must be nonnegative, we first introduce a variable yn+1 so large that the quantities ⎪ constraints: ⎪ aijxj −ε 􏶞 −bi ⎪⎩ 17.3 Approximate Solution of Inconsistent Linear Systems 679 y j = x j + yn+1 are positive. Next, we solve the linear programming problem ⎧ ⎪ minimize: ⎪ ⎪ ⎪⎨ ⎪ ⎪ constraints: ⎪ ⎪ ⎩ ε ⎩ε􏶟0 yj􏶟0 Here, we have again defined ai,n+1 = − 􏰍nj =1 ai j . y1 = 8 From it, the l∞ solution of (1) is recovered as follows: ⎧ n+1 ⎪ 􏰃 aijyj −ε 􏶞bi aijyj −ε 􏶞 −bi (1􏶞i􏶞m) (1􏶞i􏶞m) (1􏶞j􏶞n+1) ⎪ ⎪⎨ j=1 (5) n+1 ⎪ 􏰃 ⎪ − ⎪ j=1 For our System (1), the solution that minimizes the quantity max{|2x1 + 3x2 − 4|,|x1 − x2 − 2|,|x1 + 2x2 − 7|} is obtained from the linear programming problem The solution is ⎧⎪ minimize: constraints: ε ⎧⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ 2y +3y −5y 1 2 3 −ε􏶞4 −ε􏶞2 −ε􏶞7 − ε 􏶞 −4 −ε􏶞−2 −ε􏶞−7 ⎪ ⎨ ⎪ ⎪ ⎪ ⎪⎨ ⎪ −2y1 − 3y2 + 5y3 ⎪ ⎪ ⎪ −y1 + y2 ⎪ −y1 −2y2 +3y3 ⎪⎩y1,y2,y3􏶟0 ε􏶟0 ⎪ ⎪⎩ y1−y2 y1 +2y2 −3y3 (6) y2 = 5 y3 = 0 ε = 25 939 x1 = y1 − y3 = 8 x2 = y2 − y3 − 5 93 We can use mathematical software systems such as Matlab, Maple, or Mathematica to solve the linear programming problem (6). For example, we obtain the solution y1 = 8 , 9 y2 = 5 , y3 = 0, and ε = 25 from two of these systems. But for one of the mathematical 39 systems, we obtain the solution corresponding to y1 = 1.0423 × 103, y2 = 1.0431 × 103. y = 1.0414 × 103, and ε = 2.778. We do obtain the same results as before (0.8889, 1.6667) ≈ 8 , 5 . 93 3 􏰉􏰈 680 Chapter 17 Linear Programming In problems like (6), m is often much larger than n. Thus, in accordance with remarks made in Section 17.2, it may be preferable to solve the dual problem because it would have 2m variables but only n + 2 inequality constraints. To illustrate, the dual of Problem (6) is ⎧⎪maximize: ⎪ 4u1 + 2u2 + 7u3 − 4u4 − 2u5 − 7u6 ⎧⎪ 2u1+u2+ u3−2u4−u5− u6 􏶟 0 ⎪⎨ 3u1 −u2 +2u3 −3u4 +u5 −2u6 􏶟 0 ⎪ −5u1 − 3u3 + 5u4 + 3u6 􏶟 0 ⎪ −u1−u2− u3− u4−u5− u6 􏶟 −1 ⎪⎩ u i 􏶟 0 ( 1 􏶞 i 􏶞 6 ) The three types of approximate solution that have been discussed (for an overdetermined system of linear equations) are useful in different situations. Broadly speaking, an l∞ solution is preferred when the data are known to be accurate. An l2 solution is preferred when the data are contaminated with errors that are believed to conform to the normal probability distribution. The l1 solution is often used when data are suspected of containing wild points—points that result from gross errors, such as the incorrect placement of a decimal point. Additional information can be found in Rice and White [1964]. The l2 problem is discussed in Chapter 12 also. Summary (1) We consider an inconsistent system of m linear equations in n unknowns 􏰃n aijxj =bi (1􏶞i􏶞m) j=1 For the residuals ri = 􏰍nj =1 ai j x j − bi , the l1 problem for this system is to minimize the ⎪⎨ ⎪ constraints: ⎪ ⎩ expression 􏰍m i=1 |ri |. A direct restatement of the problem is ⎧⎪ ⎪ minimize: ⎪ ⎪⎨ ⎪ ⎪ constraints: ⎪ ⎪⎩ 􏰃m εi i=1 ⎧⎪ 􏰃n a i j x j − b i 􏶞 ε i a i j x j + b i 􏶞 ε i ( 1 􏶞 i 􏶞 m ) ( 1 􏶞 i 􏶞 m ) ⎪ ⎪⎨ ⎪ 􏰃n ⎪⎩ − j=1 j=1 where εi = |ri |. If the variables must be nonnegative, we introduce a variable yn+1 and write xj = yj − yn+1. Define ai,n+1 = −􏰍nj=1 aij; an equivalent linear programming problem is ⎧⎪ minimize: ⎪ ε ⎧⎪ 􏰃n ⎪⎨ aijxj −ε 􏶞bi ⎨ (1􏶞i􏶞m) (1􏶞i􏶞m) ⎧ ⎪ minimize: ⎪ ⎪ ⎪⎨ ⎪ ⎪ constraints: ⎪ ⎪⎩ ε 17.3 Approximate Solution of Inconsistent Linear Systems 681 ⎧⎪ ⎪ maximize: ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ constraints: ⎪ ⎪ ⎩ 􏰃m − εi ⎧⎪ 􏰃m 􏰃m ε􏶟0 yj􏶟0 wherewedefinedai,n+1 =−􏰍nj=1aij. ⎧i=1 n+1 ⎪ 􏰃 ⎪ aijyj −εi 􏶞bi (1􏶞i􏶞m) (1􏶞i􏶞m) ⎪⎨j=1 n+1 ⎪ 􏰃 ⎪− ⎪ j=1 aijyj −εi 􏶞 −bi ⎩ y􏶟0 ε􏶟0 which is in first primal form with m + n + 1 variables and 2m inequality constraints. (2) Another technique is to replace the 2m inequality constraints by a set of m equality constraints.Wewriteεi =|ri|=ui +vi,whereui =ri andvi =0ifri 􏶟0butvi =−ri andui =0ifri <0.Theresultinglinearprogrammingproblemis ⎪ maximize: − ⎪⎨ ⎧ i = 1 n+1 ⎪ ⎪⎨􏰃 (3) For an inconsistent system, the problem of making ε = max1 􏶞 i 􏶞 m |ri | a minimum is the l∞ problem for the system. An equivalent linear programming problem is ⎪constraints: ⎪ ⎪⎩ ⎪⎩ j = 1 (1􏶞i􏶞m) ui − vi i = 1 aijyj −ui +vi =bi u􏶟0 v􏶟0 y􏶟0 j=1 ⎪ 􏰃n ⎪⎩− j=1 If the variables must be nonnegative, we introduce a large variable yn+1 so that the quantities yj = xj + yn+1 are positive and we have an equivalent linear programming problem: ⎪ constraints: ⎪ aijxj −ε 􏶞 −bi ⎪⎩ ⎧ ⎪ 􏰃 n+1 ⎪ aijyj−ε􏶞bi (1􏶞i􏶞m) ⎨ j=1 n+1 ⎪ 􏰃 ⎪ − ⎪⎩j=1 aijyj −ε 􏶞 −bi (1􏶞i􏶞m) (1􏶞j􏶞n+1) 682 Chapter 17 Linear Programming Additional References See Armstrong and Godfrey [1979], Barrodale and Phillips [1975], Barrodale and Roberts [1974], Bartels [1971], Bloomfield and Steiger [1983], Branham [1990], Ca ̈rtner [2006], Cooper and Steinberg [1974], Dantzi, Orden, and Wolfe [1963], Huard [1979], Nering and Tucker [1992], Orchard-Hays [1968], Rabinowitz [1968], Roos et al. [1997], Schrijver [1986], Wright [1997], Ye [1997], and Zhang [1995]. 1. Considertheinconsistentlinearsystem Problems 17.3 2. a 3. a 4. a a. The equivalent linear programming problem for solving the system in the l1 sense. ab. Theequivalentlinearprogrammingproblemforsolvingthesysteminthel∞sense. (Continuation) Repeat the preceding problem for the system ⎧⎪ ⎪ ⎪ 3 x + y = 7 ⎨ x− y= 11 ⎪⎩ x+6y= 13 −x + 3y = −12 We want to find a polynomial p of degree n that approximates a function f as well as possiblefrombelow;thatis,wewant0􏶞 f−p􏶞εforminimumε.Showhowpcould be obtained with reasonable precision by solving a linear programming problem. To solve the l1 problem for the system of equations ⎧ ⎪⎨ x − y = 4 ⎪⎩ 2 x − 3 y = 7 x+ y=2 we can solve a linear programming problem. What is it? ObtainnumericalanswersforPartsaandbofProblem17.3.1. (Continuation)RepeatforProblem17.3.2. 6x1 Write the following with nonnegative variables: ⎧⎪ ⎪ ⎪ 5x1 + 2x2 =6 ⎨ x1 + x2 + x3 = 2 ⎪⎩ 7x2 −5x3 =11 +9x3=9 Computer Problems 17.3 a1. 2. 17.3 Approximate Solution of Inconsistent Linear Systems 683 a3. Find a polynomial of degree 4 that represents the function ex in the following sense: Select 20 equally spaced points xi in interval [0,1] and require the polynomial to minimize the expression max1 􏶞 i 􏶞 20 |exi − p(xi )|. Hint: This is the same as solving 20 equationsinfivevariablesinthel∞ sense.Theithequationis A+Bxi +Cxi2+Dxi3+ Exi4 = exi , and the unknowns are A, B, C, D, and E. 4. Use built-in routines in mathematical software systems such as Matlab, Maple, or Mathematica to solve the linear programming problem with the equation numbers below in first primal form, in second primal form, and the dual: a. (4) b. (6) A Advice on Good Programming Practices Because the programming of numerical schemes is essential to under- standing them, we offer here a few words of advice on good programming practices. A.1 Programming Suggestions The suggestions and techniques given here should be considered in context. They are not intended to be complete, and some good programming suggestions have been omitted to keep the discussion brief. Our purpose is to encourage the reader to be attentive to considerations of efficiency, economy, readability, and roundoff errors. Of course, some of these suggestions and admonitions may vary depending on the particular programming language that is being used and features in the language. Be Careful and Be Correct Strive to write programs carefully and correctly. This is of utmost importance. Use Pseudocode Before beginning the coding, write out in complete detail the mathemat- ical algorithm to be used in pseudocode such as that used in this text. The pseudocode serves as a bridge between the mathematics and the computer program. It need not be defined in a formal way, as is done for a computer language, but it should contain sufficient detail that the implementation is straightforward. When writing the pseudocode, use a style that is easy to read and understand. For maintainability, it should be easy for a person who is unfamiliar with the code to read it and understand what it does. Check and Double-Check Check the code thoroughly for errors and omissions before beginning to edit on a computer terminal. Spend time checking the code before running it to avoid executing the program, showing the output, discovering an error, correcting the error, and repeating the process ad nauseam.∗ ∗In 1962, the rocket carrying the Mariner I space probe to Venus went off course after only five minutes of flight and was destroyed. An investigation revealed that a single line of faulty Fortran code caused the disaster. A period was typed in the code DO 5 I=1,3 instead of the comma, resulting in the loop being executed once instead of three times. It has been estimated that this single typographical error cost the United States National Aeronautics and Space Administration $18.5 million dollars! For additional details, see material available online such as www-aix.gsi.de/∼giese/swr/mariner1.html and www-aix.gsi.de/∼giese/swr/ literatur1.html for a general reference. 684 Modern computing environments may allow the user to accomplish this process in only a few seconds, but this advice is still valid if for no other reason than that it is dangerously easy to write programs that may work on a simple test but not on a more complicated one. No function key or mouse can tell you what is wrong! Use Test Cases After writing the pseudocode, check and trace through it using pencil- and-paper calculations on a typical yet simple example. Checking boundary cases, such as the values of the first and second iterations in a loop and the processing of the first and last elements in a data structure, will often reveal embarrassing errors. These same sample cases can be used as the first set of test cases on the computer. Modularize Code Build a program in steps by writing and testing a series of segments (subprograms, procedures, or functions); that is, write self-contained subtasks as separate routines. Try to keep these program segments reasonably small, less than a page whenever possible, to make reading and debugging easier. Generalize Slightly If the code can be written to handle a slightly more general situation, then in many cases, it is worth the extra effort to do so. A program that was written for only a particular set of numbers must be completely rewritten for another set. For example, only a few additional statements are required to write a program with an arbitrary step size compared with a program in which the step size is fixed numerically. However, one should be careful not to introduce too much generality into the code because it can make a simple programming task overly complicated. Show Intermediate Results Print out or display intermediate results and diagnostic mes- sages to assist in debugging and understanding the program’s operation. Always echo-print the input data unless it is impractical to do so, such as with a large amount of data. Using the default read and print commands frees the programmer from errors associated with misalignment of data. Fancy output formats are not necessary, but some simple labeling of the output is recommended. Include Warning Messages A robust program always warns the user of a situation that it is not designed to handle. In general, write programs so that they are easy to debug when the inevitable bug appears. Use Meaningful Variable Names It is often helpful to assign meaningful names to the variables because they may have greater mnemonic value than single-letter variables. There is perennial confusion between the characters O (letter “oh”) and 0 (number zero) and between l (letter “ell”) and 1 (number one). Declare All Variables All variables should be listed in type declarations in each program or program segment. Implicit type assignments can be ignored when one writes declaration statements that include all variables used. Historically, in Fortran, variables beginning with I/i, J/j, K/k, L/l, M/m, and N/n are integer variables, and ones beginning with other letters are floating-point real variables. It may be a good idea to adhere to this scheme so that one can immediately recognize the type of a variable without looking it up in the type A.1 Programming Suggestions 685 686 Appendix A Advice on Good Programming Practices declarations. In this book, we present algorithms using pseudocode and therefore do not always follow this advice. Include Comments Comments within a routine are helpful for revealing at some later time what the program does. Extensive comments are not necessary, but we recommend that you include a preface to each program or program segment explaining the purpose, the input and output variables, and the algorithm used and that you provide a few comments between major segments of the code. Indent each block of code a consistent number of spaces to improve readability. Inserting blank comment lines and blank spaces can greatly improve the readability of the code as well. To save space, we have not included any comments in the pseudocode in this book. Use Clean Loops Never put unnecessary statements within loops. Move expressions and variables outside a loop from inside a loop if they do not depend on the loop or do not change. Also, indenting loops can add to the readability of the code, particularly for nested loops. Use a nonexecutable statement as the terminator of a loop so that the code may be altered easily. Declare Nonchanging Constants Use a parameter statement to assign the values of key constants. Parameter values correspond to constants that do not change throughout the routine. Such parameter statements are easy to change when one wants to rerun the program with different values. Also, they clarify the role key constants play in the code and make the routines more readable and easier to understand. Use Appropriate Data Structures Use data structures that are natural to the problem at hand. If the problem adapts more easily to a three-dimensional array than to several one-dimensional arrays, then a three-dimensional array should be used. Use Arrays of All Types The elements of arrays, whether one-, two-, or higher-dimensional, are usually stored in consecutive words of memory. Since the compiler may map the value of an index for two- and higher-subscripted arrays into a single subscript value that is used as a pointer to determine the location of elements in storage, the use of two- and higher- dimensional arrays can be considered a notational convenience for the user. However, any advantage in using only a one-dimensional array and performing complicated subscript calculation is slight. Such matters are best left to the compiler. Use Built-in Functions In scientific programming languages, many built-in mathematical functions are available for common functions such as sin, log, exp, arcsin, and so on. Also, numeric functions such as integer, real, complex, and imaginary are usually available for type conversion. One should utilize these and others as much as possible. Some of these intrinsic functions accept arguments of more than one type and return a result whose type may vary depending on the type of the argument used. Such functions are called generic functions, for they represent an entire family of related functions. Of course, care should be taken not to use the wrong argument type. Use Program Libraries In preference to one that you might write yourself for a pro- gramming project, a preprogrammed routine from a program library should be used when A.1 Programming Suggestions 687 applicable. Such routines can be expected to be state-of-the-art software, well tested, and, of course, completely debugged. Do Not Overoptimize Students should be primarily concerned with writing readable code that correctly computes the desired results. There are any number of tricks of the trade for making code run faster or more efficiently. Save them for use later on in your program- ming career. We are primarily concerned with understanding and testing various numerical methods. Do not sacrifice the clarity of a program in an effort to make the code run faster. Clarity of code may be preferable to optimization of code when the two criteria conflict. Case Studies We present some case studies that may be helpful. Computing Sums When a long list of floating-point numbers is added in the computer, there will generally be less roundoff error if the numbers are added in order of increasing magnitude. (Roundoff errors are discussed in detail in Chapter 2.) Mathematical Constants Some students are surprised to learn that in many programming languages, the computer does not automatically know the values of common mathematical constants such as π and e and must be explicitly told their values. Since it is easy to mistype a long sequence of digits in a mathematical constant, such as the real number π, pi ← 3.14159 26535 89793 the use of simple calculations involving mathematical functions is recommended. For ex- ample, the real numbers π and e can be easily and safely entered with nearly full machine precision by using standard intrinsic functions such as pi ← 4.0 arctan(1.0) e ← exp(1.0) Another reason for this advice is to avoid the problem that arises if one uses a short approx- imation such as pi ← 3.14159 on a computer with limited precision but later moves the code to another computer that has more precision. If you overlook changing this assignment statement, then all results that depend on this value will be less accurate than they should be. Exponents In coding for the computer, exercise some care in writing statements that in- volve exponents. The general function xy is computed on many computers as exp(y ln x) whenever y is not an integer. Sometimes this is unnecessarily complicated and may con- tribute to roundoff errors. For example, it is preferable to write code with integer exponents such as 5 rather than 5.0. Similarly, using exponents such as 1 or 0.5 is not recommended 2 because the built-in function sqrt may be used. There is rarely any need for a calculation such as j ← (−1)k because there are better ways of obtaining the same result. For example, in a loop, we can write j ← 1 before the loop and j ← − j inside the loop. Avoid Mixed Mode In general, one should avoid mixing real and integer expressions in the computer code. Mixed expressions are formulas in which variables and constants of 688 Appendix A Advice on Good Programming Practices different types appear together. If the floating-point form of an integer variable is needed, use a function such as real. Similarly, a function such as integer is generally available for obtaining the integer part of a real variable. In other words, use the intrinsic type conversion functions whenever converting from complex to real, real to integer, or vice versa. For example, in floating-point calculations, m/n should be coded as real(m)/real(n) when m and n are integer variables so that it computes the correct real value of m/n. Similarly, 1/m should be coded as 1.0/real(m) and 1/2 as 0.5 and so on. Precision In the usual mode of representing numbers in a computer, one word of storage is used for each number. This mode of representation is called single precision. In calculations that require greater precision (called double precision or extended precision), it is possible to allot two or more words of storage to each number. On a 32-bit computer, approximately seven decimal places of precision can be obtained in single precision, and approximately 17 decimal places of precision can be obtained in double precision. Double precision is usually more time-consuming than single precision because it may use software rather than hardware to carry out the arithmetic. However, if more accuracy is needed than single precision can provide, then double or extended precision should be used. This is particularly true on computers with limited precision, such as a 32-bit computer, on which roundoff errors can quickly accumulate in long computations and reduce the accuracy to only three or four decimal places! (This topic is discussed in Chapter 2.) Usually, two words of memory are used to store the real and imaginary parts of a complex number. Complex variables and arrays must be explicitly declared as being of complex type. Expressions involving variables and constants of complex type are evaluated according to the normal rules of complex arithmetic. Intrinsic functions such as complex, real, and imaginary should be used to convert between real and complex types. Memory Fetches When using loops, write the code so that fetches are made from adjacent words in memory. To illustrate, suppose we want to store values in a two-dimensional array (ai j ) in which the elements of each column are stored in consecutive memory locations. Using i and j loops with the i th loop as the innermost one would process elements down the columns. For some programs and computer languages, this detail may be of only secondary concern. However, some computers have immediate access to only a portion or a few pages of memory at a time. In this case, it is advantageous to process the elements of an array so that they are taken from or stored in adjacent memory locations. When to Avoid Arrays Although the mathematical description of an algorithm may indi- cate that a sequence of values is computed, thus seeming to imply the need for an array, it is often possible to avoid arrays. (This is especially true if only the final value of a sequence is required.) For example, the theoretical description of Newton’s method (Chapter 3) reads xn+1=xn− f(xn) f′(xn) but the pseudocode can be written within a loop simply as for n = 1 to 10 do x ← x − f(x)/f′(x) end for where x is a real variable and function procedures for f and f ′ have been written. Such an assignment statement automatically effects the replacement of the value of the old x with the new numerical value of x − f (x)/f ′(x). Limit Iterations In a repetitive algorithm, one should always limit the number of permis- sible steps by the use of a loop with a control variable. This will prevent endless cycling due to unforeseen problems (e.g., programming errors and roundoff errors). For example, in Newton’s method above, one might write If the function involves some erratic behavior, there is a danger here in not limiting the number of repetitions. It is better to use a loop with a control variable: where n and n max are integer variables and the value of n max is an upper bound on the number of desired repetitions. All others are real variables. Floating-Point Equality The sequence of steps in a routine should not depend on whether two floating-point numbers are equal. Instead, reasonable tolerances should be permitted to allow for floating-point arithmetic roundoff errors. For example, a suitable branching statement for n decimal digits of accuracy might be if |x − y| < ε then . . . end if provided that it is known that x and y have magnitude comparable to 1. Here, x, y, and ε are real variables with ε = 1 × 10−n . This corresponds to requiring that the absolute error 2 between x and y be less than ε. However, if x and y have very large or small orders of magni- tude, then the relative error between x and y would be needed, as in the branching statement if |x − y| < εmax{|x|,|y|} then ... end if Equal Floating-Point Steps In some situations, notably in solving differential equations (see Chapter 8), a variable t assumes a succession of values equally spaced a distance of h apart along the real line. One way of coding this is A.1 Programming Suggestions 689 d← f(x)/f′(x) while |d| > 1 × 10−6 do
2
x←x−d output x
d← f(x)/f′(x)
end while
for n = 1 to n max do d← f(x)/f′(x)
x←x−d
output n, x
if|d|􏶞 1 ×10−6 thenexitloop 2
end for

690 Appendix A Advice on Good Programming Practices
t ← t0
output 0, t
for i.= 1 to n do
.
t←t+h
output i, t end for
Here, i and n are integer variables, and t0, t, and h are real variables. An alternative way is
In the first pseudocode, n additions occur, each with possible roundoff error. In the second, this situation is avoided but at the added cost of n multiplications. Which is better depends on the particular situation at hand.
Function Evaluations When values of a function at arbitrary points are needed in a program, several ways of coding this are available. For example, suppose values of the function
f (x) = 2x + ln x − sin x
are needed. A simple approach is to use an assignment statement such as
y ← 2x + ln(x) − sin(x)
at appropriate places within the program. Here, x and y are real variables. Equivalently, an
internal function procedure corresponding to the pseudocode f (x) ← 2x + ln(x) − sin(x)
could be evaluated at 2.5 by
y ← f (2.5)
or whatever value of x is desired. Finally, a function subprogram can be used such as in the
following pseudocode:
Which implementation is best? It depends on the situation at hand. The assignment state- ment is simple and safe. An internal or external function procedure can be used to avoid
for i.= 0 to n do .
t ← t0 + real(i)h
output i, t end for
real function f (x) real x
f ←2x+ln(x)−sin(x) end function f

duplicating code. A separate external function subprogram is the best way to avoid diffi- culties that inadvertently occur when someone must insert code into another’s program. In using program library routines, the user may be required to furnish an external function procedure to communicate function values to the library routine. If the external function procedure f is passed as an argument in another procedure, then a special interface must be used to designate it as an external function.
On Developing Mathematical Software
Fred Krogh [2003] has written a paper listing some of the things he has learned from a career at the Jet Propulsion Laboratory involving the development and writing of mathematical software used in application packages. Some of his helpful hints and random thoughts to remember in code development are as follows: Include internal output in order to see what your algorithm is doing; support debugging by including output at the interfaces; provide detailed error messages; fine-tune your code; provide understandable test cases; verify results with care; take advantage of your mistakes; keep units consistent; test the extremes; the algorithm matters; work on what does work; toss out what does not work; do not give up too soon on ideas for improving or debugging your code; your subconscious is a powerful tool, so learn to use it; test your assumptions; in the comments, keep a dictionary of variables in alphabetical order because it is quite helpful when looking at a code years after it was written; write the user documentation first; know what performance you should expect to get; do not pay too much, but just enough, attention to others; see setbacks as learning opportunities and as the staircase for keeping one’s spirits up; when comparing codes, do not change their features or capabilities in order to make the comparison fair, since you may not fully understand the other person’s code; keep action lists; categorize code features; organize things into groups; the organization of the code may be one of the most important decisions the developer makes; isolate the linear algebra parts of the code in an application package so that the user may make modifications to them; reverse communication is a helpful feature that allows users to leave the code and carry out matrix- vector operations using their own data structures; save and restore variables when the user is allowed to leave the code and return; portability is more important than efficiency. This is just a random sampling of some of the items in this paper.
A.1 Programming Suggestions 691

B
Representation of Numbers in Different Bases
In this appendix, we review some basic concepts on number representation in different bases.
B.1 Representation of Numbers in Different Bases
We begin with a discussion of general number representation but move quickly to bases 2, 8, and 16, as they are the bases primarily used in computer arithmetic.
The familiar decimal notation for numbers uses the digits 0, 1, 2, 3, 4, 5, 6, 7, 8, and 9. When we write a whole number such as 37294, the individual digits represent coefficients of powers of 10 as follows:
37294 = 4+90+200+7000+30000
= 4×100 +9×101 +2×102 +7×103 +3×104
Thus, in general, a string of digits represents a number according to the formula anan−1…a2a1a0 =a0 ×100 +a1 ×101 +···+an−1 ×10n−1 +an ×10n
This takes care of only the positive whole numbers. A number between 0 and 1 is represented by a string of digits to the right of a decimal point. For example, we see that
0.7215 = 7 + 2 + 1 + 5 10 100 1000 10000
= 7×10−1 +2×10−2 +1×10−3 +5×10−4 In general, we have the formula
0.b1b2b3…=b1 ×10−1 +b2 ×10−2 +b3 ×10−3 +···
Note that there can be an infinite string of digits to the right of the decimal point; indeed, there must be an infinite string to represent some numbers. For example, we note that
√
2 = 1.41421 35623 73095 04880 16887 24209 69 . . .
e = 2.71828 18284 59045 23536 02874 71352 66 . . . π = 3.14159 26535 89793 23846 26433 83279 50 . . . ln 2 = 0.69314 71805 59945 30941 72321 21458 17 . . .
1 =0.33333333333333333333333333333333… 3
692

For a real number of the form
(anan−1 …a1a0.b1b2b3 …)10 =
􏰃n
ak10k +
􏰃∞
bk10−k
B.1 Representation of Numbers in Different Bases 693
k=1
second summation. If ambiguity can arise, a number represented in base β is signified by
enclosing it in parentheses and adding a subscript β. Base β Numbers
The foregoing discussion pertains to the usual representation of numbers with base 10. Other bases are also used, especially in computers. For example, the binary system uses 2 as the base, the octal system uses 8, and the hexadecimal system uses 16.
In the octal representation of a number, the digits that are used are 0, 1, 2, 3, 4, 5, 6, and 7. Thus, we see that
(21467)8 = 7+6×8+4×82 +1×83 +2×84 = 7+8(6+8(4+8(1+8(2))))
= 9015
A number between 0 and 1, expressed in octal, is represented with combinations of 8−1,
8−2, and so on. For example, we have
(0.36207)8 = 3×8−1 +6×8−2 +2×8−3 +0×8−4 +7×8−5 = 8−5(3×84 +6×83 +2×82 +7)
= 8−5(7 + 82(2 + 8(6 + 8(3)))) = 15495
32768
= 0.47286 987 . . .
We shall see presently how to convert easily to decimal form without having to find a com- mon denominator.
If we use another base, say, β, then numbers represented in the β-system look like this:
k=0
the integer part is the first summation in the expansion and the fractional part is the
􏰃n (anan−1 …a1a0.b1b2b3 …)β =
akβk +
􏰃∞
bkβ−k
k=1
introduce symbols for 10, 11, . . . , β − 1. The separator between the integer and fractional
part is called the radix point, since decimal point is reserved for base-10 numbers. Conversion of Integer Parts
We now formalize the process of converting a number from one base to another. It is advisable to consider separately the integer and fractional parts of a number. Consider, then, a positive integer N in the number system with base γ :
􏰃n k=0
k=0
The digits are 0,1,…,β −2, and β −1 in this representation. If β > 10, it is necessary to
N=(anan−1…a1a0)γ =
akγk

694
Appendix B Representation of Numbers in Different Bases
EXAMPLE 1
Solution
Convert the decimal number 3781 to binary form using the division algorithm.
As was indicated above, we divide repeatedly by 2, saving the remainders along the way. Here is the work:
Quotients Remainders
2 ) 3781
2)1890 1=c0 ↓ ̇
2)945 0=c1 2)472 1=c2 2)236 0=c3 2)118 0=c4
2)59 0=c5 2)29 1=c6 2)14 1=c7
2)7 0=c8 2)3 1=c9 2)1 1=c10
Suppose that we wish to convert this to the number system with base β and that the calcu- lations are to be performed in arithmetic with base β. Write N in its nested form:
N = a0 +γ(a1 +γ(a2 +···+γ(an−1 +γ(an))···))
and then replace each of the numbers on the right by its representation in base β. Next, carry out the calculations in β-arithmetic. The replacement of the ak’s and γ by equivalent base-β numbers requires a table showing how each of the numbers 0, 1, . . . , γ − 1 appears in the β-system. Moreover, a base-β multiplication table may be required.
To illustrate this procedure, consider the conversion of the decimal number 3781 to binary form. Using the decimal binary equivalences and longhand multiplication in base 2, we have
(3781)10 =1+10(8+10(7+10(3)))
= (1)2 + (1 010)2 ((1 000)2 + (1 010)2 ((111)2 + (1 010)2(11)2)) = (111 011 000 101)2
This arithmetic calculation in binary is easy for a computer that operates in binary but tedious for humans.
Another procedure should be used for hand calculations. Write down an equation containingthedigitsc0,c1,…,cm thatweseek:
N =(cmcm−1…c1c0)β =c0 +β(c1 +β(c2 +···+β(cm)···))
Next, observe that if N is divided by β, then the remainder in this division is c0, and the
quotient is
If this number is divided by β, the remainder is c1, and so on. Thus, we divide repeatedly
c1 +β(c2 +···+β(cm)···) byβ,savingremaindersc0,c1,…,cm andquotients.
0 1=c11

EXAMPLE 2
Solution
B.1 Representation of Numbers in Different Bases 695 Here, the symbol ↓ ̇ is used to remind us that the digits ci are obtained beginning with the
digit next to the binary point. Thus, we have
(3781.)10 =(111011000101.)2
and not the other way around: (101 000 110 111.)2 = (2615)10. ■ Convert the number N = (111 011 000 101)2 to decimal form by nested multiplication.
N = 1×20 +0×21 +1×22 +0×23 +0×24 +0×25 +1×26 +1×27 +0×28 +1×29 +1×210 +1×211
= 1+2(0+2(1+2(0+2(0+2(0+2(1+2(1+2(0 + 2(1 + 2(1 + 2(1)))))))))))
= 3781
The nested multiplication with repeated multiplication and addition can be carried out on a
hand-held calculator more easily than can the previous form with exponentiation. ■
Another conversion problem exists in going from an integer in base γ to an integer in base β when using calculations in base γ . As before, the unknown coefficients in the equation
N = c0 +c1β +c2β2 +···+cmβm
are determined by a process of successive division, and this arithmetic is carried out in the γ -system. At the end, the numbers ck are in base γ , and a table of γ -β equivalents is used. For example, we can convert a binary integer into decimal form by repeated division by (1 010)2 [which equals (10)10 ], carrying out the operations in binary. A table of binary- decimal equivalents is used at the final step. However, since binary division is easy only for
computers, we shall develop alternative procedures presently.
Conversion of Fractional Parts
We can convert a fractional number such as (0.372)10 to binary by using a direct yet naive approach as follows:
(0.372)10 = 3×10−1 +7×10−2 +2×10−3 1􏰔 1􏰔 1 􏰕􏰕
=10 3+10 7+10(2)
1􏰔 1􏰔 1 􏰕􏰕
= (1 010) (011)2 + (1 010) (111)2 + (1 010) (010)2 222
Dividing in binary arithmetic is not straightforward, so we look for easier ways of doing this conversion.
Suppose that x is in the range 0 < x < 1 and that the digits ck in the representation x= are to be determined. Observe that 􏰃∞ k=1 ckβ−k =(0.c1c2c3...)β βx = (c1.c2c3c4 . . .)β 696 Appendix B Representation of Numbers in Different Bases EXAMPLE 3 Solution etc. In this algorithm, the arithmetic is carried out in the decimal system. Use the preceding algorithm to convert the decimal number x = (0.372)10 to binary form. The algorithm consists in repeatedly multiplying by 2 and removing the integer parts. Here because it is necessary to shift the radix point only when multiplying by base β. Thus, the unknown digit c1 can be described as the integer part of βx. It is denoted by I(βx). The fractional part, (0.c2c3c4 ...)β, is denoted by F(βx). The process is repeated in the following pattern: is the work: d0 = x d1 =F(βd0) d2 = F(βd1) c1 =I(βd0) ↓ ̇ c2 = I(βd1) 0.372 2 ↓ ̇ c1=.744 2 c2 = .488 2 c3 = .976 2 c4 = .952 2 c5 = .904 2 c6 = .808 etc. 0 1 0 1 1 Thus, we have (0.372)10 = (0.010 111 . . .)2 . Base Conversion 10 ↔ 8 ↔ 2 ■ 1 Most computers use the binary system (base 2) for their internal representation of numbers. The octal system (base 8) is particularly useful in converting from the decimal system (base 10) to the binary system and vice versa. With base 8, the positional values of the numbers are80 =1,81 =8,82 =64,83 =512,84 =4096,andsoon.Thus,forexample,wehave and (26031)8 = 2×84 +6×83 +0×82 +3×8+1 = ((((2)8+6)8+0)8+3)8+1 = 11289 (7152.46)8 = 7×83 +1×82 +5×8+2+4×8−1 +6×8−2 = (((7)8+1)8+5)8+2+8−2[(4)8+6] = 3690 + 38 64 = 3690.59375 EXAMPLE 4 Solution When numbers are converted between decimal and binary form by hand, it is convenient to use octal representation as an intermediate step. In the octal system, the base is 8, and, of course, the digits 8 and 9 are not used. Conversion between octal and decimal proceeds according to the principles already stated. Conversion between octal and binary is especially simple. Groups of three binary digits can be translated directly to octal according to the following table: Binary 000 001 010 011 100 101 110 111 Octal 01234567 This grouping starts at the binary point and proceeds in both directions. Thus, we have (101101001.110010100)2 =(551.624)8 To justify this convenient sleight of hand, we consider, for instance, a fraction expressed in binary form: x = (0.b1b2b3b4b5b6 . . .)2 = b12−1 + b22−2 + b32−3 + b42−4 + b52−5 + b62−6 + ··· =(4b1 +2b2 +b3)8−1 +(4b4 +2b5 +b6)8−2 +··· In the last line of this equation, the parentheses enclose numbers from the set {0, 1, 2, 3, 4, 5, 6, 7} because the bi ’s are either 0 or 1. Hence, this must be the octal representation of x . Conversion of an octal number to binary can be done in a similar manner but in reverse order. It is easy! Just replace each octal digit with the corresponding three binary digits. Thus, for example, (5362.74)8 =(101011110010.111100)2 What is (2576.35546 875)10 in octal and binary forms? We convert the original decimal number first to octal and then to binary. For the integer part, we repeatedly divide by 8: B.1 Representation of Numbers in Different Bases 697 Thus, we have 8 ) 2576 8 ) 322 0 ↓ ̇ 8 ) 40 2 8)50 05 2576.=(5020.)8 =(101000010000.)2 using the rules for grouping binary digits. For the fractional part, we repeatedly multiply by 8 ↓ ̇ 0.35546875 8 .84375000 8 .75000000 8 .00000000 2 6 6 698 Appendix B Representation of Numbers in Different Bases so that 0.35546875=(0.266)8 =(0.010110110)2 Finally, we obtain the result 2576.35546 875 = (101 000 010 000.010 110 110)2 Although this approach is longer for this example, we feel that it is easier, in general and less likely to lead to error because one is working with single-digit numbers most of the time. ■ Base 16 Some computers whose word lengths are multiples of 4 use the hexadecimal system (base 16) in which A, B, C, D, E, and F represent 10, 11, 12, 13, 14, and 15, respectively, as given in the following table of equivalences: Hexadecimal 0 1 2 3 4 5 6 7 Binary 0000 0001 0010 0011 0100 0101 0110 0111 Hexadecimal 8 9 A B C D E F Binary 1000 1001 1010 1011 1100 1101 1110 1111 Conversion between binary numbers and hexadecimal numbers is particularly easy. We need only regroup the binary digits from groups of three to groups of four. For example, we have (010101110101101)2 =(0010101110101101)2 =(2BAD)16 (111101011110010.110010011110)2 =(101011110010.110010011110)2 = (7AF2.C9E)16 More Examples Continuing with more examples, let us convert (0.276)8 , (0.C8)16 , and (492)10 into different number systems. We show one way for each number and invite the reader to work out the details for other ways and to verify the answers by converting them back into the original base. (0.276)8 = 2×8−1 +7×8−2 +6×8−3 = 8−3[((2)8 + 7)8 + 6] = (0.37109 375)10 (0.C8)16 =(0.110010)2 = (0.62)8 = 6×8−1 +2×8−2 = 8−2[(6)8 + 2] = (0.78125)10 and because Summary (492)10 = (754)8 = (111 101 100)2 = (1EC)16 8) 492 8) 61 4 ↓ ̇ 8) 7 5 07 B.1 Representation of Numbers in Different Bases 699 (1) It might seem that there are several different procedures for converting between number systems. Actually, there are only two basic techniques. The first procedure for converting the number (N)γ to base β can be outlined as follows: • Express(N)γ innestedformusingpowersofγ. • Replaceeachdigitbythecorrespondingbase-βnumbers. • Carryouttheindicatedarithmeticinbaseβ. This outline holds whether N is an integer or a fraction. The second procedure is either the divide-by-β and remainder-quotient-split process for N an integer or the multiply-by-β and integer-fraction-split process for N a fraction. The first procedure is preferred when γ < β and the second when γ > β. Of course, the 10 ↔ 8 ↔ 2 ↔ 16 base conversion procedure should be used whenever possible because it is the easiest way to convert numbers between the decimal, octal, binary, or hexadecimal systems.
1. Findthebinaryrepresentationandcheckbyreconvertingtodecimalrepresentation. a a. e ≈ (2.718)10 b. 7 c. (592)10
8
2. Convertthefollowingdecimalnumberstooctalnumbers.
a. 27.1 b.12.34 c.3.14 d.23.58 e.75.232 a3. Converttohexadecimal,tooctal,andthentodecimal.
aa. (110111001.101011101)2 a4. Convertthefollowingnumbers:
f.57.321
b.(1001100101.01101)2 a. (100101101)2 = ( )8 = ( )10
b. (0.782)10 =( ac. (47)10 =(
)8 =( )8 =(
)2 )2
d. (0.47)10 =(
)8 =(
)2
Problems B.1

700 Appendix B
Representation of Numbers in Different Bases
5. a6.
7.
8. a 9.
10. 11.
a12. 13. 14.
h. (361.4)8 =( )2 =( )10
Convert (45653.127664)8 to binary and to decimal.
Convert (0.4)10 first to octal and then to binary. Check by converting directly to binary.
Prove that the decimal number 1 cannot be represented by a finite expansion in the 5
binary system.
Do you expect your computer to calculate 3 × 1 with infinite precision? What about
15. 16.
Explain the algorithm for converting an integer in base 10 to one in base 2, assuming that the calculations will be performed in binary arithmetic. Illustrate by converting (479)10 to binary.
Justify mathematically the conversion between binary and hexadecimal numbers by regrouping.
Justifyforintegerstherulegivenfortheconversionbetweenoctalandbinarynumbers. Provethatarealnumberhasafiniterepresentationinthebinarynumbersystemifand
only if it is of the form ±m/2n, where n and m are positive integers.
Prove that any number that has a finite representation in the binary system must have
a finite representation in the decimal system.
SomecountriesmeasuretemperatureinFahrenheit(F),whileothercountriesuseCel- sius (C). Similarly, for distance, some use miles and others use kilometers. As a frequent traveler, you may be in need of a quick approximate conversion scheme that you can do in your head.
a. Fahrenheit and Celsius are related by the equation F = 32 + (9/5)C. Verify the following simple conversion scheme for going from Celsius to Fahrenheit: A rough approximation is to double the Celsius temperature and add 32. To refine your approximation, shift the decimal place to the left in the doubled number (2C) and subtract it from the approximation obtained previously: F = [(2C ) + 32] − (2C )/10.
b. Determine a simple scheme to convert from Fahrenheit to Celsius.
c. Determine a simple scheme to convert from miles to kilometers.
d. Determine a simple scheme to convert from kilometers to miles.
Convert fractions such as 1 and 1 into their binary represention. 3 11
(Mayanarithmetic)TheMayacivilizationofCentralAmerica(2600B.C.to1200A.D.) understood the concept of zero hundreds of years before many other civilizations. For their calculations, the vigesimal (base 20) system was used, not the decimal (base 10) system. So instead of 1, 10, 100, 1000, 10000, they used 1, 20, 400, 8000, 16000. They used a dot for 1 and a bar for 5, and zero was represented by the shell symbol. For
ae. (51)10 =( )8 =( )2 f. (0.694)10 =( )8 =(
)2
ag. (110011.1110101101101)2 = ( )8 = ( )10
2 × 1 or 10 × 1 ? 2 10
3

B.1 Representation of Numbers in Different Bases 701 example, the calculations 11131 + 7520 = 18651 and 11131 − 7520 = 3611 was as
follows:
8000s
400s
20s 1s
11131 7520
18651 3611
Here, as an aid, some of our numbers are included; on the left, they indicate the powers used, and above, they are the numbers represented by the columns.
Do these calculations using Mayan symbols and arithmetic:
a. 92819+56313=149132, 92819−56313=36506
b. 3296+853 = 4149, 3296−853 = 2443
c. 2273+729 = 1544, 2273−729 = 1544
d. Investigate how the Mayans might have done multiplication and division in their number system. Work out some simple examples.
17. (Babylonianarithmetic)BabyloniansofancientMesopotania(nowIraq)usedasex- agesimal (base 60) positional number system with a decimal (base 10) system within it. The Babylonians based their number system on only two symbols! The influence of Babylonian arithmetic is still with us today. An hour consists of 60 minutes and is divided into 60 seconds, and a circle is measured in divisions of 360 degrees. Numbers are frequently called digits, from the Latin word for “finger.” The base-10 and base-20 systems most likely arose from the fact that ten fingers and ten toes could be used in counting. Investigate the early history of numbers and doing aritmetic calculations in different number systems.
1. Read into your computer x = 1.1 (base 10), and print it out using several different formats. Explain the results.
√
2. Show that eπ
3. Writeandtestaroutineforconvertingintegersintooctalandbinaryforms.
163 is incredibly close to being the 18-digit integer 262 53741 26407 68744. Hint: More than 30 decimal digits will be needed to see any difference.
Computer Problems B.1

702 Appendix B
Representation of Numbers in Different Bases
4. (Continuation)Writeandtestaroutineforconvertingdecimalfractionsintooctaland binary forms.
5. (Continuation) Using the two routines of the preceding problems, write and test a program that reads in decimal numbers and prints out the decimal, octal, and binary representations of these numbers.
6. Seehowmanybinarydigitsyourcomputerhasfor(0.1)10.Seetheintroductoryremarks at the beginning of this chapter.
7. Somemathematicalsoftwaresystemshavecommandsforconvertingnumbersbetween binary, decimal, hex, octal, and vice versa. Explore these commands using various numerical values. Also, see whether there are commands for determining the precision (the number of significant decimal digits in a number) and the accuracy (the number of significant decimal digits to the right of the decimal point in a number).
8. Write a computer program to verify the conclusions in evaluating f (x) = x − sin x for various values of x near 1.9, say, over the interval [0.1, 2.5] with increments of 0.1. For these values, compute the approximate value of f , the true calculated value, and the absolute error between them. Single-precision and double-precision computations may be necessary.

C
Additional Details on IEEE Floating-Point Arithmetic
In this appendix, we summarize some additional features in IEEE standard floating-point arithmetic. (See Overton [2001] for additional details.)
C.1 More on IEEE Standard Floating-Point Arithmetic
In the early 1980s, a working committee of the Institute for Electrical and Electronics Engineers (IEEE) established a standard floating-point arithmetic system for computers that is now known as the IEEE floating-point standard. Previously, manufacturers of different computers each developed their own internal floating-point number systems. This led to inconsistencies in numerical results in moving code from machine to machine, for example, in porting source code from an IBM computer to a Cray machine. Some impor- tant requirements for all machines adopting the IEEE floating-point standard include the following:
• Correctlyroundedarithmetic
• Consistentrepresentationoffloating-pointnumbersacrossmachines • Consistent and sensible treatment of exceptional situations
Suppose that we are using a 32-bit computer with IEEE standard floating-point arith- metic. There are exactly 23 bits of precision in the fraction field in a single-precision normalized number. By counting the hidden bit, this means that there are 24 bits in the sig- nificand and the unit roundoff error is u = 2−24. In single precision, the machine epsilon is εsingle = 2−23 because 1 + 2−23 is the first single-precision number larger than 1. Since 2−23 ≈ 1.19 × 10−7, we can expect only approximately six accurate decimal digits in the output. This accuracy may be reduced further by errors of various types, such as roundoff errors in the arithmetic, truncation errors in the formulas used, and so on.
For example, when computing the single-precision approximation to π, we obtain six accurate digits: 3.14159. Converting and printing the 24-bit binary number result in an actual decimal number with more than six nonzero digits, but only the first six digits are considered accurate approximations to π.
The first double-precision number larger than 1 is 1 + 2−52. So the double-precision machineepsilonisεdouble =2−52.Since2−52 ≈2.22×10−16,thereareonlyapproximately 15 accurate decimal digits in the output in the absence of errors. The fraction field has exactly 52 bits of precision, and this results in 53 bits in the significand when the hidden bit is counted.
703

704 Appendix C Additional Details on IEEE Floating-Point Arithmetic
For example, when approximating π in double precision, we obtain 15 accurate digits: 3.14159 26535 8979. As in the case with single precision, converting and printing the 54-bit binary significand results in more than 15 digits, but only the first 15 digits are accurate approximations to π.
There are some useful special numbers in the IEEE standard. Instead of terminating with an overflow when dividing a nonzero number by 0, the machine representation for ∞ is stored, which is the mathematically sensible thing to do. Because of the hidden bit representation, a special technique for storing zero is necessary. Note that all zeros in the fraction field (mantissa) represent the significand 1.0 rather than 0.0. Moreover, there are two different representations for the same number zero, namely, +0 and −0. On the other hand, there are two different representations for infinity that correspond to two quite different numbers, +∞ and −∞. NaN stands for Not a Number and is an error pattern rather than a number.
Is it possible to represent numbers smaller than the smallest normalized floating-point number 2−126 in IEEE standard floating-point format? Yes! If the exponent field contains a bit string of all zeros and the fraction field contains a nonzero bit string, then this representation is called a subnormal number. Subnormal numbers cannot be normalized because this would result in an exponent that does not fit into the exponent field. These subnormal numbers are less accurate than normal numbers because they have less room in the fraction field for nonzero bits.
By using various system inquiry functions (such as those in Table C.1 from Fortran 90), we can determine some of the characteristics of the floating-point number system on a typical PC with 32-bit IEEE standard floating-point arithmetic. Table C.2 contains the results. In most cases, simple programs can also be written to determine these values.
In Table C.3, we show the relationship between the exponent field and the possible single-precision 32-bit floating-points numbers corresponding to it. In this table, all lines except the first and the last are normalized floating-point numbers. The first line shows that zero is represented by +0 when all bits bi = 0, and by −0 when all bits are zero except b1 = 1. The last line shows that +∞ and −∞ have bit strings of all ones in the exponent field except for possibly the sign bit together with all zeros in the mantissa field.
TABLE C.1
Some Numeric Inquiry Functions in Fortran 90
EPSILON(X)
TINY(X)
HUGE(X)
PRECISION(X)
Machine epsilon (number almost negligible compared to 1) Smallest positive number
Largest number
Decimal precision (number of significant decimal digits in output)
TABLE C.2
Results with IEEE Standard Floating-Point on 32-Bit Machine
EPSILON(X)
TINY(X)
HUGE(X)
PRECISION(X)
X Single Precision 1.192 × 10−7 ≈ 2−23
1.175 × 10−38 ≈ 2−126 3.403 × 1038 ≈ 2128 6
X Double Precision 2.220 × 10−16 ≈ 2−52
2.225 × 10−308 ≈ (2 − 2−23) × 2127 1.798 × 10308 ≈ 21024
15

C.1 More on IEEE Standard Floating-Point Arithmetic 705
TABLE C.3 Single-Precision 32-Bit Word withSignBitb1 =0for+andb1 =1for−.
(b2 b3 . . . b9 )2 Exponent Field (00000000)2 = (0)10
(00000001)2 = (1)10 (00000010)2 = (2)10 (00000011)2 = (3)10 (00000100)2 = (4)10
.
(01111101)2 = (125)10 (01111110)2 = (126)10 (01111111)2 = (127)10 (10000000)2 = (128)10 (10000001)2 = (129)10
.
(11111011)2 = (251)10 (11111100)2 = (252)10 (11111101)2 = (253)10 (11111110)2 = (254)10
(11111111)2 = (255)10
Numerical Representation
±0, ifb10 =b11 =···=b32 =0
subnormal, otherwise ±(1.b10b11b12···b32)2×2−126 ±(1.b10b11b12···b32)2×2−125 ±(1.b10b11b12···b32)2×2−124 ±(1.b10b11b12···b32)2×2−123
.
±(1.b10b11b12 · · · b32)2 × 2−2
±(1.b10b11b12 · · · b32)2 × 2−1 ±(1.b10b11b12 · · · b32)2 × 20 ±(1.b10b11b12 · · · b32)2 × 21 ±(1.b10b11b12 · · · b32)2 × 22
.
±(1.b10b11b12 · · · b32)2 × 2124
±(1.b10b11b12 · · · b32)2 × 2125 ±(1.b10b11b12 · · · b32)2 × 2126 ±(1.b10b11b12 · · · b32)2 × 2127
􏰘
±∞, ifb10 =b11 =···=b32 =0 NaN, otherwise
􏰘
In the IEEE floating-point standard, the round to nearest or correctly rounded value of the real number x, denoted round(x), is defined as follows. First, let x+ be the closest floating-point number greater than x, and let x− be the closest one less than x. If x is a floating-point number, then round(x) = x. Otherwise, the value of round(x) depends on the rounding mode selected:
• Round to nearest: round(x) is either x− or x+, whichever is nearer to x. (If there is a tie, choose the one with the least significant bit equal to 0.)
• Round toward 0: round(x) is either x− or x+, whichever is between 0 and x. • Round toward −∞/round down: round(x) = x−.
• Round toward +∞/round up: round(x) = x+.
Round to nearest is almost always used, since it is the most useful and gives the floating-point number closest to x.
b1 b2b3b4 ···b9 b10b11 ···b32

D
Linear Algebra Concepts and Notation
In this appendix, we review some basic concepts and standard notation used in linear algebra.
D.1 Elementary Concepts
The two concepts from linear algebra that we are most concerned with are vectors and matrices because of their usefulness in compressing complicated expressions into a compact notation. The vectors and matrices in this text are most often real, since they consist of real numbers. These concepts easily generalize to complex vectors and matrices.
Vectors
A vector x ∈ Rn can be thought of as a one-dimensional array of numbers and is written
as
⎡⎤
x1
⎢ x2 ⎥
x = ⎢⎣ . . . ⎥⎦ xn
where xi is called the ith element, entry, or component. An alternative notation that is useful in pseudocodes is x = (xi )n . Sometimes the vector x displayed above is said to be a column vector to distinguish it from a row vector y written as
y = [ y1 , y2 , . . . , yn ] For example, here are some vectors:
⎡1⎤
⎢ 35⎥ 􏰙1􏰚
⎢−5⎥ [π, e, 5,−4] 21
⎣6⎦ 3 2
7
To save space, a column vector x can be written as a row vector such as
706
x = [x1,x2,…,xn]T or xT = [x1,x2,…,xn]

D.1 Elementary Concepts 707 by adding a T (for transpose) to indicate that we are interchanging or transposing a row or
column vector. As an example, we have
[1 2 3 4]T =⎢2⎥
Many operations involving vectors are component-by-component operations. For vec-
tors x and y
⎡⎤ ⎡⎤
the following definitions apply.
Equality x=yifandonlyifxi =yi foralli(1􏶞i􏶞n) Inequality x j . Examples of identity, diagonal, tridiagonal, lower triangular, and upper triangular matrices, respectively, are as follows:
⎡1 0 0 0⎤ ⎡3 0 0 0⎤ ⎡⎢5 3 0 0 0⎤⎥ ⎢0 1 0 0⎥ ⎢0 5 0 0⎥ ⎢2 5 3 0 0⎥
⎣0010⎦ ⎣0070⎦ ⎢02920⎥
0001 0009 ⎣00372⎦ 00037
⎡⎤⎡⎤
6000 1−121 ⎢3 6 0 0⎥ ⎢0 5 −5 1⎥ ⎣4 −2 7 0⎦ ⎣0 0 9 −3⎦ 5−3921 0002
As with vectors, many operations involving matrices correspond to component opera- tions. For matrices A and B,
⎡a a ···a⎤ ⎡b b ···b⎤
11 12
⎢a21 a22 ···
11 12
⎢b21 b22 ···
Inequality A 0 for all nonzero vectors x.
2. ∥ax∥ = |a|∥x∥ for all vectors x and all scalars a. 3. ∥x+y∥􏶞∥x∥+∥y∥forallvectorsxandy.
■ PROPERTIES
Here, xi denotes the ith component of the vector. Any norm can be thought of as assigning a length to each vector. It is the Euclidean norm that corresponds directly to our usual concept of length, but other norms are sometimes much more convenient for our purposes. For example, if we know that ∥x − y∥∞ < 10−8, then we know that each component of x differs from the corresponding component of y by at most 10−8 and that the converse is also true. When we solve a system of linear equations Ax = b numerically, we shall want to know (among other things) how big the residual vector is. That is conveniently measured by ∥Ax − b∥, where some norm has been specified. For n × n matrices, we can also have matrix norms, subject to the following requirements: Properties of Matrix Norms We usually prefer matrix norms that are related to a vector norm. When a vector norm has been specified on Rn, there is a standard way of introducing a related matrix norm for n × n matrices, namely, ∥A∥=sup{∥Ax∥:x∈Rn, ∥x∥􏶞1} We say that this matrix norm is the subordinate norm to the given vector norm or the norm induced by the given vector norm. The close relationship between the two is useful, ∥x∥2 = (l1-vector norm) (Euclidean/l2-vector norm) (l∞-vector norm) 1.∥A∥>0ifA≠ 0
2. ∥αA∥ = |α|∥A∥
3. ∥A+B∥􏶞∥A∥+∥B∥ (triangularinequality)
for matrices A, B and scalars α.

722 Appendix D Linear Algebra Concepts and Notation
because it leads to the following inequality, which is true for all vectors x: ∥Ax∥􏶞 ∥A∥ ∥x∥
The matrix norms subordinate to the vector norms discussed above are, respectively,
∥A∥ = max 􏰍n |a | 1 1􏶞j􏶞n i=1 ij
(l -matrix norm) 1
(Spectral/l2-matrix norm) (l -matrix norm)
∥ A∥2 = max1􏶞k􏶞 n σk
∥A∥ = max 􏰍n |a ∞ 1􏶞i􏶞n j=1 ij
|
∞
Here, σk are the singular values of A. (Refer to Section 8.2 for definitions.) Note from above that the matrix norm subordinate to the Euclidean vector norm is not what most students think that it should be, namely,
􏰎􏰃n􏰃n 􏶬1/2
∥ A∥ = a2 (Frobenius norm)
Fij i=1 j=1
This is indeed a matrix norm; however, it is not the one induced by the Euclidean vector norm.
Gram-Schmidt Process
The projection operator is defined to be
projy x = ⟨x, y⟩ y
⟨y, y⟩
that projects the vector x orthogonally onto the vector y. The Gram-Schmidt process can
be written as
q = z1 1 ||z1||
q2= z2 ||z2||
q3 = z3 ||z3||
z =v, 1 1
z2=v2−projzv2, 1
z3 = v3 −projz v3 −projz v3, 1 2
In general, the k step is
zk = vk −
k−1
􏰃 zk
projvj vk, qk = ||zk||
j=1
Here {z1,z2,z3,…,zk} is an orthogonal set and {q1,q2,q3,…,qk} is an orthonormal
set. When implemented on a computer, the Gram-Schmidt process is numerically unstable because the vectors zk may not be exactly orthogonal due to roundoff errors. By a minor modification, the Gram-Schmidt process can be stabilized. Instead of computing the vectors uk as above, it can be computed a term at a time. A computer algorithm for the modified Gram-Schmidt process
for j = 1 to k
for i = 1 to j − 1
s ← ⟨v j , vi ⟩
vj ←vj −svi end for
vi ← vj/||vj|| end for

EXAMPLE 1
Solution
Herethevectorsv1,v2,…,vk arereplacedwithorthonormalvectorsthatspanthesame subspace.Thei-loopremovescomponentsinthevi directionfollowedbynormalizationof the vector. In exact arithmetic, this computation gives the same results as the original form above. However, it produces smaller errors in finite-precision computer arithmetic.
Consider the vectors v1 = (1,ε,0,0), v1 = (1,0,ε,0), and v1 = (1,0,0,ε). Assume ε is a small number. Carry out the standard Gram-Schmidt procedure and the modified Gram- Schmidt procedure. Check the orthogonality conditions of the resulting vectors.
Using the classical Gram-Schmidt process, we obtain u = (1, ε, 0, 0), u
=
√√
(0, −1, 1, 0)/ 2, and u
D.2 Abstract Vector Spaces 723
12
= (0, −1, 0, 1)/ 2. Using the modified Gram-Schmidt process,
3
√√
we find z1 = (1,ε,0,0), z2 = (0,−1,1,0)/ 2, and z3 = (0,−1,−1,2)/ 6. Checking
orthogonality, we find ⟨u2, u3⟩ = 1 and ⟨z2, z3⟩ = 0. ■
2

Answers for Selected Problems*
Problems 1.1
2. x = 6032 ; x = 6032 9990 10010
5a. sum←0
for i = 1 to n do
3. 6 × 10−5
4. Two other ways: pi ← 2.0 arcsin(1.0) or pi ← 2.0 arccos(0.0)
for j = 1 to n do
sum ← sum + ai j
end for end for
5d. sum ← 0.0
for i = 1 to n do
sum ← sum + aii end for
for j = 2 to n do
for i = j to n do
sum ← sum + ai,i− j+1 + ai− j+1,i end for
end for
6. n multiplications and n additions/subtractions
8a. fori =1to5do x←x·x
end for
p←x
8c. z←x+2
p ← z3 􏰉6 + z4 􏰉9 + z8 􏰉3 − z10􏰈􏰈􏰈
10. z←an/bn
fori =1ton−1do
z ← an−i (z + 1/bn−i ) end for
*Answers to problems marked in the text with the symbol a are given here and in the Student’s Solution Manual with more details. 724

11b. z←1 v←1
fori =1ton−1do v ← vx
z ← vz + 1 end for
for i = 1 to n do
aij ←1.0/real(i+j−1)
end for end for
Computer Problems 1.1
z ← vxz
12b. v=􏰍n aixi
12e. v=anxn +x􏰍n i=1
an−ixn−i
13. z=1+􏰍n i=2
􏶥i bj j=2
14. n(n+1)/2
i=0
15b. for j = 1 to n do
4. exp(1.0)≈2.71828182846
9. Computation deviates from theory when a1 = 10−12, 10−8, 10−4, 1020, for example. 10. x may underflow and be set to zero. 12. 40 different spellings
20a. The computation m/n may result in truncation so that x ≠ y.
Problems 1.2
4a. First derivative +∞ at 0. 4b. First derivative not continuous. 4e. Function −∞ at 0.
􏰔x2 􏰕 6a.ecosx =e 1− 2 +···
+···
9. Yes. By using this formula, we avoid the series for e−x and use the one for ex .
x2 x3
15a. sinx+cosx=1+x− − +···; sin(0.001)+cos(0.001)≈1.00099949983
26
􏰃∞x2k
(2k)!; cosh0.7≈1.25517
5. coshx =
6b. sin(cosx) = (sin1)−(cos1)
Answers for Selected Problems 725
k=0
􏰔x2 􏰕 2
7. m = 2 8. At least 18 terms 􏰃∞ x k 􏰒 1 + x 􏰓 􏰃∞ x 2 k − 1
k ; ln 1−x =2 (2k−1) k=1 k=1
11. ln(1−x)=−
12. x = 1 , ln 2 = 0.69313 (four terms); At least 10 terms.
3
15b. (sinx)(cosx)=x−2×3+ 2 x5− 4 x7+···; sin(0.0006)cos(0.0006)≈0.000599999857 3 15 315
􏰃∞ 1 􏰒 x 􏰓 n
16. ln(e + x) = 1 + (−1)n−1
ne
n=1 􏰉􏰈
17. At least seven terms.
24. s ← 0
for i = 2 to n do
s ← s + log(i)
output i, s end for
18. At least 100 terms.
20. − 5 h4 23. 1 x − 17 884
􏶨􏶨 􏰔 x2􏰕􏶨􏶨 1 1
28.􏶨􏶨cosx− 1− 􏶨􏶨< = 2 16×24 384 726 Answers for Selected Problems 32. Maclaurin series: f (x) = 3 + 7x − 1.33x2 + 19.2x4; (x −2)2 (x −2)3 (x −2)4 f (x) = 318.88 + (x − 2)616.08 + 918.94 + 921.6 + 460.8 2! 3! 4! 35. 400 terms. √ 􏰒π􏰓1􏰃∞ h2k 3􏰃∞ h2k−1 38. cos + h 3 = (−1)k + 2 (2k)! 2 (−1)k ; cos(60.001◦ ) ≈ 0.49998 488 (2k − 1)! k=0 39. sin(45.0005 )≈0.70711295 k=1 42. f(x−h)=(x−h) =x −mhx ◦ 47. n=16orn=17 50b. lim arctanx =1 50c. lim cosx+1 =0 mmm−1 h2m−2 +m(m−1) x +··· 􏰖 x→0 x 􏰗 x→π sinx 2x3x5x7 2! 51. Atleast38terms. 10 5 53. 10 54. 10 17. α50 = 2 81437 53123 52. erf(x)=√π x− 3 +5(2!)−7(3!)+··· ; erf(1)≈0.8382 Computer Problems 1.2 1. c = 1 c = 108 x1 0 −1 x2 −108 −108 14. g converges faster (in five iterations) Problems 2.1 1c. [B5000000]16 16. λ50 = 1 25862 69025 2d. [3FA 0000000000000]16; [BFA 0000000000000]16 4d. [3E7 00000]16,[3FCE 0000000000000]16 5d. −∞ 8a. −3.131968×106 8d. 9.992892×106 8g. −3.39×103 11c. m = −1,0,1. Nonnegative machine numbers: 0, 1, 1, 3, 1, 3, 1, 3 84824 2 15. 1 17. 1.00005; 1.0 18. |x| < 5 × 10−5 21. ≈3×2−25 25. ≈3×2−24 26. ≈2−22 19. β1−n 30. ≈n×2−24;n=1000,≈2−14 39. The relative error cannot exceed 5 × 2−24. 37. 1 × 10−12 rounding; 10−12 chopping 2 38. 9% 42. 􏰉􏰉q − 2−25􏰈2m, 􏰉q + 2−25􏰈2m􏰈 Problems 2.2 4. y = cos2 x 1+sinx 6. f(x)=−1x3 − 1x4; 2 2 f(0.0125)≈−9.888×10−7 8. f(x)= √ 1 1+x2 +1 10. f(x)= √ 1 x2 +1+x +3−1.7x2; ⎧􏰉√􏰈 ⎨ ln x+ x2+1 11.f(x)=⎩ 0􏰉 √2 x2 x3 16. f(x)≈1−x+ 3 − 6; f(0.008)≈0.992020915 20.arctanx−x≈x f(0)=3.5 x>0 􏰈x=0 −ln−x+x+1 x<0 x4 13.z=√x4+4+2 3􏰉 1 2􏰉1 −3+x 5+x 2􏰉 1􏰈􏰈􏰈 −7 Answers for Selected Problems 727 22. 􏰉e2x − 1􏰈􏶭2x ≈ 1 + x(1 + (x/3)(2 + x)) 24a. Near π/2, sine curve is relatively flat. 26b. ln x − 1 = ln(x/e) 26d. x−2(sin x − ex + 1) ≈ −1 − x when x → 0 √23 28. |x| < 6ε, where ε machine precision 29. x1 ≈ 105, x2 ≈ 10−5 30. Not much. Expect to compute b2 − 4ac in double precision. Computer Problems 2.2 1. No solution; (0, 0); (0, 0); Any solution; (−1., 0.); (−0.10208 42383, −4.89791 57617); 10. x 1 −1 0.5 −0.123 −25.5 −1776 3.14159 Series 1.0 2.71828 18285 0.36787 94412 1.64872 12707 0.88426 36626 8.42346 37545 0 (4.00000 00001, 4.0009 99999); (1.99683 77223, 2.00316 22777) (−0.10208 42383, −4.89791 57617); n (1.0000 00000, 1.00000 0000E34); Problems 3.1 1. 4. 0 1 10 10 8 5 × 10−12 25 25 17 23.14063 12270 14. |x| < 10−15 15. ρ50 = 2.85987 0.61906; 1.51213 􏶺π π 3π 5π 􏶻 − − δ, 0, + ε, + ε, + ε, . . . , where δ ≈ 0.2 and ε starts at approximately 0.4 and decreases. 4444 9. 12. 0,±π,±π,±3π,±2π,... 10. x =0 22 If the original interval is of width h, then after, say, k steps, we have reduced the interval containing the root to width h2−k. From then on, we add one bit at each step. About three steps are needed for each decimal digit. 20 steps 18b. This could be false because if r is close to bn then r − an ≈ bn − an = 2−n(b0 − a0). 􏰘􏶪 17. 18d. This is true because 0 􏶞 r − an (obvious) and r − an 􏶞 bn − an = 2−n (b0 − a0 ). 19e. True. 21. n􏶟23. 23. No;No. 19a. False in some cases. Computer Problems 3.1 10. 1,2,3,3−2i,3+2i,5+5i,5−5i,16 Problems 3.2 11. 2.365 3. xn+1 = 1[xn +1/(Rxn)] 1.6 7. y = √2 x + π 􏰓 4. 0.79; 11. xn+1 =2xn xn2R+1 ;−0.49985 1− 9. π √2 􏰒 √2 2 4􏰉 􏰈 2􏶭􏰉 􏰈 12a. Yes,−3 R. 13a. xn+1 = 1 2xn +R/xn2 3 728 Answers for Selected Problems 13c. x =x 􏰉x3+2R􏰈􏶭􏰉2x3+R􏰈 13e. x = xn 􏰉4R−x3􏰈 13g. x = R􏰉2x6+1􏰈􏶭􏰉2Rx3+1􏰈 n+1 n n n 15. x1 = 1 17. xn+1 = −1 n+1 3R n n+1 xn2 n n 19. |x0| < √3 27. x =􏰠(m−1)xm +R􏰡/􏰉mxm−1􏰈; x =x 􏰠(m+1)R−xm􏰡/(mR) 22 21. Newton’s method cycles if x0 ≠ 0. 22. x ← R for n = 1 to n max do x ← (2x + Rx2)/3 end for n+1 n n n+1n n 29. Diverges. 31. xn+1 = xn − f (xn) f ′(xn) ′􏶣′2 ′′ 32. xn+1 = xn − f (xn) + [f (xn)] −2f(xn)f (xn) [ f ′(xn)]2 − f (xn) f ′′(xn) 35. en+1 =en2⎢ ⎣ 1 2 f′′ 36. en+1 = 2en g (m − 1)! m! ′ ⎡ f′′(xn) f′′(xn) ⎤ f (m+1)(ξn) f (m+1)(ηn) − m! (m+1)(m−1)!⎥ f(m)(r) + en f(m+1)(ηn) ⎦ Computer Problems 3.2 37. |g(r)|<1if0<ω<2 41. 4thorder 4. 0.3279677853318183622377546 5. 2.094551481542326591482386540579 8. 1.8392867552 9. 0.47033169 10a. 1.8954942670340 10b. 1.9926668631307 10c. 0.51097 34293 8857 10d. 2.58280 14730 552 14. 3.13108; 3.15145 (two nearby roots) 17. x=4.510187 Problems 3.3 1.2.7385 12. en+1 = 1 − 3.−3 􏰔2 9.xn+1=xn− xn2−R 4.ln2 xn − x0 Computer Problems 3.3 1. −0.45896; 3.73308 6a. 1.53209 6b. 1.23618 7. Problems 4.1 1. p3(x)=7−2x+x3 3. l2(x)=−(x−4)(x2−1)/8 7a. p3(x)=2+(x+1)(−3+(x−1)(2+(x−3)(−11/24))) 􏰖 􏰕 􏰗 xn + xn−1 13a. Linear convergence f ′(ξn ) en 13c. Quadraticconvergence 15. Show|ξ−xn+1|􏶞c|ξ−xn|. f(xn)− f(x0) √ 16. 2 8. p4(x)=−1+(x−1)􏰉2 +(x−2)􏰉1 +(x−2.5)􏰉3 +(x−3)11􏰈􏰈􏰈 3846 1.36880 81078 21373 9. 20.80485 4 Answers for Selected Problems 729 9a. 0 1 8 13 1 223 0 35 493 12 83 9 14 7 1 6 259 9b. f(4.2)=104.49 13a. x3 − 3x2 + 2x − 1 12. q(x)=x4−x3+x2−x+1− 31(x+2)(x+1)(x)(x−1)(x−2) 120 14. p(x) = x − 2.5 16. α0 = 1 2 18. 2+x(−1+(x −1)(1−(x −3)x)) 19. p4(x)=−1+2(x+2)−(x+2)(x+1)+(x+2)(x+1)x; p2(x)=1+2(x+1)x 22. p(x) = 0.76(x − 1.73)(x − 1.82)(x − 5.22)(x − 8.26) 25. 1.5727; No advantage 27. p(x) = −3 x3 − 2 x2 + 1 28. 0.38099; 0.077848 55 39. 0.85527; 0.87006 40. Divisions: 1 (n − 1); additions/subtractions: n(n − 1) 2n 42. zero 45. False, only unique for polynomial p of degree 􏶞 n − 1. Computer Problems 4.1 1. p(x)=2+46(x−1)+89(x−1)(x−2)+6(x−1)(x−2)(x−3)+4(x−1)(x−2)(x−3)(x+4) Problems 4.2 1. f [x0, x1, x2, x3, x4] = 0 6. 1.25 × 10−5 7. 9. 4.105 × 10−14 (Thm 1), 1.1905 × 10−23 (Thm 2) Errors: 8.1 × 10−6, 10. 2.6 × 10−6 6.1 × 10−6 13. n 􏶟 7 8. 497 table entries n−1 􏶦 hn(2n)! |x−xi|􏶞 22nn! 16. Yes. Problems 4.3 14. i=0 1. −hf′′(ξ) 2. Errorterm=−hf′′(ξ)forξ ∈(0,2h) 4. Nosuchformulaexists. 6. The point ξ for the first Taylor series is such that ξ ∈ (x, x + h), while the second is ξ ∈ (x − h, x). Clearly, they are h2 (6) 9b. − 6 f (ξ) not the same. 2 2 ′′′ h2 (5) 8a. −3h f (ξ) 9a. − 4 f (ξ) h2 ′′′ h ′′ 11. α=1,errorterm=− f (ξ); α≠ 1,errorterm=−(α−1) f (ξ) 2􏰖6􏰗2􏰒􏰓 12.Errorterm=−h f′′′(ξ1)+1f(4)(ξ2) forsomeξi∈(x−h,x+h). 13. p′ x0+x1 = f(x1)−f(x0) 6 2􏰎 􏶬􏶰 2 x1−x0 􏰔h􏰕 􏰖 􏰔h􏰕􏰗2 􏰔h􏰕 􏰘 􏰔h􏰕 􏰔h􏰕􏶪 16. L≈2φ −φ(h) 20. L≈ φ −φ(h)φ 2φ −φ(h)−φ 2 2323 730 Answers for Selected Problems Computer Problems 4.3 3. 0.2021158503 Problems 5.1 1. 7 3. 0.00010000250006 6. n􏶟56738 7. U −L = 1[f(1)− f(0)] 18 n 11. L(f;P)􏶞M(f;P)􏶞U(f;P) Computer Problems 5.1 2. 0.94598385; 0.94723395 4. 4.84422 Problems 5.2 1. ≈ 0.70833 􏶩 1 dx 2. T(f;P)=0.775; x2 +1 ≈0.7854;Error=0.0104 0 6.h 2 1 1/2 1/4 L 0 0 1/2 3/4 U 2 2 3/2 5/4 T2111 7. n􏶟1607439; toosmall 􏰖􏰗 8. T = 1 1(n−1)(2n−1)n + 1 n3 6 2n 12. 0.000025 13. T( f ; P) ≈ 4.37132 14. T( f ; P) ≈ 0.43056 15. | error term |􏶞 0.3104 16. T(f;P)=7.125;No,theycannotbecomputedfromthegivendata. (b−a)h ′ (b−a)h2 ′′ 17a. =− f (ξ)forsomeξ ∈(a,b). 17b. =− f (ξ)forsomeξ ∈(a,b). 24 30. 􏶧b f(x)dx=h􏰍2n f(a+ih)+EwhereE=1(b−a)hf′(ξ)forξ∈(a,b) a i=0 2 Computer Problems 5.2 2a. 2 2b. 1.71828 2c. 0.43882 Problems 5.3 1. 13 3. −136 5. 4.267 7. Notwell. 15 8. R(1,1)= h{f(−h)+4f(0)+ f(h)} Simpson’srule 10. 1+2m−1 3 13. R(2,2)= 2h [7f(a)+32f(a+h)+12f(a+2h)+32f(a+3h)+7f(b)] 45 26 18a. 1 h3 f′′(ξ) 18b. 1 􏰍n h3 f′′(ξi) 18c. b−ah2 f′′(ξ) 24 24 i=1 i 24. f(x)=xn (n>3)on[0,1],withpartition{0,1}
􏶧b′
25. L􏶞 a f(x)dx􏶞T􏶞U 26. n􏶟1155 29. −(b−a)hf (ξ)/2

Answers for Selected Problems 731 􏰔􏰕􏰔􏰕􏰔􏰕
14. X=(27v−u)/26 15. Z=4096f h −1344f h + 84 f h − 1 f(h) 2835 8 2835 4 2835 2 2835
17. xn+1+n3(xn+1−xn)/(3n2+3n+1) 18. |I−R(n,m)|=O(h2m)ash→0
23. Show􏶧b f(x)dx−R(n,0)≈c4−(n+1). 24. Letm=1andletn→∞inFormula(2). a 􏰔2π􏰕2m
27. E = A2m(2π) 4 [±42m cos(4ξ)] ± (2π)2m+142m+1 A2m cos(4ξ) Computer Problems 5.3
1. R(7,7) = 0.499969819 5. R(5,0) = 1.813799364 6. 2 = 0.22222 … 9
7. 0.62135732 11. R(7,7)=0.765197687 Problems 6.1
1. π 2a. h<0.03orn>33.97. 4
3a. 7.1667 3b. 7.0833 3c. 7.0777 7. 􏶧b f(x)dx = 16S2(n−1) − 1 Sn−1
22. R(n+1,m+1)=R(n+1,m)+[R(n+1,m)−R(n,m)]/(8m−1)
2b. h<0.15orn>7.5.
􏶧2 dx −4
Problems 6.2
􏶢
4. 1 x = 0.6933; Bound is 5.2 × 10 . 8. − 3 h5 f(4)(ξ)
a 15 15 80
1. ≈ 0.91949 4a. x = ± 1 4b. x = ±0.861136, ±0.339981 3
5.α=γ=4, β=−2 6. A=(b−a), B=1(b−a)2 332
5h2hh 􏶢
7. f(a)+ f(a+h)− f(a+2h) 9. α= 5, a=c= 7 , b= 8
12312 72575 hh3 h3
10.w1=w2=2, w3=w4=−24 11. A=2h, B=0; C= 3 12. A=8, B=−4, C=8 Yes.Exactforpolynomialsofdegree􏶞3.
333
13. A = h , D = 0, C = h , B = 4 h 333
Computer Problems 6.2
2a. 1.4183 8a. 2.034805318577
Problems 7.1
14. True for n 􏶞 3
8b. 0.892979511569
8c.0.43398771
1. Homogeneous: α = 0, zero solution; α = ±1, infinite number of solutions
2. For α ≈ 1, erroneous answer is produced.
3a. No solution
3b. Infinite number of solutions
4.
􏰘􏰘
x1 = −697.3 x2 = 343.9
x1 = −720.79976 x2 = 356.28760
􏰛 −0.001343 􏰜
5. r= −0.001572 ,􏶤r= 0.0000000 ,e= −0.001 ,􏶤e=
6a. x2 =1, x1 =0 6b. x2 =1, x1 =1 6c. Letb1 =b2 =1.Thenx2 =1, x1 =0,whichisexact.
􏰛 −0.0000001 􏰜
􏰛 −0.001 􏰜
􏰛 −0.659 􏰜 0.913

732 Answers for Selected Problems
7a. x1=2, x2=1, x3=0 7b. x1=x2=x3=1
7c. x1 ≈ −7.233, x2 ≈ 1.133, x3 ≈ 2.433, x4 = 4.5 Computer Problems 7.1
6. z=[2i,i,i,i]T,λ=1+5i; z=[1,2,1,1]T,λ=2+6i; z=[−i,−i,0,−i]T,λ= −3−7i; z = [1, 1, 1, 0]T , λ = −4 − 8i
7a. (3.75, 90◦ ); (3.27, −65.7◦ ); (0.775, 172.9◦ ) 7b. (2.5, −90◦ );
Problems 7.2
⎡1/2 5/2 −4 −1⎤
1. ⎢⎣ 1/4 −1/2 −5/19 −62/19 ⎥⎦ 2. x = [1/3, 3, 1/3]T
3/4 9/10 38/5 9/10 4104
⎡1 0 3 0⎤ ⎡0 1 3−2⎤ ⎡0 1 3.⎢⎣0 1 3−1⎥⎦⇒⎢⎣0 1 3−1⎥⎦⇒⎢⎣0 0
(2.08, 56.3◦);
3−2⎤ 0 1⎥⎦
0 6 −2 −2
(1.55, −60.2◦)
3 −3 0 6 0 2 4 −6
⎡1/4 5/2 7/4 1/2⎤ 5.⎢⎣4 2 1 2⎥⎦
1/2 0 5/9 17/9 1/4 3/5 27/10 1/5
3 −3 0 6 0 2 4 −6
3 −3 0 0
6. l=(1,3,2),thesecondpivotrowisthethirdrow.
10. x4=−1, x3=0, x2=2, x1=1 13b. x3=1, x2=1, x1=1 13d. x1 ≈ 4.267, x2 ≈ −4.133, x3 ≈ −2.467 17. n(n + 1)
18. 􏰠29(n2 −1)+ 7 n(n−1)(2n−1)􏰡10−6 seconds
8. x3 =−1, x2 =1, x1 =0
10
19. n Time
30
10 102
1 × 10−3 sec 1 sec 33
103 5.56 min
104 3.86 days
0.005¢ 5¢
21. Solve these: U T y = b, L T x = y 23a. x1 = 5 , x2 = 2 , x3 = 1 × 10−9
Cost
$46.30
$46,296.30 999
Computer Problems 7.2
2. [3.4606,1.5610,−2.9342,−0.4301]T 3. [6.7831,3.5914,−6.4451,−1.5179]T
4. 2􏶞n􏶞10,xi ≈1foralli; forlargen,manyxi ≠ 1 6.x2=1, xi=0, fori≠ 2
Problems 7.3
5. bi =n2+2(i−1)
2a. 5n−4 3. n+2nk−k(k+1) 6. Yes,itdoes.
7. D−1AD=tridiagonal􏰠±√a c , d, ±√ac 􏰡 i−1 i−1 i i i

Computer Problems 7.3
􏰘􏰘
3.
xn ←bn
12.
di ←di −1/di−1
bi ←bi −bi−1/di−1 (2􏶞i􏶞n) xi ←(bi −xi+1)/xi
(n−1􏶟 j􏶟1) 􏰘 ⎪⎨⎛⎞􏶰
x =1 n−1 4.1 −1 11a. 􏰃
aijxj⎠ aii (1=n−1,…,1)
xi =1−(4xi−1) ⎧
⎪⎩di+1 ←di+1−ai+1ci bi+1 ←bi+1−ai+1bi
⎪⎩xi ←⎝bi −
⎪⎨ c i ← c i / d i bi ← bi/di
j=1
Problems8.1
5a.M=⎢⎣ 0 1 0 0⎥⎦ 0 −x/b 1 0
−w/a (xy)/(bc) ⎡a 0 0 0
5b.L′=⎢⎣0 b 0 0 0xc 0
−y/c 1 ⎤
⎥⎦
6c. L′=⎢⎣−1 −1
⎡ 0
15/4 0 0⎥⎦ −1/4 56/15 0
−1 −16/15 24/7 ⎤
⎡0 0 y d−(wz)/a
⎤
⎤ U = ⎢⎣ 0 15/4 −1/4 −1 ⎥⎦
0 0 56/15 −16/15 0 0 0 24/7
4000
(2􏶞i􏶞100)
(2􏶞i􏶞n)
2√000 ⎢−1/2 (1/2) 15 0 0⎥ 6d. L′′ =⎢⎣−1/2 −1/􏰉2√15􏰈 2√14/15 0⎥⎦
6e. 192
(1􏶞i􏶞n−1)
⎧⎪x1 ←b1/a11
􏰘
bn ← bn/dn
bi ←bi −cibi+1
⎡ ⎤ ⎡ ⎤ 􏰙100􏰚 􏰙303􏰚 1000 1002
1/3 −3 1 0 0 8 0 −3 1 0 −5 0 −2 1
1a.L= 0 10,U=0−13 2a.M=⎢⎣0 1 00⎥⎦ U=⎢⎣0300⎥⎦
⎡⎤⎡⎤
0 0 4 0 0 0 0 0
1 0 0 0 0 ⎢0 1 0 0 0⎥ 3a. M=⎢⎣ 0 −2 1 0 0⎥⎦
0 0 −2 1 0
⎡−40001⎤ 0⎡00020⎤
1000a00z
25 0 0 0 1 􏰙 ⎢027 4 3 2⎥
􏰚
321 U=⎢⎣ 0 0 50 −6 −4⎥⎦ 4b. A= 2 2 1
1000 4−1−10
6a. L = ⎢⎣ −1/4 1 0 −1/4 −1/15 1 0 −4/15 −2/7
0 ⎥⎦ 0
1
⎡4 0 0 0⎤ ⎡1−1/4−1/4 0⎤ 6b. D = ⎢⎣ 0 15/4 0 0 ⎥⎦ U ′ = ⎢⎣ 0 1 −1/15 −4/15 ⎥⎦
0 0 56/15 0 0 0 1 −2/7 ⎡0 0 024/7 ⎤ 0 0 0 1
0 0 0 0 0
1 1 1
U=⎢⎣0 b 0 0 ⎥⎦ 0 0 c 0
0 0 0 d − (wz)/a
⎡1 0 0 z/a⎤ U′=⎢⎣0 1 0 0 ⎥⎦
0010 0⎡0 0 1
Answers for Selected Problems 733
0 −2/􏰉√15􏰈 −(4/7)√14/15 2√6/7

734 Answers for Selected Problems ⎡1001⎤ ⎡1000⎤
8.U=⎢⎣0 1 0 −2⎥⎦ 0 0 1 4
0 0 0 −8
L=⎢⎣ 1 1 0 0⎥⎦ −1 1 1 0
1 −1 1 1
􏰙1 0 0􏰚
1 1 0 D=
3−11 003 001
9a. L=
10a.L= 2 1 0 , D= 0 1 0 , U′=
9b. x=[−1,2,1]T
􏰙1−1/2 1􏰚
0 1 1 10b.x=[−1,1,1]T 001 ⎤
0 0
l22u23 0 ⎥⎦
l32u23 +l33 l33u34
12. A−1 = 1 15
⎡ 16a. X−1 =⎢⎣
􏰙 11 −5 −7􏰚
l11 l11u12
14a. ⎢⎣l21 l21u12 +l22
􏰙1 0 0􏰚 􏰙−2 0 0􏰚 −131 0⎡0−1
−13 −8
10 5
11 1
0 l32 0 0⎡
l43 ⎤ l43u34 +l44 0 −1 −1 1
􏰙2 0 0􏰚 􏰙1−1/2 1􏰚 0 −2 0 U′= 0 1 −1/2
⎤
1 1 −1 1⎥⎦ −1 0 1 −1
0 0 0 1
1 0 0 −1
Computer Problems 8.1
⎡ 536 −668 3. Case 4: p5(A) = ⎢⎣−668 994
Problems 8.2
3. d. 5. e. 9. b. Problems 8.3
9. c. 11. d.
Computer Problems 8.3
458 −854
−186 ⎤ 458⎥⎦
458 −854 994 −668 −186 458 −668 536
16b. X−1 =⎢⎣−1 0 −1 1⎥⎦ −1 −1 0 1
1 1 1 −1
11. Eigenvalues/eigenvectors:1,(−1,1,0,0); 2,(0,0,−1,1); 5,(−1,1,2,2) Problems 8.4
1. a. Problems 9.1
1. Yes
6. In Problem 9.1.5, the bracketed expression is f ′(ξ1) − f ′(ξ2) and in magnitude does not exceed 2C.
10. 􏰍n f (t )S is a linear combination of 1st-degree spline functions having knots t , . . . , t . Hence, it is also such i=1ii 􏰍n 0n
a function. Its value at tj is i=1 f(ti)Si(tj) = f(tj). Si(x) = 0 if x < ti−1 or x > ti+1. On (ti−1,ti), Si(x) is given by (x − ti−1)/(ti − ti−1). On (ti , ti+1), Si (x) is given by (x − ti+1)/(ti − ti+1). S0 and Sn are slightly different.
9. Knots 􏶟 50π108 ≈ 1.57 × 1010.

12.
17. 19.
If S is piecewise quadratic, then clearly S′ is piecewise linear. If S is a quadratic spline then S ∈ C1. Hence, S′ ∈ C.
Hence, S′ is piecewise linear and continuous.
􏰎 2 􏰉 1􏰈2 􏰉 1􏰈 Q0(x)=−(x+1) +2, Q1(x)=−2x+1, Q2(x)=8 x−2 −2 x−2
Q3(x)=−5(x−1)2+6(x−1)+1, Q4(x)=12(x−2)2−4(x−2)+2
The answer is given by Equation (8). 20a. Yes 20b. No 20c. No 21. Yes
32. S0(x)=􏰉−5􏰈(x−1)3+􏰉12􏰈(x−1) 77
Answers for Selected Problems 735
Problems 9.2
1. No 2. No 4. a=−4, b=−6, c=−3, d=−1, e=−3 5. a=−5, b=−26, c=−27, d=27 6. No
8a. (m+1)n 8b. 2n 8c. (m−1)(n−1) 8d.m−1 ⎧
⎨x2 [0,1] 10. S=⎩1+2(x−1)+(x−1)2+(x−1)3 [1,2] 5+7(x −2)+4(x −2)2 [2,3]
12. a=3, b=3, c=1 13. No 15. a=−1, b=3,
2
7a. S(x) is not continuous at x = −1. S′′(x) is not continuous at x = −1, 1.
c=−2, d=2
No 26. S is linear.
S2(x) = 􏰉−5􏰈(x −3)3 +􏰉6􏰈(4−x)3 +􏰉12􏰈(x −3)−􏰉6􏰈(4−x) 􏰉7􏰈 􏰉7􏰈 7 7
S3(x)= −5 (5−x)3+ 12 (5−x) 77
33. The conditions on S make it an even function. If S(x) = S0(x) in [−1, 0] and S(x) = S1(x) in [0, 1], then S1(0) = 1, S′ (0) = 0, S′′(1) = 0, and S1(1) = 0. An easy calculation yields S1(x) = 1 − 3 x2 + 1 x3.
19. f is not a cubic spline 22. p3(x) = x − 0.0175×2 + 0.1927×3;
17. n+3
S1(x) = 􏰉6􏰈(x −2)3 −􏰉5􏰈(3−x)3 −􏰉6􏰈(x −2)+􏰉12􏰈(3−x) 7777
11 22 38. 5n,n+4 39. Yes
Problems 9.3
2. Chebyshev polynomials recurrence relation. See Section 12.2. ⎧⎪ ⎪ ( x − t i ) 2
⎪ (ti+2 − ti)(ti+1 − ti), on [ti,ti+1]
⎪⎨ (x−ti)(ti+2−x) + (ti+3−x)(x−ti+1) , on[ti+1,ti+2] 3. Bi2(x)=⎪(ti+2−ti)(ti+2−ti+1) (ti+3−ti+1)(ti+2−ti+1)
⎪ (ti+3 − x)2
⎪ (ti+3 − ti+1)(ti+3 − ti+2), on [ti+2,ti+3]
⎩ 0, elsewhere
5. 􏰍∞ f(ti)B0(x) 14. n−k􏶞i􏶞m−1
i=−∞ i
15. Use induction on k and Bk+i (x) = 0 on [ti , ti+1].
􏰍
19. ∞ dx i=1 i i=1 i
24. Use Equation (14) with all A’s zero except A j = 1. Next, take all A’s zero except A j +1 = 1. 28. No 30. Let Ci2 = ti+1ti+2, then Ci1 = xti+1, and Ci0 = x2.
32. Bik(tj)=0ifftj 􏶟ti+k+1 ortj 􏶞ti 33. x =(ti+3ti+2 −titi+1)/(ti+3 +ti+2 −ti+1 −ti)
17. No
20. In Equation (9), take all ci = 1. Then di = 0. Hence, d n Bk(x) = 0 and n Bk(x) is constant.
i+i
16. No
􏰍 􏰍i=−∞i
ti+1 B1(x)

736 Answers for Selected Problems
Computer Problems 9.3
7. 47040
Problems 10.1
1a. x=1t4+7t3−2t3/2+c 1b. x=cet 433
1e. x=c1et+c2e−t
or x=c1cosht+c2sinht
2a. x=1t3+3t4/3+7 34
2c.
5a.
f(t,x)=+ x/ 1−t2
real function f (t , x ) real t, x
f ←t2/(1−t+2x) end function f
3. x(−0.2)=1.92
􏰃∞
(−1)n
t2n+1
(2n + 1)(2n + 1)!
2 􏰛􏶧 2 􏰜 +c 3d. x=e−t /2 t2et /2dt+c
3c. x=
4. x =a0 +a0 n=1(−1) 2n−1(2n)! t + n=1(−1) (2n+1)!
n=0 􏰔􏰕 􏰔􏰕 􏰍∞ n (2n − 1)! 2n 􏰍∞ n−1 n!2n
t
2n+1
6. Letp(t)=a0+a1t+a2t2+···anddetermineai.
9. t =10,Error=2.2×104ε; t =20,Error=4.8×108ε
10. x(4) = 18xx′x′′ + 6(x′)3 + 3x2x′′′
11a. x′=x+ex;x′′=(1+ex)x′;x′′′=(1+ex)x′′+ex(x′)2;x(4)=(1+ex)x′′′+3exx′x′′+ex(x′)3. 12. x(0.1) = 1.21633
14. n←20 s ← x(n)
for i =1to n−1do
s ← x(n−i) + [h/real(n + 1 − i)]s
end for
s ← x + h[s]
Computer Problems 10.1
1. x(2.77)=385.79118 3. x(10) = 22026.47
7. x(1)=1.6487212691
Problems 10.2
2b. x(1.75)=0.632999983 4a. Error at t = 1 is 1.8 × 10−10.
2c. x(5)=−0.2087351554 5. x(0) = 0.03245 34427
􏶢
􏰉􏰈
9. x(0)=1.6798409205×10−3
10. x(0)=−3.7594073450
8. Solve df =e−x2, f(0)=0. 􏰒dx􏰓 3
2 2 2 10.h3 1−α D2f+h fxDf where D=∂+f∂ and D2=∂ +2f ∂ +f2∂
646 ∂t∂x∂t2∂x∂t∂x2 11.h= 1
1024
12. Let’s make local truncation error 􏶞 10−13. Thus, 100h5 􏶞 10−13 or h 􏶞 10−3. So take h = 10−3 and hope that the three
extra digits will be enough to preserve 10-digit precision.

(4)3 2 2 3∂3 ∂3 2∂3 3∂3 14b.x =D f+fxD f+3DfxDf+fx Df where D =∂t3+3f∂x∂t2+3f ∂t∂x2+f ∂x2
15. f(x+th,y+tk)= f(x,y)+t[f1(x,y)h+ f2(x,y)k]+(1/2)t2􏰠f11(x,y)h2 +2f12(x,y)hk+ f22(x,y)k2􏰡+···. Now let t = 1 to get the usual form of Taylor’s series in two variables.
17. Taylor series of f (x, y) = g(x) + h(y) about (a, b) is equal to the Taylor series of g(x) about a plus that of h(y) about b.
18. f(1+h,k)≈−3h+3h2+k2 19. e1−xy ≈3−x−y 20. A=1+k+1k2,B=h(1+k) 22
21. A=1, B=h−k, C=(h−k)2
22. f(x+h,y+k)≈􏰉1+2xh+k+􏰉1+2×2􏰈h2+2hkx+1k2􏰈f; f(0.001,0.998)≈2.7128534
2
Computer Problems 10.2
2. x (1) = 1.5708 3b. n = 7; x (2) = 0.82356 78972 (RK),
3c. n = 7; x (2) = −0.49999 99998 (RK), −0.50000 00012 (TS) 4. x (1) = 0.60653 = x (3) 5. x(3) = 1.5 6. x(0) = 1.0 = x(1.6) 8. x(1) = 3.95249 9. x(10) = 1.344 × 1043
Problems 10.3
1. Let h = 1/n. Then x(1) = e−1 (true solution) and xn = {[1 − 1/(2n)]/[1 + 1/(2n)]}n approximate solution. 2. x(t+h)=x(t−h)+h[f(t−h,x(t−h))+4f(t,x(t))+ f(t+h,x(t+h))]
Answers for Selected Problems 737
3
4.a=24, b=−11, c=2, d=10, e=−2h2 13 13 13 13 39
5.a=1,b=c=h; ErrortermisO(h3). 2
0.82356 78970 (TS)
8. ∂ x(9, s) = e252 ≈ 10109 9a. All t. ∂s
Computer Problems 10.3
5. x􏰉1􏰈=2.25 6. x􏰉−1􏰈=−4.5 22
12. 0.2193839244 13. 0.9953087432 Problems 11.1
9c. Positive t.
9e. No t.
11. Divergent for all t.
y(x)=[1−lnv(x)]v(x)
9. y(e)=−6.3890560989 15. Si(1)=0.9460830703
where
1. x(t+h)=x􏰉1+1h2+ 1 h4􏰈+y􏰉h+1h3+ 1 h5􏰈,y(t+h)=y􏰉1+1h2+ 1 h4􏰈+x􏰉h+1h3+ 1 h5􏰈 2 24 6 120 2 24 6 120
2. Since system is not coupled, solve two separate problems.
3. System is not coupled so each differential equation can be solved individually by the program.
⎡⎤
1
4. X′=⎣x12+logx2+x02 ⎦, X(0)=[0,1,3]T
ex2 −cosx1+sin(x0x1)−(x1x2)7 Computer Problems 11.1
1. x(1)=2.4686939399, y(1)=1.2873552872 2. x(0.38)=1.90723×1012, y(0.38)=−8.28807×104 􏰒π􏰓 􏰒π􏰓 􏰒π􏰓 􏰒π􏰓
4. x(−1)=3.36788, y(−1)=2.36788 5. x1 2 =x4 2 =0, x2 2 =1, x3 2 =−1 7. x(6) = 4.39411, y(6) = 3.10378

738 Answers for Selected Problems
Problems 11.2
􏰙x􏰚 2
1.X′= x3 X(0)=[1,−3,5]T 2×2 +logx3 +cosx1
3. Solve each equation separately since they are not coupled.
⎡x
2 􏰉
􏰈−3/2
⎤ ⎡0.5 ⎤
4.X′=⎢−x1 x12+x32
⎣x4􏰉􏰈⎦ 0.25
⎡⎤
1
⎢x2 ⎥ 5. X′ =⎢⎣x3 ⎥⎦
x4
x42 + cos(x2x3) − sin(x0x1) + log(x1/x0) ⎡⎤⎡⎤
x4 3
⎢x5 ⎥ ⎢3 ⎥
⎥ X(0)=⎢⎣0.75⎥⎦ − x 3 x 12 + x 32 − 3 / 2 1 . 0
X(0)=[0,1,3,4,5]T
2tx6 +2tex1x3 3 􏰙x􏰚
2 7a. Letx1=x,x2=x′,x3=x′′.ThenX′= x3
6. X′=⎢x6 ⎥ X(1)=⎢2 ⎥ ⎢2x1x3x4 +3x12x2t2⎥ ⎢−79/12⎥
⎣ex2x5 +4x1t2x3 ⎦ ⎣2 ⎦
􏰖􏰗
8.X′= x2
x2 − x1
X(0)=[0,1]T
−x3 sin x1 − t x2 − x3 ⎡⎤
1
⎢x3 ⎥ 9. Letx0=t,x1=x,x2=y,x3=x′,x4=y′.ThenX′=⎢⎣x4 ⎥⎦
X(0) = [0,1,3,2,4]T Problems 11.3
1. xj(t)=eλjtxj(0) Problems 12.1
1. y(x)=1
1 􏰃m
maxima exists—only a minima.
12. c = 10**􏰠(m + 1)−1 􏰍mk=0(yk − log xk)􏰡. 13. y = (6x − 5)/10 16. a ≈ 2.5929, b ≈ −0.32583, c ≈ 0.022738
x1 +x2 −2×3 +3×4 +logx0 2×1 −3×2 +5×3 +x0x2 −sinx0
yk=(y0+···+ym)/(m+1),theaverageoftheyvalueswhichdoesnotinvolveanyxi. 􏰠􏰍m 􏰡􏶭􏰠􏰍m 􏰡
2. f(x)=m+1
3. a = (1 + 2e)/(1 + 2e2), b = 1 5. a = 2.1, b = 0.9 7. c = k=0 yk log xk k=0 (log xk)2
k=0
11. φ involves the sum of m + 1 polynomials of degree two in c which is either concave upward or a constant. Thus, no

􏰙 􏰃m 18. a=1, b=1 19. y(x)=2×2+29 20. y=x+1 21. c=
k=0
􏰚 􏶰 􏰙 􏰃m exk f(xk)
k=0
􏰚
Problems 12.2
2. 3.
5. 6. 7.
8.
􏰎wn+2 =wn+1 =0
wk =ck+3xwk+1+2wk+2 (k=n,n−1,…,0)
3 735
e2xk
f (x) = w0 − (1 + 2x)w1
Since cos(n − 2)θ = cos[(n − 1)θ − θ ] = cos(n − 1)θ cos θ + sin(n − 1)θ sin θ , we have 2 cos θ cos(n − 1)θ −
cos(n − 2)θ = cos(n − 1)θ cos θ − sin(n − 1)θ sin θ = cos(n θ ). Note if gn (θ ) = cos n θ , then gn (θ ) = 2 cos θ gn −1 (θ ) − gn−2(θ).
By the previous problem, the recursive relation is the same as (2) so that Tn(x) = fn(x) = cos(n arccos x). Tn(Tm(x)) = cos(n arccos(cos(m arccos x))) = cos(nm arccos x) = Tnm(x).
|Tn(x)|=|cos(narccosx)|􏶞1forallx ∈[−1,1]since|cosy|􏶞1andforarccosx toexistx mustbe|x|􏶞1. 􏰎g0(x) = 1
g1(x) = (x + 1)/2
gj(x)=(x+1)gj−1(x)−gj−2(x) (j􏶟2)
10. n + 2 multiplications, 2n + 1 additions/subtractions if 2x is computed as x + x 12. n multiplications, 2n additions/subtractions
13. T6(x)=32×6 −48×4 +18×2 −1
y1x13 − y2x13
17. α = 2 1 . α is very sensitive to perturbations in y1.
Computer Problems 12.2
Answers for Selected Problems
739
x12x12(x2 − x1) 12
􏰎0 7.aij= (m+1)
(m + 1)/2 Problems 12.3
(i≠ j) (i=j=1) (i = j > 1)
2. Coefficient matrix for the normal equations has elements ai j = 1 i+j−1
3. c = 0 4. y = bx 6. c = ln 2 8. x = −1, y = 20 13
15.
16.
y≈ 1 .Changeto1≈a+bx.
􏰙a+bx 􏰚􏰙􏰚y⎡ 2π ⎤ π02a (1/2)(e−1)
0 π/2 2 0
0 b π/2 c
=⎣−(2/5)(e2π +1)⎦ (1/5)(e2π +1)
c=3
Computer Problems 12.3
17.
1.a=2, b=3
Problems 13.1
y sinx􏰡􏶭􏰠􏰍n (sinx)2􏰡
20. c=􏰠􏰍n
i=1i i i=1 i
1. l0 = 123456; x1 = .96621 2243; x2 = .12917 3003; x3 = .01065 6910
by (5). 9a. c = 24
9b. c = 3
14. No.
π3

740 Answers for Selected Problems
Computer Problems 13.1
8. 32.5% 11. Sequence not periodic. 0123456789
13.
97 93 97 107 90 115 88 101 113 99 15. 5.6% 16. 200
Problems 13.2
1. m > 4 million
Computer Problems 13.2
2. 1.71828 4. 8 5. 49.9 7. 0.518 9. 1.11 10. 2.000346869 14. 0.635 17b. 8.3
Computer Problems 13.3
1. 2 2. 0.898 4. 7 6. 1.05 3 16
14. 11.6 kilometers 15. 0.14758 Problems 14.1
7. 5.24 17. 0.009
9. 0.996 12. 0.6394
21. 24.2 revolutions 23. 0.6617
2. c1 =􏰉1−2e􏰈􏶭􏰉1−e2􏰈, c2 =􏰉2e−e2􏰈􏶭􏰉1−e2􏰈 3a. x(t)=􏰉eπ+t −eπ−t􏰈􏶭􏰉e2π −1􏰈 3b. x(t)=􏰉t4−25t+12􏰈􏶭12 4a. x(t)=βsint+αcostforall(α,β)
√
4b. x(t)=c1sint+αcostforallα+β=0withc1arbitrary 6. φ(t)=z 7. φ(z)=z 8. φ(z)= 9+6z
9. φ(z)=􏰉e5+e+ze4−z􏰈􏶭􏰉2e2􏰈 10. Twoways:Usex′′(a)=zorx′(b)=z,x′′(b)=w.
11. x(t)=−et +2ln(t+1)+3t
14a. This is a linear problem. So two initial-value problems can be solved as in the text to obtain the solution. The two 􏰘􏰘
sets of initial values would be x(0) = 0 and x(0) = 1 . x′(0) = 1 x′(0) = 0
15. Solutionofx′′ =−x,x(0)=1,x′(0)=zisx(t)=cost+zsint.Soφ(z)=x(π)=−1.Sinceφisconstant,we cannot get φ(z) = 3 by any choice of z!
2. x1 ≈0.29427, x2 ≈0.57016, x3 ≈0.81040
1′1
z1 = x (0) = B. So x1 = 3 cos t + z1 sin t, x2 = 3 cos t + z2 sin t. By Equation (10), x = λx1 + (1 − λ)x2 and
λ = [β − x2(b)]/[x1(b) − x2(b)] = [7 − (−3)]/[(−3) − (−3)] = 10/0.
Problems 14.2
􏰔􏰕 􏰔􏰕
1. − 1− h xi−1 +2(1+h2)x1 − 1− h xi+1 =−h2t 2 􏰠 2􏰡
4.x′(0)=5 8.−xi−1+2+(1+ti)h2 xi−xi+1=0 3
9. x(t) = [7/u(2)]u(t)
11. x′′ = −x1,x1(0) = 3,x′(0) = z1 implies x = Acost + Bsint,3 = x(0) = A,x′ = −Asint + Bcost. Let

Computer Problems 14.2
2a. x =1/(1+t) 2b. x =−log(1+t) Problems 15.1
1 ∂ 􏰒 ∂u􏰓 1 ∂2u
1a. Elliptic. 1c. Parabolic. 1f. Hyperbolic. 2. r ∂r r ∂r + r2 ∂θ2 = 0
4. Equation (3) shows that u(x, t + k) is a convex combination of values of u(x, t) in the interval [0, 1]. So it remains in the interval.
5. a=[1+2kh−2(cosπh−1)]1/k
6. The right-hand side is changed by b1 + c0 in place of b1 and bn−1 + cn replacing bn−1 for both (5) and (7).
7. In(6),b1 isreplacedbyb1 +g(t),bn−1 bybn−1 +g(t).Atthelevelzero,bi = f(ih)for1􏶞i􏶞n−1.
8. u(x,t+k)=h2(1−h)u(x+h,t)+h2 ⎡01 ⎤
⎢−1 0 1 ⎥
9. A = ⎢ … … … ⎥
⎣ −101⎦ −2 2
Problems 15.2
k +h−2 u(x,t)+h2u(x−h,t)
Answers for Selected Problems 741
k k􏰔h2 􏰕 k
1. −0.21 2. uxx = f ′′(x + at) + g′′(x − at), utt = a2 f ′′(x + at) + a2g′′(x − at) = a2uxx
3. u(x,t)=1[F(x+t)−F(−x+t)]+1􏰠G(x+t)−G(−x+t)􏰡whereGistheantiderivativeofG
Computer Problems 15.2
1. real function fbar(x) real x, xbar
xbar ← x + 2 real(integer(−(1 + x )/2)) if xbar < 0 then fbar ← − f (−xbar) else fbar ← f (xbar) end if end function fbar Problems 15.3 22 􏰔􏰕􏰔􏰕􏰔􏰕 5. 20+ 2.5h ui+1,j + 20− 2.5h ui−1,j + −30+0.5h ui,j+1+ xi +yj xi +yj yj 􏰔􏰕 −30+0.5h ui,j−1+20uij =69h2 yj ⎡−4 1 1 0⎤ 6. u􏰉0,1􏰈≈−8.932×10−3; u􏰉1,1􏰈≈4.643×10−1 7. A=⎢⎣ 1 −4 0 1⎥⎦ 2 2 2 1 0 −4 1 0 1 1 −4 742 Answers for Selected Problems Computer Problems 15.3 5. 18.41◦ 13.75◦ 41.47◦ 36.60◦ 24.41◦ 69.41◦ 66.77◦ 61.05◦ 53.01◦ 51.00◦ Problems 16.1 1. F(2,1,−2) = −15; F(0,0,−2) = −8; F(2,0,−2) = −12 2. F 􏰉9, 9􏰈 = −20.25 􏰘88 􏶤x =(3a+b)/4+δ if a􏶞x∗􏶞b′ 4. Casen=2: 􏶤x=(a+3b)/4−δ if a′􏶞x∗􏶞b 5a. ExactsolutionF(3)=−7. √√ 7.A=α/5, A=−β/5 9. By(6),y+rb=a+r2(b−a)+rb=ar+bsincer2+r=1.Moreover,r(y+rb)=a+r(b−a)=x.Thus, yr + r2b = x or y + r2(b − y) = x. 10. n􏶟1+(k +logl−log2)/|logr| 11. n􏶟48 13. Minimum point of F is a root of F′. Newton’s method to find root of F′: xn+1 = xn − F′(xn) . Formula does not involve F itself. 14. To find minimum of F, look for root of F′. Secant method to find root of F′ is F′′(xn) 􏰖􏰗 xn+1 = xn − F′(xn) xn − xn−1 . Formula does not involve F. F′(xn) − F′(xn−1) 2􏶣√ 15b. Squarebothsidestoobtainr =1+ 1+ 1+···=1+r. 15d. 1+r−1+r−2+···=(1−r−1)−1byseriesexpansion.Hence,r=(1−r−1)−1−1= Problems 16.2 1a. Yes 1b. No 2. 􏰉1,9􏰈 3. F(x,y)=1+x−xy+1x2−1y2+··· 4422 1 orr2=r+1. r−1 6.Theslopeofthetangentisdy=−Fx≡m1.ThegradienthasdirectionnumbersFxandFy,anditsslopeisFy ≡m2. dx Fy The condition of perpendicularity m1m2 = −1 is met. 7b. F(x)= 3 − 1x2 +3x1x2 +x2x3 +2x2 − 1x2 +··· 2212325 9a. G(1,0)= Fx 􏰙 􏰚 􏰖−2􏰗 5 9b. G(1,2,1)= 2 ⎡2y2z2 sinx cosx ⎤ 10. G=⎣2yz2(1+sin2x)+2(y+1)(z+3)2⎦ 2y2z(1+sin2 x)+2(y+1)2(z+3) Problems 17.1 2. maximize: −5x −6x +2x ⎧123 ⎪−2x1 + 3x2 􏶞 −5 ⎨ x1 + x2 􏶞 15 constraints: ⎪ 2x1 − x2 + x3 􏶞 25 ⎪ ⎪⎩ − x 1 − x 2 + x 3 􏶞 − 1 x1 􏶟 0, x2 􏶟 0, x3 􏶟 0 4a. Minimum value 1.5 at (1.5, 0). 12. 􏰉−19,−1􏰈 30 5 5b. maximize: −3x+2y−5z ⎧⎪⎨ − x − y − z 􏶞 − 4 constraints: x−y−z􏶞 2 ⎪⎩ − x + y + z 􏶞 − 2 x􏶟0, y􏶟0, z􏶟0 6a. maximize: 2x +2x −6x −x 6b. minimize: 25y +20y −5y ⎧1 2 3 7. Maximumof36at(2,6) 8. Minimumof36at(0,3,1) 13c. Unbounded solution y1, y2, y3, y4􏶟0 11. Minimum2for(x,x−2)wherex􏶟3 13f. No solution 13a. Maximum of 18 at (9, 0) 13h. Maximum of 54 at 􏰉 18 , 0􏰈 55 14. Maximum of 100 at (24, 32, −124) 17. Its feasible set is empty. ⎧1 2 3 4 + x4 ⎨x1+x2+x3+x4 =20 = 25 constraints: ⎪−4x1 − 6x3 + x5 = −5 ⎪ 3x1 ⎪3y1 + y2 − 4y3 − 2y4 􏶞 2 ⎪⎨ y2 􏶞2 y2 − 6y3 − 3y4 􏶞 −6 ⎪⎩ y1 +y2 −2y4 􏶞 −1 ⎪⎩−2x1 −3x3 −2x4 +x6 = 0 x1, x2, x3, x4, x5, x6􏶟0 constraints: Answers for Selected Problems 743 Computer Problems 17.1 1. Felt Straw 0 200 150 0 150 0 Texas Hatters Lone Star Hatters Lariat Ranch Wear 3. $13.50 5. Cost 50¢ for 1.6 ounces of food f1, 1 ounce of food f3, and none of food f2. Problems 17.2 1. maximize: 􏰍nj=0 cj yj Here c0 = −􏰍nj=1 cj and ai0 = −􏰍nj=1 aij. 􏰘 􏰍 nj = 0 a i j y j 􏶞 b i yi􏶟0 (0􏶞i􏶞n) 5. First primal form: maximize: −bT y 􏰘−AT y􏶞 −c constraints: y􏶟0 constraints: 2. At most 2n . 6. GivenAx=b.Letyj =xj +yn+1.Now minimize: yn+1 􏰃n 􏰃n 􏰃n aijxj −bi = aijyj −yn+1 constraints: ⎧⎪⎨􏰃n 􏰔􏰃n􏰕 aijyj + − aij ⎪⎩ j = 1 y􏶟0 j = 1 Computer Problems 17.2 1b. x=􏰠0,0,5,2,0􏰡T 1c. x=􏰠0,8,5􏰡T 33 33 j=1 yn+1 =bi j=1 (1􏶞i􏶞n+1) aij −bi. j=1 744 Answers for Selected Problems Problems 17.3 1a. maximize: −􏰍4 (ui+vi) ⎧ i=1 ⎪5y1 +2y2 − 7y4 −u1 +v1 = 6 ⎨ y1+ y2+ y3− 3y4−u2+v2= 2 constraints:⎪ 7y2 −5y3 − 2y4 −u3 +v3 =11 ⎪⎩6y1 +9y3 −15y4 −u4 +v4 = 9 u􏶟0 v􏶟0 y􏶟0 1b. minimize: ε ⎧⎪ 5y1+2y2 − 7y4−ε􏶞 6 ⎪ y1+ y2+ y3− 3y4−ε􏶞 2 ⎪ 7y2 −5y3 − 2y4 −ε 􏶞 11 ⎪⎨ 6y1 +9y3−15y4−ε􏶞 9 constraints: ⎪−5y1 − 2y2 + 7y4 − ε 􏶞 −6 ⎪−y1− y2− y3+ 3y4−ε􏶞 −2 ⎪ −7y2 +5y3 + 2y4 −ε 􏶞 −11 ⎪⎩−6y1 − 9y3 + 15y4 − ε 􏶞 −9 ε􏶟0 yj􏶟0 (1􏶞i􏶞4) 􏰃n minimize: constraints: ε ⎧⎪􏰃n ajxj. ⎪ axj 􏶞 f(x) (1􏶞i􏶞m) 3. Takempointsxi (i=1,2,...,m).Letp(x)= ⎪ji i ⎨ j=0 ⎪ 􏰃n ⎪ ajxj+ε􏶟 f(xi) (1􏶞i􏶞m) ⎪ i ⎪⎩ j = 0 ε􏶟0 4. minimize: u1 +v1 +u2 +v2 +u3 +v3 ⎧⎪⎨ y 1 − y 2 − u 1 + v 1 = 4 constraints: 2y1 −3y2 + y3 −u2 +v2 =7 ⎪⎩ y 1 + y 2 − 2 y 3 − u 3 + v 3 = 2 y1, y2, y3􏶟0, u1, u2, u3􏶟0, v1, v2, v3􏶟0 Computer Problems 17.3 1a. x1 = 0.353, x2 = 2.118, x3 = 0.765 1b. x1 = 0.671, x2 = 1.768, 3. p(x) = 1.0001 + 0.9978x + 0.51307x2 + 0.13592x3 + 0.071344x4 x3 = 0.453 3b. (613.40625)10 9. (479)10 =(111011111)2 Problems B 1a. e ≈ (2.718)10 = (010.101 101 111 100 111 . . .)2 2e. (113.1666213...)8 2f. (71.24426416...)8 4c. (101111)2 4e. (110011)2 4g. (33.72664)8 2d. (27.45075 341 . . .)8 3a. (441.681640625)10 6. (0.31463146...)8 j=0 12. A real number R has a finite representation in binary system. ⇔ R = (amam−1 ...a1a0.b1b2 ...bn)2. ⇔ R = (am...a1a0b1b2...bn)2×2−n =m×2−n wherem=(amam−1...a1a0b1b2...bn)2. Bibliography Abell, M. L., and J. P. Braselton. 1993. The Mathematical Handbook. New York: Academic Press. Abramowitz, M., and I. A. Stegun (eds.). 1964. Hand- book of Mathematical Functions with Formulas, Graphs, and Mathematical Tables. National Bureau of Standards. New York: Dover, 1965 (reprint). Acton, F. S. 1959. Analysis of Straight-Line Data. New York: Wiley. New York: Dover, 1966 (reprint). Acton, F. S. 1990. Numerical Methods That (Usually) Work. Washington, D.C.: Mathematical Association of America. Acton, F. S. 1996. Real Computing Made Real: Prevent- ing Errors in Scientific and Engineering Calculations. Princeton, New Jersey: Princeton University Press. Ahlberg, J. H., E. N. Nilson, and J. L. Walsh. 1967. The Theory of Splines and Their Applications. New York: Academic Press. Aiken, R. C., ed. 1985. Stiff Computation. New York: Oxford University Press. Ames, W. F. 1992. Numerical Methods for Partial Dif- ferential Equations, 3rd Ed. New York: Academic Press. Ammar, G. S., D. Calvetti, and L. Reichel, 1999. “Com- putation of Gauss-Kronrod quadrature rules with non- positive weights,” Electronic Transactions on Numerical Analysis 9, 26–38. http://etna.mcs.kent.edu Anderson, E., Z. Bai, C. Bischof, S. Blackford, J. Demmel, J. Dongarra, J. Du Croz, A. Greenbaum, S. Hammarling, A. McKenney, and D. Sorensen. 1999. LAPACK User’s Guide, 3rd Ed.. Philadelphia: SIAM. Armstrong, R. D., and J. Godfrey. 1979. “Two linear pro- gramming algorithms for the linear discrete l1 norm problem.” Mathematics of Computation 33, 289–300. Ascher, U. M., R. M. M. Mattheij, and R. D. Russell. 1995. Numerical Solution of Boundary Value Prob- lems for Ordinary Differential Equations. Philadelphia: SIAM. Ascher, U. M., and L. R. Petzold. 1998. Computer Meth- ods for Ordinary Differential Equations and Differential Algebraic Equations. Philadelphia: SIAM. Atkinson, K. 1993. Elementary Numerical Analysis. New York: Wiley. Atkinson, K. A. 1988. An Introduction to Numerical Anal- ysis, 2nd Ed. New York: Wiley. Axelsson, O. 1994. Iterative Solution Methods. New York: Cambridge University Press. Axelsson, O., and V.A. Barker. 2001. Finite Element Solu- tion of Boundary Value Problems: Theory and Compu- tations. Philadelphia: SIAM. Azencott, R., ed. 1992. Simulated Annealing: Paralleliza- tion Techniques. New York: Wiley. Bai, Z., J. Demmel, J. Dongarra, A. Ruhe, and H. van der Vorst. 2000. Templates for the Solution of Algebraic Eigenvalue Problems: A Practical Guide. Philadelphia: SIAM. Baldick, R. 2006. Applied Optimization. New York, Cam- bridge University Press. Barnsley, M. F. 2006. SuperFractals. New York, Cambridge University Press. Barrett, R., M. Berry, T. F. Chan, J. Demmel, J. Donato, J. Dongarra, V. Eijkhout, R. Pozo, C. Romine, and H. van der Vorst. 1994. Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods Philadelphia: SIAM. Barrodale, I., and C. Phillips. 1975. “Solution of an overde- termined system of linear equations in the Chebyshev norm.” Association for Computing Machinery Transac- tions on Mathematical Software 1, 264–270. Barrodale, I., and F. D. K. Roberts. 1974. “Solution of an overdetermined system of equations in the l1 norm.” Communications of the Association for Computing Machinery 17, 319–320. Barrodale, I., F. D. K. Roberts, and B. L. Ehle. 1971. Ele- mentary Computer Applications. New York: Wiley. Bartels, R. H. 1971. “A stabilization of the simplex method.” Numerische Mathematik 16, 414–434. Bartels, R., J. Beatty, and B. Barskey. 1987. An Introduction to Splines for Use in Computer Graphics and Geometric Modelling. San Francisco: Morgan Kaufmann. 745 746 Bibliography Bassien, S. 1998. “The dynamics of a family of one- dimensional maps.” American Mathematical Monthly 105, 118–130. Bayer, D., and P. Diaconis. 1992. “Trailing the dovetail shuffle to its lair.” Annals of Applied Probability, 2, 294–313. Beale, E. M. L. 1988. Introduction to Optimization. New York: Wiley. Bjo ̈rck, Å. 1996. Numerical Methods for Least Squares Problems. Philadelphia: SIAM. Bloomfield, P., and W. Steiger. 1983. Least Absolute De- viations, Theory, Applications, and Algorithms. Boston: Birkha ̈user. Bornemann, F., D. Laurie, S. Wagon, and J. Waldvogel. 2004. The SIAM 100-Digit Challenge: A Study in High- Accuracy Numerical Computing. Philadelphia: SIAM. Borwein, J. M., and P. B. Borwein. 1984. “The arithmetic- geometric mean and fast computation of elementary functions.” Society for Industrial and Applied Mathe- matics Review 26, 351–366. Borwein, J. M., and P. B. Borwein. 1987. Pi and the AGM: A Study in Analytic Number Theory and Computational Complexity. New York: Wiley. Boyce, W. E., and R. C. DiPrima. 2003. Elementary Differ- ential Equations and Boundary Value Problems, 7th Ed. New York: Wiley. Branham, R. 1990. Scientific Data Analysis: An Introduc- tion to Overdetermined Systems. New York: Springer- Verlag. Brenner, S., and R. Scott. 2002. The Mathematical Theory of Finite Element Methods. New York: Springer-Verlag. Brent, R. P. 1976. “Fast multiple precision evaluation of elementary functions.” Journal of the Association for Computing Machinery 23, 242–251. Briggs, W. 2004. Ants, Bikes, and Clocks: Problems Solving for Undergraduates. Philadelphia: SIAM. Buchanan, J. L., and P. R. Turner. 1992. Numerical Methods and Analysis. New York: McGraw-Hill. Burden, R. L., and J. D. Faires. 2001. Numerical Analysis, 7th Ed. Pacific Grove, California: Brooks/Cole. Bus, J. C. P., and T. J. Dekker. 1975. “Two efficient algo- rithms with guaranteed convergence for finding a zero of a function.” Association for Computing Machinery Transactions on Mathematical Software 1, 330–345. Butcher, J. C. 1987. The Numerical Analysis of Ordinary Differential Equations: Runge-Kutta and General Linear Methods. New York: Wiley. Calvetti, D., G. H. Golub, W. B. Gragg, and L. Reichel. 2000. “Computation of Gauss-Kronrod quadrature rules.” Mathematics of Computation 69, 1035–1052. Carrier, G., and C. Pearson. 1991. Ordinary Differential Equations. Philadelphia: SIAM. Ca ̈rtner, B. 2006. Understanding and Using Linear Pro- gramming. New York: Springer. Cash, J. “Mesh selection for nonlinear two-point boundary- value problems.” Journal of Computational Methods in Science and Engineering, 2003. Chaitlin, G. J. 1975. “Randomness and mathematical proof.” Scientific American May, 47–52. Chapman, S. J. 2000. MATLAB Programming for Engineer- ing, Pacific Grove, California: Brooks/Cole. Cheney, E. W. 1982. Introduction to Approximation Theory, 2nd Ed. Washington, D.C.: AMS. Cheney, E. W. 2001. Analysis for Applied Mathematics, New York: Springer. Chicone, C. 2006. Ordinary Differential Equations with Applications. 2nd Ed. New York: Springer. Clenshaw, C. W., and A. R. Curtis. 1960. “A method for numerical integration on an automatic computer.” Numerische Mathematik 2, 197–205. Colerman, T. F. and C. Van Loan. 1988. Handbook for Matrix Computations. Philadelphia: SIAM. Collatz, L. 1966. The Numerical Treatment of Differential Equations, 3rd Ed. Berlin: Springer-Verlag. Conte, S. D., and C. de Boor. 1980. Elementary Numerical Analysis, 3rd Ed. New York: McGraw-Hill. Cooper, L., and D. Steinberg. 1974. Methods and Applica- tions of Linear Programming. Philadelphia: Saunders. Crilly, A. J., R. A. Earnshaw, H. Jones, eds. 1991. Fractals and Chaos. New York: Springer-Verlag. Cvijovic, D., and J. Klinowski. 1995. “Taboo search: An approach to the multiple minima problem.” Science 267, 664–666. Dahlquist, G., and A. Bjo ̈rck. 1974. Numerical Methods. Englewood Cliffs, New Jersey: Prentice-Hall. Dantzi, G. B., A. Orden, and P. Wolfe. 1963. “Generalized simplex method for minimizing a linear from under linear inequality constraints.” Pacific Journal of Mathematics 5, 183–195. Davis, P. J., and P. Rabinowitz. 1984. Methods of Numerical Integration, 2nd Ed. New York: Academic Press. Davis, T. 2006. Direct Methods for Sparse Linear Systems. Philadelphia: SIAM. de Boor, C. 1971. “CADRE: An algorithm for numerical quadrature.” In Mathematical Software, edited by J. R. Rice, 417–449. New York: Academic Press. de Boor, C. 1984. A Practical Guide to Splines. 2nd Ed. New York: Springer-Verlag. Dekker, T. J. 1969. “Finding a zero by means of succes- sive linear interpolation.” In Constructive Aspects of the Fundamental Theorem of Algebra, edited by B. Dejon and P. Henrici. New York: Wiley-Interscience. Dekker, T. J., and W. Hoffmann. 1989. “Rehabilitation of the Gauss-Jordan algorithm.” Numerische Mathematik 54, 591–599. Dekker, T. J., W. Hoffmann, and K. Potma. 1997. “Stability of the Gauss-Huard algorithm with partial pivoting.” Computing 58, 225–244. Dekker, K., and J. G. Verwer. 1984. “Stability of Runge-Kutta methods for stiff nonlinear differential equations.” CWI Monographs 2. Amsterdam: Elsevier Science. Demmel, J. W., 1997. Applied Numerical Linear Algebra. Philadelphia: SIAM. Dennis, J. E., and R. Schnabel. 1983. Quasi-Newton Meth- ods for Nonlinear Problems. Englewood Cliffs, New Jersey: Prentice-Hall. Dennis, J. E., and R. B. Schnabel. 1996. Numerical Methods for Unconstrained Optimization and Nonlinear Equa- tions. Philadelphia: SIAM. Dennis, J. E., and D. J. Woods. 1987. “Optimization on microcomputers: The Nelder-Mead simplex algorithm.” In New Computing Environments, edited by A. Wouk. Philadelphia: SIAM. de Temple, D. W. 1993. “A quicker convergence to Euler’s Constant.” American Mathematical Monthly 100, 468–470. Devitt, J. S. 1993. Calculus with Maple V. Pacific Grove, California: Brooks/Cole. Dixon, V. A. 1974. “Numerical quadrature: a survey of the available algorithms.” In Software for Numerical Math- ematics, edited by D. J. Evans. New York: Academic Press. Dongarra, J. J., I. S. Duff, D. C. Sorenson, and H. van der Vorst. 1990. Solving Linear Systems on Vector and Shared Memory Computers. Philadelphia: SIAM. Dorn, W. S., and D. D. McCracken. 1972. Numerical Meth- ods with FORTRAN IV Case Studies. New York: Wiley. Edwards, C., and D. Penny. 2004. Differential Equations and Boundary Value Problems, 5th Ed. Upper Saddle River: New Jersey: Prentice-Hall. Ellis, W., Jr., E. W. Johnson, E. Lodi, and D. Schwalbe. 1997. Maple V Flight Manual: Tutorials for Calculus, Linear Algebra, and Differential Equations. Pacific Grove, California: Brooks/Cole. Ellis, W., Jr., and E. Lodi. 1991. A Tutorial Introduction to Mathematica. Pacific Grove, California: Brooks/Cole. Elman, H., D. J. Silvester, and A. Wathen. 2004. Finite Element and Fast Iterative Solvers. New York: Oxford University Press. England, R. 1969. “Error estimates for Runge-Kutta type solutions of ordinary differential equations.” Computer Journal 12, 166–170. Enright, W. H. 2006. “Verifying approximate solutions to differential equations.” Journal of Computational and Applied Mathematics 185, 203–311. Epureanu, B. I., and H. S. Greenside. 1998. “Fractal basins of attraction associated with a damped Newton’s method.” SIAM Review 40, 102–109. Evans, G., J. Blackledge, and P. Yardlay. 2000. Numerical Methods for Partial Differential Equations. New York: Springer-Verlag. Evans, G. W., G. F. Wallace, and G. L. Sutherland. 1967. Simulation Using Digital Computers. Englewood Cliffs, New Jersey: Prentice-Hall. Farin, G. 1990. Curves and Surfaces for Computer Aided Geometric Design: A Practical Guide, 2nd Ed. New York: Academic Press. Fauvel, J., R. Flood, M. Shortland, and R. Wilson (eds.). 1988. Let Newton Be! London: Oxford University Press. Feder, J. 1988. Fractals. New York: Plenum Press. Fehlberg, E. 1969. “Klassische Runge-Kutta formeln fu ̈nfter und siebenter ordnung mit schrittweitenkon- trolle.” Computing 4, 93–106. Flehinger, B. J. 1966. “On the probability that a ran- dom integer has initial digit A.” American Mathematical Monthly 73, 1056–1061. Fletcher, R. 1976. Practical Methods of Optimization. New York: Wiley. Floudas, C. A., and P. M. Pardalos (eds.). 1992. Recent Ad- vances in Global Optimization. Princeton, New Jersey: Princeton University Press. Flowers, B. H. 1995. An Introduction to Numerical Methods in C++. New York: Oxford University Press. Ford, J. A. 1995. “Improved Algorithms of Ilinois-Type for the Numerical Solution of Nonlinear Equations.” Techni- cal Report, Department of Computer Science, University of Essex, Colchester, Essex, UK. Forsythe, G. E. 1957. “Generation and use of orthogonal polynomials for data-fitting with a digital computer.” Society for Industrial and Applied Mathematics Journal 5, 74–88. Forsythe, G. E. 1970. “Pitfalls in computation, or why a math book isn’t enough,” American Mathematical Monthly 77, 931–956. Forsythe, G. E., M. A. Malcolm, and C. B. Moler. 1977. Computer Methods for Mathematical Com- putations. Englewood Cliffs, New Jersey: Prentice- Hall. Bibliography 747 748 Bibliography Forsythe, G. E., and C. B. Moler. 1967. Computer Solu- tion of Linear Algebraic Systems. Englewood Cliffs, New Jersey: Prentice-Hall. Forsythe, G. E., and W. R. Wasow. 1960. Finite Difference Methods for Partial Differential Equations. New York: Wiley. Fox, L. 1957. The Numerical Solution of Two-Point Boundary Problems in Ordinary Differential Equations. Oxford: Clarendon Press. Fox, L. 1964. An Introduction to Numerical Linear Algebra, Monograph on Numerical Analysis. Oxford: Clarendon Press. Reprinted 1974. New York: Oxford University Press. Fox, L., D. Juskey, and J. H. Wilkinson, 1948. “Notes on the solution of algebraic linear simultaneous equations,” Quarterly Journal of Mechanics and Applied Mathemat- ics. 1, 149–173. Frank, W. 1958. “Computing eigenvalues of complex ma- trices by determinant evaluation and by methods of Danilewski and Wielandt.” Journal of SIAM 6, 37–49. Fraser, W., and M. W. Wilson. 1966. “Remarks on the Clenshaw-Curtis quadrature scheme.” SIAM Review 8, 322–327. Friedman, A., and N. Littman. 1994. Industrial Mathe- matics: A Course in Solving Real-World Problems. Philadelphia: SIAM. Fro ̈berg, C.-E. 1969. Introduction to Numerical Analysis. Reading, Massachusetts: Addison-Wesley. Gallivan, K. A., M. Heath, E. Ng, B. Peyton, R. Plemmons, J. Ortega, C. Romine, A. Sameh, and R. Voigt. 1990. Par- allel Algorithms for Matrix Computations. Philadelphia: SIAM. Gander, W., and W. Gautschi. 2000. “Adaptive quadra- ture—revisited.” BIT 40, 84–101. Garvan, F. 2002. The Maple Book. Boca Raton, Florida: Chapman & Hall/CRC. Gautschi, W. 1990. “How (un)stable are Vandermonde systems?” in Asymptotic and Computational Analysis, 193–210, Lecture Notes in Pure and Applied Mathemat- ics, 124. New York: Dekker. Gautschi, W. 1997. Numerical Analysis: An Introduction. Boston, Massachusetts: Birkha ̈user. Gear, C. W. 1971. Numerical Initial Value Problems in Ordinary Differential Equations. Englewood Cliffs, New Jersey: Prentice-Hall. Gentle, J. E. 2003. Random Number Generation and Monte Carlo Methods, 2nd Ed. New York: Springer-Verlag. Gentleman, W. M. 1972. “Implementing Clenshaw-Curtis quadrature.” Communications of the ACM 15, 337–346, 353. Gerald, C. F., and P. O. Wheatley 1999. Applied Numeri- cal Analysis, 6th Ed. Reading, Massachusetts: Addison- Wesley. Ghizetti, A., and A. Ossiccini. 1970. Quadrature Formulae. New York: Academic Press. Gill, P. E., W. Murray, and M. H. Wright. 1981. Practical Optimization. New York: Academic Press. Gleick, J. 1992. Genius: The Life and Science of Richard Feynman. New York: Pantheon. Gockenbach, M. S., 2002. Partial Differential Equations: Analytical and Numerical Methods. Philadelphia: SIAM. Goldberg, D. 1991. “What every computer scientist should know about floating-point arithmetic.” ACM Computing Surveys 23, 5–48. Goldstine, H. H. 1977. A History of Numerical Analysis from the 16th to the 19th Century. New York: Springer- Verlag. Golub, G. H., and J. M. Ortega. 1992. Scientific Computing and Differential Equations. New York: Harcourt Brace Jovanovich. Golub, G. H., and J. M. Ortega. 1993. An Introduction with Parallel Scientific Computing. New York: Academic Press. Golub, G. H., and C. F. Van Loan. 1996. Matrix Compu- tations, 3rd Ed. Baltimore: Johns Hopkins University Press. Good, I. J. 1972. “What is the most amazing approxi- mate integer in the universe?” Pi Mu Epsilon Journal 5, 314–315. Greenbaum, A. 1997. Iterative Methods for Solving Linear Systems. Philadelphia: SIAM. Greenbaum, A. 2002. “Card Shuffling and the Polynomial Numerical Hull of Degree k,” Mathematics Department, University of Washington, Seattle, Washington. Gregory, R. T., and D. Karney, 1969. A Collection of Matrices for Testing Computational Algorithms. New York: Wiley. Griewark, A. 2000. Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation. Philadelphia: SIAM. Groetsch, C. W. 1998. “Lanczos’ generalized derivative.” American Mathematical Monthly 105, 320–326. Haberman, R. 2004. Applied Partial Differential Equa- tions with Fourier Series and Boundary Value Problems. Upper Saddle River: New Jersey: Prentice-Hall. Hageman, L. A., and D. M. Young. 1981. Applied Itera- tive Methods. New York: Academic Press; Dover 2004 (reprint). Ha ̈mmerlin, G., and K.-H. Hoffmann. 1991. Numerical Mathematics. New York: Springer-Verlag. Hammersley, J. M., and D. C. Handscomb. 1964. Monte Carlo Methods. London: Methuen. Hansen, T., G. L. Mullen, and H. Niederreiter. 1993. “Good parameters for a class of node sets in quasi-Monte Carlo integration.” Mathematics of Computation 61, 225–234. Haruki, H., and S. Haruki. 1983. “Euler’s Integrals.” Amer- ican Mathematical Monthly 7, 465. Hastings, H. M. and G. Sugihara. 1993. Fractals: A User’s Guide for the Natural Sciences. New York: Oxford University Press. Havie, T. 1969. “On a modification of the Clenshaw-Curtis quadrature formula.” BIT 9, 338–350. Heath, J. M. 2002. Scientific Computing: An Introductory Survey, 2nd Ed. New York: McGraw-Hill. Henrici, P. 1962. Discrete Variable Methods in Ordinary Differential Equations. New York: Wiley. Heroux, M., P. Raghavan, and H. Simon. 2006. Paral- lel Processing for Scientific Computing. Philadelphia: SIAM. Herz-Fischler, 1998. R. A Mathematical History of the Golden Number. New York: Dover Hestenes, M. R., and E. Stiefel. 1952. “Methods of con- jugate gradient for solving linear systems.” Journal Research National Bureau of Standards 49, 409–436. Higham, D., and N. J. Higham. 2006. MATLAB Guide, 2nd Ed. Philadelphia: SIAM. Higham, N. J. 2002. Accuracy and Stability of Numerical Algorithms, 2nd Ed. Philadelphia: SIAM. Hildebrand, F. B. 1974. Introduction to Numerical Analysis. New York: McGraw-Hill. Hodges, A. 1983. Alan Turing: The Enigma. New York: Simon & Schuster. Hoffmann, W. 1989. “A fast variant of the Gauss-Jordan algorithm with partial pivoting. Basic transformations in linear algebra for vector computing.” Doctoral disserta- tion, University of Amsterdam, The Netherlands. Hofmann-Wellenhof, B., H. Lichtenegger, and J. Collins. 2001. Global Positioning System: Theory and Practice, 5th Ed. New York: Springer-Verlag. Horst, R., P. M. Pardalos, and N. V. Thoai. 2000. Introduc- tion to Global Optimization, 2nd Ed. Boston: Kluwer. Householder, A. S. 1970. The Numerical Treatment of a Single Nonlinear Equation. New York: McGraw-Hill. Huard,P.1979.“Lame ́thodedusimplexesansinverse explicite.” Bull. E.D.F. Se ́rie C 2. Huddleston, J. V. 2000. Extensibility and Compressibility in One-Dimensional Structures. 2nd Ed. Buffalo, NY: ECS Publ. Hull, T. E., and A. R. Dobell. 1962. “Random number gen- erators.” Society for Industrial and Applied Mathematics Review 4, 230–254. Hull, T. E., W. H. Enright, B. M. Fellen, and A. E. Sedg- wick. 1972. “Comparing numerical methods for ordi- nary differential equations.” Society for Industrial and Applied Mathematics Journal on Numerical Analysis 9, 603–637. Hundsdorfer, W. H. 1985. “The numerical solution of non- linear stiff initial value problems: an analysis of one step methods.” CWI Tract, 12. Amsterdam: Stichting Mathematisch Centrum, Centrum voor Wiskunde en Informatica. Isaacson, E., and H. B. Keller. 1966. Analysis of Numerical Methods. New York: Wiley. Jeffrey, A. 2000. Handbook of Mathematical Formulas and Integrals. Boston: Academic Press. Jennings, A. 1977. Matrix Computation for Engineers and Scientists. New York: Wiley. Johnson, L. W., R. D. Riess, and J. T. Arnold. 1997. Introduction to Linear Algebra. New York: Addison- Wesley. Kahaner, D. K. 1971. “Comparison of numerical quadra- ture formulas.” In Mathematical Software, edited by J. R. Rice. New York: Academic Press. Kahaner, D., C. Moler, and S. Nash. 1989. Numerical Methods and Software. Englewood Cliffs, New Jersey: Prentice-Hall. Keller, H. B. 1968. Numerical Methods for Two-Point Boundary-Value Problems. Toronto: Blaisdell. Keller, H. B. 1976. Numerical Solution of Two-Point Bound- ary Value Problems. Philadelphia: SIAM. Kelley, C. T. 1995. Iterative Methods for Linear and Non- linear Equations. Philadelphia: SIAM. Kelley, C. T. 2003. Solving Nonlinear Equations with Newton’s Method. Philadelphia: SIAM. Kincaid, D., and W. Cheney. 2002. Numerical Analysis: Mathematics of Scientific Computing, 3rd Ed. Belmont, California: Thomson Brooks/Cole. Kincaid, D. R., and D. M. Young. 1979. “Survey of iter- ative methods.” In Encyclopedia of Computer Science and Technology, edited by J. Belzer, A. G. Holzman, and A. Kent. New York: Dekker. Kincaid, D. R., and D. M. Young. 2000. “Partial dif- ferential equations.” In Encyclopedia of Computer Science, 4th Ed., edited by A. Ralston, E. D. Reilly, D. Hemmendinger. New York: Grove’s Dictionaries. Kinderman, A. J., and J. F. Monahan. 1977. “Computer gen- eration of random variables using the ratio of uniform deviates.” Association of Computing Machinery Trans- actions on Mathematical Software 3, 257–260. Bibliography 749 750 Bibliography Kirkpatrick, S., C. D. Gelatt, Jr., and M. P. Vecchi. 1983. “Optimization by simulated annealing.” Science 220, 671–680. Knight, A. 2000. Basics of MATLAB and Beyond. Boca Raton, Florida: CRC Press. Knuth, D. E. 1997. The Art of Computer Programming, 3rd Ed. Vol. 2, Seminumerical Algorithms. New York: Addison-Wesley. Krogh, F. T. 2003. “On developing mathematical software.” Journal of Computational and Applied Mathematics 185, 196–202. Kronrod, A. S. 1964. “Nodes and Weights of Quadra- ture Rules.” Doklady Akad. Nauk SSSR, 154, 283–286. [Russian] (1965. New York: Consultants Bureau.) Krylov, V. I. 1962. Approximate Calculation of Integrals, translated by A. Stroud. New York: Macmillan. Lambert, J. D. 1973. Computational Methods in Ordinary Differential Equations. New York: Wiley. Lambert, J. D. 1991. Numerical Methods for Ordinary Differential Equations. New York: Wiley. Lapidus, L., and J. H. Seinfeld. 1971. Numerical Solution of Ordinary Differential Equations. New York: Academic Press. Laurie, D. P. 1997. “Calculation of Gauss-Kronrod quadrature formulae.” Mathematics of Computation, 1133–1145. Lawson, C. L., and R. J. Hanson. 1995. Solving Least- Squares Problems. Philadelphia: SIAM. Leva, J. L. 1992. “A fast normal random number genera- tor.” Association of Computing Machinery Transactions on Mathematical Software 18, 449–455. Lindfield, G., and J. Penny. 2000. Numerical Methods Us- ing MATLAB, 2nd Ed. Upper Saddle River: New Jersey: Prentice-Hall. Lootsam, F. A., ed. 1972. Numerical Methods for Nonlinear Optimization. New York: Academic Press. Lozier, D. W., and F. W. J. Olver. 1994. “Numerical eval- uation of special functions.” In Mathematics of Com- putation 1943–1993: A Half-Century of Computational Mathematics 48, 79–125. Providence, Rhode Island: AMS. Lynch, S. 2004. Dynamical Systems with Applications. Boston: Birkha ̈user. MacLeod, M. A. 1973. “Improved computation of cubic natural splines with equi-spaced knots.” Mathematics of Computation 27, 107–109. Maron, M. J. 1991. Numerical Analysis: A Practical Ap- proach. Boston: PWS Publishers. Marsaglia, G. 1968. “Random numbers fall mainly in the planes.” Proceedings of the National Academy of Sci- ences 61, 25–28. Marsaglia, G., and W. W. Tsang. 2000. “The Ziggurat Method for generating random variables.” Journal of Statistical Software 5, 1–7. Mattheij, R. M. M., S. W. Rienstra, and J. H. M. ten Thije Boonkkamp. 2005. Partial Differential Equa- tions: Modeling, Analysis, Computation. Philadelphia: SIAM. McCartin, B. J. 1998. “Seven deadly sins of numerical computations,” American Mathematical Monthly 105, No. 10, 929–941. McKenna, P. J., and C. Tuama. 2001. “Large torsional oscil- lations in suspension bridges visited again: Vertical forc- ing creates torsional response.” American Mathematical Monthly 108, 738–745. Mehrotra, S. 1992. “On the implementation of a primal-dual interior point method.” SIAM Journal on Optimization 2, 575–601. Metropolis, N. et al. 1953. “Equation of state calcula- tions by fast computing machines.” Journal of Physical Chemistry 21, 1087–1092. Meurant, G. 2006. The Lanczos and Conjugate Gradient Algorithms: From Theory to Finite Precision Computa- tions. Philadelphia: SIAM. Meyer, C. D., 2000. Matrix Analysis and Applied Linear Algebra. Philadelphia: SIAM. Miranker, W. L. 1981. “Numerical methods for stiff equa- tions and singular perturbation problems.” In Mathe- matics and its Applications, Vol. 5. Dordrecht-Boston, Massachusetts: D. Reidel. Moler, C. B., 2004. Numerical Computing with MATLAB. Philadelphia: SIAM. More ́, J. J., and S. J. Wright. 1993. Optimization Software Guide. Philadelphia: SIAM. Moulton, F. R. 1930. Differential Equations. New York: Macmillan. Nelder, J. A., and R. Mead. 1965. “A simplex method for function minimization.” Computer Journal 7, 308–313. Nerinckx, D., and A. Haegemans. 1976. “A comparison of nonlinear equation solvers.” Journal of Computational and Applied Mathematics 2, 145–148. Nering, E. D., and A. W. Tucker. 1992. Linear Programs and Related Problems. New York: Academic Press. Niederreiter, H. 1978. “Quasi-Monte Carlo methods.” Bulletin of the American Mathematical Society 84, 957–1041. Niederreiter, H. 1992. Random Number Generation and Quasi-Monte Carlo Methods. Philadelphia: SIAM. Nievergelt, J., J. G. Farrar, and E. M. Reingold. 1974. Com- puter Approaches to Mathematical Problems. Engle- wood Cliffs, New Jersey: Prentice-Hall. Noble, B., and J. W. Daniel. 1988. Applied Linear Algebra, 3rd Ed. Englewood Cliffs, New Jersey: Prentice-Hall. Nocedal, J., and S. Wright. 2006. Numerical Optimization. 2nd Ed. New York: Springer. Novak, E., K. Ritter, and H. Woz ́niakowski. 1995. “Average-case optimality of a hybrid secant- bisection method.” Mathematics of Computation 64, 1517–1540. Novak, M., ed. 1998. Fractals and Beyond: Complexities in the Sciences. River Edge, NJ: World Scientific. O’Hara, H., and F. J. Smith. 1968. “Error estimation in Clenshaw-Curtis quadrature formula.” Computer Journal 11, 213–219. Oliveira, S., and D. E. Stewart. 2006. Writing Scientific Software: A Guide to Good Style. New York: Cambridge University Press. Orchard-Hays, W. 1968. Advanced Linear Programming Computing Techniques. New York: McGraw-Hill. Ortega, J., and R. G. Voigt. 1985. Solution of Partial Dif- ferential Equations on Vector and Parallel Computers. Philadelphia: SIAM. Ortega, J. M. 1990a. Numerical Analysis: A Second Course. Philadelphia: SIAM. Ortega, J. M. 1990b. Introduction to Parallel and Vector Solution of Linear Systems. New York: Plenum. Ortega, J. M., and W. C. Rheinboldt. 1970. Iterative Solu- tion of Nonlinear Equations in Several Variables. New York: Academic Press. (2000. Reprint. Philadelphia: SIAM.) Ostrowski, A. M. 1966. Solution of Equations and Sys- tems of Equations, 2nd Ed. New York: Academic Press. Overton, M. L. 2001. Numerical Computing with IEEE Floating Point Arithmetic. Philadelphia: SIAM. Otten, R. H. J. M., and L. P. P. van Ginneken. 1989. The Annealing Algorithm. Dordrecht, Germany: Kluwer. Pacheco, P. 1997. Parallel Programming with MPI. San Francisco: Morgan Kaufmann. Patterson, T. N. L. 1968. “The optimum addition of points to quadrature formulae.” Mathematics of Computations 22, 847–856, and in 1969 Mathematics of Computations 23, 892. Parlett, B. N. 1997. The Symmetric Eigenvalue Problem. Philadelphia: SIAM. Parlett, B. 2000. “The QR Algorithm,” Computing in Sci- ence and Engineering 2, 38–42. Pessens, R., E. de Doncker, C. W. Uberhuber, and D. K. Kahaner, 1983. QUADPACK: A Subroutine Package for Automatic Integration. New York: Springer-Verlag. Peterson, I. 1997. The Jungles of Randomness: A Mathe- matical Safari. New York: Wiley. Phillips, G. M., and P. J. Taylor. 1973. Theory and Applica- tions of Numerical Analysis. New York: Academic Press. Press, W. H., S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery. 2002. Numerical Recipes in C++, 2nd Ed. New York: Cambridge University Press. Quinn, M. J. 1994. Parallel Computing: Theory and Prac- tice. New York: McGraw-Hill. Rabinowitz, P. 1968. “Applications of linear programming to numerical analysis.” Society for Industrial and Applied Mathematics Review 10, 121–159. Rabinowitz, P. 1970. Numerical Methods for Nonlinear Algebraic Equations. London: Gordon & Breach. Raimi, R. A. 1969. “On the distribution of first sig- nificant figures.” American Mathematical Monthly 76, 342–347. Ralston, A. 1965. A First Course in Numerical Analysis. New York: McGraw-Hill. Ralston, A., and C. L. Meek (eds.) 1976. Encyclopedia of Computer Science. New York: Petrocelli/Charter. Ralston, A., and P. Rabinowitz 2001. A First Course in Numerical Analysis, 2nd Ed. New York: Dover. Recktenwald, G. 2000. Numerical Methods with MATLAB: Implementation and Applications. New York: Prentice- Hall. Reid, J. 1971. “On the method of conjugate gradient for the solution of large sparse systems of linear equations.” In Large Sparse Sets of Linear Equations, J. Reid (ed.), London: Academic Press. Rheinboldt, 1998. Methods for Solving Systems of Nonlin- ear Equations, 2nd Ed. Philadelphia: SIAM. Rice, J. R. 1971. “SQUARS: An algorithm for least squares approximation.” In Mathematical Software, edited by J. R. Rice. New York: Academic Press. Rice, J. R. 1983. Numerical Methods, Software, and Anal- ysis. New York: McGraw-Hill. Rice, J. R., and R. F. Boisvert. 1984. Solving Elliptic Prob- lems Using ELLPACK. New York: Springer-Verlag. Rice, J. R., and J. S. White. 1964. “Norms for smooth- ing and estimation.” Society for Industrial and Applied Mathematics Review 6, 243–256. Rivlin, T. J. 1990. The Chebyshev Polynomials, 2nd Ed. New York: Wiley. Roger, H.-F. 1998. A Mathematical History of the Golden Number. New York: Dover. Bibliography 751 752 Bibliography Roos, C., T. Terlaky, and J.-Ph. Vial. 1997. Theory and Algorithms for Linear Optimization: An Interior Point Approach. New York: Wiley. Saad, Y., 2003. Iterative Methods for Sparse Linear Sys- tems. Philadelphia: SIAM. Salamin, E. 1976. “Computation of π using arithmetic- geometric mean.” Mathematics of Computation 30, 565–570. Sauer, T. 2006. Numerical Analysis. New York: Pearson, Addison-Wesley. Scheid, F. 1968. Theory and Problems of Numerical Anal- ysis. New York: McGraw-Hill. Scheid, F. 1990. 2000 Solved Problems in Numerical Analysis. Schaum’s Solved Problem Series. New York: McGraw-Hill. Schilling, R. J., and S. L. Harris. 2000. Applied Numerical Methods for Engineering Using MATLAB and C. Pacific Grove, California: Brooks/Cole. Schmidt 1908. Title unknown. Rendiconti del Circolo Matematico di Palermo 25, 53–77. Schoenberg, I. J. 1946. “Contributions to the problem of approximation of equidistant data by analytic func- tions.” Quarterly of Applied Mathematics 4, 45–99, 112–141. Schoenberg, I. J. 1967. “On spline functions.” In Inequali- ties, edited by O. Shisha, 255–291. New York: Academic Press. Schrage, L. 1979. “A more portable Fortran random number generator.” Association for Computing Machinery Trans- actions on Mathematical Software 5, 132–138. Schrijver, A. 1986. Theory of Linear and Integer Program- ming. Somerset, New Jersey: Wiley. Schultz, M. H. 1973. Spline Analysis. Englewood Cliffs, New Jersey: Prentice-Hall. Schumaker, L. L. 1981. Spine Function: Basic Theory. New York: Wiley. Shampine, J. D. 1994. Numerical Solutions of Ordinary Differential Equations. London: Chapman and Hall. Shampine, L. F., R. C. Allen, and S. Pruess. 1997. Funda- mentals of Numerical Computing. New York: Wiley. Shampine, L. F., and M. K. Gordon. 1975. Computer Solu- tion of Ordinary Differential Equations. San Francisco: W. H. Freeman. Shewchuk, J. R. 1994. “An introduction to the conjugate gradient method without the agonizing pain,” online Wikipedia. Skeel, R. D., and J. B. Keiper. 1992. Elementary Numerical Computing with Mathematica. New York: McGraw-Hill. Smith, G. D. 1965. Solution of Partial Differential Equa- tions. New York: Oxford University Press. Sobol, I. M. 1994. A Primer for the Monte Carlo Method. Boca Raton, Florida: CRC Press. Southwell, R. V. 1946. Relaxation Methods in Theoretical Physics. Oxford: Clarendon Press. Spa ̈th, H. 1992. Mathematical Algorithms for Linear Re- gression. New York: Academic Press. Stakgold, I., 2000. Boundary Value Problems of Mathemat- ical Physics. Philadelphia: SIAM. Steele, J. M., 1997. Random Number Generation and Quasi-Monte Carlo Methods. Philadelphia: SIAM. Stetter, H. J. 1973. Analysis of Discretization Methods for Ordinary Differential Equations. Berlin: Springer- Verlag. Stewart, G. W. 1973. Introduction to Matrix Computations. New York: Academic Press. Stewart, G. W. 1996. Afternotes on Numerical Analysis. Philadelphia: SIAM. Stewart, G. W. 1998a. Afternotes on Numerical Analy- sis: Afternotes Goes to Graduate School. Philadelphia: SIAM. Stewart, G. W. 1998b. Matrix Algorithms: Basic Decompo- sitions, Vol. 1. Philadelphia: SIAM. Stewart, G. W. 2001. Matrix Algorithms: Eigensystems, Vol. 2. Philadelphia: SIAM. Stoer, J., and R. Bulirsch. 1993. Introduction to Numerical Analysis, 2nd Ed. New York: Springer-Verlag. Strang, G. 2006. Linear Algebra and Its Applications. Belmont, California: Thomson Brooks/Cole. Strang, G., and K. Borre. 1997. Linear Algebra, Geodesy, and GPS. Cambridge, MA: Wellesley Cambridge Press. Street, R. L. 1973. The Analysis and Solution of Par- tial Differential Equations. Pacific Grove, California: Brooks/Cole. Stroud, A. H. 1974. Numerical Quadrature and Solution of Ordinary Differential Equations. New York: Springer- Verlag. Stroud, A. H., and D. Secrest. 1966. Gaussian Quadrature Formulas. Englewood Cliffs, New Jersey: Prentice-Hall. Subbotin, Y. N. 1967. “On piecewise-polynomial approxi- mation.” Matematicheskie Zametcki 1, 63–70. (Transla- tion: 1967. Math. Notes 1, 41–46.) Szabo, F. 2002. Linear Algebra: An Introduction Using MAPLE. San Diego, California: Harcourt/Academic Press. Torczon, V. 1997. “On the convergence of pattern search methods.” Society for Industrial and Applied Mathemat- ics Journal on Optimization 7, 1–25. To ̈rn, A., and A. Zilinskas. 1989. Global Optimization. Lecture Notes in Computer Science 350. Berlin: Springer-Verlag. Traub, J. F. 1964. Iterative Methods for the Solution of Equations. Englewood Cliffs, New Jersey: Prentice-Hall. Trefethen, L. N., and D. Bau. 1997. Numerical Linear Algebra. Philadelphia: SIAM. Turner, P. R. 1982. “The distribution of leading significant digits.” Journal of the Institute of Mathematics and Its Applications 2, 407–412. van Huffel, S. and J. Vandewalle. 1991. The Total Least Squares Problem: Computational Aspects and Analsyis. Philadelphia: SIAM. Van Loan, C. F. 1997. Introduction to Computational Sci- ence and Mathematics. Sudbury, Massachusetts: Jones and Bartlett. Van Loan, C. F. 2000. Introduction to Scientific Computing, 2nd Ed. Upper Saddle River: New Jersey: Prentice-Hall. Van der Vorst, H. A. 2003. Iterative Krylov Methods for Large Linear Systems. New York: Cambridge University Press. Varga, R. S. 1962. Matrix Iterative Analysis. Englewood Cliffs: New Jersey: Prentice-Hall. (2000. Matrix Itera- tive Analysis: Second Revised and Expanded Edition. New York: Springer-Verlag.) Wachspress, E. L. 1966. Iterative Solutions to Elliptic Systems. Englewood Cliffs: New Jersey: Prentice-Hall. Watkins, D. S. 1991. Fundamentals of Matrix Computation. New York: Wiley. Westfall, R. 1995. Never at Rest: A Biography of Isaac Newton, 2nd Ed. London: Cambridge University Press. Whittaker, E., and G. Robinson. 1944. The Calculus of Ob- servation, 4th Ed. London: Blackie. New York: Dover, 1967 (reprint). Wilkinson, J. H. 1965. The Algebraic Eigenvalue Problem. Oxford: Clarendon Press. Reprinted 1988. New York: Oxford University Press. Wilkinson, J. H. 1963. Rounding Errors in Algebraic Proc- esses. Englewood Cliffs, New Jersey: Prentice-Hall. New York: Dover 1994 (reprint). Wood, A. 1999. Introduction to Numerical Analysis. New York: Addison-Wesley. Wright, S. J. 1997. Primal-Dual Interior-Point Methods. Philadelphia: SIAM. Yamaguchi, F. 1988. Curves and Surfaces in Computer Aided Geometric Design. New York: Springer-Verlag. Ye, Yinyu. 1997. Interior Point Algorithms. New York: Wiley. Young, D. M. 1950. Iterative methods for solving par- tial difference equations of elliptic type. Ph.D. thesis. Cambridge, MA: Harvard University. See www.sccm .stanford.edu/pub/sccm/david young thesis.ps.gz. Young, D. M., 1971. Iterative Solution of Large Linear Sys- tems. New York: Academic Press: Dover 2003 (reprint). Young, D. M., and R. T. Gregory. 1972. A Survey of Numer- ical Mathematics, Vols. 1–2. Reading, Massachusetts: Addison-Wesley. New York: Dover 1988 (reprint). Ypma, T. J. 1995, “Historical development of the Newton- Raphson method.” Society for Industrial and Applied Mathematics Review 37, 531–551. Zhang, Y. 1995. “Solving large-scale linear programs by interior-point methods under the MATLAB envi- ronment.” Technical Report TR96–01, Department of Mathematics and Statistics, University of Maryland, Baltimore County, Baltimore, MD. Bibliography 753 Index Absolute errors, 5 Abstract vector spaces in linear algebra, 716–723 bases for, 718 change in similarity of, 719–720 eigenvalues and eigenvectors in, 719 Gram-Schmidt process for, 722–723 linear independence in, 717–718 linear transformations for, 718–719 norms for, 721–722 orthogonal matrices and spectral theorem in, 720–721 subspaces in, 717 Accelerated steepest decent procedure, 655 (CPb 16.2.2) Accuracy first-degree polynomial, 375 first-degree spline, 375 in ordinary differential equation (ODE) solutions, 435 precision and, 5–6 A−1 computation, 307 A-conjugate vectors, 332 Adams-Bashforth-Moulton methods adaptive scheme for, 488 example of, 488–489 for first-order ordinary differential equations, 455–456 predictor-corrector scheme in, 483–484 problems on, 241 (Pb 6.2.15), 461 (CPb 10.3.2–4) pseudocode for, 484–488 stiff equations and, 489–491 Adaptive Runge-Kutta methods, 450–454 Adaptive Simpson’s rule, 221–225 Adaptive two-point Gaussian integration, 242 (CPb 6.2.7) Advection equation, 601–602 Aiken acceleration formula, 363 A-inner product, of vectors, 332 Airy differential equation, 483 (CPb 11.2.2) Algebra. See Linear algebra Algorithms Berman, 638 (16.1.5) complete Horner’s, 7, 23–24 conjugate gradient, 334 converting bases of numbers, 696 Fibonacci search, 628–631 Gauss-Huard, 279–280 (CPb 7.2.24) Gaussian, 248, 250–251 golden section search, 631–633 Gram-Schmidt process, 519 linear least squares, 497 Moler-Morrison, 122 (CPb 3.3.14) multivariate case of minimization of functions, 644–646 natural cubic spline functions, 388–392 Neider-Mead, 647–648 Neville’s, 142–144 Newton, 129 normalized tridiagonal, 289 (CPb 7.2.12) orthogonal systems, 508–510 polynomial interpolation, 136–138 power method, 361–362 quadratic interpolation, 633–635 random numbers, 533–535, 535 Romberg, 165, 168, 204–215 description of, 204–205 Euler-Maclaurin formula and, 206–209 pseudocode for, 205–206 Richardson extrapolation of, 209–211 secant method for roots of equations, 112–113 shooting method for ordinary differential equations, 565–567 simplex, 672–673 variable metric, 647 Alternating series theorem, 28–30, 32 (Pb 1.2.13) Antiderivative, 181. See also Integration, numerical Approximation. See Least squares method; Spline functions Area and volume estimation, 544–552 computing, 547–548 “ice cream cone” example of, 548 numerical integration for, 544–545 pseudocode for, 545–547 Arithmetic Babylonian, 701 IEEE standard floating-point, 703–705 Mayan, 700–701 partial double-precision, 492 (CPb 11.3.2) Arithmetic mean, 15 (CPb 1.1.7) Arrays, 686, 688–689 Attraction, fractile basins of, 99–100, 108 (CPb 3.2.27) Autonomous ordinary differential equations, 471–472, 479–480 Back substitution, in Gaussian algorithm, 248, 250–251 Backward error analysis, 52 Banded storage mode, 291 (CPb 7.2.19) Banded systems of linear equations, 280–292 block pentadiagonal, 285–286 pentadiagonal, 283–285 strictly diagonal dominance in, 282–283 tridiagonal, 280–282 Banker’s rounding, 6 Bases for numbers, 692–702 β, 693 conversion between, 693–696 16, 698 10, 692–693 from 10 to 8 to 2, 696–698 Basic Simpson’s rule, 216–220, 228 (Pb 6.1.8) Basic trapezoid rule, 190 Basins of attraction, 99–100, 108 (CPb 3.2.27) Basis functions, 500–501, 505–508 Berman algorithm, 638 (CPb 16.1.5) Bernoulli numbers, 208 Bernstein polynomials, 416 Bessel functions, 42 (CPb 1.2.23), 186, 215 (CPb 5.3.11) Best-step steepest descent procedure, 643 Bézier curves, 416–418 Big O notation, 27 Biharmonic equation, 583 Binary search, for intervals, 384 (CPb 9.1.2) Binary system, 693, 696–697. See also Bases for numbers Binomial series, 31 (Pb 1.2.1) Birthday problem, 553–555 754 Bisection method for locating roots of equations, 76–85 convergence analysis in, 81–83 example of, 79–81 false position method in, 83–84 pseudocode in, 78–79 secant method and Newton’s method versus, 117 Bivariate functions, 144–145 Block pentadiagonal systems of linear equations, 285–286 Boundary cases, 685 Boundary-value problems. See Ordinary differential equations, boundary-value problems in Bratu’s problem, 581 (CPb 14.2.7) B spline functions, 404–425 for Bézier curves, 416–418 interpolation and approximation by, 410–412 pseudocode and example of, 412–413 Schoenberg’s process for, 414–415 theory of, 404–410 Bucking of a circular ring project, 581 (CPb 14.2.8) Buffon’s needle problem, 555–556 Calculus, Fundamental Theorem of, 181, 195 Cantilever beam, 341 (CPb 8.1.10) Cardinal polynomials, 126–127 Case studies in programming, 687–691 Cauchy-Riemann equation, 105 (Pb 3.2.40) Cauchy-Schwartz inequality, 503 (Pb 12.1.9), 643 Cayley-Hamilton Theorem, 358 (CPb 8.2.5) Central difference formula, 15 (CPb 1.1.3), 166, 171 Centroids, 648 Chapeau functions of B splines, 406 Characteristic equations, 719 Characteristic polynomials, 343 Chebyshev nodes, 155–156, 158, 163 (CPb 4.2.10), 174 Chebyshev polynomials orthogonal systems and, 505–518 algorithm for, 508–510 orthonormal basis functions in, 505–508 polynomial regression in, 510–515 properties of, 140–141 Checkerboard ordering, 620 (Pb 15.3.3) Cholesky factorization, 305–306, 315 (Pb 8.1.24) Chopping numbers, 6, 51 Clamped cubic splines, 387 Clean loops, 686 Code, modularizing, 685, 687–688 Coefficients aj, 131–136 Collocation method, 618 Column vectors, 671–672, 706 Companion matrix, 358 (CPb 8.2.3) Complete Horner’s algorithm, 23–24 Complete partial pivoting, 261–264 Components, in vectors, 706 Composite Gaussian three-point rule, 243 (CPb 6.2.11) Composite midpoint rule for equal subintervals, 188 (Pb 5.1.12) Composite (left) rectangle rule, 202 (Pb 5.2.28) Composite rectangle rule with uniform spacing, 202–203 (Pb 5.2.29) Composite Simpson’s rule, 220–221, 228 (Pb 6.1.6), 243 (CPb 6.2.11) Composite trapezoid rule, 191, 194, 243 (CPb 6.2.11) Composite trapezoid rule with unequal spacing, 203 (Pb 5.2.32) Computation, noise in, 174 Computer-aided geometric design, 425 (CPb 9.3.19) Condition number, in linear equations, 321–322 Conjugate gradient method, 332–335 Constrained minimization problems, 625–626 Continuity of functions, 373–375 Contour diagrams, 644 Control points, in drawing curves, 371, 416 Convergence analysis in bisection method, 81–83 in Newton’s method, 93–96 in secant method, 114–116 Convergence theorems, 328–331 Convex hull, of vectors, 417 Corollaries on divided differences, 160 Correctly rounded value, 705 Correct rounding, 50 Cramer’s Rule, 715 Crank-Nicolson method, 588–591 Crout factorization, 317 (CPb 8.1.2) Cubic B spline, 423 (Pb 9.3.38) Cubic interpolating spline, 371. See also Spline functions Curves. See Ordinary differential equations; Spline functions Dawson integral, 439 (CPb 10.1.12) Decimal places, accuracy to, 5 Decimal point, 693 Decomposition, in matrix factorizations, 296 Deflation of polynomials, 8, 11 Delay ordinary differential equations, 450 (CPb 10.2.17) Derivatives, 164–179 of B splines, 408 divided differences and, 159 of functions, 9–10 Lanczos’ generalized, 178 (Pb 4.3.21) Index 755 noise in computation and, 174 polynomial interpolation estimating of, 170–174 Richardson extrapolation for, 166–170 Taylor series estimating of, 164–166 Determinants, 278 (CPb 7.2.14) Diagonal dominance, 282–283, 330 Diagonal matrices, 346–347, 709 Diet problem, 670 (CPb 17.1.5) Differential equations, 353–355. See also Ordinary differential equations; Partial differential equations Differentiation, 718 Diffusion equation, 584 Dimension, 718 Direct error analysis, 52 Direction vectors, 333 Direct method, for eigenvalues, 343 Dirichlet function, 154, 184, 584, 593, 618 Discretization method, 570–572 Divergent curves, 458 Divided differences for calculating coefficients aj, 131–136 corollary on, 160 derivatives and, 159 Doolittle factorization, 300, 317 (CPb 8.1.2) Dot product of vectors, 708 Double-precision floating-point representation, 48–49 Dual problem, in linear programming, 661–663, 673 Economical version of singular value decomposition, 356 (Pb 87.2.5) Eigenvalues and eigenvectors, 258 (CPb 7.1. 6), 342–360. See also Power method for linear equations calculating, 343–344 Gershgorin’s Theorem and, 347–348 in linear algebra, 719 in linear differential equations, 353–355 in mathematical software, 344 matrix spectral theory of, 349–351 properties of, 345–347 singular value decomposition of, 348–349, 351–353 Elements, in vectors, 706, 708 Elliptic integrals, 39 (CPb 1.2.14), 180, 186 Elliptic problems, in differential equations, 584, 594 (Pb15.1.1), 605–624 finite-difference method for, 606–609 finite-element methods for, 613–619 Gauss-Seidel iterative method for, 610 Helmholtz equation model, 605–606 pseudocode for, 610–613 756 Index Entry, in vectors, 706, 708 Epsilon, machine, 47–48 Equal oscillation property, 141 Equations, roots of. See Roots of equations, locating Error. See also Polynomial interpolation absolute and relative, 5 in ordinary differential equations (ODE), 435 roundoff, 50, 52, 54, 63, 253, 687 single-step, 453 trapezoid rule analysis of, 192–196 truncation, 165–166, 174 unit roundoff, 703 vectors of, 254–255, 279 (CPb 7.2.19) Error function, 34 (Pb 1.2.52), 185–186 Error term, 25, 27, 174 Euclidean/l2-vector norm, 721 Euler-Bernoulli beam, 340 (CPb 8.1.10) Euler-Maclaurin formula, 206–209, 214 (Pb 5.3.26) Euler’s constant, 59–60 (CPb 2.1.7) Euler’s method, 432–433, 437 (Pb 10.1.15) European Space Agency, 54 Expanded reflected points, 648 Expansion, finite, 44 Explicit method for partial differential equations, 587, 591, 595 (Pb 15.1.12) Exponents, 44, 544 (CPb 13.1.20), 687 Factorial notation, 21 Factoring, 296. See also Matrix factorizations Fairing curves, 371 False position method, 83–84 Feasible set, of vectors, 658 Fehlberg method of order 4, 451 Fibonacci numbers, 40 (CPb 1.2.16), 115, 628–631 Finite-difference method, 570–571, 574, 606–609 Finite-dimensional number, 718 Finite-element methods, 613–619 Finite expansion, 44 First bad case, of quadratic interpolation algorithm, 635 First-degree polynomial accuracy theorem, 375 First-degree spline accuracy theorem, 375 First-derivative formulas, 164–166, 170–174 First primal form, in linear programming, 657–658, 660–661, 673 Five-point formula for Laplace’s equation, 606–607 Fixed point iteration, 117–118 Flatness test, 648 Floating-point numbers, 43–55, 102 (Pb 3.2.24) computer errors in, 50–51, 54, 687 double-precision, 48–49 equality of, 689–690 floating-point machine number [fl(x)] and, 51–55 IEEE standard arithmetic for, 703–705 normalized, 44–46 single-precision, 46–47 standard, 46 Forward elimination, in Gaussian algorithm, 248, 250 Fourier series, 73 (CPb 2.2.15) Fractile basins of attraction, 99–100, 108 (CPb 3.2.27) Fractional numbers, converting bases of, 695–696 Fractional parts, 696 French curves, 371 French railroad system problem, 559 (CPb 13.3.3) Fresnel integral, 186, 204 (CPb 5.2.5) Frobenius norm, 338 (Pb 8.1.10) Fully implicit method for partial differential equations, 595 (Pb 15.1.13) Functions, minimization of, 625–658 multivariate case of, 639–656 advanced algorithms for, 644–646 contour diagrams for, 644 minimum, maximum and saddle points in, 646 Neider-Mead algorithm for, 647–648 positive definite matrix and, 647 quasi-Newton methods for, 647 simulated annealing method for, 648–649 steepest descent procedure for, 643 Taylor Series for F in, 640–642 one-variable case of, 625–639 Fibonacci search algorithm and, 628–631 golden section search algorithm and, 631–633 quadratic interpolation algorithm and, 633–635 special case of, 626–627 unconstrained and constrained problems in, 625–626 unimodal functions F as, 627–628 Fundamental Theorem of Calculus, 181, 195 Galerkin equation, 617 Gauss-Huard algorithm, 279–280 (CPb 7.2.24) Gaussian continued functions, 73 (CPb 2.2.18) Gaussian elimination naive, 245–258 algorithm for, 248–250 example of, 247–248 failure of, 259–260 in matrix factorizations, 295–296, 311 (Pb 8.1.1) pseudocode for, 250–254 residual and error vectors in, 254–255 with scaled partial pivoting, 259–280 complete partial pivoting versus, 261–264 example of, 265–266 long operation count for, 269–270 numerical stability of, 271 pseudocode for, 266–269 Gaussian method for elliptic integrals, 39 (CPb 1.2.14) Gaussian quadrature formulas, 230–244 change of intervals in, 231 composite three-point, 243 (CPb 6.2.11) description of, 230–231 integrals with singularities in, 237–239 Legendre polynomials in, 234–237 nodes and weights in, 232–234 Gauss-Jordan algorithm, 279–280 (CPb 7.2.24) Gauss-Legendre quadrature formulas, 232 Gauss-Seidel method, 323–325, 330–331, 610 Generalized Neumann equation, 584 Generalized Newton’s method, 104 (Pb 3.2.36) General quadratic functions, 652 (Pb 16.2.15) Gershgorin’s Theorem, 347–348 Global positioning systems, 111 (CPb 3.2.41) Golden ratio, 115, 638 (CPb 16.1.5) Golden section search algorithm, 631–633 Goodness of fit, 374 Gradient of quadratic forms, 333 Gradient vector matrix, 640–641 Gram-Schmidt process, 506, 519, 722–723 Greatest lower bound, in integration, 182 Great Internet Mersenne Prime Search (GIMPS), 541 Halley’s method, 122 (CPb 3.3.13) Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables (Abramowitz and Stegun), 186 Harmonic functions, 607, 618 Harmonic series, 59–60 (CPb 2.1.7) Hat functions of B splines, 406 Heat equation model, 583–586 Helmholtz equation model, 584, 605–606 Hermitian matrices, 345 Hessian matrix, 640–641 Heun’s method, 437 (Pb 10.1.15) Hexadecimal system, 693, 698. See also Bases for numbers Hidden bits, 47 Hilbert matrix, 276 (CPb 7.2.4), 527 (Pb 12.3.2) Histograms, 560 (CPb 13.3.13) Horner’s algorithm, 7, 23–24 Hyperbolic problems, in differential equations, 584, 594 (Pb15.1.1), 596–605 advection equation as, 601 analytical solution for, 597–598 Lax method for, 602 Lax-Wendroff method for, 602–603 numerical solution for, 598–599 pseudocode for, 600–601 upwind method for, 602 wave equation model as, 596–597 Idemtity matrix, 709 IEEE floating-point standard arithmetic, 703–705 Ill-conditioning, 321–322, 448 (CPb 10.2.5) Improved Euler’s method, 437 (Pb 10.1.15) IMSL mathematical library, 10 Incompatible systems, 519 Inconsistent systems, 519 Index vector, 262, 266 Inductive definition, in Newton’s method, 91 Initial-value problem, 426–428, 431, 463 (CPb 10.3.17) Inner product, 332, 512, 708 Integer parts, 696 Integrals Dawson, 439 (CPb 10.1.12) elliptic, 39 (CPb 1.2.14), 180, 186 sine, 189 (CPb 5.1.2), 204 (CPb 5.2.5), 463 (CPb 10.3.15) Integration, numerical, 180–244 for area and volume estimation, 544–545 definite and indefinite, 180–181 Gaussian quadrature formulas in, 230–244 change of intervals in, 231 description of, 230–231 integrals with singularities in, 237–239 Legendre polynomials in, 234–237 nodes and weights in, 232–234 lower and upper sums in, 181–183 of ordinary differential equations (ODE), 428–429 pseudocode and examples of, 184–187 Riemann-integrable functions in, 183–184 Romberg algorithm in, 204–215 description of, 204–205 Euler-Maclaurin formula and, 206–209 pseudocode for, 205–206 Richardson extrapolation of, 209–211 Simpson’s rule in, 216–229 adaptive, 221–225 basic, 216–220 composite, 220–221 Newton-Cotes rules and, 225–226 trapezoid rule in, 190–204 error analysis in, 192–197 multidimensional integration in, 198–199 uniform spacing in, 191–192 Intermediate-value theorem, 78, 194 Interpolation. See B spline functions; Polynomial interpolation; Quadratic interpolation algorithm Invariance theorem, 135 Inverse polynomial interpolation, 141–142, 567 Inverse power method, 364–365 Irregular five-point formula for Laplace’s equation, 607 Iterations. See also Linear equations, systems of fixed point, 117–118 limiting, 689 Newton-Raphson, 89 Richardson, 322–323 Jacobean matrix, 97–98, 100 Jacobi method, 323–325, 330–331 Jacobi overrelaxation (JOR) method, 332 Kepler’s equation, 106 (CPb 3.2.6) Knots, in spline theory, 372, 378 Kronecker delta equation, 145 kth residual, 519 Lagrange form of polynomial interpolation, 25, 126–128, 144 Lanczos’ generalized derivative, 178 (Pb 4.3.21) LAPACK mathematical software, 344, 351 Laplace’s equations, 286, 583–584, 605–606, 618 Laws of Motion, Newton’s, 428, 465 Lax method, 602 Lax-Wendroff method, 602–603 LDLT factorizations, 302–304, 315 (Pb 8.1.24) Least lower bound, in integration, 182 Least squares method, 495–505, 518–531, 652 (Pb 16.1.20) basis function in, 500–501 linear example of, 521–522 nonlinear example of, 520–522 nonpolynomial example of, 499–500 Index 757 singular value decomposition (SVD) and, 522–527 weight function in, 519–520 Least upper bound, of number set, 374 Lebesgue constants, 73 (CPb 2.2.15) Legendre polynomials, 234–237 Legendre’s elliptic integral relation, 39 (CPb 1.2.14) Lemma, upper bound, 157 Length of vectors, 320 L’Hôpital’s rule, 34 (Pb 1.2.49) Libraries, program, 10, 686–687 Linear algebra, 706–723 abstract vector spaces in, 716–723 bases for, 718–720 change in similarity of, 719–720 eigenvalues and eigenvectors in, 719 Gram-Schmidt process for, 722–723 linear independence in, 717–718 linear transformations for, 718–719 norms for, 721–722 orthogonal matrices and spectral theorem in, 720–721 subspaces in, 717 Cramer’s Rule and, 715 matrices in, 708–710 matrix product in, 711–713 matrix-vector product in, 711 symmetric matrices in, 714–715 transpose matrices in, 713–714 vectors in, 706–708 Linear B spline, 422 (Pb 9.3.36) Linear combinations, 707 Linear convergence, 82 Linear equations, systems of, 245–370 banded, 280–292 block pentadiagonal, 285–286 pentadiagonal, 283–285 strictly diagonal dominance in, 282–283 tridiagonal, 280–282 eigenvalues and eigenvectors in, 342–360 calculating, 343–344 Gershgorin’s Theorem and, 347–348 in linear differential equations, 353–355 in mathematical software, 344 matrix spectral theory of, 349–351 properties of, 345–347 singular value decomposition of, 348–349, 351–353 Gaussian elimination with scaled partial pivoting of, 259–280 complete partial pivoting versus, 261–264 example of, 265–266 long operation count for, 269–270 numerical stability of, 271 pseudocode for, 266–269 inconsistent, 675–683 iterative solutions of, 319–341 758 Index Linear equations, systems of (continued) basic methods of, 322–327 condition number and ill-conditioning in, 321–322 conjugate gradient method of, 332–335 convergence theorems for, 328–331 matrix formulation for, 331–332 overrelaxation in, 332 pseudocode for, 327–328 vector and matrix norms in, 319–320 matrix factorizations in, 293–319 Cholesky factorization as, 305–306 derivation of, 296–300 example of, 294–296 A−1 in, 307 LDLT factorization as, 302–304 LU factorization as, 300–302 multiple right-hand sides in, 306–307 pseudocode for, 300 software package example of, 307–309 naive Gaussian elimination of, 245–258 algorithm for, 248–250 example of, 247–248 failure of, 259–260 pseudocode for, 250–254 residual and error vectors in, 254–255, 279 (CPb 7.2.19) power method for, 360–370 Aiken acceleration formula for, 363 algorithms for, 361–362 inverse, 364–365 in mathematical software, 363 shifted inverse, 365–366 Linear functions, 361, 641 Linear interpolation, 162 (Pb 4.2.8) Linearize and solve approach to solving nonlinear equations, 96, 117 Linearly independent sets, 501 Linear polynomial interpolation, 125–126 Linear programming, 657–683 approximate solution of inconsistent linear systems from, 675–683 l∞ problem for, 678–680 l1 problem for, 676–678 dual problem in, 661–663 first primal form in, 657–658, 660–661 optimization example of, 658–660 second primal form in, 663–664 simplex method for, 670–675 l∞-matrix norm, 320 l∞ problem, 678–680 l∞-vector norm, 320, 721 l∞-matrix norm, 722 l1-matrix norm, 722 Loaded die problem, 552–553 Localization theorems, 347 Local minimum points of functions, 626 Local truncation error, 435 Logarithmic integral, 186, 189 (CPb 5.1.3) l1 approximation, 496 l1-matrix norm, 320 l1 problem, 676–678 l1-vector norm, 320, 721 Loops, clean, 686 Lower and upper sums, in integration, 181–183 Lower triangular matrix, 710 Lucas-Lehmer test, 540 LU factorization derivation of, 296–300 description of, 294 problems in, 314–315 (Pb 8.1.18), 319 (CPb 8.1.14) solving linear systems with, 300–302 Machine epsilon, 47–48, 703 Machine numbers, 44, 51. See also Floating-point numbers Maclaurin series, 31 (Pb 1.2.1), 41 (CPb 1.2.21) Macsyma mathematical software, 10 Magnitude of vectors, 320 Main diagonal matrix, 710 Mantissa, normalized, 44, 47 Maple mathematical software, 10 boundary-value problem, 577 differential equations, 427 eigenvalues, 343–344 error function in, 186 linear programming, 678–679 LU factorization in, 308 minimal solution, 526 minimization problems, 626 nonlinear equations, 99, 111 (CPb 3.2.42), 123 (CPb 3.3.19) partial differential equations, 592 polynomial interpolation in, 153 (CPb 4.1.11), 164 (CPb 4.2.12) random numbers, 533, 535 roots of equations in, 81, 88 (CPb 3.1.12), 93 singular value decomposition, 351 splines, 409–410, 418 symbolic verification in, 20 (CPb 1.1.26) Marching problem/method, 586 March of B splines, 424 (CPb 9.3.6) Mathematica mathematical software, 10 boundary-value problem, 577 differential equations, 427 eigenvalues, 343–344 error function in, 186 linear programming, 678–679 LU factorization in, 308 minimal solution, 526 minimization problems, 626 nonlinear equations, 99, 111 (CPb 3.2.42), 123 (CPb 3.3.19) partial differential equations, 592 polynomial interpolation in, 153 (CPb 4.1.11), 164 (CPb 4.2.12) random numbers, 533, 535 roots of equations in, 81, 88 (CPb 3.1.12), 93 splines, 418 symbolic verification in, 20 (CPb 1.1.26) Matlab mathematical software, 10 boundary-value problem, 577 eigenvalues, 343–344 error function in, 186 linear programming, 678–679 LU factorization in, 308 minimal solution, 526 minimization problems, 626 nonlinear equations in, 99, 111 (CPb 3.2.42), 123 (CPb 3.3.19) not-a-knot condition, of splines, 394 PDE Toolbox, 584, 592–593, 612 polynomial interpolation in, 153 (CPb 4.1.11), 164 (CPb 4.2.12) random numbers, 533, 535 roots of equations in, 81, 88 (CPb 3.1.12), 93 singular value decomposition, 351 splines, 409 vector fields, 430 Matrices. See also Linear algebra; Singular value decomposition (SVD) companion, 358 (CPb 8.2.3) diagonal, 346–347 Gershgorin’s Theorem and, 348 gradient vector, 640–641 Hermitian, 345–346 Hessian, 640–641 Hilbert, 276 (CPb 7.2.4), 527 (Pb 12.3.2) Jacobean, 97–98 of near-deficiency in rank, 526 permutation, 307 positive definite, 305, 332–333, 345, 647 pseudo-inverse of, 525–526 row-equilibrated, 275 (Pb 7.2.23) similar, 345 singular values of, 349 symmetric, 332, 345, 640 symmetric positive definite (SPD), 305, 330 transpose of, 345 triangular, 346 unitarily similar, 345–346 Vandermonde, 139–141, 152 (Pb 4.1.47), 254 Matrix factorizations, 293–319 Cholesky, 305–306 derivation of, 296–300 example of, 294–296 A−1 in, 307 LDLT , 302–304 LU, 300–302 multiple right-hand sides in, 306–307 pseudocode for, 300 software package example of, 307–309 Matrix formulations, 331–332 Matrix norms, 319–320, 721–722 Matrix spectral theory, 349–351 Maximal linearly independent basis, 718 Maximum points of functions, 646 Mayan arithmetic, 700–701 Mean, arithmetic, 15 (CPb 1.1.7) Mean-Value Theorem, 26, 193, 397 Memory fetches, 688 Mersenne prime number, 534 Midpoint method, 188 (Pb 5.1.10), 188 (Pb 5.1.12), 201 (Pb 5.2.18), 462 (CPb 10.3.8) Minimal solution, to linear equations, 524–526 Minimization of functions. See Functions, minimization of Minimum points of functions, 626, 646 Mixed Dirichlet/Neumann equation, 584 Mixed mode coding, 687–688 Modified false position method, 84 Modified Newton’s method, 104 (Pb 3.2.35) Modularizing code, 685 Modulus of continuity in spline functions, 374–375 Molecular conformation, 655 (CPb 16.2.2), 655 (CPb 16.2.10) Moler-Morrison algorithm, 122 (CPb 3.3.14) Monte Carlo methods. See also Simulation area and volume estimation by, 544–552 computing, 547–548 “ice cream cone” example of, 548 numerical integration for, 544–545 pseudocode for, 545–547 random numbers and, 532–544 algorithms and generators for, 533–535 examples of, 535–537 pseudocode for, 537–541 Muller’s method, 123 (CPb 3.3.17) Multidimensional integration, 198–199 Multiple zero, 96, 104 (Pb 3.2.35) Multiplication, nested, 7–9, 12 (Pb 1.1.6), 131 Multipliers, in Gaussian algorithm, 249 Multistep methods, 483 Multivariate case of minimization of functions advanced algorithms for, 644–646 contour diagrams for, 644 minimum, maximum and saddle points Neider-Mead algorithm for, 647–648 positive definite matrix and, 647 quasi-Newton methods for, 647 simulated annealing method for, 648–649 steepest descent procedure for, 643 Taylor Series for F in, 640–642 NAG mathematical library, 10 NaN (Not a Number), 704 Natural cubic spline functions algorithm for, 388–392 introduction to, 385–387 pseudocode for, 392–394 smoothness property from, 396–398 space curves from, 394–396 Natural logarithm (ln), 1 Natural ordering, 262–264, 609 Navler-Stokes equation, 583–584 Near-deficiency in rank, matrix with, 526 Neider-Mead algorithm, 647–648 Nested form of polynomial interpolation, 130–131 Nested multiplication, 7–9, 12 (Pb 1.1.6), 131 Neumann equation, 584 Neutron shielding simulation, 557–558 Neville’s algorithm, 142–144 Newton-Cotes rules, 225–226, 229 (CPb 6.1.7) Newton-Raphson iteration, 89 Newton’s form of polynomial interpolation, 128–130, 133, 150–151 (Pb 4.1.38), 164 (CPb 4.2.14) Newton’s Laws of Motion, 428, 465 Newton’s method for locating roots of equations, 89–100 bisection method and secant method versus, 117 convergence analysis in, 93–96 fractile basins of attraction in, 99–100 generalized, 104 (Pb 3.2.37) interpretation of, 90–91 modified, 104 (Pb 3.2.35) nonlinear equation systems in, 96–99 pseudocode in, 92–93 Newton’s method for nonlinear systems, 98 Nine-point formula for Laplace’s equation, 607, 621 (Pb 15.3.10) Nodes Chebyshev, 155–156, 158, 163 (CPb 4.2.10), 174 Gaussian, 230, 232–234 in polynomial interpolation, 125 in spline theory, 378 Noise in computation, 174 Nonlinear equation systems, 83, 96–99, 104 (Pb 3.2.39) Nonlinear least squares problems, 520–522 Index 759 Nonperiodic spline filter, 291 (CPb 7.2.22) Normal equations, 497, 499, 501, 617 Normalized floating-point representation, 44–46 Normalized mantissa, 44, 47 Normalized scientific notation, 43 Normalized tridiagonal algorithm, 289 (CPb 7.2.12) Norm induced, 721 Norms, 319–320, 721–722 n-simplex sets, 648 Number representation. See Floating-point numbers Objective functions, 658 Octal system, 693, 696–697. See also Bases for numbers Octave mathematical software, 10 Odd periodic functions, 598 Olver’s method, 122 (CPb 3.3.12) One-variable case of minimization of functions, 625–639 Fibonacci search algorithm and, 628–631 golden section search algorithm and, 631–633 quadratic interpolation algorithm and, 633–635 special case of, 626–627 unconstrained and constrained problems in, 625–626 unimodal functions F as, 627–628 Optimization example, of linear programming, 658–660 Ordering, natural, 262–264, 609 Ordering, red-black (checkerboard), 620 (Pb 15.3.3) Ordinary differential equations (ODE), 426–464 Adams-Bashforth-Moulton formulas for, 455–456 error types in, 435 Euler’s method pseudocode for, 432–433 initial-value problem in, 426–428 integration and, 428–429 Runge-Kutta methods for, 439–450 adaptive, 450–454 example of, 454–455 of order 4, 442–443 of order 2, 441–442 pseudocode for, 443–444 Taylor series in two variables and, 440–441 stability analysis for, 456–459 Taylor series methods for, 431–435 vector fields in, 429–431 Ordinary differential equations, boundary-value problems in, 563–581 discretization method for, 570–572 in, 646 760 Index Ordinary differential equations (continued) shooting method for algorithm for, 565–567 in linear case, 574–575 overview of, 563–565 pseudocode for, 575–577 refinements to, 567 Ordinary differential equations, systems of, 465–494 Adams-Bashforth-Moulton methods for, 483–494 adaptive scheme for, 488 example of, 488–489 predictor-corrector scheme in, 483–484 pseudocode for, 484–488 stiff equations and, 489–491 first order methods for, 465–477 for autonomous ODE, 471–471 Runge-Kutta, 469–471 Taylor series, 466–469 uncoupled and coupled systems in, 465–466 vector notation for, 467–469 higher order, 477–483 Orthogonal matrices, 720–721 Orthogonal systems. See also Chebyshev polynomials algorithm for, 508–510 orthonormal basis functions in, 505–508 polynomial regression in, 510–515 Overflow, of range, 45 Overrelaxation, 324, 326–327, 331–332 Padé interpolation, 153 (CPb 4.1.17) Padé rational approximation, 41 (CPb 1.2.22), 73 (CPb 2.2.17) Parabolic problems, in differential equations, 582–596, 594 (Pb15.1.1) applied, 582–585 Crank-Nicolson alternative method for, 590–591 Crank-Nicolson method for, 588–589 heat equation model as, 585–586 pseudocode for Crank-Nicolson method for, 589–590 pseudocode for explicit model of, 587 stability and, 591–593 Parametric representation, of curves, 394 Partial differential equations, 582–624 elliptic problems in, 605–624 finite-difference method for, 606–609 finite-element methods for, 613–619 Gauss-Seidel iterative method for, 610 Helmholtz equation model, 605–606 pseudocode for, 610–613 hyperbolic problems in, 596–605 advection equation as, 601 analytical solution for, 597–598 Lax method for, 602 Lax-Wendroff method for, 602–603 numerical solution for, 598–599 pseudocode for, 600–601 upwind method for, 602 wave equation model, 596–597 parabolic problems in, 582–596 applied, 582–585 Crank-Nicolson alternative method for, 590–591 Crank-Nicolson method for, 588–589 heat equation model as, 585–586 pseudocode for Crank-Nicolson method for, 589–590 pseudocode for explicit model of, 587 stability and, 591–593 Partial double-precision arithmetic, 492 (CPb 11.3.2) Partition of unity on interval, 417 Pascal’s triangle, 37 (CPb 1.2.10c) Penrose properties, 526–527 Pentadiagonal systems of linear equations, 280, 283–285 Periodic cubic splines, 387, 401 (Pb 9.2.23) Periodicity, 67, 598 Periodic sequences of random numbers, 535 Periodic spline filter, 292 (CPb 7.2.23) Permutation matrices, 307 Piecewise bilinear polynomial, 384 (CPb 9.1.3) Piecewise linear functions, 372 Pierce decomposition, 356 (Pb 8.2.6) 􏶜, computing value of, 12 (Pb 1.1.1, Pb 1.1.4) Pivoting, 246 pivot element for, 249, 271 pivot equation for, 247, 249 scaled partial, 259–280 complete partial pivoting and, 261 example of, 265–266 Gaussian elimination with, 262–264 long operational count and, 269–270 numerical stability and, 271 pseudocode for, 266–269 Poisson equation, 584, 605, 613, 615 Polygonal functions, 372 Polyhedral set, 671 Polynomial(s), 8, 11, 343 Polynomial interpolation, 124–164 algorithms and pseudocode for, 136–138 of bivariate functions, 144–145 derivative estimating by, 170–174 divided differences for calculating coefficients aj in, 131–136 errors in, 153–164 Dirichlet function as, 154 Runge function as, 154–156 theorems on, 156–160 inverse, 141–142 Lagrange form of, 126–128 linear, 125–126, 162 (Pb 4.2.8) nested form of, 130–131 Neville’s algorithm for, 142–144 Newton form of, 128–130 Vandermonde matrix for, 139–141 Polynomial regression, 510–515 Positive definite matrices, 305, 332–333, 345, 647 Power method for linear equations. See also Eigenvalues and eigenvectors Aiken acceleration formula for, 363 algorithms for, 361–362 inverse, 364–365 in mathematical software, 363 shifted inverse, 365–366 Precision, 3–6, 63–64, 688. See also IEEE floating-point standard arithmetic Preconditioning, 335 Predator-prey models, 465 Predictor-corrector scheme, 461 (CPb 10.3.4), 483–484 Prime numbers, 534, 540 Probability integral, 204 (CPb 5.2.5) Product, matrix, 711–713 Program libraries, 686–687 Programming derivatives, 9–10 Programming suggestions, 684–691 Projection, 356 (Pb 8.2.6) Projection operator, 722 Prony’s method, 530 (CPb 12.3.2) Protein folding, 655 (CPb 16.2.10) Pseudocode Adams-Bashforth-Moulton methods, 484–488 area and volume estimation, 545–547 bisection method, 78–79 as bridge, 684 B spline functions, 412–413 conjugate gradient algorithm, 334 Crank-Nicolson method, 589–590 elliptic problems, 610–613 Euler’s method, 432–433 explicit model of partial differential equations, 587 Gaussian elimination with scaled partial pivoting, 266–269 Gauss-Seidel method, 327, 610 hyperbolic problems, 600–601 Jacobi method, 327 linear equations, 327–328 loaded die problems, 552–553 matrix factorizations, 300 naive Gaussian elimination, 250–254 natural cubic spline functions, 392–394 Newton’s method, 92–93 numerical integration, 184–187 polynomial interpolation, 136–138 power method, 361–362 random numbers, 535, 537–541 Romberg algorithm, 205–206 Runge-Kutta-Fehlberg methods, 452 Runge-Kutta methods, 443–444, 453–454 Schoenberg’s process, 415 secant method, 112 shooting method for ordinary differential equations (ODE), 575–577 successive overrelaxation (SOR) method, 327 Taylor series of order 4, 468–469 Pseudo-inverse, of matrices, 525–526 Pseudo-random numbers, 533 Quadratic B spline, 423 (Pb 9.3.37) Quadratic convergence, 93, 100 Quadratic form, 333 Quadratic functions, 642, 652 (Pb 16.2.15) Quadratic interpolation algorithm, 633–635 Quadratic splines, 376–378 Quadrature rules, 187 Quasi-Newton methods for minimization of functions, 647 Quasi-random number sequences, 540 Radix point, 693 Random numbers, 532–544 algorithms and generators for, 533–535 examples of, 535–537 pseudocode for, 537–541 Random walk problem, 561 (CPb 13.3.17–18) Range, of computer, 45 Range reduction, 67–68 Rationalizing numerators, 64 Rayleigh quotient, 368 (Pb 8.3.7) Reciprocals of numbers, 102 (Pb 3.2.23) Recursive definition, in Newton’s method, 91 Recursive property of divided differences theorem, 134 Recursive trapezoid formula for equal subintervals, 196–197 Red-black ordering, 620 (Pb 15.3.3) Reflected points, 648 Regression, polynomial, 510–515 Regula falsi method, 83–84 Relative errors, 5 Relaxation factor, 326. See also Overrelaxation Remainder, 25 Residual, 254–255, 279 (CPb 7.2.19), 519, 619 Richardson extrapolation estimating derivatives and, 166–170, 177 (Pb 4.3.19) Euler-Maclaurin formula and, 207 of Romberg algorithm, 209–211 Richardson iteration, 322–323 Riemann-integrable functions, 183–184 Riffle shuffles, 562 (CPb 13.3.27) Rising sequences, 562 (CPb 13.3.27) Robust software, 269 Rolle’s Theorem, 156–157 Romberg algorithm convergence in, 165 description of, 204–205 Euler-Maclaurin formula and, 206–209 notation for, 196 pseudocode for, 205–206 Richardson extrapolation and, 168, 209–211 Roots of equations, locating, 76–123 bisection method for, 76–85 convergence analysis in, 81–83 example of, 79–81 false position method in, 83–84 pseudocode for, 78–79 Newton’s method for, 89–100 convergence analysis in, 93–96 fractile basins of attraction in, 99–100 interpretation of, 90–91 nonlinear equation systems in, 96–99 pseudocode in, 92–93 secant method for, 111–119 algorithm for, 112–113 bisection and Newton’s methods versus, 117 convergence analysis in, 114–116 fixed point iteration and, 117–118 Rounding modes, 705 Rounding numbers, 6, 50 Roundoff error, 50, 52, 54, 63, 253, 435, 687, 703 Round-to-even method, 6 Round to nearest value, 705 Row-equilibrated matrix, 275 (Pb 7.2.23) Row vectors, 706 Runge function, 125, 154–156 Runge-Kutta-England method, 463–464 (CPb 10.3.19) Runge-Kutta methods, 439–450 adaptive, 450–454 example of, 454–455 of order 5, 451 of order 4, 442–443 of order 3, 445–446 (Pb 10.2.7) of order 2, 441–442 pseudocode for, 443–444 for systems of ordinary differential equations, 469–472 Index 761 Taylor series in two variables and, 440–441 Saddle points of functions, 646 Scale vector, 262 Scaling, 271 Schoenberg’s process, 414–415 Scientific notation, normalized, 43 Secant method for locating roots of equations, 111–119 algorithm for, 112–113 bisection and Newton’s methods versus, 117 convergence analysis in, 114–116 fixed point iteration and, 117–118 Second bad case, of quadratic interpolation algorithm, 635 Second-derivative formulas, 173–174 Second primal form, in linear programming, 663–664 Seed, for random number sequence, 534 Serpentine curves, 395 Shifted inverse power method, 365–366 Shooting method for ordinary differential equations (ODE), 563–570 algorithm for, 565–567 in linear case, 574–575 overview of, 563–565 pseudocode for, 575–577 refinements to, 567 Shure’s Theorem, 346 Significance loss of, 61–68 avoiding in subtraction, 64–67 computer-caused, 62–63 range reduction and, 67–68 theorem for, 63–64 significant digits in, 3–5, 61 Significands, 47 Similar matrices, 345 Simplex method, 670–675 Simple zero, 93 Simpson’s rule, 216–229 adaptive, 221–225 basic, 216–220, 228 (Pb 6.1.8) composite, 220–221, 228 (Pb 6.1.6), 243 (CPb 6.2.11) Simulated annealing method, 648–649 Simulation, 552–562. See also Monte Carlo methods birthday problem as, 553–555 Buffon’s needle problem as, 555–556 loaded die problem as, 552–553 neutron shielding, 557–558 two dice problem as, 556–557 Simultaneous nonlinear equations, 104 (Pb 3.2.39) Sine integral, 189 (CPb 5.1.2), 204 (CPb 5.2.5), 463 (CPb 10.3.15) Single-precision floating-point representation, 46–47 Single-step error, 453 762 Index Single-step methods, 483 Singular value decomposition (SVD) economical version of, 356 (Pb 8.3.5) eigenvalues and eigenvectors and, 348–349 least squares method and, 519, 522–527 matrix spectral theory and, 350 numerical examples of, 351–353 Singular values, 320 sin x, periodicity of, 67 Smoothing data, 396–398. See also Chebyshev polynomials; Least squares method Software, mathematical, 10–11 boundary-value problem, 577 development of, 691 differential equations, 427 eigenvalues and eigenvectors, 343–344 error function in, 186 linear programming, 678–679 LU factorization, 308 matrix factorizations, 307–309 minimal solution, 526 minimization problems, 626 nonlinear equations, 99, 111 (CPb 3.2.42), 123 (CPb 3.3.19) partial differential equations, 584, 592 polynomial interpolation, 153 (CPb 4.1.11), 164 (CPb 4.2.12) power method for linear equations, 363 random numbers, 533, 535 robust, 269 roots of equations, 81, 88 (CPb 3.1.12), 93 singular value decomposition, 351 splines, 394, 409–410, 418 symbolic verification, 20 (CPb 1.1.26) vector fields, 430 Solution case, of quadratic interpolation algorithm, 634 Solutions for differential equations, 426 Sparse factorization, 315 (Pb 8.1.24) Spectral/l2-matrix norm, 320. See also Matrix spectral theory Spectral/l2-vector norm, 722 Spectral radius, 320, 329 Spectral theorem, 720–721 Spline functions, 371–425 B, 404–425 for Bézier curves, 416–418 interpolation and approximation by, 410–412 pseudocode and example of, 412–413 Schoenberg’s process for, 414–415 theory of, 404–410 first-degree, 371–374 interpolating quadratic, Q(x), 376–378 modulus of continuity in, 374–375 natural cubic, 385–404 algorithm for, 388–392 introduction to, 385–387 pseudocode for, 392–394 smoothness property from, 396–398 space curves from, 394–396 second-degree, 376 Subbotin quadratic, 378–380 Spurious zeros, 62 Stability numerical, 271 in ordinary differential equations (ODE), 456–459 in partial differential equations, 591–593 Standard deviation, 15 (CPb 1.1.7) Standard floating-point representation, 46 Stationary points of functions, 646 Statistician’s rounding, 6 Steady state of systems, 489 Steepest descent procedure, 643, 655 (CPb 16.2.2) Steffensen’s method, 104 (Pb 3.2.36) Stiff equations, 489–491 Stirling’s formula, 34 (Pb 1.2.47) Subbotin quadratic spline functions, 378–380 Subdiagonal matrix, 280, 710 Subnormal numbers, 704 Subordinate norms, 721 Subtraction, significance and, 64–67 Successive overrelaxation (SOR) method, 324, 326, 331–332 Superdiagonal matrix, 280, 710 Superlinear convergence, 84, 115 Supremum (least upper bound), 374 Symbolic computations, 435 Symbolic verification, 20 (CPb 1.1.26) Symmetric banded storage mode, 291 (CPb 7.2.20) Symmetric matrices, 332, 345, 640, 714–715 Symmetric positive definite (SPD) matrices, 305, 330 Symmetric storage mode, 278 (CPb 7.2.13) Synthetic division, 7 Tacoma Narrows Bridge project, 493 (CPb 11.3.9) Taylor series, 20–31, 177 (Pb 4.3.19) alternating series and, 28–30 complete Horner’s algorithm in, 23–24 derivative estimating by, 164–166 examples of, 20–22 of f at the point c, 22–23 for F in minimization of functions, 640–642 machine precision and, 70 (Pb 2.2.28) in Mean-Value Theorem, 26 for natural logarithm (ln), 1 for ordinary differential equations, 431–435, 466–469 Runge-Kutta methods and, 440–441 Taylor’s Theorem in terms of h and, 27–28 Taylor’s Theorem in terms of (x − c) and, 24–26 Telescoped rational functions, 73 (CPb 2.2.18) Tensor-product interpolation, 144 Tent function, 122 (CPb 3.3.15) Test cases, 685 Theorems alternating series, 28–30, 32 (Pb 1.2.13) axioms for a vector space, 716 bisection method, 82 Cayley-Hamilton, 358 (CPb 8.2.5) Cholesky factorizations, 305 cubic spline smoothness, 397 divided differences and derivatives, 159 duality, 662 eigenvlaues of similar matrices, 345 Euler-Maclaurin formula, 208 on existence of polynomial interpolation, 128 first-degree polynomial accuracy, 375 first-degree spline accuracy, 375 first primal form, 658 Fundamental Theorem of Calculus, 181, 195 Gaussian quadrature, 232 Gershgorin’s, 347 initial value problem uniqueness, 431 intermediate-value, 78, 194 on interpolation errors, 156–160 on interpolation properties, 143 invariance, 135 Jacobi and Gauss-Seidel convergence, 330 linear differential equations, 354 linear independence, 718 localization, 347 long operations, 270 on loss of precision, 63–64 LU factorization, 298 matrix spectral, 349 Mean-Value, 26, 397 Mean-Value Theorem for Integrals, 193 minimal solution, 525 Newton’s method of locating roots of equations, 94 orthogonal basis, 350 Penrose properties of pseudo-inverse, 526 on polynomial interpolation error, 156–160 primal and dual problems, 662 recursive property of divided differences, 134 recursive trapezoid formula, 197 Richardson extrapolation, 168 Riemann integral, 183 Rolle’s, 156–157 second primal form, 663 Shure’s, 346 spectral, 720–721 spectral radius, 329 spectral theorem for symmetric matrices, 720 successive overrelaxation (SOR), 331 SVD least squares, 523 Taylor’s, 166 Taylor’s Theorem in terms of h, 27–28 Taylor’s Theorem in terms of (x - c), 24–26 trapezoid rule precision, 192 vertices and column vectors, 671 Weierstrass approximation, 416 weighted Gaussian quadrature, 232 3-simplex sets, 648 Transpose of matrices, 345, 707, 713–714 Trapezoid rule, 190–204. See also Simpson’s rule composite, 191, 194, 243 (CPb 6.2.11) composite with unequal spacing, 203 (Pb 5.2.32) error analysis in, 192–196 multidimensional integration in, 198–199 recursive formula for equal subintervals in, 196–197 uniform spacing in, 191–192 Triangular inequality, 320, 721 Triangular matrix, 346, 710 Tridiagonal matrix, 709 Tridiagonal systems of linear equations, 280–282, 289 (CPb 7.2.12) Troesch’s problem, 581 (CPb 14.2.7) Truncated series, 25, 28 Truncation error, 165–166, 174, 435 Two dice problem, 556–557 Two-dimensional integration over the unit square, 198 2-simplex sets, 648 Unconstrained minimization problems, 625–626 Underflow, of range, 45 Undetermined coefficients, method of, 233 Uniformly distributed numbers, 533 Unimodal functions F, 627–628 Unitarily similar matrices, 345–346 Unit roundoff error, 50, 703 Unit vectors, 708 Unstable functions, roots as, 88 (CPb 3.1.12) Upper bound lemma, 157 Upper triangular matrix, 710 Upper triangular system, 248 Upwind method, 602 Usual case, of quadratic interpolation algorithm, 634 Vandermonde matrix, 139–141, 152 (Pb 4.1.47), 254 Variable metric algorithm, 647 Variables, declaring, 685–686 Variance, 15 (CPb 1.1.7, CPb 1.1.8) Vector norms, 319–320, 721 Vector notation, 467–469 Vectors. See also Abstract vector spaces in linear algebra; Eigenvalues and eigenvectors column, 671 A-conjugate, 332 convex hull of, 417 direction, 333 gradient, 640–641 index, 262, 266 inner product of, 332 in linear algebra, 706–708 matrix-vector product and, 711 in ordinary differential equations (ODE), 429–431 residual, 254–255, 279 (CPb 7.2.19) scale, 262 vector inequality of, 658 Verification, symbolic, 20 (CPb 1.1.26) Vertices in K , 671–672 Volume estimation. See Area and volume estimation Warning messages, 685 Wave equation model, 582, 584, 596–597 Weierstrass approximation theorem, 416 Weight function, 519–520 Weights, Gaussian, 230, 232–234 Wilkinson’s polynomial, 88 (CPb 3.1.12), 121 (CPb 3.3.9) Zeros of f , 76–77, 81 of multiplicity, 96, 104 (Pb 3.2.35) simple, 93 spurious, 62 Index 763 This page intentionally left blank Formulas from Integral Calculus 􏶩 􏶩 xa+1 xa dx = (a + 1) + C (a ̸= 1) 􏶩􏶩 ex dx = ex + C 􏶩1􏶩 eax dx = aeax +C 􏶩1􏶩 xeax dx = a2 eax (ax − 1) + C 􏶩􏶩 x−1 dx =ln|x|+C 􏶩􏶩 ln x d x = x ln |x | − x + C 􏶩x2x2 􏶩 xlnxdx= 2 ln|x|− 4 +C 􏶩 dx 1 􏶩 􏶩dx−1 􏶩 (a+bx)2 =b(a+bx)+C 􏶩 dx 1 􏶨􏶨 x 􏶨􏶨 􏶩 􏶩 dx 1 􏰒x􏰓 a2+x2 =aarctan a +C 􏶩dx􏰒x􏰓 √ = arcsin + C cosxdx=sinx+C tanxdx=ln|secx|+C secxdx=ln|secx+tanx|+C xsinxdx=sinx−xcosx+C sec2xdx=tanx+C secxtanxdx=secx+C sinhxdx=coshx+C coshxdx=sinhx+C tanhxdx=ln|coshx|+C a+bx =bln|a+bx|+C ln 􏶨􏶨 􏶨􏶨 + C ax + b cothxdx=ln|sinhx|+C 􏶩dx1􏰔1√􏰕 􏶩x1 = a+bx2 =√abarctan ax ab +C x(ax + b) b sin2 x d x = 2 − 4 sin 2x + C 􏶩 x1 arcsinxdx=xarcsinx+ 1−x2+C 􏶩 􏶣 (a̸=1) (a ̸= 1) cos2xdx=2+4sin2x+C 􏶩 􏶣 a2 −x2 a 􏶩1􏶨􏶨􏶣􏶨􏶨 √ x2 ±a2 dx= x2 ±a2 ± ln􏶨x+ x2 ±a2􏶨+C 􏶩22􏶩 sin x dx = −cos x + C arccosxdx=xarccosx− 1−x2+C 1 􏰉 2􏰈 arctanxdx=xarctanx− ln 1+x +C 2 F′(g(x))g′(x) dx = F(g(x)) + C dx=ln􏶨 x2+a2+x􏶨+C 􏶩􏶣x􏶣a2􏶨􏶨􏶣􏶨􏶨􏶩 x2 +a2 Fundamental Theorem of Calculus Integration by Parts d􏶩x 􏶩􏶩 f(t)dt= f(x) Mean Value for Integrals 􏶩b 􏶩b f(x)g(x)dx = f(ξ) g(x)dx aa udv=uv− vdu dx a (g(x)􏶟0) Series x x2x3x4x5x6 􏰃∞xk e =1+x+2!+3!+4!+5!+6!+···= (|x|<∞) ( |x| < ∞) x = 1 + x ln a + (xlna)2 2! + + (xlna)3 3! + · · · = a sin x = x − 3! x 11 11! k=0 􏰃∞ k! (−1)k x 2k +1 (2k + 1)! x2k (2k)! 􏰒 2 π2􏰓 x3 1 3 x5 1 3 5 x7 arcsinx=x+ 6 +24 5 +246 7 +··· (x2 <1) cosx =1− 2! x2 x4 x6 x8 x10 +···= (−1)k k=0 + tanx=x+ 3 + 15 + 315 +2835+··· x < 4 − + 8! − k! 􏰃∞ (xlna)k k=0 x 3 + x 5 5! − x 7 7! x 9 9! − + · · · = ( |x| < ∞) (|x|<∞) k=0 􏰃∞ x3 4! 2x5 6! 17x7 10! 62x9 x 3 x 5 arctanx=x− + (x2 <1) (−1 < x 􏶞 1) 􏰃∞ 357 k=0 (2k+1) x 2 k + 1 x2 x3 x4 􏰃∞ xk 􏰔1+x􏰕􏰖x3x5x7 􏰗􏰃∞x2k−1 x 7 (−1)k 234 k=1 k − ln(1 + x) = x − + − + · · · = (−1)k−1 +···= ln 1−x =2 x+ 3 + 5 − 7 +··· =2 n(n−1) n(n−1)(n−2) (|x|<1) xn−3y3 +···= (|x|<1) (x+y)n =xn +nxn−1y+ 2! xn−2y2 + 1 􏰃∞ 1−x =1+x+x2 +x3 +x4 +x5 +···= Formal Taylor Series for f about c k=0 k xn−kyk k=1 2k−1 3! 􏰃n 􏰔n􏰕 k=0 f(x)∼ f(c)+ f′(c)(x−c)+ f′′(c)(x−c)2 + f′′′(c)(x−c)3 +···=􏰃∞ f(k)(c)(x−c)k 2! 3! k=0k! xk Taylor Series for f (x) f (x) = 􏰃n f (k)(c)(x − c)k + En+1 where En+1 = f (n+1)(ξ)(x − c)n+1 k=0 k! (n + 1)! Taylor Series for f (x + h) f (x + h) = 􏰃n f (k)(x)hk + En+1 where En+1 = f (n+1)(ξ)hn+1 k=0 k! (n + 1)! Alternating Series Ifa1􏶟a2􏶟 ···􏶟an􏶟 ···􏶟0forallnandlimn→∞an =0then 􏰃∞ 􏰃n k=1 (−1)k−1ak = lim (−1)k−1ak = lim Sn = S. Moreover, |S − Sn|􏶞 an+1 for all n. n→∞ n→∞ k=1 Mean-Value Theorem f(b)= f(a)+(b−a)f′(ξ) forsomeξ in(a,b)