Web Building Data
Probability of being caught with web size w
Probability of catching prey with web size w
Energy cost of building web of size w with current weight s Basal metabolic expenditure
Energy value of prey
Weight, st Actions
Websize,w∈W ={4,8,12} Value Function
Vt(st) is the maximum probability of reaching 80 mg by the end of Day 10 starting Day t with weight st.
1, ifs10≥80 V10(s10) = 0, if s10 < 80
Vt(st)=1ifst ≥80
β w × 0 Vt(st) = max +(1 − βw)λwVt+1(st − αm + b − αw(sw))
w∈W +(1 − βw)(1 − λw)Vt+1(st − αm − αw(sw))
We want to explore the optimal action associated with V0(s0) for varying val- ues of s0.
1
βw
λw αw (s) αm b
Stages
Days, t State
Democracy Data
dj Desirable representatives for city j Stages
Cities, j ∈ {1, 2, 3} State
Number of unallocated representatives, sj Actions
Number to allocate to city j, aj Value Function
Vj (sj ) is minimum of the maximum discrepancy between the desired and ac- tualnumberofrepresentativesforcitiesj,...,3withsj available.
We want V1(3).
V3(s3) = |d3 − s3|
Vj(sj)= min {max(|dj −aj|,Vj+1(sj −aj))}
0≤aj ≤sj
2
Altitude Data
hij maximum altitude of road from i to j Stages
Cities, i ∈ {A, B, . . . , J} State
None
Actions
Next city to visit, ai Value Function
Vi is minimum maximum altitude of driving from i to J. VJ =0
WewantVA.
Vi = min {max(hij,Vj)} j∈D(i)
3
Betting Data
p Probability of winning Stages
Games, j ∈ {1, 2, 3} State
How much money she has, $sj Actions
How much to bet on game j, bj Value Function
Vj(sj) is maximum probability of having at least $5 after three games if we start game j with $sj .
Vj(sj) = We want V1(2).
1, if s4 ≥ 5 V4(s4) = 0, if s4 < 5
max {pVj+1(sj + bj) + (1 − p)Vj+1(sj − bj)} 0≤bj ≤sj
4
Advertising Data
psa probability of high sales if sales were s and we take action a rs revenue ($) if sales are s
c cost ($) of changing production d cost ($) of advertising
Stages
Weeks, t ∈ {1, 2, 3, 4} State
Sales level in previous week, st ∈ {H, L} Actions
Advertise,at ∈{Y,N} Value Function
Vt(st) is maximum expected profit for weeks t, . . . , 4 if we start week t with st sales in the previous week.
V5(s5) = 0
pHY(rH −d+Vt+1(H))+(1−pHY)(rL −d−c+Vt+1(L)) [at =Y]
Vt(H)=max pHN(rH +Vt+1(H))+(1−pHN)(rL −c+Vt+1(L)) [at =N] pLY(rH −d−c+Vt+1(H))+(1−pLY)(rL −d+Vt+1(L)) [at =Y]
Vt(L)=max pLN(rH −c+Vt+1(H))+(1−pLN)(rL +Vt+1(L)) [at =N] WewantV1(s1)fors1 ∈{H,L}.
5