Transcript CPU Utilization Control in Distributed Real-Time Systems Chenyang Lu Department of Computer Science and Engineering SIGMETRICS 2008: Introduction to Control Theory.

CPU Utilization Control in
Distributed Real-Time Systems
Chenyang Lu
Department of Computer Science and Engineering
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
Why CPU Utilization Control?
Overload protection
CPU over-utilization  system crash
Nightmare for mission-critical applications and
“always-on” E-businesses
Meet deadlines
CPU utilization < schedulable utilization bound
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
2
End-to-End Task Model
in Distributed Real-Time Systems
 Periodic task Ti = a chain of subtasks {Tij} located on
different processors
 Subtasks run at a same rate
 Task rate can be adjusted within a range
 Higher rate  higher utility
T1
T11
T12
T13
T3
T2
P1
Remote Invocation
Subtask
P2
P3
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
3
Problem Formulation
 Bi: Utilization set point of processor Pi (1 ≤ i ≤ n)
 ui(k): Utilization of Pi in kth sampling period
 rj(k): Rate of task Tj (1 ≤ j ≤ m) in kth sampling period
n
min
{r j ( k )|1 j  n}
2
(
B

u
(
k
))
 i i
i 1
subject to rate constraints:
Rmin,j  rj(k)  Rmax,j (1 ≤ j ≤ m)
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
4
Challenge: Uncertainties
 Execution times?
 Unknown sensor data or user input
 Request arrival rate?
 Aperiodic events
 Bursty service requests
 Disturbance?
 Denial of Service Attacks
Control-theoretic approaches to adaptive software
 Robust performance in face of workload uncertainty
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
5
Single-Processor Solution:
Feedback Control Real-Time Scheduling
 Adaptation based on single-input-single-output control
Sensor Inputs
FCS
Set point
Us = 69%
Task Rates
R1: [1, 5] Hz
R2: [10, 20] Hz
{r(k+1)}
Controller
Actuator
u(k)
Application
Middleware
Monitor
OS
Processor
C. Lu, X. Wang, and C. Gill, Feedback Control Real-Time Scheduling in
ORB Middleware, IEEE Real-Time and Embedded Technology and
Applications Symposium (RTAS'03), May 2003.
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
6
What’s New in Distributed Systems?
 Need to control utilization of multiple processors
 Utilization of different processors are coupled with each
other due to end-to-end tasks
 Replicating FCS on all processors does not work!
 Constraints on task rates
T1
T11
T12
T13
T3
T2
P1
P2
P3
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
7
EUCON: Multi-Input-Multi-Output Control
Measured
Output
 B1 
 ,
 
 Bn 
 Rmin,1 Rmax,1 




 Rmin,m Rmax,m 


 u1 (k ) 
  


u n (k )
Utilization
Monitor
Model
Predictive
Controller
Control
Input
Distributed System
(m tasks, n processors)
Rate
Modulator
 r1 ( k ) 
  


 rm ( k ) 
UM
UM
RM
RM
Feedback Loop
Precedence Constraints
Subtask
C. Lu, X. Wang and X. Koutsoukos, Feedback Utilization Control in
Distributed Real-Time Systems with End-to-End Tasks, IEEE Transactions
on Parallel and Distributed Systems, 16(6): 550-561, June 2005.
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
8
Control Theoretic Methodology
1.
2.
3.
Derive a dynamic model of the controlled system
Design a controller
Analyze stability
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
9
Dynamic Model: One Processor
ui (k )  ui (k  1)  g i
c
T jl S i
jl
rj (k  1)
 Si: set of subtasks on Pi
 cjl: estimated execution time of Til running on Pi
may not be correct
 gi: utilization gain of Pi
unknown ratio between actual and estimated change
in utilization
models uncertainty in execution times
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
10
Dynamic Model: Multiple Processors
u(k) = u(k-1) + GFr(k-1)
 G: diagonal matrix of utilization gains
 F: subtask allocation matrix
models the coupling among processors
 fij = cjl task Tj has a subtask Tjl on processor Pi
 fij = 0 if Tj has no subtask on Pi
T1
T11
T22
T3
T2 T21
P1
c11
F
0
c 21
c 22
0

c31 
T31
P2
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
11
Model Predictive Control
 Advanced control technique for coupled MIMO control
problems with actuator constraints.
 Minimize a cost function over an interval in the future.
 Compute an input trajectory that minimizes cost subject to
actuator constraints.
 Predict cost based on a system model and feedback.
 Combines optimization, model-based prediction, and
feedback.
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
12
Model Predictive Controller
At a sampling instant
 Compute inputs in future sampling periods
r(k), r(k+1), ... r(k+M-1)
to minimize a cost function
 Cost is predicted using
(1) feedback u(k-1)
(2) approximate dynamic model
 Apply r(k) to the system
At the next sampling instant
 Shift time window and re-compute r(k+1), r(k+2), ...
r(k+M) based on feedback
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
13
Model Predictive Controller in EUCON
 B1 

 
 Bn 
System
Model
Rate
Constraints
Least Squares Solver
 u1 (k ) 
  


u n (k )
Difference with
reference
trajectory
Cost
Function
Constrained
optimization
solver
 r11 (k )1) 
   
  


rrmm ((kk) 1) 
Reference
Trajectory
Model Predictive Controller
Desired trajectory for
u(k) to converge to
B
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
14
Stability Analysis
 Stability: system converges to equilibrium point from any
initial condition
Equilibrium point = utilization set points B
If stable, utilization of all processors converge to their
set points whenever feasible
 Derive stability condition  tolerable range of G
tolerable variation of execution times
 Stability analysis establishes analytical guarantees on
utilization despite uncertainty
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
15
Simulation: Stable System
CPU utilization
1
0.8
0.6
0.4
0.2
0
0
50
100
150
200
250
300
Time (sampling period)
P1
P2
Set Point
execution time factor = 0.5
(actual execution times = ½ of estimates)
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
16
Simulation: Unstable System
CPU utilization
1
0.8
0.6
0.4
0.2
0
0
100
200
300
Tim e (sam pling period)
P1
P2
Set Point
execution time factor = 7
(actual execution times = 7 times estimates)
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
17
C PU uilization
Stability
 Stability: system converges to desired utilizations from any initial
condition
 Derive stability condition  tolerable range of execution times
Analytical assurance on utilizations despite uncertainty
1
Deviation
Average
Set point
0.95
0.9
0.85
0.8
0.75
Overestimation
of execution
times prevents
oscillation
0.7
0.65
0.6
0.55
Predicted
bound for
stability
0.5
0
2
4
6
execution-time factor
8
10
actual execution time / estimation
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
18
FC-ORB Middleware
Measured
Output
Feedback lane
 u1 ( k ) 
u ( k ) 
 2 
u 3 ( k ) 
Model
Predictive
Controller
Control
Input
 r1 (k ) 
r (k )
2 
Rate
Modulator
Rate
Modulator
Rate
Modulator
Priority
Manager
Priority
Manager
Priority
Manager
Utilization
Monitor
Utilization
Monitor
Utilization
Monitor
Remote request lanes
Remote request lanes
X. Wang, C. Lu and X. Koutsoukos, Enhancing the Robustness of
Distributed Real-Time Middleware via End-to-End Utilization Control, IEEE
Real-Time Systems Symposium (RTSS'05), December 2005.
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
19
Workload Uncertainty
disturbance from periodic tasks
time-varying execution times
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
20
Processor Failure
1.
2.
3.
4.
Norbert fails.
move its tasks to other processors.
reconfigure controller
control utilization by adjusting
task rates
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
21
Summary: Model Predictive Control
 Robust utilization control for distributed systems
 Handles coupling among processors
 Enforce constraints on task rates
 Analyze tolerable range of execution times
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
22
References
 Centralized control: EUCON
 C. Lu, X. Wang and X. Koutsoukos, Feedback Utilization Control in Distributed
Real-Time Systems with End-to-End Tasks, IEEE Transactions on Parallel and
Distributed Systems, 16(6): 550-561, June 2005.
 Decentralized control: DEUCON
 X. Wang, D. Jia, C. Lu and X. Koutsoukos, DEUCON: Decentralized End-to-End
Utilization Control for Distributed Real-Time Systems, IEEE Transactions on
Parallel and Distributed Systems, 18(7): 996-1009, July 2007.
 Middleware: FC-ORB
 X. Wang, C. Lu and X. Koutsoukos, Enhancing the Robustness of Distributed
Real-Time Middleware via End-to-End Utilization Control, IEEE Real-Time
Systems Symposium (RTSS'05), December 2005.
 Controllability and feasibility
 X. Wang, Y. Chen, C. Lu and X. Koutsoukos, On Controllability and Feasibility of
Utilization Control in Distributed Real-Time Systems, Euromicro Conference on
Real-Time Systems (ECRTS'07), July 2007.
 Project page: http://www.cse.wustl.edu/~lu/aqc.htm
SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.
23