Transcript A Hierarchical Approach to POMDP Planning and Execution

Hierarchical Methods for
Planning under Uncertainty
Thesis Proposal
Joelle Pineau
Thesis Committee:
Sebastian Thrun, Chair
Matthew Mason
Andrew Moore
Craig Boutilier, U. of Toronto
Integrating robots in living environments
The robot’s role:
- Social interaction
- Mobile manipulation
- Intelligent reminding
- Remote-operation
- Data collection / monitoring
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
A broad perspective
Belief
state
OBSERVATIONS
STATE
USER + WORLD + ROBOT
ACTIONS
GOAL = Selecting appropriate actions
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Why is this a difficult problem?
UNCERTAINTY
Cause #1: Non-deterministic effects of actions
Cause #2: Partial and noisy sensor information
Cause #3: Inaccurate model of the world and the user
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Why is this a difficult problem?
UNCERTAINTY
Cause #1: Non-deterministic effects of actions
Cause #2: Partial and noisy sensor information
Cause #3: Inaccurate model of the world and the user
A solution: Partially Observable Markov
Decision Processes (POMDPs)
S1
o1, o2
a1
a2
S2
o1, o 2
S3
o1, o2
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
The truth about POMDPs
• Bad news:
– Finding an optimal POMDP action selection policy is
computationally intractable for complex problems.
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
The truth about POMDPs
• Bad news:
– Finding an optimal POMDP action selection policy is
computationally intractable for complex problems.
• Good news:
– Many real-world decision-making problems exhibit structure
inherent to the problem domain.
– By leveraging structure in the problem domain, I propose an
algorithm that makes POMDPs tractable, even for large domains.
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
How is it done?
• Use a “Divide-and-conquer” approach:
– We decompose a large monolithic problem into a collection of
loosely-related smaller problems.
Dialogue
manager
Health
manager
Social
manager
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Reminding
manager
Joelle Pineau
Thesis statement
Decision-making under uncertainty
can be made tractable for complex problems
by exploiting hierarchical structure
in the problem domain.
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Outline
• Problem motivation
 Partially observable Markov decision processes
• The hierarchical POMDP algorithm
• Proposed research
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
POMDPs within the family of Markov models
Uncertainty in sensor input?
no
yes
no
Markov Chain
Hidden Markov Model
(HMM)
yes
Markov Decision Process
(MDP)
Control
problem?
Partially Observable MDP
(POMDP)
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
What are POMDPs?
0.5
Pr(o1)=0.9
Pr(o2)=0.1
0.5
S1
Pr(o1)=0.5
Pr(o2)=0.5
a1
a2
Components:
Set of states: sS
Set of actions: aA
Set of observations: oO
S2
1
S3
Pr(o1)=0.2
Pr(o2)=0.8
POMDP parameters:
Initial belief: b0(s)=Pr(so=s)
Observation probabilities: O(s,a,o)=Pr(o|s,a)
Transition probabilities: T(s,a,s’)=Pr(s’|s,a)
Rewards: R(s,a)
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
HMM
MDP
Joelle Pineau
A POMDP example: The tiger problem
S1
“tiger-left”
S2
“tiger-right”
Pr(o=growl-left)=0.85
Pr(o=growl-right)=0.15
Pr(o=growl-left)=0.15
Pr(o=growl-right)=0.85
Actions={ listen,
open-left,
open-right}
Reward Function: R(a=listen)
= -1
R(a=open-right, s=tiger-left) = 10
R(a=open-left, s=tiger-left) = -100
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
What can we do with POMDPs?
1) State tracking:
Not so hard.
– After an action, what is the state of the world, st ?
2) Computing a policy:
Very hard!
– Which action, aj, should the controller apply next?
World:
St-1
at-1
Control layer:
Robot:
bt-1
...
st
ot
??
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
??
...
Joelle Pineau
The tiger problem: State tracking
b0
Belief vector
S1
“tiger-left”
S2
“tiger-right”
Belief
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
The tiger problem: State tracking
b0
Belief vector
S1
“tiger-left”
S2
“tiger-right”
Belief
obs=growl-left
action=listen
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
The tiger problem: State tracking
b1
b0
b1 si  
Po | si , a   Psi | s j , a b0 s j 
s j S
Po | a, b
Belief vector
S1
“tiger-left”
S2
“tiger-right”
Belief
obs=growl-left
action=listen
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Policy Optimization
• Which action, aj, should the controller apply next?
– In MDPs:
• Policy is a mapping from state to action, : si  aj
– In POMDPs:
• Policy is a mapping from belief to action, : b  aj
• Recursively calculate expected long-term reward for each state/belief:
N


V ( si )  max R( si , a)    Pr(s j | si , a)V ( s j ) 
a
j 1


• Find the action that maximizes the expected reward:
N


 ( si )  arg max R( si , a)    Pr(s j | si , a)V ( s j ) 
a
j 1


Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
The tiger problem: Optimal policy
Optimal policy:
Belief vector:
S1
“tiger-left”
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
S2
“tiger-right”
Joelle Pineau
Complexity of policy optimization
Complexity
MDP
(per step of
value iteration)
POMDP
recursive
upper-bound
Time
| S |2 | A |
| S |2 | A || n1 ||O|
| S |2 | A ||O|
Space
|S|
| A || n1 ||O|
| A ||O|
n
n
| n |
• Finite-horizon POMDPs are in worse-case doubly exponential:
• Infinite-horizon undiscounted stochastic POMDPs are EXPTIME-hard,
and may not be decidable (|n|).
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
The essence of the problem
• How can we find good policies for complex POMDPs?
• Is there a principled way to provide near-optimal policies
in reasonable time?
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Outline
• Problem motivation
• Partially observable Markov decision processes
 The hierarchical POMDP algorithm
• Proposed research
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
A hierarchical approach to POMDP planning
• Key Idea: Exploit hierarchical structure in the problem domain to
break a problem into many “related” POMDPs.
subtask
• What type of structure?
Action set partitioning
Act
InvestigateHealth
CheckPulse CheckMeds
abstract
action
Move
Navigate
AskWhere
Left Right Forward Backward
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Assumptions
• Each POMDP controller has a subset of Ao.
• Each POMDP controller has full state set S0, observation set O0.
• Each controller includes discriminative reward information.
• We are given the action set partitioning graph.
• We are given a full POMDP model of the problem: {So,Ao,Oo,Mo}.
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
The tiger problem: An action hierarchy
act
open-left
investigate
listen
open-right
Pinvestigate={S0, Ainvestigate, O0, Minvestigate}
Ainvestigate={listen, open-right}
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Optimizing the “investigate” controller
Locally optimal
policy:
Belief vector:
S1
“tiger-left”
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
S2
“tiger-right”
Joelle Pineau
The tiger problem: An action hierarchy
Pact={S0, Aact, O0, Mact}
Aact={open-left, investigate}
act
But... R(s, a=investigate)
is not defined!
open-left
investigate
listen
open-right
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Modeling abstract actions
Insight: Use the local policy of corresponding low-level controller.
General form: R( si, ak) = R ( si, Policy(controllerk,si) )
Example: R(s=tiger-left,ak =investigate) =
Policy (investigate,s=tiger-left) = open-right
open-right
tiger-left
tiger-right
listen
open-left
10
-1
-100
-100
-1
10
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Optimizing the “act” controller
Locally optimal
policy:
Belief vector:
S1
“tiger-left”
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
S2
“tiger-right”
Joelle Pineau
The complete hierarchical policy
Hierarchical policy:
Belief vector:
S1
“tiger-left”
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
S2
“tiger-right”
Joelle Pineau
The complete hierarchical policy
Optimal policy:
Hierarchical policy:
Belief vector:
S1
“tiger-left”
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
S2
“tiger-right”
Joelle Pineau
Results for larger simulation domains
POMDP
H-POMDP
MDP
CPU Time (secs):
1119.93
2.84
0.000654
Average Reward:
12.5
12.2
0.0
>24hrs
77.99
6.46
Average Reward:
64.43
53.33
%Correct actions:
93.2
80.0
Navigation Problem:
|S|=11, |A|=6, |O|=6
Dialogue Problem:
|S|=20, |A|=30, |O|=27
CPU Time (secs):
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Related work on hierarchical methods
• Hierarchical HMMs
– Fine et al., 1998
• Hierarchical MDPs
– Dayan&Hinton, 1993; Dietterich, 1998; McGovern et al., 1998; Parr&Russell,
1998; Singh, 1992.
• Loosely-coupled MDPs
– Boutilier et al., 1997; Dean&Lin, 1995; Meuleau et al. 1998; Singh&Cohn, 1998;
Wang&Mahadevan, 1999.
• Factored state POMDPs
– Boutilier et al., 1999; Boutilier&Poole, 1996; Hansen&Feng, 2000.
• Hierarchical POMDPs
– Castanon, 1997; Hernandez-Gardiol&Mahadevan, 2001; Theocharous et al., 2001;
Wiering&Schmidhuber, 1997.
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Outline
• Problem motivation
• Partially observable Markov decision processes
• The hierarchical POMDP algorithm
 Proposed research
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Proposed research
1) Algorithmic design
2) Algorithmic analysis
3) Model learning
4) System development and application
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Research block #1: Algorithmic design
• Goal 1.1: Developing/implementing hierarchical POMDP algorithm.
• Goal 1.2: Extending H-POMDP for factorized state representation.
• Goal 1.3: Using state/observation abstraction.
• Goal 1.4: Planning for controllers with no local reward information.
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Goal 1.3: State/observation abstraction
• Assumption #2:
“Each POMDP controller has full state set S0, and observation set O0.”
• Can we reduce the number of states/observations, |S| and |O|?
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Goal 1.3: State/observation abstraction
• Assumption #2:
“Each POMDP controller has full state set S0, and observation set O0.”
• Can we reduce the number of states/observations, |S| and |O|?
Yes! Each controller only needs subset of state/observation features.
Navigate
InvestigateHealth
CheckPulse CheckMeds
Left Right Forward Backward
• What is the computational speed-up?
recursive
Time complexity:
POMDP
upper-bound
| S |2 | A || n 1 ||O|
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
2
|O|n
| S | | A|
Joelle Pineau
Goal 1.4: Local controller reward information
• Assumption #3:
“Each controller includes some amount of discriminative reward
information.”
• Can we relax this assumption?
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Goal 1.4: Local controller reward information
• Assumption #3:
“Each controller includes some amount of discriminative reward
information.”
• Can we relax this assumption?
Possibly. Use reward shaping to select policy-invariant reward function.
• What is the benefit?
– H-POMDP could solve problems with sparse reward functions.
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Research block #2: Algorithmic analysis
• Goal 2.1: Evaluating performance of the H-POMDP algorithm.
• Goal 2.2: Quantifying the loss due to the hierarchy.
• Goal 2.3: Comparing different possible decompositions of a problem.
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Goal 2.1: Performance evaluation
• How does the hierarchical POMDP algorithm compare to:
– Exact value function methods
» Sondik, 1971; Monahan, 1982; Littman, 1996; Cassandra et al, 1997.
– Policy search methods
» Hansen, 1998; Kearns et al., 1999; Ng&Jordan, 2000; Baxter&Bartlett, 2000.
– Value approximation methods
» Parr&Russell, 1995; Thrun, 2000.
– Belief approximation methods
» Nourbakhsh, 1995; Koenig&Simmons, 1996; Hauskrecht, 2000; Roy&Thrun, 2000.
– Memory-based methods
» McCallum, 1996.
• Consider problems from POMDP literature and dialogue management
domain.
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Goal 2.2: Quantifying the loss
• The hierarchical POMDP planning algorithm provides an
approximately-optimal policy.
• How “near-optimal” is the policy?
• Subject to some (very restrictive) conditions:
“The value function of top-level controller
is an upper-bound on the value
of the approximation.”
Atop
...
A1
...
• Can we loosen the restrictions? Tighten the bound?
Find a lower-bound?
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Goal 2.3: Comparing different decomposition
• Assumption #4:
“We are given an action set partitioning graph.”
• What makes a good hierarchical action decomposition?
a1
a1
a2
Manufacture Examine
a2
Replace
Inspect
Manufacture Replace
a3
Examine Inspect
• Comparing decompositions is the first step towards automatic
decomposition.
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Research block #3: Model learning
• Goal 3.1: Automatically generating good action hierarchies.
– Assumption #4: “We are given an action set partitioning graph.”
– Can we automatically generate a good hierarchical decomposition?
– Maybe. It is being done for hierarchical MDPs.
• Goal 3.2: Including parameter learning.
– Assumption #5: “We are given a full POMDP model of the problem.”
– Can we introduce parameter learning?
– Yes! Maximum-likelihood parameter optimization (Baum-Welch) can
be used for POMDPs.
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Research block #4: System development and application
• Goal 4.1: Building an extensive dialogue manager
Remote-control
command
Teleoperation module
Reminder
message Reminding
Facemail operations
module
Touchscreen input
Speech utterance
Status information
Touchscreen message
Speech utterance
User
Robot sensor readings
Robot module
Motion
command
Dialogue Manager
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
An implemented scenario
Problem size: |S|=288, |A|=14, |O|=15
State Features: {RobotLocation, UserLocation,
UserStatus, ReminderGoal,
UserMotionGoal, UserSpeechGoal}
Patient
room
Robot
home
Test subjects: 3 elderly residents in assisted living facility
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Physiotherapy
Joelle Pineau
Contributions
• Algorithmic contribution: A novel POMDP algorithm based
on hierarchical structure.
 Enables use of POMDPs for much larger problems.
• Application contribution: Application of POMDPs to
dialogue management is novel.
 Allows design of robust robot behavioural managers.
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Research schedule
1) Algorithmic design/implementation
fall 01
2) Algorithmic analysis
spring/summer 02
3) Model learning
spring/summer/fall 02
4) System development and application
ongoing
5) Thesis writing
fall 02 / spring 03
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Questions?
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
A simulated robot navigation example
Act
($$)
($$)
ReadMap
Read
GetReward(t)
Navigate(t)
GoLeft
OpenDoor
GoRight GoBack GoForward
Domain size: |S|=11, |A|=6, |O|=6
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
A dialogue management example
- SayTime
Act
CheckHealth
Greet
Move
CheckWeather
- AskHealth
- OfferHelp
DoMeds
Phone
- GreetGeneral
- GreetMorning
- GreetNight
- RespondThanks
- AskGoWhere
- GoToRoom
- GoToKitchen
- GoToFollow
- VerifyRoom
- VerifyKitchen
- VerifyFollow
- AskWeatherTime
- SayCurrent
- SayToday
- SayTomorrow
- AskCallWho
- Call911
- CallNurse
- CallRelative
- Verify911
- VerifyNurse
- VerifyRelative
- StartMeds
- NextMeds
- ForceMeds
- QuitMeds
Domain size: |S|=20, |A|=30, |O|=27
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Action hierarchy for implemented scenario
Act
Remind
Assist
RemindPhysio
PublishStatus
Contact
RingBell
GotoRoom
Rest
VerifyBring
VerifyRelease
Move
Recharge
GotoHome
Inform
BringtoPhysio
CheckUserPresent
DeliverUser
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
SayTime
SayWeather
VerifyRequest
Joelle Pineau
Sondik’s parts manufacturing problem
Decomposition2:
Decomposition1:
a1
a2
a1
Manufacture Examine
a2
Manufacture Examine
a3
Inspect
Replace
Replace
Inspect
+5 more decompositions
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Manufacturing task results
0.5
Avg. Reward
0.4
0.3
0.2
0.1
P
M
D
7
D
6
D
5
D
4
D
3
D
2
D
1
D
PO
M
D
P
0
Planning Method
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Using state/observation abstraction
Action Set:
CheckHealth
- AskHealth
- OfferHelp
DoMeds
State Set:
ReminderGoal={none, medsX}
CommunicationGoal={none, personX}
UserHealth={good, poor, emergency}
Phone
Phone
- AskCallWho
- CallHelp
- CallNurse
- CallRelative
- VerifyHelp
- VerifyNurse
- VerifyRelative
CommunicationGoal={none, nurse, 911, relative}
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Related work on robot planning and control
• Manually-scripted dialogue strategies:
– Denecke&Waibel, 1997; Walker et al., 1997.
• Markov decision processes (MDPs) for dialogue management
– Levin et al., 1997; Fromer, 1998; Walker et al., 1998; Goddeau&Pineau, 2000; Singh
et al., 2000; Walker, 2000.
• Robot interface:
– Torrance, 1996; Asoh et al., 1999.
• Classical planning
– Fikes&Nilsson, 1971; Simmons, 1987; McAllester&Rosenblitt, 1991;
Penberthy&Weld, 1992; Kushmerick, 1995; Veloso&al., 1995; Smith&Weld, 1998.
• Execution architectures
–
Firby, 1987; Musliner, 1993; Simmons, 1994; Bonasso&Kortenkamp, 1996;
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
Decision-theoretic planning models
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
Joelle Pineau
The tiger problem: Value function solution
40
20
V
0
-20
-40
-60
-80
-100
0
belief
S=tiger-left
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
1
S=tiger-right
Joelle Pineau
Optimizing the “investigate” controller
0
-20
-40
V
-60
-80
-100
-120
0
belief
S=tiger-left
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
1
S=tiger-right
Joelle Pineau
Optimizing the “act” controller
80
60
40
V
20
0
-20
-40
-60
0
belief
S=tiger-left
Thesis Proposal: Hierarchical Methods for Planning under Uncertainty
1
S=tiger-right
Joelle Pineau