SUBTLE: Overview

(Situation Understanding Bot Through Language and Environment) UMass Amherst, UMass Lowell, Penn, Stanford,

Cornell, George Mason

Mitch Marcus University of Pennsylvania

Our Guiding Vision

Standing orders: clear the building: search for any bombs or weapons, find any injured persons and report them and their location so the medical team can find them, and tell any other persons to leave the building. 0:00 Commander: Jr, I want you to search the room you are in first and when you go to leave head for the North end of the building, continue around East and we will meet up at the end of a long hallway.

0:32 Jr: 10-4 3:13 Jr: I have cleared the first room and am proceeding North down a hallway.

3:20 Commander: Mark that room as clear and make certain to check all the rooms in the hall.

3:45 Jr: Room is marked. Proceeding to the next room.

3:56 Commander: Wait. Shouldn’t you be going into the hallway?

4:02 Jr: No, I went directly into the next room.

4:05 Commander: Can you show me the map and mark your position.

4:15 Jr: I am displaying the map, I am the yellow dot.

5:23 Commander: Alright I see now, carry on.

10/16/2009 SUBTLE Year 2 Overview 2

Scientific Objective: Central Hypothesis

Effective communications with autonomous bots in real-time situations requires that bots understand not only what is

literally said

, but also what is

intended

, the

implicit meaning

Example: Exchange between firefighters (Worcester Cold Storage Fire 1999):

  

P2 (a)

answers question

P1’s

implicit

(b) appears to

give

command question: How can we find you?

• Linguists call this the

Question Under Discussion (QUD) P2 answers the QUD

10/16/2009 SUBTLE Year 2 Overview 3

Technical Approach: Key Ideas

   

We must develop techniques to analyze not only what sentence a speaker used, but also

what that speaker’s implicit meaning.

The communication system must exploit the

broad context of the environment

. The linguistic specification should incorporate

formal models of language

• To guarantee

computationally efficient

analysis methods • To facilitate study of the

habitability

and effectiveness of the result

The adequacy of specifications should be determined by

empirical, corpus-based methods

• Our research must proceed by collecting example corpora of interactions in increasingly complex (simulated) environments 10/16/2009 SUBTLE Year 2 Overview 4



Potential Breakthroughs

New frameworks and theories for

• Robust automated understanding of linguistic intention • Linking linguistic intention to appropriate robot control 

New machine learning algorithms for structured problems such as Natural Language



A formal computational specification of a significant subset of English



A testbed system for investigating HRI using natural language

• Should ultimately enable military designers to develop powerful communication methods between bots and humans. 10/16/2009 SUBTLE Year 2 Overview 5

     

The SUBTLE team: senior participants

Natural Language Processing/Computer Science

•

Aravind Joshi

, Penn •

Mitch Marcus

, Penn • •

David Smith, UMass Amherst (Research Scientist To join Y3)

Fernando Peirera

, Penn (On leave: Director of Research, Google)

Robotics/Electrical Engineering, Computer Science

•

George Pappas

, Penn • •

Holly Yanco

, UMass Lowell

Hadas Kress-Gazit, Cornell (Penn grad student



Cornell)

Human Simulation & Graphics

•

Norm Badler, Penn

•

Jan Allbeck, George Mason (Penn grad student



George Mason)

Machine Learning: Andrew McCallum,

UMass Amherst

Linguistics

:

Chris Potts,

Stanford

Current Grad Students: 8

10/16/2009 SUBTLE Year 2 Overview 6

People Actively Involved Year 2

   

Faculty

• • • • • • • Norm Badler Aravind Joshi Mitch Marcus Andrew McCallum George Pappas Chris Potts Holly Yanco

New PhDs

• • Jan Allbeck Hadas Kress-Gazit

Limited involvement:

• Fernando Pereira

Newly Involved

• • David Smith Florian Schwarz 10/16/2009  

Grad Students

• • • • • • • • Chris Czyzewicz Munjal Desai Kuzman Ganchev Dan Hestand Karl Schultz Qiuye Zhao Pengfei Huang Dan Brooks

Undergraduates

• Victoria Schwanda  HCI, Cornell • • • Jessica Ouyang Dan Keller Phil Kovac SUBTLE Year 2 Overview 7

Framework – Year 1 Review

Parsing

Parse tree, indices, semantic tags

Semantics

Underspecified predicate logic

Pragmatics 10/16/2009 World Model Parameterized Action Representations SUBTLE Year 2 Overview Linear Temporal Logic 8 Natural Language Processing Linguistics Robotics Graphics/ Human Simulation Machine Learning

Year 1: Primary Accomplishments

   

SUBTLE architecture intensively reworked Key components built and tested

• • LTL →FSA transducer PAR (Procedural Action Representation) for PragBot

“PragBot 1” corpus collection tool built, initial corpus collected, NLP analyzer built New results: Probabilistic pragmatics meets probabilistic inference

10/16/2009 SUBTLE Year 2 Overview 9

Evolution of Framework

Parsing

Parse tree, indices, semantic tags

Semantics

Underspecified predicate logic

Pragmatics 10/16/2009 World Model Parameterized Action Representations SUBTLE Year 2 Overview Linear Logic 10 Natural Language Processing Linguistics Robotics Graphics/ Human Simulation Machine Learning

Evolution of Framework

Parsing

Parse tree, indices, semantic tags

Semantics

Underspecified predicate logic

Pragmatics PAR + LTL 10/16/2009 World Model SUBTLE Year 2 Overview PAR + LTL 11 Natural Language Processing Linguistics Robotics Graphics/ Human Simulation Machine Learning

Evolution of Framework

Parsing

Parse tree, indices, semantic tags

Semantics

Underspecified predicate logic

Pragmatics PAR + LTL 10/16/2009 World Model SUBTLE Year 2 Overview 12 Natural Language Processing Linguistics Robotics Graphics/ Human Simulation Machine Learning

Accomplishments to date I



PhD dissertations completed

• Hadas Kress-Gazit,

Transforming high level tasks to low level controllers,

Dec 2008 (Asst Prof, Cornell) — Uses Linear Temporal Logic Model Checking to compile complex constraints and requirements into real-time hybrid robot controller.

• Jan Allbeck,

Creating 3D Animated Human Behaviors for Virtual Worlds,

June 2009 (Asst Prof, George Mason) — Extended Parameterized Action Representation (PAR) for robust, multi-agent setting. Developed extensive authoring tools.

 

LTL controller is now outputting PAR representations to drive virtual animation for corpus collection, system development

PAR simulation handles multi-agent setting needed for team exercises of Years 4-5.

10/16/2009 SUBTLE Year 2 Overview 13

LTL Example: “Find Nemo”



Task spec in English

•

“Nemo can only be in Regions 1, 3, 5 and 8. Look for Nemo and if you find him, turn your video camera on and stay where you are. If he disappears again, turn the camera off and resume the search.”

(There are12 regions which define the robot propositions {r 1 , …, r 12 }). 

Fragment encodes possible moves that enforce:

• if R sees Nemo, stay in that region at next step with camera on. Otherwise, camera off.

  ◊ - Next - Always - Eventually 10/16/2009 SUBTLE Year 2 Overview 14

The resulting controller can control virtual robot & actual robot

10/16/2009 SUBTLE Year 2 Overview 15

Talk 5

CAROSA: Authoring PARs for NL Predicates, for multiple agents, …

10/16/2009 SUBTLE Year 2 Overview 16

Talk 6

Accomplishments II

 

A new formal account of Grice’s maxims, a foundation of linguistic pragmatics Embedded and tested in a computation implementation of a rich account of formal linguistic pragmatics

• In communicating — — — Be truthful Be relevant Be brief • Encoding (for the moment in Markov Logic):

Assert(p) => True(p). True(p) =>Assert(p). //Truthful // Relevance Assert(p) => Qud(q) ^ About(p,q)). 10 (Qud(q) ^ About(p,q)) => Assert(p)

10/16/2009

Assert(p) => !CommanderBelieve(p). 10 !Assert(p) => CommanderBelieve(p)

SUBTLE Year 2 Overview

// Brief

17

Short term application: Response Relevance



Pragmatics of Relevance, Brevity for Bot’s responses formalized within Probabilistic Markov Logic

// 1. General description of what it means to have completed the task.

Report(TaskComplete) <=> (FORALL x (Relevant(x) => Found(x))).

// 2. If you find something, report it.

5 Found(x) => Report(x)

// 3. Report only relevant things.

10 Report(x) => Relevant(x) 10 !Relevant(x) => !Report(x)

// 4. If you can report that the task is complete, report nothing else.

5 Report(TaskComplete) => ((y != TaskComplete) => !Report(y))

Talk 3

10/16/2009 SUBTLE Year 2 Overview 18

Pragmatics & Action

→

Pragmatics in Action

10/16/2009 SUBTLE Year 2 Overview 19

Talk 4

Integration and State Summarization

10/16/2009 SUBTLE Year 2 Overview 20

Other Accomplishments

  

Last Year: Initial PragBot web-based interface constructed for corpus collection

• 50 Human-Human corpus of collaborative interactions in simple isomorphic environment captured & analyzed

New framework & toolkit for joint inference about to be released

• Allows efficient machine inference of probabilistic models across very large, complex knowledge bases • Joint Segmentation & Coreference of research paper citations: mentions, 134 entities, 36487 tokens 1295 • Compare with Markov Logic Networks (Alchemy) —

~25% reduction in error (segmentation & coref)

—

3-20x faster

…

10/16/2009 SUBTLE Year 2 Overview 21

First Year Review Report – 10/2008

       

Develop a common simulation and experimentation scenario…which is demanding and sophisticated….

You must get busy on building the corpus from a major search and rescue simulation/experiment/game….

Build a robot language community….

Start to visit DoD labs and establish the groundwork for technology transfer.

Continue to assume that future…bots will have better…capabilities than current bots, yet whenever possible use both current bots and simulated future bots to test algorithms and tools.

Focus on developing how the world view (situation awareness) will influence the language and understanding of the bot.

While the first…2-3 years is focusing…mostly on the bot receiving and understanding information, the overall goal…is to provide a framework for dialog among a team of several bots and several humans … Don’t lose sight of this important goal while performing the preliminary work.

Develop more robust (human-based) performance metrics and base your models and goals on these.

10/16/2009 SUBTLE Year 2 Overview 22

First Year Review Report – 10/2008

1.

2.

3.

4.

Develop a common simulation and experimentation scenario…which is demanding and sophisticated…. You must get busy on building the corpus from a major search and rescue simulation/experiment/game….

Build a robot language community….

Start to visit DoD labs and establish the groundwork for technology transfer.

10/16/2009 SUBTLE Year 2 Overview 23

The PragBot I Data Collection Environment

  

During Year 1:



PragBot data collection interface implemented

• • Two players in world with cards (each can hold 3). Goal: pick suit, find 6-in-a-row

PragBot symmetric corpus of 50 dialogues collected NL analyzer built to map corpus to PAR

Guidance: “Develop a common simulation and experimentation scenario … which is demanding and sophisticated….”, “build the corpus from a major search and rescue simulation/…/game….

10/16/2009 SUBTLE Year 2 Overview 24

Example data: Cooperation

            

% pragbot_chat_log_2007.11.01 AD at 19.48.20 EDT.txt Player 2: i have 4H Player 1: I want it! Player 1: where is it? Player 2: should i leave it for you somewhere? Player 1: sure Player 1: where are you? Player 2: okay, where are you? Player 1: I'm near the top Player 2: i'm left side. Player 1: next to the gap near the middle Player 2: i'll leave the card in the upper left corner. Player 1: awesome

10/16/2009 SUBTLE Year 2 Overview 25

Talk 2 Last Year’s Annual Review



First Annual Review: 3 Oct 2008 – UPenn

• “Develop a common simulation and experimentation scenario … which is demanding and sophisticated….” • Response: Pragbot 2.0

10/16/2009 SUBTLE Year 2 Overview 26

PragBot 2 scenarios are our targets

  

The PragBot 2 world provides a simple yet relevant world for scenarios for testing human robot communication using language Limits in world align with limits of current robot perception One extension: We will allow Jr to pick up objects in our test scenarios.

• Will simulate in Robot demo 10/16/2009 SUBTLE Year 2 Overview 27

Workshop on Situated Understanding



First Year Review: “Build a robot language community ….”

  

When: two full days July 23-24, 2009 Where: Institute for Research in Cognitive Science, Penn Who: 26 participants, including many members of SUBTLE advisory committee



Workshop Focus:

• Human-Robot Communication through NL and other contexts requiring recognition of goals and intentions — Interactive fiction — Embodied agents — Social avatars — Avatars in games involving language communication — … 10/16/2009 SUBTLE Year 2 Overview 28

Workshop Talks

       

Four talks by SUBTLE participants Dialog as Planning with Knowledge and Sensing

•

Ron Petrick & Mark Steedman (Edinburgh)

Towards a Robotic Architecture for Natural Spoken Human-Robot Dialogues

•

Matthias Scheutz (Indiana) & Kathleen Eberhard (Notre Dame)

Intention Situated in Collaborative Dialogue

•

James Allen (Rochester)

Models and Skills for Understanding Communicative Intentions

•

Matthew Stone (Rutgers)

A Simple Computational Model of Interactive Language Comprehension

•

William Schuler (OSU)

Engagement and Deixis for a Humanoid Robot

•

Candy Sidner (BAE)

From Events to Natural Language in System Curveship

•

Nick Montfort (MIT)

the Interactive Fiction

10/16/2009 SUBTLE Year 2 Overview 29

Connections to DoD labs – Year 2

     

“Start to visit DoD labs and establish the groundwork for technology transfer.” 2007 —Visit NRL Talk: The generality of pragmatic

inference: Message enrichment in multi- agent interactions

(Potts) 5/09 HRI Workshop & Group Meeting sponsored by ARL’s Advanced Decision Architectures (ADA) Collaborative Technology (Yanco) 5/09 —Visit ARL Talk: SUBTLE (Marcus) 7/09 —Situated Understanding Workshop (All) 7/09 —Kick-off meeting for new ONR MURI & Talk (Potts)

10/16/2009 SUBTLE Year 2 Overview 30

Behind plan: Two areas

• • • •

Corpus Collection

• Implementation of Pragbot 2 more difficult than expected 3D implementation ambitious, given that game must completely download as webapp Initial game completed and limited data now collected, but scenario requires tuning. • • • • •

Connection of Pragmatics to Language

Alchemy system for Markov Logic good for small examples

but Alchemy can’t handle rich highly interconnected logical models

Expected: that’s why we were betting on McCallum’s Factorie But we didn’t expect the limitations to hit so soon Factorie beta due within a week or two!

Talk 7

10/16/2009 SUBTLE Year 2 Overview 31

Summary of progress

  

Crucial Steps Forward

• Architecture evolving as components connect — LTL  PAR — Pragmatics  Robot’s expectations • • Pragbot 2 functioning Two grad students to strong faculty positions, with continued involvement (1 undergrad to HCI PhD program) • Situated Understanding Workshop building community

Where behind

• • Just beginning to gather data Factorie is crucial for progress

Pieces about to come together

10/16/2009 SUBTLE Year 2 Overview 32

8:35 8:45 9:30 10:10 10:50 11:15 11:55 12:30 p.m.

1:10 1:50 2:05 2:45 3:00 4:00 4:30

Schedule for the Day…

Introduction

SUBTLE: Overview Pragbot 2.0:

Moving 'Pragbot' Language Interactions Toward More Realistic Situations

The interplay of linguistic & contextual inferences

Break

Integration and State Summarization

Lunch

Meaning to motion:

Transforming specifications to provably correct control

Integrating Linear Temporal Logic and a Parameterized Action Representation

& Creating 3D Animated Human Behaviors for Virtual Worlds

Break

Joint Inference for the NLP Pipeline:

Probabilistic Programming and the FACTORIE System

SUBTLE Year 3 plans

Advisory Committee Meeting Advisory Committee Initial Report

Review Adjourns

Joe Myers Mitch Marcus Chris Czyzewicz/Norm Badler Chris Potts Dan Hestand / Munjal Desai /Holly Yanco Hadas Kress-Gazit/George Pappas Jan Allbeck/Norm Badler David Smith/Andrew McCallum/Karl Schultz Mitch Marcus 10/16/2009 SUBTLE Year 2 Overview 33

BACKUP

10/16/2009 SUBTLE Year 2 Overview 34

Components built and tested

   

PAR implemented and tested for SUBTLE world High level autonomous actions added to robot

• Search, Find, Follow, Snapshot

World Model Ontology DB and API implemented LTL stress-tested in Urban Challenge enviroment

• Connection from Pseudo-English to LTL to Robot tested 10/16/2009 SUBTLE Year 2 Overview 35