The Strategic Role of Vision in Cognitive Systems and Robotics - A Perspective from the EU Research Programme

ECCV 2010, Special session on "Research Funding for Vision", September 9, 2010

Cécile Huet – [email protected]

Unit E5 - Cognitive Systems, Interaction, Robotics DG Information Society and Media European Commission http://www.cognitivesystems.eu

••• 1

Outline

    History Projects portfolio Perspectives and future opportunities Conclusion

••• 2

It all started with VISION: “Cognitive Vision” Initiative in 1998 (FP5): Rationale



Limitation of computer vision systems:

 Lack of ROBUSTNESS/VERSATILITY/SPEED 

Cognitive vision

 combine state-of-the-art image processing with cognitive reasoning: abstract representation, categorisation, memory organisation, self-adaptivity, contextual knowledge, behavioural/scene interpretation,…

••• 3

Projects portfolio



>140 projects

-

€480 million EU funding

.

 FP5 (1998-2002) Cognitive vision - 9 projects  FP6 (2002-2006) Cognitive Systems + Advanced robotics - 45 projects  FP7 (2007-2013) Cognitive systems, Interaction, Robotics 68 running projects + 22 projects under negotiation

••• 4

FP5 Cognitive Vision - achievements



FP5 – DETECT Real Time Detection of Motion Picture Content in Live Broadcasts

-> detect/track brand appearance in sport videos

••• 5

FP5 Cognitive Vision - achievements



FP5 – ACTIPRET Interpreting and Understanding Activities of Expert Operators for Teaching and Education

Recognition of hand and arm gestures: original image with overlay and 3D representation Long term goal: giving hints on how to clear the paper jam in a copy machine

••• 6

FP5 Cognitive Vision - achievements



LAVA - Learning for adaptable visual assistants

 Categorization and interpretation of large numbers of objects, scenes and events, in real settings.

• Recognition/Categorization: Occlusion – cluttered background – viewpoint-rotation-scale,… invariance, lighting conditions,… • Categorization - Cope with different forms of the same object (shape, colours, size, angle etc) and an open set of instances of the object • Scene understanding – disambiguation –> multi cue/information fusion – context,… Source: www

••• 7

FP6: Cognitive systems research - motivation

Beyond vision….

Achieving naturalness, versatility, robustness at system level     from constrained to real-world environments from application-specific to general solutions from component methods to a systems approach from a ‘monodisciplinary’ research effort to working across disciplines RESULTS: Integration of perception (attention mechanisms, affordances, multi-sensory/cue…), representation, learning, reasoning, decision-making, action and communication, cooperation (HRI/RRI), anticipation,…

FP6: Cognitive systems projects - classification

     Robotics (navigation – manipulation) - action-perception(COSPAL SPARK), affordances (MACS-PACO-PLUS), anticipation(MINDRACES), reasoning - understanding (COSY), attention, knowledge-reasoning (GNOSYS), observation (MATHESIS), decision-making (DECISIONS-IN MOTION), reasoning & self-preservation (ICEA) , Joint Action (JAST), Active haptic(SENSOPAC), developmental (ROBOT-CUB) Interfaces - audio-visual- HHI/HRI(POP), HRI (PACO-PLUS, COSY), HRI RRI (JAST) – ECAS (RASCALLI) Assistive technologies - Blind (DECISIONS-IN-MOTION- CASBLIP) – driving assistance (BACS) – Augmented human action (PACO-PLUS) Scene analysis/digital content - audio-visual + attention (POP-DIRAC), language & information discovery (RASCALLI), description image/video + text (CLASS), man-made scenes interpretation (eTrims), description of human behaviour(movement-facial expression) from video sequences (HERMES) + euCognition / PASCAL

••• 9

FP6: Advanced Robotics projects - classification

HRI

• FEELIX GROWING: FEEL, Interact, eXpress: a Global appRoach to develOpment With INterdisciplinary Grounding • INDIGO: Interaction with Personality and Dialogue Enabled Robots • COMMROB: Advanced Behaviour and High-level Multimodal Communication With and Among Robots • PHRIENDS: Physical Human-Robot Interaction: DepENDability and Safety

ROBOT – FOR HUMAN ASSISTANCE (environments: Firefighter/Urban pedestrian area/home)

• VIEWFINDER: Vision and Chemiresistor Equipped Web-connected Finding Robots / GUARDIANS • URUS: Ubiquitous Networking Robotics in Urban Settings • Robots@home: An Open Platform for Home Robotics (platform-perception-navigation)

SWARM – for human assistance or autonomous tasks

• IWARD: Intelligent Robot Swarm for Attendance, Recognition, Cleaning and Delivery (+HRI) • ROBOSWARM: Knowledge Environment for Interacting ROBOt SWARMs (IT infrastr./Cleaning Hospital) • GUARDIANS: Group of Unmanned Assistant Robots Deployed In Aggregative Navigation supported by Scent detection Network of Excellence / Coordination and Support Actions • EURON: European Robotics Research Network • CARE: Coordination Actions for Robotics in Europe • RoSta: Robot Standards and Reference Architectures (+benchmarking) • RAWSEEDS: Robotics Advancement through Web-publishing of Data Sets

••• 10

FP6 – Vision: Examples



CASBLIP: Cognitive Aid System for Blind People



Sensory substitution: from vision to audio information



DECISIONS-IN-MOTION: Neural Decision-Making in Motion

 understand the neural mechanisms underlying fast target selection and sensory-guided decisions and actions in humans in motion  transfer this knowledge into an artificial visual system for automated robotic navigation.

••• 11

FP7 - Challenge 2: Cognitive Systems, Robotics, Interaction

Challenges

  greatly improving robustness etc. requires rethinking the way systems are engineered -> theories are needed Scientific approach: understanding of what both natural and artificial systems can and cannot do, and how and why

Equally importantly

 we need to be able to support claims that progress is being made  identify would greatly benefit from such systems real applications that  integration of disciplines philosophy of mind,… : robotics, AI, computer vision, natural language, cognitive science, psychology,… mathematics,…

••• 12

FP7 – CALL1 – PROJECTS CLASSIFICATION

Medical – Rehab./Surgery Dexterous Manipulation GRASP Perception & Sensors CHIROPING Cognition & Language emergence Cognition for Human Assistance DIPLECS in robots ALEAR ITALK ROSSI ROBOCAST MIMICS CHRIS DEXMART EYESHOTS CogX HRI LIREC SEMAINE PINVIEW SF SPARK II REPLICATOR SEARISE PROMETHEUS CoFRIEND SCOVIS POETICON CLASSIC EMIME HMI Autonomous Robots Sensor network Scene understanding Spoken language PASCAL 2 ••• 13 Machine Learning

FP7 – CALL3 – PROJECTS CLASSIFICATION: SCENE UNDERSTANDING

MULTI-ROBOT / SWARM UNDERWATER AERIAL URBAN Co3AUVs EUROPA SCANDLE ACOUSTIC SCENE ANALYSIS CA in Cog Sys FILOSE NAVIGATION SCENE UNDERSTANDING EUCOGII ••• 14 HRI

FP7 – CALL3 – PROJECTS CLASSIFICATION FROM AN APPLICATION ORIENTED PERSPECTIVE

INDUSTRY / MANUFACTURING BRICS MONITORING SURVEILLANCE SECURITY Co3AUVs SCANDLE FILOSE SHOAL SFLY ROSETTA IM_CLEVER HANDLE MANIPULATION SORTING OBJECTS ASSEMBLY ECHORD ECCEROBOT ENVIRONMENTAL MONITORING MOBILE MANIPULATION PROSTHETICS SEARCH & RESCUE SOCIAL COMMUNICATIVE SKILLS (multimodal interaction – TV-Host) STIFF SERVICE ROBOTICS & HRI EUROPA HUMANOBS ROBOSKIN GUIDANCE DELIVERY TRANSPORT HUMOUR REHABILITATION MOTOR SKILL LEARNING PROGRAMMING BY DEMO/ TOUCH-BASED

FP7 – CALL4 – PROJECTS CLASSIFICATION

MULTIMODAL LANGUAGE HAPTICS + VISION NIFTI IURO RADHAR ROBOSOM AMARSI AIROBOTS ALIZ-e HUMAVIPS TRIDENT COGNITO FIRST-MM ROBOEARTH BIO-INSPIRED LOCOMOTION MOTOR CONTROL THE DEXTEROUS MANIPULATION GERT GARNICS TACO AUDIO VISUAL HRI MASH euROBOTICS NAVIGATION VANAHEIM SENSOR NETWORK/ HUMAN ACTION RECO CA in ROBOTICS WWW SHARED DB: Robots actions ••• 16 PERCEPTION: VISION

Shared resources: Contribute to and benefit from:

RoboEarth

– Robots sharing a knowledge base for world modelling and learning of actions http://www.roboearth.org/

  World Wide Web for robots: a giant network and database repository where robots around the world access and continually update to share information and learn from each other about their behaviour and their environment.

Improve any robot’s 3D sensing, acting and learning capabilities.

••• 17

Shared resources: Contribute to and benefit from: MASH - Massive Sets of Heuristics for Machine Learning

 Design of complex learning systems integrating large and heterogeneous families of feature extraction modules.

  Open web-based platform where contributors will find tools to interact and communicate with each other, and the means to integrate their feature extractors in continuously running experiments.

Quantitative evaluation of submitted modules  Performance will be measured on action selection with a robotic arm and in a simulated 3d environment, and object detection .

 http://mash-project.eu/

••• 18

FP7 – CALL FoF – Smart Factories: ICT for

agile

manufacturing – Target outcome c) Robotics-enabled production processes tested and validated in real-world environment and environmentally friendly

PACKAGING HRI + HR COOP.: WORKER DETECTION + INTENTION RECO SKILL LEARNING PACKAGING ELECTRONIC CONSUMER GOODS MANIPULATION/ HANDLING SORTING OBJECTS ROBOFOOT VISUAL INSPECTION VISUAL SERVOING SHOE TAPAS NAVIGATION / PLANNING: MOBILE MANIPULATION / MOBILE ASSISANT: LOGISTICS + ASSEMBLY PRECISION FLEXIBILITY/SCALABILITY RELIABILITY ROBUSTNESS VERSATILITY

FP7 – CALL6 – PROJECTS CLASSIFICATION (! Subject to successful negotiation)

NAVIGATION SENSOR NETWORK GOAL-DIRECTED/ANTICIPATION VISION + Object/scene understanding HRI/SOCIAL INTERACT

What next?

The 2010 Call for Proposals – FP7 - ICT Call 7 -

http://www.cognitivesystems.eu

    INDICATIVE - BASED ON DRAFT TEXT ! INFORMATION TO BE CONFIRMED TARGET a): Research projects on Robotic systems operating in real-world environments: + functionalities + autonomy, safety, robustness, efficiency, and ease of use AS APPROPRIATE: + new materials and advanced sensor, actuator, effector, memory and control technologies.

->> validation through REAL-WORLD scenarios TYPE OF PROJECTS

:

STReP (Specific Targeted Research Projects) High risk/Focussing on specific research topic/component IP (Integrated Projects) System oriented BUDGET: 70M€ DEADLINE: 18.01.2011

••• 21

What next?

The 2010 Call for Proposals – FP7 - ICT Call 7 -

http://www.cognitivesystems.eu

    INDICATIVE - BASED ON DRAFT TEXT ! INFORMATION TO BE CONFIRMED TARGET d) Coordinated Action(CA) Fostering communication

and co-operation between robotics and cognitive systems

research communities: + knowledge sharing: EU, national, and international; + open source; R&D roadmaps, market potential, user acceptance, standardisation, education, ethics, socioeconomic impacts; outreach to relevant professional and general audiences.

TYPE OF PROJECTS

:

CA BUDGET: 3M€ DEADLINE: 18.01.2011

••• 22

What next?

The 2011/12 Call for Proposals – FP7 - ICT Call 9 -

http://www.cognitivesystems.eu

   INDICATIVE - BASED ON DRAFT TEXT ! INFORMATION TO BE CONFIRMED Cognition and control in complex systems (IP and STReP) Cross-fertilisation between academic and industrial robotics research (IP) Targeted competitions for smarter robots (CA)  INDICATIVE DEADLINE: 17.04.2012

••• 23

Perspectives



Outcome so far and next steps

 New research questions -> need for further scientific progress  Also technological progress and application motivated research, building on previous results – innovation in integration and implementation of results to improve robotics functionalities and features.

 Real world scenarios: source of research questions + validate research results  Demonstrate progress: Benchmarking

••• 24

CONCLUSION FP7 – Workprogramme text « Research and development […] will be guided by demanding, yet pragmatic application scenarios » Role of vision?

••• 25

CONCLUSION FP7 – Workprogramme text « Research and development […] will be guided by demanding, yet pragmatic application scenarios » Critical!!

••• 26

THANK YOU!

http://www.cognitivesystems.eu

••• 27