Transcript Computer Vision Technologies for Remote Collaboration Using Physical Whiteboards, Projectors and Cameras

Computer Vision Technologies for
Remote Collaboration Using Physical
Whiteboards, Projectors and Cameras
Zhengyou Zhang
R
Microsoft Research
mailto: [email protected]
http://research.microsoft.com/~zhang/
Motivations
 WB
provides a large shared space
for the participants to focus their attention and
express their ideas spontaneously
 Many
meetings use whiteboard heavily
brainstorming sessions, lectures, project planning
meetings, patent disclosures, etc
 WB
is ubiquitous in knowledge workers’ offices
→ Whiteboard will still exist for long time
Motivations (cont’d)
 Difficulties:
– Content is hard to archive or share
– Busy in note taking, instead of spending time
sharing and absorbing ideas
 Need
tools to increase productivities
– Don’t lose any sparks
– Share with others
– Keep for later review
– Integrate with your office document
Motivations (cont’d)
 Projector
– A great tool for presentation
– Not so convenient for discussions
 Whiteboard-Whiteboard-Camera
System
– Whiteboard: Projecting surface (Output) and
Writing surface (Input)
– Seamless integration of computer
presentation with whiteboard discussions
– Enable remote collaboration on a shared
workspace
Whiteboard Scenarios
 Note
taking with a high-res digital camera
– Whiteboard It!
 Note
taking with a low-res PC camera
– Whiteboard Scanning
 Real-time
collaboration
– Live Whiteboard
 Meeting
(WB + Audio / Video) Archiving
– Whiteboard Archiving
♥ Whiteboard It! ♥

Convert WB Content into an Electronic Document
Overview of the System
♥ Whiteboard Scanning ♥
 Get
a High-res Scan With an Inexpensive
PC Camera
 How
to do it?
– Take multiple pictures with overlap
– Stitch them automatically into a high-res image
– Image enhancement (Whiteboard It!)
System Overview
Example
+
+
Document Scanning w/ Low-Res camera
♥ Live Whiteboard ♥
 Real-time
distributed collaboration
Whiteboard
Display
Projector
Table
Mic (array)
TabletPC
TabletPC
Wire
Speaker
or wireless
Camera
Meeting Room/Office
PC
Windows
Communications Desktop
Offices
A typical image sequence
Segmentation of the person and WB background is needed
Demo
 Integration
with Windows Messenger
Local WB client
Video
Remote WB client
Annotations
USB
2
Whiteboard
changes
Real-Time WB
Processing
Video
Whiteboard
update
Windows
Messenger
T120
Windows
Messenger
♥ Whiteboard Archiving ♥
 Capture
whiteboard content + audio/Video
– high-resolution digital still camera
 Produce
key frames
– A KF usually corresponds to a major topic
– Print as notes, or cut & paste into documents
 Record
time stamps of each stroke
 Efficient meeting browsing
– Key frames to navigate between sections
– Strokes to bring up the audio at the moment
when they were written
A Typical System Installation
Example: Sample Images
Key Frame Extraction

Number of strokes
Key Frame 2
1000
900
Key Frame 1
800
700
600
500
400
300
200
100
0
1
21
41
Chapter 1
61
81
101
121
141
Chapter 2
161
181
Browsing Interface: Demo
Current
Strokes
Key Frame
Thumbnails
Raw
Image
VCR &
Timeline
Control
Future
Strokes
♥ Projector-Whiteboard-Camera System ♥
P
Presentation
W
P
P+A
A
Video
Mixer
A
A
Communi cation
Network
P
Visual Echo
Cancellation
Camera
P+W
P
Projector
Whiteboard
Video
Mixer
Annotation
Display
Remote Room
W
P+A+W
Remote Room
Local Meeting Room
Visual Echo Cancellation
Color Lookup Table
(given by color calibration
)
captured image I
Color Mapping &
Geometric Warping
Homography H
(given by geometric calibration
)
estimated visual echo E
 Prerequisites
– Geometric calibration
– Color calibration
albedo
estimation
&
color
clustering
recover writings W
VEC Results
Conclusions
 Tools
to use whiteboard and projector
 Whiteboard It! for anyone who has a
digital camera and it is easy to use
 Whiteboard Scanning for anyone who
has a PC camera and it is easy to use
 Whiteboard Archiving and Streaming for
meeting rooms and collaboration
 Integration of Projector and Whiteboard
for seamless remote collaboration
Acknowledgements
 Thanks
to Li-wei He, Zicheng Liu &
Hanning Zhou for contributions