Transcript Document

Capstone: Vision, Video and VR
CSC 59866CD
Fall 2004
Lecture 2
Project Topic 1:
Augmented NYC
Zhigang Zhu, NAC 8/203A
http://www-cs.engr.ccny.cuny.edu/~zhu/
Capstone2004/Capstone_Sequence2004.html
Augmented NYC:
Interaction, Modeling and Visualization of
a Human-In-the-Loop Traffic Sensor Network
Or CityEyes: City Traffic Sensor Network for
Everyone and by Everyone for a Safer Life
Zhigang Zhu
Department of Computer Science
The City College / CUNY
7/16/2015
Copyright @Zhigang Zhu
2
Outline

A Bigger Picture


The Real-World Problem
Capstone Project


7/16/2015
Scale-down project
Visualization and Vehicle Extraction parts
Copyright @Zhigang Zhu
3
A BIGGER Picture
7/16/2015
Copyright @Zhigang Zhu
4
Related Research Efforts at CUNY





Prof.
Prof.
Prof.
Prof.
Prof.
7/16/2015
Robert Haralick, PR Lab
Brian Schwartz, New Media Lab
Robert Paaswell, UTRC II
George Wolberg, Vision & Graphics
Zhu – Video Computing
Copyright @Zhigang Zhu
5
Background

NYC DOT’s Traffic Management Center


DOT’s Real-Time Traffic Cameras


86 closed circuit TV cameras on major arteries of
NYC
44 video streams or updated still images on the
web
City Drive Live on Crosswalks Channel 74

7/16/2015
22 live streams, weekday mornings from 6:30 to
9:00 am
Copyright @Zhigang Zhu
6
NYC DOT’s
Traffic Management Center




86 cameras
NOW
more in the
future
11,000 signals
in NYC
ITS
7/16/2015
Copyright @Zhigang Zhu
7




Real Player plug-in
At times when demand is high,
video streaming may not be
available to each visitor
Live camera locations will be
limited during the introductory
phase of this new feature
Still web cam photos are
available at all locations.
7/16/2015
Copyright @Zhigang Zhu
8
City Drive Live
on Crosswalks Channel 74



Twenty-two of DOT's traffic cameras
Channel 74 on weekday mornings
from 6:30 to 9 AM

separated views

cannot give global information
raw video streams






Problems (Internet/TV):
bandwidth
camera failures
unavoidable shaking
NO (little) processing/ extraction
Solutions

New ways for Information




extraction/compression
enhancement
integration and
presentation
7/16/2015
Storm Feb 18 2003
Copyright @Zhigang Zhu
9
Sensor Network for NYC Traffic

Sensor network in 4D (3D space + Time)





Traffic Cameras (above ground)
Inductive Loops (underground)
Inspection Cameras (under vehicles)
Aerial Cameras (high in air)
Sensors (GPS, EZ-Pass) on Vehicles (mobile)
7/16/2015
Copyright @Zhigang Zhu
10
Research Issues
– in cope with NSF Sensors and Sensor Networks





localization versus distributed versus
centralized processing
information extraction
on-sensor self-calibration
Adaptive management and control
of sensor nodes
human-in-the-loop or close-loop
adaptive feedback
7/16/2015
Copyright @Zhigang Zhu
11
Human In the Loop

DARPA Total Information Awareness


Google Search Engine


Relevance feedback
Amozon.com


Humans in all the three stages – sensors, alarm
and response
recommends books based upon what other
customers have selected
Can we do it in ITS?
7/16/2015
Copyright @Zhigang Zhu
12
Human Involvement in ITS

How to get user feedbacks?

Web browsers


Drivers

7/16/2015
As of Feb 18, 2003, the hit of the San Diego
Area Traffic Page of Caltrans since 1996 is
385,379,722, which means more than 100
visits every minute!
Real traffic information from sites via wireless
connection
Copyright @Zhigang Zhu
13
Information Sharing

Homeland security Department


“ would make this a top priority ”
Lockheed Martin Transportation
Systems


7/16/2015
one of the biggest challenges confronting
ITS today is information sharing
Data and control
Copyright @Zhigang Zhu
14
How to make these happen?

Advanced HCI

Interaction ( by everyone)



Visualization (for everyone)


7/16/2015
Not only provide info to, but also collect info
from people
User Internet !!
3D immersive and augmented interface
Instead of locator map + video streams
Copyright @Zhigang Zhu
15
Proposed Research

Traffic Information Extraction (Front End)



Traffic Information Analysis (in Background)



Vision-based vehicle detection and tracking
Human “sensors” - feedbacks
For traffic management and advisory
Distributed inference model (DBNs)
Traffic Visualization (Human Interface)

Augmented and immersive user interface


7/16/2015
3D model + real video
2D Map + animation
Copyright @Zhigang Zhu
16
Outcome of the Research




Traffic Control and Management (DOT)
 signal timing, city evacuation,...
City and Traffic Show (Web, TV)
 realtime, interactive, 3D walkthrough
Traffic Forecasting/Advisory (Web, TV)
 predicting the future traffic, personalized advisory
Traffic Open Source for Education and Research (Internet)
 interdisciplinary – sensors, vision, graphics, mm,
transportation, networking, virtual reality, decision
making system...
 open for students and researchers!
7/16/2015
Copyright @Zhigang Zhu
17
Camera views registered with
map…and take a virtual walk..
7/16/2015
Copyright @Zhigang Zhu
18
Grid Traffic Project

Brian Schwartz and
Lei Zhou


Gridlock simulation
Cellular automaton
7/16/2015
Copyright @Zhigang Zhu
19
Virtual Fly-through + Real Video
Zhu & Schwartz (CUNY) 2003
7/16/2015
Copyright @Zhigang Zhu
20
Traffic Information Extraction
Z. Zhu, et al, VISATRAM: A real-time vision system for automatic traffic
monitoring, Image and Vision Computing Journal, 18(10), July 2000: pp 781-794
7/16/2015
Copyright @Zhigang Zhu
21
Security – Under Vehicle Inspection
Full FOV coverage when a car drives
over an a array of cameras ...
One of the
cameras
P. Dickson, J. Li, Z. Zhu, A. Hanson, E. Riseman, H. Sabrin, H. Schultz and G. Whitten,
Mosaic generation for under-vehicle inspection. IEEE Workshop on Applications of
Computer Vision, Orlando, Florida, Dec 3-4, 2002, p 251- 256.
7/16/2015
Copyright @Zhigang Zhu
22
Video Mosaics and
Bird-Eye View
Z. Zhu, E. M. Riseman, A. R. Hanson, Parallel-perspective stereo
mosaics, the Eighth IEEE International Conference on Computer Vision,
Vancouver, Canada, July 9-12, 2001, vol I, 345-352.
7/16/2015
Copyright @Zhigang Zhu
23
Dynamic Mosaic from 2 Cams
Wide FOV
movie
7/16/2015
Copyright @Zhigang Zhu
24
Sensors on Vehicles
Video
Stabilization
and GeoRegistration
J. S. Jin, Z. Zhu, G. Xu, A stable vision system for moving vehicles,
IEEE Transactions on Intelligent Transportation Systems, vol. 1, no. 1,
March 2000, pp 32-39
7/16/2015
Copyright @Zhigang Zhu
25
Vehicle Extraction
[YeMing Ye, Robert M. Haralick and Linda G. Shapiro, " Estimating
Optical flow Using a Global Matching Formulation and Graduated
Optimization ", I6th International Conference on Image Processing, 2002
7/16/2015
Copyright @Zhigang Zhu
26
System Diagram
7/16/2015
Fig.6. System diagram of the human-in-loop distributed computing
Copyright @Zhigang Zhu
27
Traffic Information Extraction

from 4D distributed environment (PAST)





spatially distributed sensors
over time
provide real-time information (NOW)
predict the future traffic (FUTURE)
Challenging Research Issues

7/16/2015
various road situations, realtime optical flow,
traffic volume/velocity, 24/7, 4 season, rain &dry
Copyright @Zhigang Zhu
28
Traffic Information Analysis

Problems


Traffic Information



Gridlock, traffic jams ...
NOT ONLY space-time data from sensors
BUT events, people in NYC!
Research Issues


7/16/2015
Mathematic model – differential equations
or Simulation – cellular automata
Copyright @Zhigang Zhu
29
City and Traffic Show

Three modes




2D map + cartoon-like animation (w/ info)
3D model + real video
3D model + (real video) + traffic info
Research Issues




7/16/2015
camera calibration...static and dynamic
real-virtual registration... geo-referenced
fast rendering ... realistic??
new media presentation – enhanced video??
Copyright @Zhigang Zhu
30
Traffic Forecasting (Advisory)

Two modes


On TV (like weather report)
Internet (interactive)
give the best route
 WHEN start ... end


WHERE start ... end
Research Issues


show the animation
prediction
 data from sensors
 data from models (events, facilities)
presentation
 pseudo-colored
 real video on map
 interactive and personalized
7/16/2015
Copyright @Zhigang Zhu
31
Traffic Education Open Source

interdisciplinary






ubiquitous sensors in NYC
vision, graphics, mm ,virtual reality
networking, video transmission
transportation, traffic, city, tourism
control, decision making system...
open for students and researchers!

study various traffic problems
7/16/2015
Copyright @Zhigang Zhu
32
Related Work

Blending simulations with data from sensors



Schreckenberg in Germany for up-to-the minute
traffic show and 1-hour traffic forecast
(http://traf2.uniduisburg.de/OLSIM)
Texas A&M for computerized traffic control
Source : 1999 Science News Online
7/16/2015
Copyright @Zhigang Zhu
33
Come Down to the Capstone…
7/16/2015
Copyright @Zhigang Zhu
34
What you need to do

Augmented NYC demo code: starting point


CCNY\Working\NYC_VC\AugmentedNYC\AugmentedNYC.exe
C++ and OpenGL

NeHe OpenGL Tutorials (with code samples)
http://nehe.gamedev.net/

To-Do List – Real Work of the Senior Project





2D or 3D NYC map and virtual walk-through (basics)
Multiple Real Video Streams in 3D (basics)
Vehicle Extraction (advanced)
Traffic Analysis and Prediction (optional)
Reports and Papers
7/16/2015
Copyright @Zhigang Zhu
35
Next…

Project Topic 1 Update


Everyone contributes to the update of the
system design
Project Topic 2

7/16/2015
Virtualized Classroom
Copyright @Zhigang Zhu
36