SHRP 2

Strategic Highway Research Program

Accelerating solutions for highway safety and performance

Charles Fay, Sr. Program Officer Big Data Meets Computer Vision Dec. 7, 2012

         Like challenges? Then you should be excited by SHRP 2 NDS ~ 4 petabytes of data that need to be post-processed ~ 1 million hrs of video ~ 3000 subjects, 5 million trips, >18 million miles driven, 4 billion GPS points Real world - automotive conditions (daylight variance; nighttime IR); low quality cameras & images Data compressed(H264) and saved at 15 Hz PII (personal identifiable information) & protection of privacy Patience with getting access to data- working out details 

National Academy of Sciences

Advisors to the Nation on Science, Engineering, and Medicine To: "investigate, examine, experiment,

and report upon any subject of science or art - whenever called upon to do so by

any department of the government” Transportation Research Board (TRB) is one of six major divisions

Est. 1863

Content

• What’s the Problem(s)? • Preview video data • Naturalistic Driving Study (NDS) • Roadway Information Database(RID) • FHWA Exploratory Advanced Research Program • Goal today: •

promote interest in mining these data

o

making these data more usable

Ultimately saving lives/ reducing severity of injury

Public Health & Highway Safety: Crashes leading cause of death for ~ 4-34 year old (US)* ~ 40,000 total deaths in US/year* ~ 2.5-3.0 million injuries /yr in US Estimated costs: $230 billion/ yr in US wot.motortrend.com

Driver behavior has been identified as the major factor in 90% -95% of roadway crashes (know very little about behavior ) Major issue around the world; Naturalistic driving studies in EU, China, Australia; others in development- way of the future

Computer Vision:

 Before analysts can use the full NDS dataset – more usable form – that is where you come in  Lots of data from ~ 3000 participants ▪ ~ 4 petabytes; 1 million hrs video + other sensor data; 5 million trips; > 18 million miles  Saved video poorer quality relative to what you are used to analyzing.

 PII (personal identifiable information) & data access (working out details-patience please) ▪ recording continuously: GPS; face video

DRIVER RELATED

 Driver behavior  Distraction  Head pose  Eye gaze  Fatigue/drowsiness  Mobile device use  Hand position  Foot/pedal

CONTEXT RELATED

 Traffic signal state  Roadside information  Weather, pavement conditions  Bike/ Pedestrian  Other vehicles (brake lights) & traffic

“Your challenge should you choose to accept… …working on post processing these data in an efficient manner to gain meaningful information” http://kellypuffs.wordpress.com

kmnnz.wordpress.com

Benefits of the Study (safety related ) These data are not available – one of a kind database(s): decades of use

Almost Everyone (OEMs, DOTs, researchers) eager to get hands on these data •

Intelligent / automated/connected vehicles & transportation

•

Improved understanding of baseline driving behaviors:

 Trip characteristics  Driver performance profiles  Adherence to laws and basic safety practices •

Improved understanding of unsafe behaviors and traffic events:

 Assess circumstances and motivations for speeding, red light running, etc.

 Deconstruct crashes and near-misses and examine causality  How do driver, vehicle, roadway, and environmental factors influence behavior and impact crash risk?

•

Improved ability to develop safety countermeasures for:

 Education and training  Roadway design and traffic engineering  Vehicle design  Regulation and enforcement  Ability to direct countermeasures at driver subgroups

What can be done post-processed?

Camera Image Samples

Video saved @ 15Hz; H 264 compression Forward View - color Driver Face – Rotated for max pixel efficiency Right-Rear View Center stack – Pedal Interactions; hands Periodic still cabin image, permanently blurred for passenger anonymity (child safety seat use?)

480x360

Scaled full to 480x360

360x120

Scale Vertical by ¼ horizontal by 1/2

360x120

Crop 25% off top and Bottom then Scale by 1/2 11

Camera view Forward view Face Rear Cabin Instrument Cluster Horizontal FOV 82 75 92 92 92 sensor 1/3 DPS CMOS lens F= 3.6/F1.4

Lines effective pixels 540 1/4" BW CCIQ II camera 3.3mm/F2.0

400 720 (H) X 540 (V) 648X488 (EIA/NTSC) stored size 480X360 240X360 Image alterations Measured Full scaled full scaled rotate 90 degrees 83.5

78 1/4" BW CCIQ II camera 2.1mm/F2.5

400 1/4" BW CCIQ II camera 2.1mm/F2.5

400 1/4" BW CCIQ II camera 2.1mm/F2.5

400 648X488 (EIA/NTSC) 648X488 (EIA/NTSC) 648X488 (EIA/NTSC) 360X120 360X120 Crop 25% off top and Bottom then Scale by 1/2 Scale Vertical by 1/4 horizontal by 1/2 Blurred Jpeg 100 95 92

NDS RID

~ 1950 DAS ~3000 participants ~ 5 million trips Passenger Car, Van, SUV, Pickup

NDS Data

(DAS GPS is Link)

RID (GIS) Existing Data characterize the environment in which the participant/ DAS operates: roadway, crash, safety campaigns, laws, traffic, weather, work zones… linked to roadway segment New Roadway Data Collected and QA

Six NDS Data Collection Sites across the U.S. One Coordinator

WA Data Collection IN Data Collection NC Data Collection

NDS Data

NY Data Collection PA Data Collection FL Data Collection

NDS Data  Driving data: from instrumentation on vehicle  Driver data: from questionnaires, tests  Vehicle data: vehicle inspection; CANbus-vehicle network  Crash data: detailed investigation of selected crashes  Will include both restricted and non-restricted data requiring various levels of access  Restricted data: that which may be used to identify a participant, such as face video or GPS. Requires high level of physical and electronic security, data access agreements, ethics review, oversight. Working on specifics

for data access (remote enclave(s) being considered)

 Non-restricted data can be disseminated more widely via web access, summarized data sets, numerical variables

18

DAS Overview  Multiple Videos  Machine Vision  Eyes Forward Monitor  Lane Tracker  Accelerometer Data (3 axis)  Rate Sensors (3 axis)   GPS  Latitude, Longitude, Elevation, Time, Velocity Forward Radar  X and Y positions   X and Y Velocities Cell Phone  ACN, health checks, location notification  Health checks, remote upgrades  Illuminance sensor  Infrared illumination  Passive alcohol sensor  Incident push button  Audio (only on incident push button)  Turn signals  Vehicle network data  Accelerator  Brake pedal activation  ABS  Gear position  Steering wheel angle  Speed  Horn  Seat Belt Information  Airbag deployment  Many more variables…

400 350 300 250 200 150 100 50 0 3500-3900 total vehicle years F M F M Vehicle Years Accumulated F M F M F M F M F M F Total Projected Vehicle Years Accumulated with Extensions and Reinstalls M N 20

Paved Shoulder Median

Flush Paint.

Lanes

3’ Mix/Combo N/A Thru Lane: 1 (12’) N/A 3’ Mix/Combo 4’ Mix/Combo Flush (Painted) N/A Thru Lane: 1 (11’) Right Turn: 1 Flush (Painted) N/A Flush Paint.

Thru Lane: 1 (12’)

Lanes

Flush Paint.

Median Paved Shoulder

Thru Lane: 1 (12’) Left Turn Lane: 1 N/A 2’ Mix/Combo Unpaved Shoulder: N/A

Rumble Strips

: N/A Lighting: N/A Thru Lane: 1 (14’) Deccel. Lane: 1 Flush (Painted) Thru Lane: 2 (11’) Deccel. Lane: 1 N/A 0’ Mix/Combo Flush Thru Lane: 1 (12’) Accel. Lane: 1 Thru Lane: 1 (21’) (Painted) 3’ Mix/Combo N/A Flush Paint.

2’ Mix/Combo

Grade, Cross Slope

• Horizontal Curvature: Radius , Length ,PC , PT ,Direction • Grade • Cross Slope/ Super Elevation • Lane in terms of the number, width, and type ( turn, passing, acceleration, car pool, etc…) • Shoulder type/curb; paved width if exists • Intersection location , number of approaches, and control (uncontrolled, all-way stop, two-way stop, yield, signalized, roundabout). Ramp termini are considered intersections • Posted speed limit sign and location (R2-4 Series) • Median presence(Y/N), type (depressed, raised, flush, barrier) • Rumble Strip presence(Y/N) location (centerline, edgeline, shoulder) • Lighting presence( Y/N) • FHWA determining if additional data types will be processed (e.g., All MUTCD signs; barriers - TBD)

Front ROW Images

Route Name Direction Chainage State Collection Date

14 15 16 5

#

1 2 8 9 10 11 12 13 17 4 3 24 18 19 21 22 6 20 7 23 25 Crash Data Traffic Information - AADT Aerial Imagery Speed Limit Data Speed Limit Laws Cell phone and text messaging laws

Item

Automated enforcement laws Alcohol-Impaired and Drugged Drivers laws Graduated driver licensing (GDL) laws State motor cycle helmet use laws Seat Belt Use laws Local Climatological Data (LCD) NOAA Cooperative Weather Observer/Other Sources Winter Road Conditions (DOT) Work Zone 511 Information Traffic Data - Continuous Counts (ATR) Traffic Data -Short Duration Counts Changes to existing infrastructure condition Roadway Capacity Improvements Nonrecurring Congestion Automated Enforcement Travel Time Data Innovative Treatments Recurring Congestion

2 3 4 Priority 1

25

       Working on providing data from 24 individuals ~ 45 min per driver Variety of facial features Glasses/sunglasses Daytime/ nighttime conditions IRB; consent form allow data to be shared for research purposes May need your IRB approval- most likey expedited review

 Exploratory Advanced Research Program  Video analytics workshop: 10/10-11/2012  summary report by January 2013  http://www.fhwa.dot.gov/advancedresearch/  [email protected]

 [email protected]

   Charles Fay cfay @nas.edu

202-334-1817