Transcript CSA 2010 Slides for CVSA September 2009

Comprehensive Safety Analysis
CSA 2010
National Rollout and Implementation
CVSA Fall Conference
September 21, 2009
U.S. Department of Transportation
U.S. Department of Transportation
Federal
Motor
Safety Administration
Federal
Motor
CarrierCarrier
Safety Administration
CVSA Annual Conference, September 21, 2009
Presentation Overview
•
•
•
•
•
•
•
CSA 2010 Defined
Op-Model Test Results
National Implementation
Panel Discussion
State Programs Impacts
Roadside Uniformity Update
SMS Methodology Details
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 2
CSA 2010 Defined
Gary Woodford, FMCSA
CSA 2010 Program Manager
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
Comprehensive Safety Analysis 2010
CSA 2010 is a pro-active initiative to improve
the efficiency and effectiveness of FMCSA’s
enforcement and compliance program.
• Uses ALL roadside inspection results and crash reports
to identify safety deficiencies
• Employs a wider array of interventions tailored to
problems instead of solely the time-intensive Compliance
Review process
• Enables more carriers to be contacted earlier
• Requires sustained accountability of carriers AND
increases accountability of drivers
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 4
New Operational Model: 3 components
1. New Safety Measurement System
(SMS)
Improved ability to identify demonstrated
safety problems
2. New intervention process
Employs an array of interventions instead of
the single option, labor-intensive compliance
review
3. New approach to the Safety Fitness Determination (SFD)
SFD tied to current safety performance; not limited to acute/critical violations
from a Compliance Review
• SFD requires rulemaking, not necessarily part of rollout in July
2010
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 5
New Safety Measurement System
CSA 2010 introduces a new safety measurement
system (SMS) that…
• Uses crash records and ALL roadside inspection safetybased violations to determine carrier/driver safety
• Weights time and severity of violations based on
relationship to crash risk
• Triggers the intervention process (eventually would feed
the proposed Safety Fitness Determination)
• Calculates safety performance based on 7 Behavior
Analysis and Safety Improvement Categories (BASICs)
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 6
SMS BASICs
SMS BASICs focus on behaviors linked to crash risk
1. Unsafe Driving (Parts 392 & 397)
2. Fatigued Driving (Hours-of-Service)
(Parts 392 & 395)
3. Driver Fitness (Parts 383 & 391)
4. Controlled Substances/Alcohol
(Parts 382 & 392)
5. Vehicle Maintenance (Parts 393 & 396)
6. Cargo Related
(Parts 392, 393, 397 & HM)
7. Crash Indicator
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 7
SafeStat vs SMS
Today’s Measurement System: SafeStat
CSA 2010 SMS
Organized by four broad categories - Safety
Evaluation Areas (SEAs): Accident, Driver,
Vehicle, and Safety Management
Organized by seven specific Behavior Analysis
Safety Improvement Categories (BASICs)
Identifies carrier for a compliance review (CR)
Identifies safety problems to determine who to
investigate and where to focus the investigation
From roadside inspections, uses only out-ofservice (OOS) and moving violations
Emphasizes on-road safety performance, using all
safety-based road-side inspection violations
No impact on safety rating
Used to propose adverse safety fitness
determination based on carriers’ current on-road
safety performance (future)
Violations are not weighted based on relationship
to crash risk
Violations are weighted based on relationship to
crash risk
Assesses carriers only
Two distinct safety measurement systems-one for
individual carriers and one for individual
commercial motor vehicle (CMV) drivers
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
New Intervention Process
The New Intervention Process addresses the…
• WHAT
Discovering violations and
defining the problem (similar to
current model), but also expanding
to include the why and how
• WHY
Identifying the cause or
where the processes broke down
• HOW
Determining how to fix it/prevent it through use of
Safety Management Cycle and Safety Improvement
Resources
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 9
Safety Management Cycle
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 10
New Intervention Tools
New intervention tools reach more carriers
and influence safety compliance earlier
•Roadside Inspections
•Warning Letters
•Investigations
− Offsite Investigations
− Onsite Investigations – Focused
− Onsite Investigations – Comprehensive
•Follow-on corrective actions
−
−
−
−
Cooperative Safety Plan (CSP)
Notice of Violation (NOV)
Notice of Claim (NOC)
Operations Out-of-Service Order (OOS)
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 11
Current vs CSA 2010 Intervention Process
Current CR Process
CSA 2010 Intervention Process
Broad one-size fits-all investigation
regardless of extent or scope of safety
deficiencies
Array of interventions can be tailored to
address extent and scope of specific safety
deficiencies
Resource intensive for agency and time
consuming for carrier/fewer carriers
contacted
Less resource intensive for agency and less
time consuming for carrier/more carriers
contacted
Focuses on broad compliance based on
rigid set of acute/critical violations
Focuses on improving behaviors that are
linked to crash risk
Discover what violations exist
Discover what safety problem(s) are and
why they exist, to facilitate corrective action
Major safety problems result in fines
(Notice of Claim (NOC))
When problems found, major focus on
carrier proving corrective action; significant
problems continue to result in fines
Focuses on carrier
Expands focus to include investigating
individual
drivers
CVSA
Annual Conference, September 21, 2009
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CSA 2010
Operational Model
Test Results
Bryan Price
Office of Enforcement & Compliance
Federal Motor Carrier Safety Administration
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
Operational Model Field Test
• Began February 2008
• Planned completion June 2010
• Designed to test validity, efficiency
and effectiveness of new model
• Evaluation to be conducted by
independent 3rd party
• Original test states include:
Colorado, Georgia, Missouri, New
Jersey
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 14
CSA 2010 Test
• Spring 2009
– Added Montana and Minnesota
• 100% of the State participates in CSA 2010
– Offers a more accurate picture of efficiencies, capabilities
and benefits
– Tests integration with national program goals and
Congressional mandates
– Provides more data to evaluate test including workload
and workforce analyses
• Fall 2009:
Two additional states: Kansas and Maryland
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 15
Preliminary OM Test Results (10/1/08-8/31/09)
WARNING LETTERS
CSA Warning Letters / PRISM Warning Letters
Test
951
Non-Test
535
Warning letters are having a positive impact:
• About 3,000 sent since Feb. 08
• 45% of recipients logged in to view safety scores
• Feedback from test states indicate that carriers appreciate the early
alert
“…carrier officials thanked us for notifying them of their safety
problems… once carrier officials understand that the new system
enables them to identify their problem drivers, a light goes on. They
see CSA 2010 as a tool that they can use to stress the importance of
roadside inspections with their drivers, to hold their drivers
accountable for their on-road safety performance, and to thereby
improve their companies’ overall safety performance.”
- Daniel Drexler, Division Administrator in Minnesota
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 16
Preliminary OM Test Results (10/1/08-8/31/09)
Test Group
25
Resources (Safety Investigator: Full Time Equivalents)
INVESTIGATIONS
Offsite Investigations
294 (9 hours per)
Onsite – Focused
431 (13 hours per)
Onsite - Comprehensive / Compliance Review (CR)
221 (19 hours per)
Total Carrier Investigations
946
Total Carriers Investigated
931
Carrier Investigations per investigator per month
3.5
Non-Test Group
39
Not Applicable
Not Applicable
1,181 (20 hours per)
1,181
1,117
2.8
• Reaching its goal of contacting more carriers
– One objective of CSA 2010 was to conduct more investigations
• Employing the full array of investigations to achieve efficiency and
effectiveness
– Investigations in test states have been done in the following proportions:
• Onsite Investigations – Comprehensive (~25%)
• Onsite Investigations – Focused (~45%)
• Offsite Investigations (~30%)
U.S. Department of Transportation
CVSA Annual Conference, September 21, 2009
Federal Motor Carrier Safety Administration
| 17
Preliminary OM Test Results (10/1/08-7/31/09)
FOLLOW-ON ACTIONS
Test Group
126
19
43
Non-Test Group
348
Not Applicable
Not Applicable
64
348
Notice of Violation (NOV)
12
No Data Available
Enforcement (NOV, NOC) per investigator per month
0.5
0.8
337
475
469 (50%)
Not Applicable
348
348 (29%)
Notice of Claim (NOC)
From Offsite Investigations
From Onsite–Focused
From Onsite- Comprehensive/Compliance Review (CR)
Cooperative Safety Plan (CSP)
Total Follow-On Actions
Total Investigations with Follow-On Action
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 18
Preliminary OM Test Results (10/1/08-7/31/09)
DRIVER COMPLETED ACTIVITIES
INTERVENTIONS
Red Flag Driver Reviews
Driver Notice of Violation (NOV)
Driver Notice of Claim (NOC)
Driver Enforcement (NOV, NOC) per investigator
Test Group
200
2
35
Non-Test Group
No Data Available
No Data Available
49
1.5
1.3
Emphasizing Red Flag driver review with every
carrier investigation
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 19
CSA 2010 National
Implementation Plans
Bryan Price
Office of Enforcement & Compliance
Federal Motor Carrier Safety Administration
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
CSA 2010 Rollout Strategy
Summer 2010: Nationwide rollout
• SMS replaces Safestat
• BASICs sent to roadside
Summer – December 2010 : State by State rollout
•Roll out interventions
•Send warning letters after Division/state training
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 21
What is Changing?
• The way FMCSA assesses carrier safety
– Identifies unsafe carrier and driver behaviors that lead to crashes
– Uses all safety-based roadside inspection violations
– Evaluates/tracks driver performance individually
• How FMCSA addresses carrier safety issues
– Reaches more carriers earlier and more frequently
– Improves efficiency of investigations
• Focuses on specific unsafe behaviors
• Identifies root causes
• Defines and requires corrective actions
• How FMCSA promotes safety
– Forces carriers/drivers to be accountable for their safety performance
• Demands and enforces safe on-road performance
– Makes more complete safety performance assessments publicly available
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 22
What Can Carriers Do To Prepare Now?
• Learn more about CSA: http://csa2010.fmcsa.dot.gov
– Understand the BASICs
– Check the site for implementation schedule
– Sign up for latest news: RSS/listserv
• Check and update records
– Motor Carrier Census (Form MCS -150)
– Inspection and crash reports
• Ensure compliance
– Review inspections and violation history over the past 2 years
– Address safety problems now
– Educate drivers about how their performance impacts their own driving record
and the safety assessment of the carrier
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 23
Implementation: The State
Partner Perspective
A Panel Discussion
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
State Considerations
Topic Area
Things to Consider
Training
•New measurement system, new interventions, new IT tools
IT
•BASICs replace SEA values at roadside (ISS)
•Impact on related state systems
•Hardware or software changes and training
Outreach
•Roadside inspectors (ISS viewchanges)
•Helping to spread word to industry
Workforce Planning
•Job or role changes?
•New or different resources needed?
•Administrative and/or DataQ increase?
Policy/Guidance
•Intrastate process
•State-specific legislation or policy
Funding
•More funding needed to implement? If so, when will I get
information on how to apply ?
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 25
CSA 2010 Implementation:
State Program Impacts
Tom Keane
Chief, State Programs Division
Federal Motor Carrier Safety Administration
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
Overview
1. Overall Impacts to State MCSAP Programs
2. Specific Changes to CVSP Requirements
3. Other Changes – Data Quality
4. Specific Changes to FMCSA State Programs (SP)
Policy Memos
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 27
1. Overall Impacts-State MCSAP Programs
•“Change is more about culture than cost”
•CR production may change – new interventions
•CR processes – still asked to follow eFOTM
•How you prepare CVSPs
•Volume of Data Qs challenges (350.211 (11))
•Questions/phone calls from industry
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 28
2. Specific Changes to CVSP
• Will ask States to identify goals for new CSA2010
interventions as part of CR National Program
Element
• CVSP National Program Element Changes
– Driver/Vehicle Inspections
– Traffic Enforcement
– Compliance Reviews
• Add CSA 2010 interventions here
– Public Education & Awareness
– Data Collection
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 29
2. Specific Changes to CVSP – Continued
• May require updates to FY2010 CVSP budget
– i.e., training & travel costs
• State-Specific Objectives should remain unchanged
– Specific strategies & monitoring plan may change
• Maintenance of Effort (MOE) levels should remain
unchanged
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 30
3. Other Changes – Data Quality
• Data quality under new system is critical
– How violations are treated by SMS
– All safety-based violations count (not just OOS)
– Roadside violations may be used to assign ratings
• Dependent upon SFD rulemaking
• Data Qs challenges will likely increase
– Inspection Uniformity will be critical
– Requests for inspection reports will increase
– Impacted by Driver Info Resource (DIR) tool
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 31
4. Specific Changes to MCSAP SP Memos
• Prior to Implementation
– May update SP Memo – Funding Eligibility of
Compliance Reviews (SP-90-014-SA)
– May update Inspection Selection System (ISS)
Memo (SP-09-001-GE)
• After Implementation
– Update to Cost Eligibility (Overtime) Memo
(SP-02-001-CE)
– Update to MOE Memo (SP-06-003-GE)
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 32
Primary Keys to Successful Implementation
• FY2010 will be transition year - Be flexible
– CSA 2010 Rollout
– FMCSA grants program changes
– Reauthorization (indirectly)
• Please articulate your concerns & challenges
• We will work with you during implementation!
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 33
For more information please contact:
Tom Keane, [email protected]
(202) 366-4025
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 34
Roadside Data Uniformity
Ad-hoc Status Report
Mark Savage, Colorado State Patrol
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
Objectives of CVSA Data Uniformity
Ad-hoc
• Consistent documentation of roadside inspection
and violation data
• Standardized processes for challenging data
• Increased awareness and understanding that all
inspections (good and bad) must be uploaded
• Uniform inspection selection processes
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 36
Ad-hoc Structure
• At the spring meeting the structure of the Data
Uniformity Ad-hoc group changed
– Much of the work was divided between the various
committees
• Programmatic issues to PIC
• Roadside data violations to individual committees
• This format shows that CVSA is committed to the goals
of the Ad-hoc for the long term
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 37
Consistent Documentation of Roadside
Inspection and Violation Data
Why?
• CSA 2010
• Dependent upon SFD rulemaking, roadside data
may be used to rate carriers as opposed to its
current use which is to prioritize carriers for
compliance reviews
• New safety measurement system uses more than
just OOS violations in carrier measurement
• Program Integrity
• Data Reciprocity
• CVSA Strategic Goals
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 38
Consistent Documentation of Roadside
Inspection and Violation Data
• Last spring CVSA and the Ad-hoc submitted a grant to
FMCSA to fund travel costs associated with work on
the Aspen violation lists. This work has been
completed and a work-product will be presented at the
various committees. The objective of the work groups
was to:
– “To develop operational policies or guidance that will promote
and/or mandate the consistent documentation of roadside inspection
and violation data.”
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 39
Consistent Documentation of Roadside Inspection
& Violation Data-Goals & Objectives
• The actual finished product is a consistent and uniform
listing of violations that will appear in
ASPEN/Safetynet
– Thank you to the subject matter experts from the
HM, Driver-Traffic Enforcement, Vehicle and
Passenger Carrier committees
– Goal is to correctly classify the data as consistently as
possible across jurisdictional lines
• Operational guidance for roadside inspector
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 40
Consistent Documentation of Roadside
Inspection and Violation Data-Plan
• The work product in the form of suggested violations
list will be presented to the full committees at this
meeting
• Next, the suggestions will go to Info Systems
Committee and for review and eventual submission to
FMCSA as the final authority for review and
consideration as a change to ASPEN/Safetynet
– How could IT support the guidance being developed?
• Simultaneously, the same suggestions will be submitted
to the training committee for review for training needs
to the roadside inspectors or data entry specialists
– How could training support the guidance being developed?
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 41
Standardized Processes for Challenging
Data
• As for the second task, standardizing the data
challenge process, FMCSA has formed a working
group that is currently addressing these issues.
Several CVSA members are on this group
• The group is developing procedural guidance on the
Data Q’s process
– Sub-committee of this group is working on a
national model for standardizing due process
among jurisdictions
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 42
Uniform Inspection Selection
Processes
• Increased awareness and understanding that all
inspections (good and bad) must be uploaded
• Uniform inspection selection processes
• Clean inspections are just as valuable as inspections
with multiple violations
– Documents carrier improvement
– Impacts resource usage and carrier safety
measurement
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 43
Program Management Issues-Uniform
Inspection Selection Processes
• Issue request has been submitted to CVSA and
routed to PIC for review. The issue request
suggests a change to Operational Guidance #5
Inspection Selection
– Currently CVSA operational guidance in this area
is to follow your jurisdiction’s policies
– Amendment suggests that jurisdictions develop
and implement inspection selection policies based
on several principles
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 44
Program Management Issues-PIC
• Program-wide management of inspectors and the
inspection process
• Examine possible metrics for measuring roadside
data accuracy and uniformity
• Promote integrity of our processes throughout the
inspection from initial contact to completion
―Inspection/Screening
• A final outcome is to provide operational guidance
to program managers in the form of best practices
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 45
Questions?
For more information please contact:
Mark Savage, [email protected]
303 273 1875
Mike Wilson, [email protected]
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 46
SMS:
Methodology Details,
Role in SFD NPRM,
& Violation Severity Weightings
Bryan Price
Office of Enforcement & Compliance
Federal Motor Carrier Safety Administration
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
Safety Measurement System (SMS)
Objectives
• SMS Methodology Details
• SMS Role in Safety Fitness Determination (SFD)
• Development of Violation Severity Weights
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 48
Entities
Two measurement systems for CSA 2010:
• Carrier Safety Measurement System (CSMS)
• Driver Safety Measurement System (DSMS)
• Potential to add additional measurement systems
in the future
• HM Shipper
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 49
Methodology Overview
1.
2.
3.
4.
Obtain on-road safety event data (e.g., inspections, crashes) and
attribute to entity to create a safety event history
Place each entity’s violations/crashes into a BASIC
Convert BASIC data to quantifiable measure/rate (Safety Fitness
Determination would be based on absolute performance)
Based on each entity’s BASIC measure, develop rank and percentile
for each entity’s BASIC performance
Safety Events
By Entity
BASIC
Data
BASIC
Measures
Percentile
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 50
Safety Events
Safety Event Data Attributed to Entity
• Carrier Safety Measurement System (CSMS)
– Includes 24 months of carrier on road safety performance
~6.6 Million inspections
~290 K crashes
~690 K carriers
• Driver Safety Measurement System (DSMS)
– Includes 36 months of driver on road performance
~9.6 Million inspection records
~440 K crash records
~3.6 Million drivers
SAFETY
EVENTS
BASIC DATA
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
BASIC
MEASURES
PERCENTILE
CVSA Annual Conference, September 21, 2009
| 51
BASIC Data
Safety Event Data Sorted by BASIC
•
•
•
•
•
•
•
Unsafe Driving (Parts 392 & 397)
Fatigued Driving (HOS) (Parts 392 & 395)
Driver Fitness (Parts 383 & 391)
Controlled Substances /Alcohol (Part 392)
Vehicle Maintenance (Parts 393 & 396)
Cargo Related / HM (Parts 392, 393, 397 & HM)
Crash Indicator
SAFETY
EVENTS
BASIC DATA
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
BASIC
MEASURES
PERCENTILE
CVSA Annual Conference, September 21, 2009
| 52
BASIC Measures
Convert BASIC Data into Quantifiable Measure
Considerations
• Time Weighting / Time Frame - More recent events more relevant
• Severity Weightings - Increase weighting of violations that have
been shown to create a greater risk of crash involvement
• Normalizing - Based on exposure: use of number of inspections and
power units
• Single Inspection Cap – limit violation weight of single poor
inspection
• Violation Cap – Cited section number only counts once per
inspection
SAFETY
EVENTS
BASIC DATA
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
BASIC
MEASURES
PERCENTILE
CVSA Annual Conference, September 21, 2009
| 53
Percentile
• Based on each BASIC measure, develop percentile indicating
entity's BASIC performance
– Provides a relative assessment of performance
– Allows for prioritizing intervention resources by behavior
• Considerations:
– Peer Grouping - compare measures of entities with similar levels of exposure
– Data Sufficiency standards – define events/exposure necessary to generate a
robust measure
– Current Inspection and Crash Data – assignment of percentile dependent on
age and result of most recent inspection (12 months)
SAFETY
EVENTS
BASIC DATA
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
BASIC
MEASURES
PERCENTILE
CVSA Annual Conference, September 21, 2009
| 54
Peer Grouping
Create percentile based on measure for carrier with
similar exposure (same peer group)
BASICs
Peer
Group
Unsafe Driving
Controlled Substances/Alcohol
Crash Indicator
Fatigued Driving (HOS)
Driver Fitness
Vehicle Maintenance
Cargo Related
1
0 < PU <= 5
5 – 10 Inspections (3-10 Fatigued)
2
5 < PU <= 15
11 – 20 Inspections
3
15 < PU <= 50
21 – 100 Inspections
4
50 < PU <= 500
101 – 500 Inspections
5
500 < PU
501+ Inspections
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 55
Data Sufficiency
• Minimum number of inspections with applicable violations
required for percentile to be assigned
• Assists in identification of patterns of carrier behavior- note safety
problems across multiple inspections
BASIC
Number of
Inspections
Unsafe Driving
3
Fatigued Driving (Hours-of-Service)
3
Driver Fitness
5
Controlled Substances / Alcohol
1
Vehicle Maintenance
5
Cargo Related
5
Crash Indicator
2 Crashes
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 56
CSMS BASIC Thresholds
Carriers’ BASIC percentile results are used to trigger
carriers for interventions in Op-Model Test
BASIC
•Unsafe Driving,
•Fatigued Driving (HOS)
•Crash Indicator
•Driver Fitness
•Drugs / Alcohol
•Vehicle Maintenance
•Cargo Related
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
Passenger
HM
Other
50%
67%
72%
55%
72%
77%
CVSA Annual Conference, September 21, 2009
CSA 2010 OM Test Phase 2 Training
| 57
How does a carrier move below the
intervention thresholds?
• “Good” Inspections
• “Get Well” Rules
– Violation time weight diminishes
– Unsafe Driving and Controlled Substances /Alcohol BASICs
• No percentile assigned if no inspections with a violation in these
BASICs in the last year
– Crash Indicator
• No percentile assigned if no crashes in last year
– Fatigued Driving (HOS), Driver Fitness, Vehicle Maintenance
and Cargo Related BASICs
• No percentile assigned if:
– No inspections with a violation in that BASIC within the past year; and
– Most recent relevant inspection does not have a violation of that BASIC
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 58
Driver Safety Measurement System (DSMS)
DSMS quantifies commercial motor vehicle
(CMV) driver performance in terms of BASICs,
using available roadside performance data
During the Operational Model Test:
• SIs examine drivers who have been cited for severe driver violations,
in conjunction with carrier interventions
• May result in driver Notice of Violation or Notice of Claim based on
driver violation history across current and previous employers
Beyond the Operational Model Test:
• DSMS may be used to identify the “worst of the worst” drivers so
that interventions may be done directly with drivers, independent of
carrier interventions
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 59
Commonly Asked Question About Driver Data
When Will Carriers Have Access to Driver Data for
Employment Decisions?
• FMSCA’s Driver Information Resource (DIR) attributes roadside
inspection and crash data to individual CMV drivers
• “Driver Profiles” from DIR that contain inspection and crash
histories for individual drivers will be made available through
FMCSA’s Commercial Driver Pre-employment Screening
Program (PSP) later in 2009; drivers would authorize release of
profiles
• FMCSA is negotiating with 3rd party vendors to provide access to
PSP data for carriers and drivers
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 60
Example of SafeStat vs SMS
The following slides provide
examples of key differences between
SafeStat and the new SMS
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 61
Carrier A Measurement: SafeStat Results
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 62
Carrier A Measurement: SMS Results
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 63
Violation Details Provided in SMS
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 64
Further Drilldown in SMS
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 65
Moving to a new SFD process
The following slides provide an example of
why we are moving to propose through
rulemaking changes to the formal safety
rating process that is currently tied to the
on-site compliance review
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 66
Carrier B Safety Rating
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 67
Carrier B Measurement: SMS Results
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 68
Limitations of Current Rating Process
Current Ratings:
• Can only be issued or downgraded with an on-site review –
resource intensive
• Represent a snapshot of carrier compliance at the moment of
the most recent compliance review
• Do not consider roadside driver inspection performance
• Are based only on violations deemed “critical” or “acute” and
vehicle out-of-service violations
• Generally require multiple areas of deficiency for adverse rating
• Only issued to small portion of carrier population
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 69
Objectives of Proposed CSA 2010 SFD
Methodology
• Make carriers accountable for sustained unsafe
operations and performance
• Assess larger portion of carrier population
• Move away from agency “seal of approval”
– Carrier can continue to operate until deficiency
identified, focus is on removing high risk carriers from
road vs. identifying “good” carriers
• Maximize use of data collected by inspection program
– ~3 million inspections performed annually
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 70
CSA 2010 SFD Methodology
Two major components considered in
determining SFD for a carrier:
1. On Road Performance - Violations identified
during roadside inspections and crash data
AND
2. Intervention Results – Violations identified and
data collected during investigations
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 71
SFD – Roadside Data
Role of On Road Performance
• 24 months of violation data used to evaluate a carrier in the
following BASICs:
–
–
–
–
–
Unsafe Driving
Fatigued Driving (Hours-of-Service)
Driver Fitness
Vehicle Maintenance
Cargo Related
• (1) Crash Indicator and (2) Controlled Substances /Alcohol BASICs cannot
fail based on roadside data alone
• Measure exceeding established “absolute” thresholds – results in
failed BASIC
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 72
SFD – Intervention Data
Role of Intervention Results
• Essential Safety Management Violations
– Tied to BASICs
– Analogous to “critical” violations of current rating process
– Discovery of at least 10% of the records checked would result
in failed BASIC
• Fundamental Violations
– Discovery of a single instance during an intervention would
result in proposed Unfit
– Largely in line with New Entrant Rule
• Accountable Crashes and VMT
– Determined onsite during investigation by SI
– Rate may result in failed Crash BASIC
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 73
CSA 2010 SFD Methodology
• Failed BASICs result from roadside and investigation data
• SFD methodology
– Classifies BASICs as “Stand Alone” or “Non Stand
Alone” according to their demonstrated relationship with
carrier crash risk
– Driven by the carrier’s failed BASICs
• Have any BASICs failed? How Many? Which One(s)?
– Would result in three potential SFDs
• Continue to Operate
• Marginal
• Unfit
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 74
CSA 2010 Safety Fitness Determination
Methodology Being Considered
SFD Methodology Being Considered
Number of BASICs:
Measure equals or exceeds BASIC Failure Threshold
or
Essential Safety Management Violations Discovered
Stand Alone BASICs:
Unsafe Driving
Fatigued Driving (HOS)
Non Stand Alone BASICs:
Driver Fitness
Cargo Related
Crash Indicator
Controlled Substances / Alcohol
Discovery of
Fundamental
Violation?
1
Resulting
Proposed SFD
Unfit
0
>1
0
0
1
Unfit
0
1
0
Marginal
0
0
0
Continue to Operate
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
Unfit
CVSA Annual Conference, September 21, 2009
| 75
Example: Marginal SFD (Roadside datadriven)
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
Example: Proposed Unfit SFD
(Investigation-driven)
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 77
Safety Fitness Determination Today vs.
CSA 2010 Methodology
Existing Safety Fitness Rating Process
CSA 2010 Safety Fitness
Determination (SFD) Under
Consideration
Rating only issued or changed with on-site review
SFD would change based on roadside data alone
Rating is a snapshot of compliance on date of
compliance review
Safety fitness would be evaluated on a monthly
basis
Rating does not consider roadside driver inspection
performance
Adverse SFD could be made based on roadside
driver inspection performance alone
Rating based on violations deemed “critical and
acute” and vehicle out-of-service violations from
inspections
SFD would be based on violations of all safetybased regulations and evaluation in 7 BASICs
NTSB Recommendation: H-07-3
Adverse rating generally only issued with multiple
areas of deficiency
Adverse SFD could be issued with a single area
of deficiency
NTSB Recommendation: H-99-006
3 ratingU.S.labels:
Unsatisfactory, Conditional,
Department of Transportation
Federal Motor Carrier Safety Administration
Satisfactory
3 SFD “labels”: Unfit, Marginal, Continue to
CVSA Annual Conference, September 21, 2009
Operate
Violation Severity Weighting
Bryan Price
Office of Enforcement & Compliance
Federal Motor Carrier Safety Administration
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
Background Study: Establish Association Between
BASIC Violations and Crash Involvement
• Identified drivers with 7+ inspections, excluding post-crash
inspection, over 3-year period for DIR data.
• Grouped drivers by crash involvement:
Crash Level
0 Crashes
1 Crash
2+ Crashes
Total Drivers
197,762
40,893
7,119
• For each crash level –
– Calculated average violation rate (no severity weights) for
each BASIC
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 80
Association Between BASICs
And Crash Involvement
% Difference from Zero Crash Baseline
Difference in Violation Rates By Crash Level
60.00%
50.00%
40.00%
1 Crash
30.00%
2 or more
Crashes
20.00%
10.00%
0.00%
Unsafe
Driving
Fatigued
Driving
(HOS)
Fitness Training
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
Fitness Physical
Controlled
Substance
Vehicle
Maint.
CargoRelated
CVSA Annual Conference, September 21, 2009
| 81
Violation Severity Overview
• Objective of SMS is to identify patterns of behavior
assessing all violations
• However, all violations do not have the same
relationship to crash risk
• Therefore, preliminary severity weights have been
assigned to each violation to reflect relevance to crash
risk
• Severity weights based on quantitative analysis
supplemented with Subject Matter Expert input
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 82
Violation Severity Overview
• Because SMS assesses each BASIC independently,
severity weights are used to differentiate crash risk for
violations classified within that BASIC only
― 1 to 10 scale where 1 represents lowest crash risk
― Can NOT compare weights across different
BASICs (e.g. 5 in one BASIC is not equivalent to a
5 in another BASIC)
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 83
Derivation of Preliminary Severity Weights
– 6 Step Process
1. BASIC Mapping
• All safety-related roadside violations mapped to
appropriate BASIC
2. Violation Grouping
• Violations in each BASIC placed in groups of like
violations
• Allows rarely cited violations to be used in statistical
analysis
• Helps ensure similar violations receive same severity
weight
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 84
Derivation of Preliminary Severity Weights
– 6 Step Process
3. Crash Occurrence Analysis
•
•
Used DIR data of crash occurrences for drivers to
analyze relationship between violation rates in each
violation group within a BASIC, and crash involvement
Applied multivariate regression:
― Showed a vast majority of violation groups yielded statistically
significant relationships between high violation rates and
increased crash occurrence
― Provided relative strength of the relationship between each
violation group and crash occurrence within a BASIC
•
Served as basis for preliminary violation group severity
weights
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 85
Derivation of Preliminary Severity Weights
– 6 Step Process
4. Crash Consequence Analysis
•
Incorporated findings from the Violation Severity
Assessment Study (VSAS) to address crash
consequence (severity of outcome)
5. Subject Matter Expert (SME) Review
•
•
Enforcement SMEs reviewed purely statistical based
results
Modifications were made based on SME input
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 86
Derivation of Preliminary Severity Weights
– 6 Step Process
6. Carrier Safety Measurement System (CSMS)
Effectiveness Test
1.
Performed a simulated CSMS run that calculates carrier percentile ranks
for each BASIC using historical data
Observed each carrier’s crash involvement over the immediate 18
months after the simulated CSMS timeframe, and
Calculated the relationship between the percentile ranks in each BASIC
and the subsequent post-CSMS carrier crash rates.
2.
3.
•
Iterative process used to optimize the ability of CSMS to identify high risk carriers
Simulated
CSMS Run
12/2005
12/2003
6/2007
Period of Events used in Simulated CSMS Run
Crash Monitoring Period
24 Months of Data
18 Months of Crash Data
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 87
Ongoing Review of Severity Weights in
Op-Model Test
• Use of SMS throughout OM Test
– Ongoing review and feedback on SMS design through
Federal and State Working Group, Safety Investigators
and SMEs involved in testing the CSA 2010
Operational Model
– SMS methodology has been continuously improved
based recommendations and experiences
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009
| 88
For more information see
http://csa2010.fmcsa.dot.gov
U.S. Department of Transportation
Federal Motor Carrier Safety Administration
CVSA Annual Conference, September 21, 2009