Transcript FSUTMS-Cube Calibration Standards

New Calibration and Validation
Standards for Travel Demand Modeling
presented to
12th TRB Transportation Planning Applications Conference, May
2009, Houston, TX
presented by
Robert G. Schiffer, AICP
Thomas F. Rossi
Cambridge Systematics, Inc.
Study prepared for
Florida Department of Transportation
January 12, 2009
Transportation leadership you can trust.
Presentation Overview
Background
Literature Review
Recommended Calibration and
Validation Guidelines and Standards
• LRTP Models with Transit
• Other Model Applications
Calibration and Validation
Best Practices
Guidelines for Model Application
Next Steps
1
Background
Follow-up to Phase I Study on model parameters
• Phase I Final report is available for downloading at
− http://www.fsutmsonline.net/images/uploads/mtf-files/FSUTMSCube_Parameters.pdf
Phase II Study on calibration standards included four
subtasks
• Literature Review
• Model Calibration/Validation Guidelines and Standards
• Best Practices for Model Calibration/Validation
• Documentation – 1) Calibration and Validation Standards;
2) Best Practices for Model Validation; 3) Guidelines for
Model Application
2
Background (continued)
Calibration versus validation
− Calibration – process where models are adjusted to simulate
or match observed travel behavior in the study area
− Validation – procedure used to adjust models to simulate
base-year traffic counts and transit ridership figures
Standards versus guidelines/benchmarks
− Standards – desirable accuracy levels for comparing estimated
versus observed metrics
− Benchmarks – documented statistical ranges from literature
review, model outputs, NHTS, etc.
Purposes of validation process
− Level of comfort to planners, agency staff, and elected officials
− Evidence that model is accurate enough for specific application
− Accounts for errors in observed comparative data
3
Literature Review
Checklist of Available Validation Standards from Literature – Trip Generation
60+ documents
reviewed
• Specific models
Statistic
Standard
Population/Employment Ratio
Benchmark
40-60%
3.64 – 3.87
Person Trips/Person
Document(s) Cited
Iowa DOT Peer Review (39)
Validation and Reasonableness (14)
Person Trips/Person (Urban)
2.54
University of Wisconsin (16): Kentucky Statewide
Model/NPTS
Person Trips/Person (Rural)
2.57
University of Wisconsin (16): Kentucky Statewide
Model/NPTS
− Technical reports
Person Trips/HH
8.5 – 10.5
University of Tennessee (59)
Person Trips/HH
6.8 – 12.4
Validation and Reasonableness (14), NCHRP 365 (15)
− Model outputs
Person Trips/DU
14.1/14.5/11.8/7.6
Person Trips/DU
9.2/9.0/8.6/8.5
Vehicle Trips/DU
9.15
VTRC (29)
78.5%/21.5%
VTRC (29)
• Reference reports
− Federal
agencies/TMIP
− State DOT
guidelines and
standards
Prepared ranges of
acceptability
Resident/Commercial
Neighborhood Trips
1.29 – 1.40
Person Trips/Employee
Calibration and Adjustment 6) – population sizes: 50-100/100250/250-750/750k+
NCHRP 365 (15) – population sizes: 50k-200k/200k500k/500k-1M/1M+
Validation and Reasonableness (14)
TAZs/Population
1 TAZ/1k Population
Iowa DOT Peer Review (39)
Person Trips/TAZ
25k or less
Iowa DOT Peer Review (39)
Percent Trips by Purpose – HBW*
18% – 27%
University of Tennessee (59)
Percent Trips by Purpose – HBNW
47% – 54%
University of Tennessee (59)
Percent Trips by Purpose – NHB
22% – 31%
University of Tennessee (59)
Percent Trips by Purpose – HBW*
17% – 23%
Validation and Reasonableness (14), NCHRP 365 (15)
Percent Trips by Purpose – HBNW
52% – 60%
Validation and Reasonableness (14), NCHRP 365 (15)
Percent Trips by Purpose – NHB
23% – 25%
Validation and Reasonableness (14), NCHRP 365 (15)
Unbalanced
Attractions/Productions
4
0.90-1.10
Validation and Reasonableness (14)
Recommended Guidelines and Standards
Checking Input Data
Socioeconomic data
• Visual comparisons
• Statistical comparisons
• Regionwide comparisons
(below)
County
− Persons per
DU (or HH)
− Employment/
population ratio
− Autos/DU (or HH)
Census Data
2000
2003
Duval
779,618
817,480
Clay
141,671
Percent
Difference
2005
5%
762,674
810,493
6%
157,502
11%
139,036
167,020
20%
57,903
61,625
6%
56,897
64,695
14%
124,458
142,869
15%
120,738
150,084
24%
1,103,650
1,179,476
7%
1,079,345
1,192,292
10%
St Johns
Benchmarks/Settings
Statistic
Percent
Difference
2000
Nassau
Total
NERPM Data
Low
High
Regionwide Persons/DU (or HH)
2.0
2.7
Regionwide Employment/Population Ratio
0.45
0.75
Regionwide Autos/DU (or HH)
1.75
2.10
Approximate Population/TAZ
N/A
5
3,000
Recommended Guidelines and Standards
Checking Input Data (continued)
Highway network data
Transit network data
Highway and transit speed data
• Logical hierarchy
• Balance highway and transit
Terminal times
• Logical hierarchy
• Phase I Report
6
Recommended Guidelines and Standards
Trip Generation
Benchmarksa
Aggregate trip rates
• Person trips/TAZ
• Person trips/person
• Person trips/DU (or HH)
a
Statistic
Low
High
Person Trips/TAZ
N/A
15,000
Person Trips/Person
3.3
4.0
Person Trips/DU (or HH)
8.0
10.0
HBW Person Trips/Employee
1.20
1.55
Generally excludes nonmotorized trips; including motorized trips could increase
person trips per DU up to 11.5.
• HBW person trips/employee
Total unbalanced attractions versus productions
by purpose
• Preferred +/-10%; acceptable in some instances +/-50%
Percent external-external trips by zone/station
• Great variation expected (4-21 percent range documented)
7
Recommended Guidelines and Standards
Trip Generation (continued)
Percent trips by purpose
Benchmarks
Statistic
Low (Percent)
High (Percent)
Percent Trips by Purpose – HBW
12
24
Percent Trips by Purpose – HBSH
10
20
Percent Trips by Purpose – HBSR
9
12
Percent Trips by Purpose – HBSC
5
8
Percent Trips by Purpose – HBOa
14
28
Percent Trips by Purpose – HBNWb
45
60
Percent Trips by Purpose – NHBc
20
33
a
HBO includes a variety of special trip purposes depending on the model (e.g., airport, college, and shop).
b
HBNW accounts for all home-based trip purposes except HBW.
c
NHB includes combined purposes for NHB Work and NHB Nonwork, where appropriate.
8
Recommended Guidelines and Standards
Trip Distribution
Benchmarks
Statistic
Average trip length by purpose
Trip length frequency
distributions by purpose
Coincidence ratios by purpose –
measures the percent of area
that coincides for two trip
length frequencies
Low
High
Average Trip Length – HBW (minutes)
12
35
Average Trip Length – HBSH (minutes)
9
19
Average Trip Length – HBSR (minutes)
11
19
Average Trip Length – HBSC (minutes)
7
16
Average Trip Length –
(minutes)
8
20
Average Trip Length – NHBb (minutes)
6
19
Average Trip Length – IE (minutes)
26
58
HBOa
Statistic
Standards
Mean Trip Length, Observed Total Trips
+/-3%
Trip Length Frequency Distribution
versus Observed
+/-5%
Percent of Total Trips
Coincidence Ratios by Purposec
8%
Coincidence Ratio = 0.82
HBO includes a variety of special trip purposes, depending
on the model (e.g., airport, college, and school).
b
NHB includes combined purposes for NHB Work and NHB
Nonwork, where appropriate.
c
Some lower coincidence ratios have been deemed
acceptable for trip purposes that had relatively few trips
and therefore higher error rates.
Estimated (ATL = 18.2 Min)
6%
Observed (ATL = 18.9 Min)
4%
2%
0%
0
10
20
30
Travel Time (in Minutes)
40
50
9
60
70%
a
Recommended Guidelines and Standards
Trip Distribution (continued)
Percent intrazonal trips by purpose
Map-based (“node-point”) charts
• Zone-based
• Number of trips
• Trip productions/attractions
by purpose
Benchmarks
Statistic
Low
High
Percent Intrazonal – HBW
1%
4%
Percent Intrazonal – HBSH
3%
9%
Percent Intrazonal – HBSR
4%
10%
Percent Intrazonal – HBSC
10%
12%
Percent Intrazonal –
HBOa
3%
7%
Percent Intrazonal –
NHBb
5%
9%
3%
5%
a
b
Percent Intrazonal – Total Trips
Standards
Statistic
Percent Intrazonal, Observed Total Trips
Acceptable
Preferable
+/-3%
+/-5%
10
HBO includes a variety of special trip purposes, depending
on the model (e.g., airport, college, and school).
NHB includes combined purposes for NHB Work and NHB
Nonwork, where appropriate.
Recommended Guidelines and Standards
Mode Choice
Mode split targets (ideal)
Mode
Zero-Vehicle
Households
One-Vehicle
Households
Two-Vehicle
Households
Three-Vehicle
Households
Walk
5,000
6,000
4,000
3,000
Bike
2,000
1,000
500
200
-
130,000
350,000
200,000
Shared Ride 2 Persons
6,000
15,000
20,000
10,000
Shared Ride 3 Persons
1,000
2,000
4,000
2,000
Local Bus, Walk
6,000
7,000
4,000
1,000
Local Bus, PNR
-
500
2,000
500
Local Bus, KNR
-
200
Express Bus, Walk
1,000
1,000
1,000
500
Express Bus, PNR
-
2,000
4,000
2,000
Express Bus, KNR
-
200
500
500
1,000
400
-
300
500
Drive Alone
• Trip purpose
• Mode
• Auto ownership level
• Geographic subarea
LRT, Walk
LRT, PNR
LRT, KNR
Polk
2000
1990
Census 2000
Trips
Percent of
Trips
Trips
Percent of
Trips
Number
Percent of
Trips
Drive Alone
244,414
79.69%
188,259
80.47%
Drive Alone
84.14%
One Passenger
51,465
16.78%
38,212
16.33%
Carpool
15.07%
Two+ Passenger
9,637
3.14%
7,023
3.00%
Total Transit
1,206
0.39%
465
0.20%
Transit
0.78%
Trip Allocation By Mode
HBW
11
Recommended Guidelines and Standards
Mode Choice (continued)
Mode splits by observed calibration targets
Total area transit trips, estimated
versus observed
Transit trips between districts
• Tabular comparisons (CTPP)
• Desire lines
Mean trip length, estimated transit trips versus observed
Standards
Statistic
Low
High
Total Area Transit Trips versus Observed
+/-1%
+/- 2%
Transit Trips between Districts
Compare model trip table against CTPP or HH survey
Mean Trip Length Transit Trips versus Observed
+/-5%
+/-15%
Mode Splits by Observed Calibration Targets
+/- 2%
+/- 2%
-0.6
-0.1
Elasticity of Demand with Respect to LOS Variables
12
Recommended Guidelines and Standards
Trip Assignment
Standards
Statistic
Volume-over-count
ratios
+/-1 lane percent error
(calculated based on
FDOT LOS Handbook)
Acceptable
Preferable
Freeway Volume-over-Count
+/- 7%
+/- 6%
Arterial Volume-over-Count
+/- 15%
+/- 10%
Collector Volume-over-Count
+/- 25%
+/- 20%
Frontage Road Volume-over-Count
+/- 25%
+/- 25%
Freeway Peak Volume-over-Count
75% of links @ +/-20%; 50% of links @ +/-10%
Major Arterial Peak Volume-over-Count
75% of links @ +/-30%; 50% of links @ +/-15%
Assigned VMT-over-Count Areawide
+/-5%
+/-2%
Assigned VHT-over-Count Areawide
+/-5%
+/-2%
Assigned VMT-over-Count by FT/AT/NL
+/- 25%
+/- 15%
Assigned VHT-over-Count by FT/AT/NL
+/- 25%
+/- 15%
Aggregate VMT
Standards
Statistic
• VMT/HH (60-75)
• VMT/person (24-32)
• VMT/commercial
vehicle (3-25%)
Acceptable
Preferable
Percent Error – LT 10,000 volume (2L road)
50%
25%
Percent Error – 10,000-30,000 (4L road)
30%
20%
Percent Error – 30,000-50,000 (6L road)
25%
15%
Percent Error – 50,000-65,000 (4-6L freeway)
20%
10%
Percent Error – 65,000-75,000 (6L freeway)
15%
5%
Percent Error – GT 75,000 (8+L freeway)
10%
5%
13
Recommended Guidelines and Standards
Trip Assignment (continued)
Standards
Statistic
Screenline volume-over-count
• By volume and location
RMSE by volume group
Transit assignment validation
Benchmarks
Statistic
Estimated-over-Observed Transit Trips
Low
High
+/- 9%
+/- 3%
Standards
Statistic
Acceptable
Preferable
Acceptable Error – Transit Screenlines
+/-20%
+/-10%
Transit Ridership – <1,000 Passengers/Day
+/-150%
+/- 100%
Transit Ridership – 1k-2k Passengers/Day
+/- 100%
+/- 65%
Transit Ridership – 2k-5k Passengers/Day
+/- 65%
+/- 35%
Transit Ridership – 5k-10k Passengers/Day
+/- 35%
+/- 25%
Transit Ridership – 10k-20k Passengers/Day
+/- 25%
+/- 20%
Transit Ridership – >20,000 Passengers/Day
+/- 20%
+/- 15%
14
Acceptable
Preferable
RMSE – LT 5,000 AADT
150%
45%
RMSE – 5,000-9,999 AADT
45%
35%
RMSE – 10,000-14,999 AADT
35%
27%
RMSE – 15,000-19,999 AADT
35%
25%
RMSE – 20,000-29,999 AADT
27%
15%
RMSE – 30,000-49,999 AADT
25%
15%
RMSE – 50,000-59,999 AADT
20%
10%
RMSE – 60,000+ AADT
19%
10%
RMSE Areawide
45%
35%
Calibration and Validation Best Practices
Steps in Model Validation and Calibration Process
Iterative process
Initiate
Inventory
Process must acknowledge
• Availability of behavioral
data (or lack thereof)
Institutional
Framework
Secondary
Data
Collection
Primary
Data
Collection
Data
Checking
Model
Estimation
Model
Implementation
• Variability in data accuracy
Iterate
Model
Calibration
• Regional issues to be
evaluated with the model
Model
Validation
Model
Application
• Need for future year
sensitivity testing
NO
Satisfactory
Results?
YES
Continual
Model
Maintenance,
Application
Must validate each step
(i.e., not just assignment)
15
Calibration and Validation Best Practices
Guidance on Validation and Adjustment
Matching base year statistics is
not sufficient to say model is
validated
Ideal combination of skills
Validation and Reasonableness Checks
Estimation
Calibration
Validation
Application
• Local area knowledge
Iterate
• Familiarity with sources for
transferable parameters
• Understanding of what
constitutes acceptable results
• Experience with cause and effect
of model adjustments
• Quality and availability of data
16
Iterate
Iterate
Calibration and Validation Best Practices
Special Validation Considerations
Statewide Model
Requirements by
study type
• FTA New Starts
projects
• Subarea and corridor
validation
Statewide
Model
Statewide Model
Urban
Model
Area
Study
Area
Study Area
Within One
Model Area
Urban
Model
Area 1
Study
Area
Urban Study
Model Area
Area 2
Study
Area
Study Areas
Within Two
Model Areas
Urban
Model
Area
Study Area
Outside Urban
Model Areas
Example of Coding Penalties for HOV Lanes
• Site impact studies
General Purpose Lanes
HOV Lanes
Other validation practices
General Purpose Lanes
• Adjustment of congested
speeds
Example of Coding Prohibitors for Proper Access
Would not
cross traffic to
take loop ramp
• Use of special generators
• Transferable parameters
• Impact of new paradigms
Cannot cross median
to access driveway
(centroid connector)
Red lines represent penalized or prohibited movements.
17
Guidelines for Model Applications
Stability of Model Parameters
DeKalb County Comparisons of Daily VMT
Vehicle Miles Traveled (1000s)
Model Year
0
5,000
10,000
15,000
20,000
Static versus dynamic
parameter settings
• Generally “locked down”
during validation
25,000
30,000
19,099
Refined 2000
Refined 2030 E+C
Comprehensive Moderate Existing (A)
Comprehensive Moderate Focused (B)
Comprehensive High Existing (C)
Comprehensive High Focused (D)
Trends Moderate Existing Pattern (E)
Trends High Existing Pattern (F)
New Visions Moderate Focused (G)
New Visions High Focused (H)
Final w/Moderate & Focused Growth
24,255
23,982
23,855
29,524
29,833
24,046
29,743
24,520
30,628
23,872
DeKalb County Comparisons of PM Peak Period V/C Ratio
PM Peak Period Volume-to-Capacity Ratio
• Limited trend data to back
up values for future years
0.0
Model Year
History has shown changes
in…
0.1
0.2
0.3
0.4
Refined 2000
Refined 2030 E+C
Comprehensive Moderate Existing (A)
Comprehensive Moderate Focused
Comprehensive High Existing (C)
Comprehensive High Focused (D)
Trends Moderate Existing Pattern (E)
Trends High Existing Pattern (F)
New Visions Moderate Focused (G)
New Visions High Focused (H)
Final w/Moderate & Focused Growth
• Nonhome-based trip rates
0.5
0.6
0.7
0.8
0.9
1.0
0.60
0.77
0.59
0.59
0.77
0.78
0.61
0.79
0.63
0.83
0.60
DeKalb County Comparisons of Average Travel Time
Average Travel Time (in minutes)
Model Year
• Trip length frequencies
Refined 2000
Refined 2030 E+C
Comprehensive Moderate Existing (A)
Comprehensive Moderate Focused
Comprehensive High Existing (C)
Comprehensive High Focused (D)
Trends Moderate Existing Pattern (E)
Trends High Existing Pattern (F)
New Visions Moderate Focused (G)
New Visions High Focused (H)
Final w/Moderate & Focused Growth
• Auto occupancy rates
• BPR curves
18
0
2
4
6
8
10
12
14
16
18
20
22
24
26
19.0
26.2
20.6
20.9
22.7
24.2
21.1
23.8
22.1
26.1
21.0
28
30
32
Guidelines for Model Applications
Typical Model Applications and Guidelines
Developed list of
14 different
model
application types
and relevant
model guidance
unique to each
(e.g., forecasting
external trips for
MPO LRTP
Updates, etc.)
Acknowledge
uncertainty in
forecasting
process
•
•
•
•
•
•
•
•
•
MPO LRTP Updates
Comprehensive Plans
SIS/FIHS Planning
Campus Master Plans
Concurrency Applications
Development of Regional Impacts (DRIs)
Congestion Management Systems
Air Quality and Climate Change
Corridor Studies
 Corridor Feasibility Studies;
 FTA New Starts/Small Starts
Applications;
 Project Development and Environment
(PD&E) Studies;
 Interstate Master Plans;
 Interchange Justification/Modification
Reports (IJR/IMR); and
 Toll Feasibility Studies.
19
Guidelines for Model Applications
Model Application Checks

Review logic of demographic forecasts at region
and subarea

Generate color-coded plots of highway network
characteristics

Compare base and future year trip productions and
attractions by purpose at the regional and subarea level

Compare base and future year trip distribution patterns

Review logic of changes in mode splits resulting from
scenario testing that would seemingly benefit one mode
over another

Compare traffic estimates on specific corridors and
screenlines between base and future years and build
and no-build conditions
20
Next Steps
FDOT is conducting peer review of Final Report
Validation checklist and new standards being tested to
• Review/comment on recent validation studies
• Ratchet up expectations for ongoing validation studies
Model Step
Input Data
Model Statistic to Evaluate
Socioeconomic Data
Acceptable Range of Values
Low
High
Accuracy
Standard
Cube, GIS Visual and Statistical
Comparisons/Checks
Recommended Comparisons and
Calculation Methods/Comments
Document checks for households and
employment
Persons/DU (or HH)
2.00
2.70
N/A
Employment/Population Ratio
0.35
0.75
N/A
Autos/DU (or HH)
1.75
2.10
N/A
Approximate Population/TAZ
N/A
3,000
N/A
NHTS > 2.46 FL – 2.59 U.S.
Recommendations from TAZ White
Paper
Highway Network Data
Cube, GIS visual and statistical
comparisons/checks
Check hwy network, prohibitors, tolls,
paths
Highway Speed Data
ensure logical hierarchy by AT/FT/NL; survey
chk
Will provide acceptable ranges in sep.
table
Transit Network Data
chk access links; chk routing against GIS data
Checks for transit network, access,
paths
21