Transcript CE 451 - Urban Transportation Planning and Modeling Iowa State University Calibration, Adjustment and Validation Sources: Calibration and Adjustment of System Planning Models Note: Date.

CE 451 - Urban Transportation Planning and Modeling
Iowa State University
Calibration, Adjustment and Validation
Sources:
Calibration and Adjustment of System Planning Models
Note: Date = 1990 (need to adjust for inflation, other changes)
Model Validation and Reasonableness Checking Manual
NHI course on Travel Demand Forecasting (152054A)
Objectives:
• Identify and interpret trends affecting travel demand
• Explain difference between calibration and validation
• Identify critical reasonableness checks
– socioeconomic
– travel survey
– network
– trip generation
– mode split
– trip assignment
Terminology
• Model Calibration
– Estimate parameters
– Match observations (OD, AADT)
• Model Validation
– Reasonableness checks
– Sensitivity checks
Is the model
sensitive to policy
options?
• Special generators
• Screen lines, cut lines, cordons
Planner responsibilities
• Actively involve all participants
–
–
–
–
Modelers
Planners
Decision makers
Public
• Fairly present all alternatives
– Timely
– Unbiased
• Identify (clearly) the decision making process
– Who, when, and how
– Allows input from all interested groups
• You must rely on the TDM
– Therefore, must be validated
– Accurate and easy to understand (documented)
www.readysetpresent.com
How do you judge a model/recommend
improvement?
Scrutinize these characteristics:
• Data requirements
• Logic of structure and conceptual appeal
• Ease of calibration
• Effectiveness of the model (accuracy, sensitivity)
• Flexibility in application
• Types of available outputs
• Operational costs
• Experience and successes to date
• Public or private domain availability
cio.com
Trends Affecting Travel Demand
• Planners should monitor the following trends:
–
–
–
–
–
–
–
–
Demographics
Composition of the labor force
Immigration and emigration
Regional economic development
Modal shares
Vehicle occupancy
Average trip length
Freight transport
Must be aware of
trends to ensure
reasonable forecasts
• Are trends consistent with assumptions made in the modeling
process?
Image sources: scu.edu; usda.gov; illinois.edu; uwex.edu; mwcog.org; fhwa.gov; transportation1.org
doe.gov
http://www.eia.doe.gov/oiaf/aeo/pdf/trend_4.pdf
Tips for building a good model*
•
•
•
•
•
•
•
Build accurate road network
Use aerial photos behind
Make sure road attributes are
correct, esp. traffic
Use hourly counts
Income and auto ownership don’t
fully explain travel
age, gender, life cycle and personal
interest come into play
Use survey data
•
Employer based surveys get good
response (but may be biased)
–
•
•
•
•
– visualize these data
– Survey can be done cheaply
– Cooperation will be good if there’s a
good reason for it – mayor sends
letter, e.g.
*Howard Slavin, Caliper Corp. 3/13/04 peer review
Some will give home addresses, customer
addresses, license plates
Use trip chaining (tour based) and
activity based trip generation
We don’t know much about
attractions – ITE sample too small –
do your own
Drive the network using GPS
Get some data and do some statistics
to derive your parameters
Tips for building a good model*
•
Some models are completely made
up except traffic counts
–
•
•
Examine individual links after model
run
See if you really believe the counts
–
Create your OD matrix from ground
counts
–
–
–
–
–
May be better than trip gen/dist if you
“made up” the whole model (no surveys)
TransCAD has a tool for this
If still want to use trip gen/dist, this
method can be used to determine K
factors
Could also use the row and column totals
as the dependent variables in your trip gen
model
*Howard Slavin, Caliper Corp. 3/13/04
–
•
Where are the trips coming from and
going to that use the link?
In TransCAD, what is the process used to
determine this (for a particular link)?
In TransCAD, what is the process used to
show where traffic from a particular zone
is going to?
Familiarity with your region is helpful
Sources of Error
• Coding
• Sampling
• Computation (if done by
Improper structure
hand)
of model, e.g., wrong
• Specification
variables
• Data Transfer
• Data aggregation
Key Concepts
• Not enough attention on model evaluation and
reasonableness checks
• Checks should be performed after each step
– reduces error propagation
Errors can
also “cancel”
Evaluation and Reasonableness Checks
Overview
Complete?
Level of Detail?
Transportation
system
(supply)
►Network Data
Number and location
of households and
employment (demand)
►Socioeconomic Data
TDF
►Model Specification
►Model validation and calibration
Sensitive?
Documentation of
calibration?
Valid for base year?
Transportation
system
performance
Reasonable?
Methodology?
Source?
Travel
survey data
Current?
Reasonable?
Model Calibration
Model Validation
Model Application
CALIBRATION and VALIDATION
are sometimes confused.
Model development is
sometimes called calibration or
estimation as we are
estimating parameters and
constants for the particular
model structure.
estimating is a statistical
process … want high
correlation coefficients and
significant parameter values
can "import" a model - or
borrow structure and
parameters from a "similar"
area
VALIDATION is checking if the
model is accurately estimating
traffic volumes by calculated
measures (like RMSE)
Model Validation
• Validation of new model
– Model applied to complete model chain
– Base year model compared to observed travel
– Judgment as to model suitability, return to calibration
if not
• Validation of a previously calibrated model
– Compare to a new base year, with new …
•
•
•
•
SE data
Special gen.
Network
Counts
“Transportation Conformity Guidelines” (Air
Quality) require model validated < 10 years
ago
Validation suggestions
- Systemwide
- compare traffic counts across …
- Screenlines
- (long lines, check major flows)
check trip interchange (distribution)
between large sections or
quadrants
- need a survey; local knowledge of
commute patterns helps
- Cordon lines (surround a major
generator, e.g. university, CBD...)
- Cutlines (shorter, verify corridor flows,
fine tuning)
if "importing" should validate all borrowed
parameters and constants
IT IS VERY IMPORTANT TO HAVE A GOOD COUNT
PROGRAM DESIGNED TO SUPPORT VALIDATION!
iowadotmaps.com
To "calibrate" the model, need an OD database from a
survey. This is time consuming and expensive. Few, if any
cities have developed OD databases since 1980, but many
have updated old ones since then using a small survey
(e.g. 1%)
The Calibration and Adjustment manual is not intended
to replace good OD data, and is intended more for small
urban areas. (and has some old data in it! – more recent
data area available in the Barton-Ashman publication).
Calibration and Adjustment Steps:
1) verify network and socioeconomic data
2) run the model
3) develop region-wide values (e.g. trips/person, vmt/person)
4) compare region wide values with “Appendix A” values
5) develop screenlines and cutlines
6) compare model results with ground counts for crossings
7) determine problems (system level, local, combination)
8) modify one or more equations, parameters or variables
according to chapters on:
- networks
- trip generation
- auto occupancy
- trip distribution
- traffic assignment
Other chapters focus on:
- transit
- external stations
- system vs. local changes
- expected vs. required accuracy
- conclusions
- trouble shooting
Network Data Reasonableness Checks
• Check Trees for 2-3 major attractions*
• Check coded facility types – how used (BPR?)?
• Verify speed and capacity look-up table (what LOS used
for capacity?)*
– Speed adjust (can lower the freeway speed if it is being
overloaded – tweak?)
• Significant transportation projects – narrative included?
Still viable?
• Consistency with MTP
• Plot (facility types, # lanes,
speeds, area types) to detect
coding errors*
* Items we can check in labs
Details
2. Network Errors
2.1 Centroid Connectors
- represent local streets
- check access (all 4 sides?)
- not connected to intersections
- make sure they are not blocked by a
physical barrier (river, etc.)
Des Moines
Model
Capacity
Look-up
Table
Des Moines Model Capacity Look-up Table (cont.)
Des Moines
Model
Capacity
Look-up
Table (cont.)
Des Moines Model Capacity Look-up Table (cont.)
2.3 Intersection Penalties (check them!)
- most congestion here
- more important in sub-area modeling
- turn penalties
- account for congestion
- speed volume function
- can include delay on approach links
- can do it manually for small networks
-check for circuity (correct with small turn penalties!)
-See TransCAD Manual B “Chapter 10: Traffic Assignment with Volume Dependent Turning
Delays”
2.4 Intrazonal times
• increasing intrazonal trips (in
distribution) decreases
interzonal trips (useful if too
many trips are being loaded on
the network)
•number of trips is a function of
travel time (gravity model)
-can adjust travel time on
intrazonals
-can adjust friction factor
curve to produce more
shorter trips (which
intrazonals usually are)
-can change definition of
zones (size, land use)
•Air quality analysis
implications???
3.1 Trip generation
- socioeconomic data can be a source of error
- initial step is to check system trip totals, compare w/ Table
4 and A1 and A2 (next pages)
- if there is a problem, check the system number of dwelling
units
- still a problem?, check production/attraction rates
Trip Generation Calibration
Reasonableness checks – compare to other cities,
check future trends
•
•
•
•
•
•
•
•
•
•
•
•
•
Population
Households
Average Household Size
Basic employment
Retail employment
Service employment
Military employment
Population per employee
Person trips per person
Person trips per household
HBW attractions per employee
HBW productions per household
HB shopping attractions per retail employee
503,345
201,116
2.50
76,795 (33%)
50,465 (24%)
101,697 (43%)
42,800
1.81
4.26
10.65
1.44
1.74
5.99
Colorado Springs 1996 Travel Demand Model Calibration
Table A2
More recent data …
From Minimum Travel Demand Model Calibration and
Validation Guidelines for the State of TN
3.2 Income
- be sure you are using “real” dollars
3.3 P and A rates
• Problems: old, borrowed, small survey
• may work OK at the system level, but not for sub-areas
• check system-wide values (see tables, next pages)
– raise or lower trip generation rates
– Person trip or vehicle trip rates used?
• we usually have person trip by purpose, but can apply occupancy factor and check
against vehicle rates (ITE)
• later, screen line counts can be adjusted by varying trip generation rates
(post assignment)
• check cutlines and cordon counts
-
coordinate all of the above
Nonhome
More recent data …
Trip Generation Calibration
Typical Values
•
•
•
•
•
•
•
Person trips per household: 8.5 to 10.5
HBW person trips per household: 1.7 to 2.3
HBO person trips per household: 3.5 to 4.8
NHB person trips per household: 1.7 to 2.9
HBW trips: 18% to 27% of all trips
HBO trips: 47% to 54% of all trips
NHB trips: 22% to 31% of all trips
Trip Generation Reasonableness Checks
• Examine trip production and attraction models
– Form?
– sensitivity?
– IMPORTANT: keep parameters reasonable (e.g. don't use negative
coefficients in regression models just because they provide the best fit.)
• If you think you need to use unintuitive parameters, check the whole
process...
• Check models for …
– External-through and external-local trips
– Truck trips
• To calibrate trip generation and trip distribution, sometimes we
may use ...
– default values from past surveys
– very limited new surveys
– census journey to work data (CTPP)
Examine trip purposes used … Use more trip purposes?
TRIP PURPOSES
Note:
each
income
class is a
purpose!
Scaling Factor
HBW low income
0.795
HBW low-middle income
0.823
HBW middle income
0.861
HBW upper middle income
0.908
HBW high income
0.936
HB elementary school
0.733
HB high school
1.991
HB university
0.895
HB shopping
0.698
HB social-recreation
0.945
HB other
0.875
NHB work-related
0.858
NHB other
0.820
Truck
0.985
Internal-external
0.591
Scale survey for
participation
(relative
participation)
Colorado Springs 1996 Travel Demand Model Calibration
Travel Survey Data
Reasonableness Checks
• Determine source of travel survey data
– Types of survey conducted
– Year of survey
• Scale survey for participation
• If no survey (borrowed)
– Check source of trip rates, lengths, TLFD
– Is area similar
• Geographic area?
• pop/HH/empl. characteristics?
• Urban density and trans system?
• Compare to similar regions and to same
region in earlier times:
– Person trip rates by trip purpose
– Mean trip lengths by trip purpose
• HBW longest? HBO shortest?
– TLFDs by trip purpose
Socioeconomic Data: Check
Reasonableness
• Review source for estimates and forecasts
• Visualize (plot) trends …
–
–
–
–
–
Population and household size
Household income
automotive availability
distribution of employment by type (basic, retail, service)
employees per household and per capita … rate of increase is
decreasing
• Check future household and employment changes by zone
3.4 Special generators
-e.g. universities, airports, malls, ...
-Use ITE or survey
3.5 trip balancing factors
4.0 Auto occupancy
• initially, Ps and As should balance to should be 0.9 to 1.1; if not, check
your PA rates and socioeconomic data
• NHB is usually out of balance
• Automobile occupancy
– by trip purpose?
– Basis?
– Constant?
• see table 6 and A9 (next pages … are these still good?)
5.0 Trip Distribution
5.1 Mean Trip Length
- recall: shape of curve affects trip length
distribution
-See below for effect of changing friction factors
F
Curve
trips
link vols.
F
tt
more long trips
more short trips
internal vols.
-varying trip length has a big impact
on assigned volumes
-portions of a friction factor table
can be adjusted (more flexible than
adjusting equations)
5.2 Estimate Trip Length
-compare average trip lengths (in
minutes) by purpose to:
HBW
HBSR
HBSh
NHB
t = 0.98 x p.19
t = 2.18 x p.12
t = 8.1
t = 0.63 x p.20
where p is population
SR = social/recreation
Sh = shopping
From Minimum Travel Demand Model Calibration and
Validation Guidelines for the State of TN
Source: Virginia Travel Demand Modeling Policies and Procedures Manual
Source: Virginia Travel Demand Modeling Policies and Procedures Manual
Source: Virginia Travel Demand Modeling Policies and Procedures Manual
5.3 Employment Distribution Problems
(large cities, mostly)
problem: match low income households
with low income jobs
solution #1: disaggregate trip purposes by
income quartile
solution #2: use k-factors (trial and error)
… yuk
5.4 Special Treatment, other trip purposes
- schools (ignore if small %?)
- trucks (calibrate with externals?)
-Taxi
normally, distortions are insignificant
Trip Distribution Reasonableness Checks
Examine …
• Mean trip length (increasing or decreasing?)
• TLFDs
• Treatment of friction factors (same?)
• Treatment of terminal times (logic?)
• Treatment of K factors
• Comparison with JTW trip length
• Comparison with JTW sector interchange volumes or
percentages.
Calibrating Friction Factors
Calibrating a Gravity Model
Adjusting Friction Factors
Travel Times
Ranges from
Skims
Observed Trip
Expanded from
Surveys
Input
Friction
Factors
Gravity
Model Trips
Adjustment
Factor
Observed
Gravity
Model
New Friction Factors
Friction
Adjustment
Factor x Friction
Factor
2.5
7,100
30.0
8,200
0.87
25.98
5.0
14,950
2.50
16,300
0.92
2.29
7.5
17,850
1.80
19,250
0.93
1.67
10.0
16,000
1.50
19,100
0.84
1.26
12.5
15,500
1.20
17,100
0.91
1.09
15.0
15,900
1.00
12,300
1.29
1.29
17.5
16,400
0.95
18,000
0.91
0.87
20.0
15,150
0.90
14,300
1.06
0.95
22.5
13,500
0.85
11,900
1.13
0.96
25.0
11,000
0.80
9,250
1.19
0.95
27.5
9,500
0.75
8,100
1.17
0.88
30.0
9,100
0.70
6,100
1.49
1.04
32.5
5,700
0.65
4,900
1.16
0.76
…
…
…
…
…
…
Trip Distribution Calibration and Validation
• Check modeled vs. household survey TLFD and mean trip lengths
• Get HBW area-to-area flows from JTW
HBW 1990 JTW TLFD and Area-to-Area Flows for Kansas City
Commute Length in
Minutes
Percent
< 15
27.87
Central-Central
County
31.49
15-29
41.63
Central-Suburban
County
7.48
30-39
17.04
Suburban-Central
County
15.13
40-59
7.70
Within Suburban
County
32.98
>60
3.00
To Other Suburban
County
10.81
Work out of area
2.11
Mean 21.44
Journey-to-Work
Flows
Percent
OD validation
Using cell phone and/or GPS location to determine travel patterns is nothing
new. But leave it to Google to make it really easy - maybe too easy.
http://googleblog.blogspot.com/2009/08/bright-side-of-sitting-in-traffic.html
Adam Shell
Office of Systems Planning
Iowa Department of Transportation
Link
POA: price of anarchy (30%?)
Nash equilibrium vs. system optimality
OD data are destroyed! (privacy)
6.0 Traffic assignment
6.1 All or nothing
- adjusting link speeds will change assigned volumes
- initial speeds should be set to LOS C speeds (0.87 x free
flow speeds)
6.2 capacity restraint
- volume = f(time)
- final volume is average of all iterations or
later iterations can be weighted more
heavily
- adjust free flow time or c (capacity) to
change volumes
IF…
Link
Capacity
THEN…
Speed
Travel
Time
Assigned
Volume
6.2.1 definition of capacity
design: LOS C (0.87c)
ultimate: LOS E (1.00c)
parameters differ depending on definition of
capacity …
if defined as LOS C, 0.15(v/c)4
if defined as LOS E, 0.80(v/c)4 (see HCM)
7.0 Transit Ridership
- for small/medium cities, may not have to build a transit network
- If not using a transit network, can use the following method (if
trip generation includes transit trips):
1. increase auto occupancy by transit percentage (e.g. if auto
occupancy is 1.05, then change to 1.05 x 1.38 = 1.45) if transit
percentage is 38%
2. decrease trip production or attraction rates (one of them only,
then balance) … if you use productions, can vary mode split by
income class
3. modify productions or attractions by zone
- get data from transit company
- adjust socioeconomic data or make direct P/A adjustments
Mode Split Reasonableness Checks
• Mode split model?
• Form?
• Variables included in the utility functions?
• Coefficients logical?
• Value of time assumptions
• Parking cost assumptions
• How do mode shares change over
time?
• Mode share comparisons
with other cities
http://www.bts.gov/publicati
ons/journal_of_transportation
_and_statistics/volume_08_n
umber_02/html/paper_05/fig
ure_05_03.html
Mode Split Calibration and Validation
• Experienced planning consultant required …
–
–
–
–
–
–
Form of LOGIT model
Variables included in utility functions
Calibration of coefficients for utility function variables
Testing for IIA properties
Analysis of household survey data
Analysis of on-board transit survey data
• Calibration tasks we can do:
• Compare highway and transit trips
• Total
• By purpose
• Compare Ridership by route
• CBD cordon line survey (if bus service is downtown only)
8. External stations
- externals have no socioeconomic data
- Ps and As are prepared by matching ground counts
- I/E treated with the gravity model
-E/E
- compare with Table 11 below
9. System vs. local checks
check
1. system wide (screenlines)
2. major movements (cutlines)
3. links
if all screenlines are high or low, vary
- auto occupancy
- trip generation rates
- trip lengths
- intrazonal times - all zones
- socioeconomic data - all zones
if corridor volumes are high or low, vary
(for zones affecting corridor…)
- auto occupancy
- trip generation rates
- intrazonal travel times
- land use
- centroid connectors
- intersection penalties
if links are high/low, vary
- speed
- intersection penalty
- centroid locations
- special generators
- local network configuration
10. Expected/Required accuracy
 We are concerned about errors that would require a design
change (e.g. number of lanes)
 Note that ground counts also contain error
 Perfectly calibrated models produce link estimates with 1/3 above
the standard error in ground counts and 2/3 below the standard
error.
 Need ground counts for 65% of freeways and arterials, and a good
sample from other facilities
From Minimum Travel Demand Model Calibration and Validation
Guidelines for the State of TN
10. Expected/Required accuracy (cont.)
 The correlation coefficient should be greater than .88
 VMT estimate (region-wide) should be within 5% (take care to
compare same roads in systems)
 VMT/person should be 17-24 for large areas, 10-16 for smaller areas
(see also Table A7, next page)
 VMT/household should be 40-60 for large areas, 30-40 for smaller
areas
From CTRE Employment Data Project:
From Minimum Travel Demand Model Calibration and
Validation Guidelines for the State of TN
Source: Virginia Travel Demand Modeling Policies and Procedures Manual
From Minimum Travel Demand Model Calibration and Validation
Guidelines for the State of TN
From Minimum Travel Demand Model Calibration and Validation
Guidelines for the State of TN
Source: Virginia Travel Demand Modeling Policies and Procedures Manual
From Minimum Travel Demand Model Calibration and Validation
Guidelines for the State of TN
Trip Assignment Reasonableness Checks
• All-or-nothing assignment
• study effect of increasing capacity
• Compare to Equilibrium assignment
• Check volume delay equation (BPR parameters)
• Compare
• screen line volumes
• Cut line volumes
• Time-of-day assignments?
• Source of factors
• Peak spreading used for future?
• If not, conversion factors source?
(peak hour to 24-hour)
• Local VMT (% assigned to
intrazonals and centroid connectors
Trip Assignment
Calibration and Validation
Overall VMT or VHT check
• 40 to 60 miles per day per HH in
large metro areas
• 30 to 40 miles per day per HH in
medium metro
• +/- 10% OK on screen lines
• Sign is important
Compute by …
- volume group
- facility type
- transit assignments
- time of day
12. TROUBLE SHOOTING
Other Factors Impacting
Forecasted Travel Demand
• Can be implied in travel surveys (but not explicit)
– Telecommuting
– Flexible work hours
– HB business
• How to account for …
– Aging population
– Internet shopping
– Roadway congestion (will it affect generation in the
future)
– New modes
Issues for modeling
• Transferability of parameters
– More research is needed
• Forensic analysis
– How well did the models work?
• Confidence and Credibility
– How to improve
• “Official” versions vs. what-if models
– Integrity of the model
• Need more transparency, documentation,
appropriateness of techniques