Teaching Simulation

Roger Grinde, [email protected]

University of New Hampshire Files: http://pubpages.unh.edu/~rbg/TMS/TMS_Support_Files.html

Teaching Simulation

   Do you teach simulation?

  In which courses?

With spreadsheets? Add-Ins?

 Monte Carlo? Discrete Event?

Do you use simulation to help teach other topics?

Do other courses at your school use simulation?

Session Overview

        Common Student Misunderstandings Simulation-Related Learning Goals Motivations Building on Other Methodologies Effects of Correlation Interpreting Results Software Issues Considerations, Recommendations

Student Misunderstandings

  What are some misunderstandings students have about decision-making in the face of uncertainty?

What are some common errors students make in simulation?

Some Considerations

   Decide which learning goals are most important, and structure coverage so those goals are attained.

  Student backgrounds Time constraints   Overall course objectives Inter-course relationships, role of course in curriculum Monte-Carlo and/or Discrete-Event? Related software selection question.

Teaching environment, class size, TA support, etc.

Learning Goals

 What are your learning goals when teaching simulation?

     Fundamental Concepts Methodology of Simulation Applications of Simulation Modeling Knowledge & Skills Critical & Analytical Thinking

Mapping: Learning Goals to Examples

Examples/Application Areas

Mapping: Goals to Examples

Learning Goal/Objective

Extension of other analaysis tools Is simulation needed?

Variety of probability distributions Model-building issues (where a simulation model would be different than a deterministic model) Output distribution as function of input distributions Historical/empirical data Summary statistics Alternate decision criteria & risk measures Sources of error Correlation and/or relationships among input variables Optimization concepts in simulation X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X

Motivations (Why is simulation useful?)

   Two investment alternatives  A: Invest $10,000.

  Probability of a $100,000 gain is 0.10

Probability of a $10,000 loss is 0.90

 B: Invest $10,000  Probability of a $500 gain is 1.0

Which would you choose?

Why?

Risk-Informed Decision Making

    Appropriate and inappropriate uses of averages.

Managers manage risk.

Simulation gives us a tool to help us evaluate risk.

Risk: The uncertainty associated with an undesirable outcome.

  Risk is not the same as just being uncertain about something, and is not just the possibility of a bad outcome.

Risk considers the likelihood of an undesirable outcome (e.g., the probability) as well as the outcome.

magnitude of that

“Flaw of Averages” (Sam Savage)

  Article by Sam Savage ( http://www.stanford.edu/~savage/faculty/savage/ ) Annuity Illustration (historical simulation)

Simulation Model Schematic

Fixed (Known) Inputs Random (Uncertain) Inputs Decision Variables Simulation Model Outputs & Performance Measures  Concept of an output “distribution.”

Foundations of Simulation

   Randomness, Uncertainty Probability Distributions Tools    Dice Roller (John Walkenbach: http://www.j-walk.com/ss ) Die Roller (modified) Interactive Simulation Tool

Extending Other Methodologies

      Spreadsheet Engineering Base Case Analysis What-If Analysis, Scenario Analysis Critical Value Analysis Sensitivity Analysis Simulation

Extending Other Methodologies

     Familiar Example/Case; Students have already developed model and done some deterministic analysis.

Students provided with some probability distribution information Develop comfort with mechanics of simulation See the “value added” of simulation Provides entry point for discussion of important questions

Example: Watson Truck

   Adapted from Lawrence & Weatherford (2001) Students have previously built base case model, done “critical value” analysis (using Goal Seek), and have done sensitivity analysis (data tables, tornado charts) Link to files: PDF , Sensitivity , Simulation

Watson Truck: Inputs

A

Watson Truck Rental

B 1 2 3 4 5 6 7 18 19 20 21 22 8 9 10 11 12 13 14 15 16 17

Parameters/Uncontrollable Inputs

Purchase Price Property Tax Var Cost/Truck # Trucks Prop Tax Growth Truck Cost Growth Base Truck Rental Rate % Trucks Rented @ $1000 Rental Rate Slope Rental Rate Inflation Business Sale Multiplier Discount Rate

Decision Variable

Rental Rate (decision variable)

Intermediate Calculations

Rental Rate Slope % Trucks Rented C $1,000,000 $35,000 $4,800 50 4% 7% $1,000 60% 7% 9% 3 10.0% per year per year per month per $100 reduction in rental rate Sales price assumed to be 3*(year 3 revenues) $1,000 -0.07% 60.0% D E per $1 increase in rental rate F G

Watson Truck: Base Case Model

24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 A

Primary Output

B Net Present Value (@ discount rate)

Cash Flow Model Cash Inflows

Truck Rental Income Business Sale Total Inflows

Cash Outflows

Purchase Price Property Tax Truck Var Cost Total Outflows

Net Cash Flow

C

$209,769 0

$0 $1,000,000 $1,000,000 ($1,000,000) D C25: =NPV(C15,D40:F40)+C40

1

$360,000 $360,000 $35,000 $240,000 $275,000 $85,000 E

2

$392,400 $392,400 $36,400 $256,800 $293,200 $99,200 F

3

$427,716 $1,283,148 $1,710,864 $37,856 $274,776 $312,632 $1,398,232 G H F30: =E30*(1+$C13) F31: =C14*F30 F32: =SUM(F30:F31) F36: =E36*(1+$C8) F37: =E37*(1+$C9) F38: =SUM(F35:F37) F40: =F32-F38

Watson Truck: Sensitivity Analysis

Tornado Sensitivity Chart

Base Truck Rental Rate % Trucks Rented @ $1000 # Trucks Purchase Price Business Sale Multiplier Var Cost/Truck Discount Rate Rental Rate Inflation Rental Rate Slope Property Tax Truck Cost Growth Prop Tax Growth $0

Output Measure

$50,000 $100,000 $150,000 $200,000 $250,000 $300,000 $350,000 $400,000 $450,000 $500,000 -10 Pct +10 Pct

Watson: Simulation

1,000 Trials

.035

.026

.018

.009

.000

($525,250)

Forecast: Net Present Value (@ discount rate) Frequency Chart

Mean = $232,119 ($139,570) $246,111 $631,792 Certainty is 82.00% from $0 to +Infinity Dollars

1,000 Displayed

35 26.25

$1,017,472 0 17.5

8.75

Learning Goals Addressed (at least partially)

        Linkage with other course/functional area What inputs should we simulate?

Useful probability distributions. Choice of parameters. Subjective versus objective estimates.

Concept of an output distribution What results are important?

Sources of error in simulation Simulation mechanics Simulation in context with other tools

Example: Single-Period Portfolio

  Simple example, but helps address a number of learning goals  Do we need to simulate?

     Effect of correlation among input quantities Confidence vs. Prediction (certainty) intervals Quantification of risk, multiple decision criteria Optimization concepts within simulation context Precision of estimates from simulation Link to file

Spreadsheet

B C D 1 2

Portfolio Allocation Model

3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Investments

Money Market fund Income fund Growth and Income fund Aggressive Growth fund

Total amount available

Decision variables

Money Market fund Income fund Growth and Income fund Aggressive Growth fund

Total expected return

Annual return

3.0% 5.0% 7.0% 11.0% $100,000

Amount invested

$25,000 $25,000 $25,000 $25,000 $6,500

Lower bound

$0 $10,000 $0 $10,000 E

Upper bound

$50,000 $25,000 $80,000 $100,000

Do we need simulation?

 Assuming we know the distributions for the returns, do we need simulation to compute the    expected return of the portfolio?

variance of the portfolio?

tail probabilities?

What if the asset returns are correlated?

 What is the effect of correlation on the distribution of portfolio returns?

Large Stocks Large Growth Stocks Large Value Stocks Small Stocks Small Growth Stocks Small Value Stocks Foreign Stocks Bonds

Large Stocks

1 0.958411

0.901159

0.720036

Large Growth Stocks Large Value Stocks

1 0.74063

1 0.606356 0.776748

Small Stocks

1

Small Growth Stocks Small Value Stocks Foreign Stocks

0.755265 0.720758 0.678815 0.921266

0.507037 0.322101 0.715396 0.875155 0.618755

0.391551 0.294704 0.454882 0.275857 0.325624 0.140232

0.366054

0.267404 0.465424

0.28663

1 1 1 0.164181 0.369522 0.112362

Bonds

1 Based on Standard & Poor Micropal, via Franklin/Templeton Investor Topics Update, Winter 2001 (Asset Returns from 1980-2000)

Results (n=1000)

  No Correlation  Mean = $6842   Standard Deviation = $5449 5% VaR = ($2165) Positive Correlation    Mean = $6409 Standard Deviation = $7386 5% VaR = ($5655)

Decision Criteria, Risk Measures

   What criteria are important for making decision as to where to invest? Average? Standard Deviation? Minimum? Maximum? Quartiles? VaR? Probability of Loss?

Measures of risk.

 Simulation gives us the entire output distribution.

Entry point for optimization within simulation context  Alternate scenarios, efficient frontier, OptQuest, RiskOptimizer, etc.

Confidence Intervals

   Students can (usually) calculate a confidence interval for the mean.

Do they know what it means?

Reconciling confidence and prediction intervals.

Sample Results (Portfolio Problem)

Statistics:

Trials Mean Median Mode Standard Deviation

Value

1000 $6,409 $6,531 -- $7,386

90% Confidence Interval

Standard Error Z Lower Limit Upper Limit $233.56

1.645

$6,025 $6,794    90% CI on Mean Dollar Return: ($6025, $6794) What does that confidence interval mean?

 Common (student) error What does the CI about an individual outcome? For example, from this year’s return?

Sample Results (cont)

Percentile

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

dollars

($16,088) ($5,655) ($3,052) ($1,008) $271 $1,277 $2,498 $3,484 $4,307 $5,365 $6,531 $7,695 $8,349 $9,310 $10,386 $11,419 $12,431 $13,896 $15,689 $18,659 $30,330    Cumulative Percentiles of the Portfolio Return Distribution What do these results mean?

What is the 90% “prediction” (or “certainty”) interval (centered around the median)?

Putting Them Together

   90% Confidence Interval for the Mean  ($6025, $6794) 90% Prediction Interval (centered around median)   (-$5655, $18,659) Note: Crystal Ball uses the term “certainty”) Students:   Understand the difference?

Understand when one is more appropriate than the other?

Precision of Simulation Results

  Since we know the true value of the mean (for the portfolio problem), this can be a good example to look at precision and sample size issues.

Confidence interval for proportion or for a given percentile sometimes makes more sense.

Crystal Ball: Precision Control

    Nice way to illustrate effect of sample size.

Precision Control stops simulation based on user-specified precision on the mean, standard deviation, and/or a percentile.

 Actually, CB stops whenever the trials, precision specifications).

first of a number of conditions occurs (e.g., maximum number of Example Example (Portfolio Allocation) (Option Pricing)

Precision: Portfolio Example

Precision Control Experiment: Summary

Number of Trials Required for Specified Precision

Precision

$800 $400 $200 $100

Mean

300 1250 5350 21,700

5th Percentile

1900 4950 20,300 95,200 Note: "Precision" is the half-width of the 95% confidence interval True Mean = $6,500

Precision: Option Pricing Example

Precision Control Experiment: Mean Call and Put Values

Number of Trials Required for Specified Precision of Mean Call and Put Values

Trials

50 200 700 2600 64400

Precision: Call

$0.259

$0.166

$0.099

$0.050

$0.010

Precision: Put

$0.390

$0.181

$0.090

$0.047

$0.009

Note: "Precision" is the half-width of the 95% confidence interval Black-Scholes Call Price = $0.733

Black-Scholes Put Price = $0.955

Crystal Ball Functions and Simple VBA Control

   Crystal Ball provides built-in functions  Distribution Functions (e.g., CB.Normal)  Functions for Accessing Simulation Results (e.g., CB.GetForeStatFN) Control through VBA   For some students, can be a hook into greater interest in simulation and/or VBA/DSS.

Allows one to prepare a simulation-based model for someone who doesn’t know Crystal Ball.

Example

VBA-Enabled Example

B C D E F 20 21 22 23 24 25 26 27 28 29 30 31 32 1 2 14 15 16 17 18 19 3 4 5 6 7 8 9 10 11 12 13

Portfolio Allocation Model

Investments

Money Market fund Income fund Growth and Income fund Aggressive Growth fund

Total amount available

Decision variables

Money Market fund Income fund Growth and Income fund Aggressive Growth fund

Total return

Summary Statistics

Number Trials Mean Standard Deviation Minimum Maximum Standard Error of Mean Value at Risk (enter %) Critical Value (enter $)

Annual return

3.5% 4.1% 6.5% 29.3% $100,000

Amount invested

$25,000 $25,000 $25,000 $25,000 $10,866

Simulation Results

400 $6,695 $5,644 ($9,718) $20,612 $282 5% $4,000 D30: =CB.GetCertaintyFN(C$17,C30)/100

Param1 Distributions/Parameters Param2 Distribution

2.0% 5.0% 7.0% 11.0% 4.0% 5.0% 12.0% 18.0% Uniform Normal Normal Normal C5: =CB.Uniform(D5,E5) C6: =CB.Normal(D6,E6) (copied down) $C$17: =SUMPRODUCT(C5:C8,C13:C16) (Forecast Cell) C22: =CB.GetForeStatFN(C$17,1) ($2,985) 0.305

Quartiles

0% 25% 50% 75% 100% ($9,718) $2,840 $6,931 $10,730 $20,612 5.0% chance return will be <= ($2,985) Probability (Return <= $4,000) = 0.305

F22: =CB.GetForePercentFN(C$17,E22*100) G H I J Enter Number of Trials 400 Run Simulation

CB. Functions and VBA

   CB. Distribution Functions  e.g., CB.Normal, CB.Uniform, CB.Triangular) CB. Functions for reporting results  CB.GetForeStatFN, CB.GetCertaintyFN, CB.GetForePercentFN

VBA: simple to automate specific processes Sub RunSimulation() CB.ResetND

CB.Simulation Range("n_trials").Value

End Sub Sub CreateReport() CB.CreateRpt

' CB.CreateRptND cbrptOK End Sub

Learning Goals Revisited

  Decide which learning goals are the most important, and structure coverage so those goals are attained.

   Student backgrounds Time constraints Overall course objectives Mapping of learning goals to examples, cases, and projects that you will use.

Mapping: Learning Goals to Examples

Examples/Application Areas

Mapping: Possible Learning Goals to Examples

Learning Goal/Objective

Extension of other analaysis tools Is simulation needed?

Variety of probability distributions Model-building issues (where a simulation model would be different than a deterministic model) Output distribution as function of input distributions Historical/empirical data Summary statistics Alternate decision criteria & risk measures Sources of error Correlation and/or relationships among input variables Optimization concepts in simulation X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X

Common Student Errors

      Thinking of simulation as the method of first Simulating too many quantities.

choice.

Too much focus on distribution/parameter selection or on the numerical results, not enough on insights/decision.

Misinterpretation of results, especially confidence intervals Modeling: Using same return, lead time, etc. for every time period/order, etc. (difference between deterministic and simulation models) Choosing the assumptions, distributions, parameters, etc. that give the “best” numerical results.

Software Issues: Monte-Carlo

   Alternatives  “Full-Service” Add-In? (e.g., @Risk , Crystal Ball , XLSim by Sam Savage, RiskSim )   “Helper” Workbook? (e.g., Interactive Simulation Tool with Random Number Function support) “Native” Excel?

All have advantages, disadvantages Back to learning objectives, role of course, student audience, etc.

Software Issues: Discrete Event

  Alternatives  Stand-alone package (e.g., Arena , Process Model , Extend )   Excel Add-In (e.g., SimQuick by David Hartvigsen) Native Excel modeling augmented by Monte Carlo tool (e.g., QueueSimon by Armann Ingolfsson) DE Simulation can be a great way to help teach concepts in other areas (e.g., queuing, inventory)  Don’t necessarily need to teach DE Simulation to be able to use it to teach other things.

Other Considerations

      Program-level, inter-course objectives Role of course in curriculum Level/background of students Monte-Carlo and/or Discrete-Event? Related software selection question.

Teaching environment, class size, TA support, etc.

How much of course can/should be devoted to simulation?

Recommendations

   Learning Goals: Figure out what you really want students to learn and be able to do, accomplish these goals?

after class is over; in other classes, internships, future jobs? How can simulation coverage help your Cases: Engage students in the business problem, let them discover relevance of simulation.

Student-Developed Projects: Students gain better awareness of all the “little” decisions involved in modeling and simulation.

Additional Slides

Concept Coverage Through Examples

   Philosophy: Expose students to a number of application areas, but at the same time covering fundamental decision-making, modeling, and analysis concepts and methodologies.

Counter to the way many of us were taught.

Key: We need to clearly understand which concepts we’re trying to convey with each example.

Examples that Work Well

    Fundamentals: Dice Roller, Interactive Simulation Tool

Personal Decisions:

Car Repair/Purchase Decision , Portfolio (single period, based on CB Model), College Funding (based on Winston & Albright) Capital Project Evaluation: Truck Rental Company (based on Lawrence & Weatherford), Project Selection/Diversification (CB Model), Product Development & Launch (CB Model)

Finance:

Stock Price Models , Option Pricing , Random Walks , Mean Reverting Processes

Examples (continued)

    

Inventory:

Antarctica DG Winter Coats (NewsVendor), (multi-period, based on Lapin & Whisler)

Queuing:

QueueSimon (Armonn Ingolfsson)

Games/Tournaments, Sports:

Simulation NCAA Tourney (based on Winston & Albright), Home Run Derby Baseball Simulation (VBA-enabled), Baseball Inning

Simulation in Teaching Other Topics:

Management Illustration , QueueSimon Ingolfsson) Revenue (Armonn

Crystal Ball Features:

CB Macros , CB Functions

Examples Posing Difficulties for Spreadsheets

    Multi-server queues and queue networks Most production systems Business process redesign However, some add-ins do exist for simple discrete-event models (e.g., SimQuick by David Hartvigsen)

Sources of Error in Simulation

 What are some of the sources of error in a spreadsheet simulation model/analysis?

Learning Objectives (Revisited)

     General Probability Distributions Statistics Relationships Among Variables Decision Making

Possible Learning Goals

  General    Use simulation as an extension of other analysis tools Apply simulation to a variety of business problems Identify when simulation is and is not needed to analyze a situation Probablilty Distributions    Understand and use probability distributions to model phenomena Describe the output distribution, understanding this to be a function of the input distributions Use historical/empirical data and subjective assessments appropriately in choosing distributions and parameters

Possible Learning Goals (cont)

  Statistics    Correctly interpret summary statistics, including percentiles/histograms Correctly interpret confidence and prediction (certainty) intervals Identify sources of error in simulation, apply to specific situations Relationships Among Variables   Include appropriate correlation and/or other relationships when model building Describe the effect of correlation and/or other relationship on simulation results

Possible Learning Goals (cont)

 Decision Making    Identify and correctly use different risk measures Use appropriate criteria in making recommendations Use optimization concepts in a simulation application

Student Project Example (MBA)

  PPT File Excel File