Transcript Slide 1
Chap. 5 Building Valid, Credible,
and Appropriately Detailed
Simulation Models
5.1 Introduction and Definitions (1)
• Verification is concerned with determining
whether the conceptual simulation model
(model assumptions) has been correctly
translated into a computer program, i.e.,
debugging the simulation computer program.
'
.
I
5.1 Introduction and Definitions(2)
• Validation is the process of determining
whether a simulation model (as opposed to the
computer program) is an accurate representation
of the system, for the particular objectives of
the study.
Validation
• A valid model can be used to make decisions.
• A validation process depends on the complexity of
the system and on whether a version of the system
currently exists.
• A model can only be an approximation.
• A model is valid for one purpose.
• The measures of performance used to validate the
model should include those that the decision maker
will actually use for evaluating system design.
5.1 Introduction and Definitions(3)
• A simulation model and its results have
credibility if the manager and other project
personnel accept them as "correct“.
• A credible model is not necessarily valid, and
vice versa.
Validation
Verification
Validation
Establish credibility
Establish
credibility
System
Analysis
and data
1, 2, 3
Conceptual
model
Simulation
Programming program
4
Make
model runs
5,6,7,8,9
“Correct”
results
available
Sell
results to
management
10
Results used
in decisionmaking
process
Figure 5.1
Timing and relationships of validation, verification, and establishing credibility
5.2 Guidelines for Determining the
Level of Model Detail (1)
• Carefully define the specific issues to be investigated
by the study and the measures of performance that
will be used for evaluation.
• The entity moving through the simulation model does
not always have to be the same as the entity moving
through the corresponding system.
• Use subject-matter experts (SMEs) and sensitivity
analyses.
• “Moderately detailed“ model.
• Regular interaction.
5.2 Guidelines for Determining the
Level of Model Detail (2)
• Do not have more detail in the model than is necessary to
address the issues of interest, subject to the proviso that the
model must have enough detail to be credible.
• The level of model detail should be consistent with the type
of data available.
• In all simulation studies, time and money constraints are a
major factor in determining the amount of model detail.
• If the number of factors (aspects of interest) for the study is
large, then use a "coarse" simulation model or analytic model
to identify what factors have a significant impact on system
performance.
5.3 Verification of Simulation
Computer Program
Tech 1: Write and debug the computer program on modules or subprograms.
Tech 2: More than one person review the computer program (structured walk
through of the program).
Tech 3: Run the simulation under a variety of settings of input parameters, and
check to see that the output is reasonable.
Tech 4: "trace", interactive debugger.
Tech 5: The model should be run under simplifying assumptions for which its
true characteristics are known or can easily be computed.
Tech 6: Observe an animation of the simulation output.
Tech 7: Compute the sample mean and variance for each simulation input
probability distribution, and compare them with the desired mean and
variance.
Tech 8: Use a commercial simulation package to reduce the amount of
programming required.
5.4 Techniques for Increasing Model
Validity and Credibility (1)
• Collect high-quality information and data on the system
– Conversation with subject matter experts
in MS, machine operators, engineers, maintenance personnel, schedulers,
managers, vendors, …
– Observation of the system
•
•
•
•
•
Data are not representative of what one really wants to model
Data are not of the appropriate type or format
Data may contain measurement, recording, or rounding errors
Data may be biased because of self interest
Data may be inconsistent
– Existing theory IID exponential random variables
– Relevant results from similar simulation study
– Experience and intuition of the modelers
5.4 Techniques for Increasing Model
Validity and Credibility (2)
• Interact with the manager on a regular basis
– There may not be a clear idea of the problem to be solved at initiation
of the study.
– The manager’s interest and involvement in the study are maintained.
– The manager’s knowledge of the system contributes to the actual
validity of the model
– The model is credible since the manager understands and accepts the
model’s assumptions.
• Maintain an assumptions document and perform a structured
walk-through
• Validate components of the model by using quantitative
techniques.
• Validate the output from the overall simulation model
• Animation
5.5. Management's Role in the
Simulation Process
• Formulating problem objectives.
• Directing personnel to provide informationand
data to the simulation modeler and to attend
the structured walk-through.
• Interacting with the simulation modeler ona
regular basis.
• Using the simulation results as an aid in the
decision-making process.
5.6 Statistical Procedures for
Comparing Real-world Observations
and Simulation Output Data
• Inspection approach.
• Confidence-interval approach based on
independent data.
• Time-series approach
5.6.1 Inspection Approach
• Statistics: sample mean, sample variance, the
sample correlation function, histograms.
• dangerous! for sample size 1.
• Correlated inspection approach
Table 5.4
Results for three experiments with the inspection approach
Experiment
ˆ X
ˆ Y
1
0.90
0.70
0.20
2
0.70
0.71
-0.01
3
1.08
0.35
0.73
ˆ X ˆ Y
Historical system
input data
Historical system
input data
Actual system
Simulation model
System output data
Compare
Model output data
Figure 5.2
The correlated inspection approach
Table 5.5
Results for the first 10 of 500 experiments with the correlated and basic
Inspection approaches, and a summary for all 500
Experiment j
X j Yj
X j Yj
2.62
-0.75
0.44
3.37
2.05
-0.58
0.74
2.21
2.61
4.56
-0.40
-2.35
4
2.54
3.59
1.86
-1.05
0.68
5
9.27
11.02
2.41
-1.75
6.86
6
3.09
3.75
1.85
-0.66
1.24
7
2.50
2.84
1.13
-0.34
1.37
8
0.31
0.71
3.12
-0.40
-2.81
9
3.17
3.94
5.09
-0.77
-1.92
10
0.98
1.18
1.25
-0.20
-0.27
Sample mean
of all 500
2.10
2.85
2.70
-0.75
-0.60
Sample variance
of all 500
2.02
2.28
2.12
0.08
4.08
Xj
Yj
Yj
1
3.06
3.81
2
2.79
3
5.6.2 Confidence-Interval Approach
based on Independent Data
• Condition: it is possible to collect a potentially
large amount of data for both the model and
the system.