Transcript APPENDIX B. SOME BASIC TESTS IN STATISTICS

Slides for Introduction to Stochastic Search
and Optimization (ISSO) by J. C. Spall
APPENDIX D
RANDOM NUMBER GENERATION
• Organization of chapter in ISSO*
– General description and linear congruential generators
• Criteria for “good” random number generator
– Random variates with general distribution
• Different types of random number generators
*Note: These slides cover some topics not included in ISSO
Uniform Random Number Generators
• Want a sequence of independent, identically
distributed U(0, 1) random variables
• However, random number generators (RNGs)
produce a deterministic and periodic sequence of
numbers
• What qualities should the generators have?
D-2
Criteria for ‘Good’ Random Number
Generators
•
•
•
•
•
Long period
Good distribution of the points (low discrepancy)
Able to pass some statistical tests
Speed/efficiency
Portability – can be implemented easily using
different languages and computers
• Repeatability – should be able to generate the same
sequence over again
D-3
Generating Random Numbers
• Given a transition function, f, the state at step n is
given by
xn  f ( xn1), n  1
• The output function, g, produces the outputs as
u n  g  xn 
• The output sequence is un , n  1
• Want the sequence period to be close to 2b, where b
corresponds to the number of bits
D-4
Types of Random Number Generators
• Linear – most commonly used
• Combined – can increase period and improve
statistical properties
• Non-linear – structure is less regular than linear
generators but more difficult to implement
D-5
Linear Congruential Generators
• U(0,1) numbers via linear congruential generators
(LCG) are calculated by
xn   axn 1  c  mod m
un  xn / m
• These are the most widely used and studied random
number generators
• The values a, c, and m should be carefully chosen
0  m, 0  a  m, 0  c  m
x0  m, xk  0,1, , m  1
D-6
Linear Congruential Generators
• Some values for a and m (assuming c = 0)
– a = 23, m = 108+1 (original implementation)
– a = 65534, m = 229 (poor because of high order correlations)
– a = 515, m = 247 (long period, good distribution, but lower
order bits should not be trusted)
– a = 16807, m = 231 –1 (this has been discussed as the
minimum standard for RNGs)
D-7
0.58
Empirical Mean
0.56
0.54
m = 231 – 1, a = 4, c = 1
0.52
0.5
m = 482, a = 13, c = 14
0.48
m = 27, a = 26, c = 5
0.46
0.44
m = 9, a = 4, c = 1
0.42
0
500
1000
1500
2000
Number of Samples
D-8
Lattice Structure (Exercise D.2)
30 points
96 points
1
1
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0
0
0.2
0.4
0.6
Uk
0.8
1
0
0
0.2
0.4
0.6
0.8
Uk
D-9
1
Fibonacci Generators
• These are generators where the current value is the
sum (or difference, or XOR) or the two preceding
elements
• Lagged Fibonacci generators use two numbers
earlier in the sequence
xn   xn  p  xn r  mod m
un  xn / m
p, q are the lags
D-10
Multiple Recursive Generators
• Multiple recursive generators (MRGs)are defined by
xn   a1xn 1   ak xn k  mod m
un  xn / m
where the ai belong to {0,1,…,m – 1} and
• For prime m and properly chosen ai’s, the maximal
period is mk-1
sn   xn k 1, , xn 
D-11
Combining Generators
• Used to increase period length and improve statistical
properties
• Shuffling: uses the second generator to choose a
random order for the numbers produced by the final
generator
• Bit mixing: combines the numbers in the two
sequences using some logical or arithmetic operation
(addition and subtraction are preferred)
D-12
Nonlinear Generators
• Nonlinearity can be introduced by using a linear
transition function with a nonlinear output function
• An example is the explicit inversive generator where
xn  an  c
zn   an  c 
m 2
mod m
un  zn / m
D-13
Random Number Generators Used in
Common Software Packages
• Important to understand the types of generators used
in statistical software packages and their limitations
• MATLAB:
– Versions earlier than 5: a linear congruential generator with
a  7  16807; c  0; m  2  1  2147483647
5
31
– Versions 5 & 6: a lagged Fibonacci generator combined with
1492
a shift register random integer generator with period ~ 2
• EXCEL: un = fractional part (9821×un –1 + 0.211327);
period ~ 223
• SAS (v6): LCG with period ~ 231  1
D-14
Inverse-Transform Method for Generating
Non-U(0,1) Random Numbers
• Let F(x) be the distribution function of X
• Define the inverse function of F by
F 1( y )  inf x : F( x)  y ,0  y  1.
• Generate X by
X  F 1(U )
• Example: exponential distribution
F ( x )  1  e x
1
X  F 1(U )   ln(1  U )

D-15
AcceptReject Method
• Let pX(x) be the density function of X
• Find a function f(x) that majorizes pX(x)
– f( x )  ch( x ), c  1, q is a density function
• Generate X by
– Generate U from U(0,1) (*)
– Generate Y from q(y), independent of U
pX (Y )
– If U 
, then set X=Y. Otherwise, go back to (*)
f (Y )
• Probability of acceptance (efficiency) = 1/c
• Related to Markov chain Monte Carlo (MCMC)
method (see Exercise 16.4)
D-16
60 x 3 (1  x )2
pX ( x )  
0
Y ~ q( y )  U (0,1)
if 0  x  1
60Y 3 (1  Y )2
U
2.0736
otherwise
2.5
f( x )  cq( x )  2.0736  U(0,1)
2.0
pX(x)
1.5
1.0
q(x) = U(0,1)
0.5
0
0.2
0.4
0.6
0.8
1.0
1.2
D-17
U ~ U(0,1): 0.9501, 0.2311, 0.6068, 0.4860, 0.8913, 
Y ~ q(y)  U(0,1): 0.7621, 0.4565, 0.0185, 0.8214, 0.4447, 
pX (Y )
: 0.7249, 0.8131, 
cq(Y )
X ~ PX(x): 0.7621, 0.4565, 
reject
accept
D-18
References for Further Study
• L’Ecuyer, P. (1998), “Random Number Generation,”in
Handbook of Simulation: Principles, Methodology,
Advances, Applications, and Practice (J. Banks, ed.),
Wiley, New York, Chapter 4.
• Neiderreiter, H. (1992), Random Number Generation
and Quasi-Monte Carlo Methods, SIAM, Philadelphia.
D-19