Lecture 2B Experimental Methods for Business Strategy

In this session you will design a game on your own laptop and have your colleagues log on as subjects. I will then provide some guidance on how to analyze experimental data.

Designing an experiment that uses the extensive or strategic form

Open a browser and visit: http://www.comlabgames.com/ Click “Old discrete and Strategic Form Module”: http://www.comlabgames.com/tree/index.html

Click “Edit a Tree” to design a game in extensive form or “Edit a Matrix” to design a game in strategic form.

The rudiments of constructing a simultaneous move game

The mechanics of designing your own two player simultaneous move game are easy: 1.

2.

3.

4.

5.

6.

7.

Determine the dimensions of the matrix.

Enumerate the strategies.

Define the payoffs in the cells.

Name the players.

Give your game a title.

Undo your work and revise your game.

Save your game in a directory.

The rudiments of constructing an extensive form

The mechanics of designing your own extensive form game are almost as easy: 1.

2.

3.

4.

5.

6.

Draw the moves of the players and nature.

Label the moves and define the probabilities.

Name the players and define the payoffs.

Draw the information sets.

Undo your work and revise your game.

Save your game in a directory

Conducting an Experiment

Disable all firewalls on your laptop. Otherwise your experimental subjects will be prevented from participating by the firewall. Use a wired connection to the internet to avoid congestion. If you use a wireless connection, your subjects may be disconnected while waiting to join your game.

Open your game in the Comlabgames module and provide your subjects with your internet IP address and port number.

Analyzing the data

The experimental results are automatically saved in the same directory as your game, and can be opened as an HTML file or in Excel. We now review methods for analyzing categorical data from finite games played in the extensive and strategic forms.

We discuss measures for evaluating performance, ways of summarizing the data, and statistical methods for forming confidence intervals and testing hypotheses. Chapter 2 of Strategic Play provides a more detailed analysis.

Learning strategic behavior using experimental methods

Applying experimental methods is a self-contained training tool for learning strategic behavior: 1. Design a variety of games that capture parts of the strategic issue you are trying to understand.

2. Conduct experiments with human subjects in the area using small monetary stakes as motivation. Your subjects do not need to have any training in game theory or experimental methods.

3. Analyze the results from the experiments seeking behavioral patterns that might apply in real life.

Four advantages of experimental methods over theory

One way of formulating strategy is to find empirical behavioral patterns that emerge from repeatedly conducting experiments: 1. The game might be too complex to solve.

2. The game might have multiple solutions .

3. Experimental subjects and also managers sometimes make irrational decisions.

4. Managers learn more from experience than theory, so managers in training might learn more quickly by artificially recreating strategic situations rather than theorizing about them.

Empirically optimal strategies for strategic form games

Given the behavior of the subject population, what is optimal play?

To answer this question for the row player in a strategic form game, we weight each cell payoff by the relative frequency its column was visited by the column player, and form the average payoff the row player would have received from playing any given strategy.

The best reply to the empirical distribution generated by the column players maximizes the average payoff calculated in this fashion.

Empirically optimal strategies for extensive form games

In the extensive form game, a similar approach is used to evaluate a any given move that taken be taken from a designated information set.

First we compute the expected payoff from making a particular move from a given node, weighting the payoffs with their relative frequency of occurrence conditional on making that move.

Then we calculate the relative frequency of arriving at any node belonging to the same information set.

In this way we form the empirical expected payoff from making any given move from any given information set.

Two uses for the data

The results from the experiment can be interpreted as a comment on the rationality of the subjects, and also the usefulness of the theory. Data from experiments can be used to: 1. Evaluate the performance of subjects, and thus link their incentives to play the game with the payoffs that face them in the game.

2. Investigate whether subjects follow the predictions of theory, whether the empirically optimally strategies match the predictions, and whether different characteristics of subjects are significant.

An example

You may recall playing this game in the first lecture:

Expected value maximization

This is an example of a game with perfect information. In such games each player sees exactly how the game progresses to her decision node. There are no dotted lines connecting decision nodes.

Perfect information games can be readily solved if the players maximize their expected value , and there are not too many nodes.

Solution

1.

2.

3.

Expected value maximizers use the principle of backwards induction to solve the problem: At node 4, NATURE selects node 6 with probability 0.5 and node 7 with probability 0.5. On reaching that node, the expected value for INNOVATOR is 5 and the expected value for VENTURE CAPITALIST is 6. Anticipating this, the VENTURE CAPITALIST would fund project at node 2 because 6 exceeds 5. At node 1, INNOVATOR will request funding because 5 is more than 2. So in the solution to this game the INNOVATOR requests funding and the VENTURE CAPITALIST will fund the project.

Conduct of the experiment

23 subjects from the 2003 undergraduate economics class participated in the experiment.

No knowledge of game theory was explained to the subjects before the experiment. The solution of the game was not explained. The subjects were randomly assigned player roles upon logging on the game, and pairs were anonymously matched.

Subjects were told that they should play the game at least once, and were permitted to play more than once.

13 subjects played the game once. The remaining 10 played it twice or three times.

Criteria for rewards and grading

In this experiment subjects were told there were no rewards from playing the game well.

An alternative scheme is to sum the points each player gets in total and pay them at the rate of a dollar a point.

Or we get reward subjects by playing the game correctly. For example we could award each subject one point every time the game in which he is participating ends in the terminal node that is solution to the game, and zero otherwise, and sum the points for each subject and divide through by the number of games played.

List of subjects

Username

mfi rcol rfe ro sal ren vic ved adi ais ali ant bhu che col eug gig jac jar jmk kev lex lin

Role assigned in the experiment

Venture capitalist Innovator, Venture capitalist Innovator twice Innovator Venture capitalist Innovator Innovator Innovator three times Venture capitalist Innovator, Venture capitalist twice Venture capitalist Innovator twice Innovator Venture capitalist three times Venture capitalist Innovator twice Venture capitalist three times Venture capitalist Innovator Innovator, Venture capitalist twice Innovator twice Venture capitalist three times Innovator

Gender

Male Male Female Male Male Female Male Female Female Male Male Male Male Male Female Female Male Male unknown male male female male

Year

Junior Junior unknown Senior Junior unknown Senior unknown Senior Junior Senior Junior Senior Senior Junior Senior Senior Senior unknown unknown Senior Junior Junior

School

CIT CIT unknown CIT CIT unknown CIT unknown HSS HSS HSS CIT SCS HSS HSS HSS SIT CMU unknown unknown HSS HSS HSS

Test results

How would subjects have fared under this alternative scheme for awarding points?

Username

kev lin mfi rco Rfe ro sal vic che eug lex ren ved adi ais ali ant bhu col gig jac jar jmk

Score

1.00 0.33 0.67 1 0 0 0.5 1 1 0 1 1 0.33 1 1 1 1 0.5 0.67 1 0 0.67 0.67

Number of games played

1 3 1 1 3 1 2 3 2 1 1 2 2 1 1 1 1 2 3 1 1 3 3

Further notes on assessment

Should we let the grade a particular subject receives be affected by other subjects’ behavior or chance?

Under the alternative grading scheme in this example the VENTURE CAPITALIST is automatically punished if the INNOVATOR makes a mistake.

In the alternative scheme, it is implicitly assumed that both players are net present value maximizers, although very risk averse subjects might rationally choose to pass on the project. If we do not make points proportional to the payoffs in the game, the game is being modified.

Trials and outcomes

We now turn to the second use of the data, for analyzing the game and its solution.

We could treat each game played by a pair of subjects as a trial, and the full history of play as an outcome.

Alternatively we could treat each time a subject plays one game as a trial. Then his moves would be the trial outcome. In both cases the number of trials is the size of the sample.

Analyzing behavior

The sample of trials and their associated outcomes is used as evidence to inform us about the underlying population pool from which the sample is drawn.

We compare the predictions of the theory to the sample behavior observed, to see whether the theory applies to the underlying population.

Similarly the behavior of different sub-samples are compared with each other, to see whether different types of subjects behave the same way or not.

Summary statistics

As a first cut a histogram shows the predicted and observed outcomes 22 20 18 16 14 12 10 8 6 4 2 0 3 5

Node number

Predicted Observed 6 and 7

Other graphics

Bar graphs, pie charts and Venn diagrams are also useful ways of graphically depicting the data.

An advantage of the pie chart is that it automatically incorporates the normalization that the proportions implied by a partitioning must sum to one. If one of K outcomes occurs each trial, their relative frequencies are easy to read off a pie chart.

A Venn diagram is quite useful in showing sets of outcomes that have nonempty intersections. For example we could illustrate the number of times the INNOVATOR maximized expected value, the number both players did, and the remaining times, when the INNOVATOR did not maximize expected value.

Statistical inference

We might consider each outcome of a trial as a random draw from a probability distribution.

The characteristics of the sample then provide us with information to estimate the parameters describing the distribution, and to test hypotheses of interests to us.

In our example, we define each trial as a move by a subject and ask what is the probability that subjects maximize expected value.

Estimating the mean of a Bernoulli random variable

Criteria for getting the right move Number of trials n Number of successes x Mean

q



x n

Unconditional probability Conditional on playing the first time Conditional on being an innovator Conditional on being junior Conditional on being female Conditional on being from HSS Conditional on being from CIT If you are a female what is the probability of playing an innovator role (test of random assignment) 40 26 20 14 13 17 8 13 26 20 13 7 9 11 5 8 0.65 0.77 0.65 0.50 0.69 0.64 0.63 0.62

Standard Deviation



q



q

( 1 

q

)

n

0.075 0.083 0.107 0.134 0.128 0.116 0.171 0.135

A 95% confidence interval for p

q

( 1 

q

)

q

 1 .

96

n

0.50

 q  0.80

0.61  q  0.93

0.44  q  0.86

0.24  q  0.76

0.44  q  0.94

0.41  q  0.87

0.30  q  0.96

0.36  q  0.88

Are there gender differences?

The numbers in brackets predict the number who would have ended up on a terminal node if both genders behaved exactly the same way.

For example, considering females who played in a game ending on node 5, note that 3 = 9 divided by 39 times 13.

Are the corresponding numbers in brackets statistically different from the actual outcomes?

Terminal node

3 5

2 (1.33) 2 (3)

6 or 7

9 (8.7)

Total

13

Female Male Total

2 (2.67) 4 7 (6) 9 17 (17.33) 26 26 39

Estimated expected cell frequency for testing gender differences

Are juniors and seniors different?

Junior Senior Total

Terminal node

3 5

2 (0.93) 5 (3.73)

6 or 7

7 (9.3) 0 (1.07) 2 3 (4.26) 8 13 (10.67) 20

Total

14 16 30 The test statistic is 4.18, which is less than the Chi square critical value, for an 0.5 test with 2 degrees of freedom, of 5.99. We cannot reject the null hypothesis that juniors and seniors are the same.

Lecture Summary

We designed games in the extensive and strategic forms.

We explained how to conduct experiments for analyzing strategic behavior with invited subjects who have internet access.

We showed how to analyze categorical data generated by experimental sessions.

Finally, we discussed the merits of using experimental methods to learn strategy.