Transcript Code Hunt: Experience with Coding Contests at Scale Judith Bishop, R Nigel Horspool, Tao Xie, Nikolai Tillmann, Jonathan de Halleux Microsoft Research, University of.

Code Hunt: Experience with
Coding Contests at Scale
Judith Bishop, R Nigel Horspool, Tao Xie,
Nikolai Tillmann, Jonathan de Halleux
Microsoft Research, University of Victoria and UIUC
Working and learning for fun
Enjoyment adds to long term retention on a task
Discovery is a powerful driver, contrasting with direct instructions
Gaming joins these two, and is hugely popular
Can we add these elements to coding?
Code Hunt can!
www.codehunt.com
Websites
Game
Project
Community
Data Release
Blogs
www.codehunt.com
research.microsoft.com/codehunt
research.microsoft.com/codehuntcommunity
github.com/microsoft/code-hunt
Linked on the Project page
The Code Hunt Game
Code Hunt
• Is a serious programming game
• Works in C# and Java (Python coming)
• Appeals to coders wishing to hone their programming skills
• And also to students learning to code
• Code Hunt has had over 300,000 users since launching in March 2014
with around 1,000 users a day
• Stickiness (loyalty) is very high
The Story
Gameplay
code
test cases
1. User writes code in browser
2. Cloud analyzes code – test cases show differences
As long as there are differences: User must adapt code, repeat
When they are no more differences: User wins level!
It’s a game!
1.
2.
3.
4.
5.
iterative gameplay
adaptive
personalized
no cheating
clear winning criterion
Score is based on
• how many puzzles solved,
• how well solved, and
• when solved
code
test cases
Dynamic Symbolic Execution
Choose next path
Solve
void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
F
F
a.Length>0
a==null
T
T
Constraints to solve
Input
Observed constraints
a!=null
null
{}
a==null
a!=null &&
!(a.Length>0)
a==null &&
a.Length>0 &&
a[0]!=1234567890
a==null &&
a.Length>0 &&
a[0]==1234567890
a!=null &&
a.Length>0
{0}
a!=null &&
a.Length>0 &&
a[0]==123456890
{123…}
Done: There is no path left.
a[0]==123…
F
Execute&Monitor
T
Powered by a IntelliTest in Visual Studio 2015
Available in Visual
Studio since 2010
(as Pex and Smart Unit
Tests)
Nikolai Tillmann, Jonathan de Halleux, Tao Xie:
Transferring an automated test generation tool to
practice: from pex to fakes and code digger. ASE
2014: 385-396
Code Hunt Contests
Code Hunt Contests
The game is fun, fresh and different.
Code Hunt Properties
Code Hunt Contests
Browser-based
Have world-wide reach on all platforms
Automatic grading
Are cost effective
Cloud based hosting
Scale to 100,000s players
Clear scoring criteria
Determine winners precisely
Results available
Can identify top coders
The ah-ha moment
Code Hunt can identify
top coders
30 Contests in the past year
What the player sees
Dashboard and Leaderboard
Visible only to the organizer
Publically visible when the contest ends
Puzzles and Analytics
Number:
Group:
Description:
Puzzles
• About 400 puzzles
• Classified and rated by difficulty
• Puzzle Bank is highly protected
Puzzle Group
Description
Source:
Difficulty:
P067
Arrays
Remove duplicates
from an array
NH
2
Difficult Calculat
Source
y
ed
Players
P249-1 numbers Compute 23*X-42
NH
2
1
335
P277
string
Convert N to base 4 string
NH
2
3
332
P115
array
Convert array of scores to letter grades
APCS
3
2
329
P002
array
Count occurrences of a string in an array
APCS
1
2
325
NH
1
2
325
P260-1 bools
Compute x && (y || !z)
Puzzle Bank Statistics
Modify the Calculated rating based on user experience.
D = a + tries * b + tries * c * distance
where a, b and c are constants with values
a=1
b = 0.02
c = 0.05
tries is the average attempts for all players who solved the puzzle
distance is the number of levels solved so far.
Tries does depend on
• Mix of students who enter
• The internet speed
• Whether students use an IDE in addition
Laser, Sept Original
2014
Difficulty
24
1.1
1.2
1.3
1.4
1.5
1.6
2.1
2.2
2.3
2.4
2.5
2.6
2.21
2
2
3
2
2
1
3
3
2
2
1
2
Average
Total Try
Calculated
Count
Difficulty s
4.60
1.79
1.42
2.42
2.43
4.05
3.29
4.29
1.87
1.84
9.05
3.10
18.10
Average Try Max Try
Count
Count
106
56
158
130
211
122
116
23
20
70
42
190
8.83
4.67
15.80
13.00
23.44
15.25
19.33
4.60
4.00
35.00
8.40
63.33
Total Solved
Users
27
13
45
31
73
52
43
7
8
68
27
138
12
12
10
10
9
8
6
5
5
2
5
3
How players perceive difficulty
Beauty of Programming
Qualification
Preliminary A
Preliminary B
Semi-Final
CSTA and TEALS
Subjective Perceived
difficulty
difficulty
1.59
2.72
2.17
1.84
2.50
1.84
2.60
2.22
Players who
started
1511
251
247
140
Subjective
difficulty
Players who
scored
Calculated
difficulty
Students
1.96
5.22
61
Teachers
1.96
4.38
14
Code Hunt - the APCS (default) Zone
• Opened in March 2014
• 129 problems covering the Advanced Placement Computer Science course
• By August 2014, over 45,000 users started.
APCS Zone, First three sectors, 45K to 1K
50000
45000
40000
Players
35000
30000
25000
20000
15000
10000
5000
0
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
1.9
1.10 1.11 1.12 1.13 1.14 1.15
2.1
2.2
Sector and Level
2.3
2.4
2.5
2.6
2.7
3.1
3.2
3.3
3.4
3.5
3.6
3.7
3.8
Effect of difficulty on drop off in sectors 1-3
Yellow – Division
Blue – Operators
Green - Sectors
Puzzle
Aug 2014 and Feb 2015
Effect of Puzzle Difficulty on Drop off
60
50
Percentage Drop Off
40
30
Level Aug Feb-A
Compute -X
1.1
17
22
Compute 4 / X
1.6
18
21
Compute X-Y
1.7
18
22
Compute X/Y
1.11
32
38
Compute X%3+1
1.13
15
18
Compute 10%X
1.14
12
16
Construct a list of numbers 0..N-1 2.1
37
48
Construct a list of multiples of N
2.2
19
23
Compute x^y
3.1
11
18
Compute X! the factorial of X
3.2
16
19
Compute sum of i*(i+1)/2
3.5
17
22
20
10
0
1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 1.10 1.11 1.12 1.13 1.14 1.15 2.1 2.2 2.3 2.4 2.5 2.6 2.7 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8
-10
Aug
Feb-A
One week 3-10 October, 2014
One year May 2014 – April 2015
Survey results – before and after contest season
How much did the puzzle aspect of Code Hunt keep you interested in reaching a solution?
Till Sept 2014
Sept 2014 – May 2015
Survey Results cont.
In your opinion, were your final solutions well-structured code?
Till Sept 2014
Sept 2014 – May 2015
Gender and Code Hunt
Female
Male
Respondents
12%
80%
Played as practice for
a course in C#
11%
18%
Played as practice for
a course in Java
40%
23%
Played for own
enjoyment
47%
57%
Open survey
with 850
respondents
Beyond contests
Towards a Course Experience
Public Data release in open source
For experimentation on how people program and reach solutions
Total Try Average Try
Count
Count
13374
363
Max Try
Count
1306
Total
Solved
Users
1581
For ImCupSept
257 users x 24 puzzles x approx. 10 tries =
about 13,000 programs
Github.com/microsoft/code-hunt
Upcoming events
PLOOC 2015 at PLDI 2015, June 14
2015, Portland, OR, USA
CHESE 2015 at ISSTA 2015, July 14,
2015, Baltimore, MD, USA
Worldwide intern and summer school
contests
Public Code Hunt Contests are over for
the summer
Special ICSE attendees Contest. Register
at aka.ms/ICSE2015
Code Hunt Workshop February 2015
Summary:
Code Hunt: A Game for Coding
1.
2.
3.
4.
Powerful and versatile platform for coding as a game
Unique in working from unit tests not specifications
Contest experience fun and robust
Large contest numbers with public data sets from cloud data
• Enables testing of hypotheses and making conclusions about how players are
mastering coding, and what holds them up
5. Has potential to be a teaching platform
• collaborators needed
Websites
Game
Project
Community
Data Release
Blogs
Office Mix
ICSE Contest
www.codehunt.com
research.microsoft.com/codehunt
research.microsoft.com/codehuntcommunity
github.com/microsoft/code-hunt
Linked on the Project page
mix.office.com
aka.ms/icse2015 to register
Aka.ms/icse2015codehunt to play
Grazie!
Appendix
Results and rewards
Detailed results are presented
Ranking is always based on score, not on time or attempts
Score is based on
• how many puzzles solved,
• how well solved, and
• when solved
Special rewards for contests
Comparing Code Hunt
Code Hunt
IPCC
A serious online coding game
Annual programming contest
Built on symbolic execution
Scored by unit testing
Addressing various audiences
Addressing university students
Data available in the cloud
Questions available on the web
BoP China 2014