Transcript Metrics in MMP Development and Operations

Metrics for
MMP Development and Operations
Lessons Learned: The Sims Online
Larry Mellon
GDC, Spring 2004
Metrics: Catch-22
Useful
Careful…
Hard Data
GI / GO
Optimization Tool
Expensive
Importance of Metrics is Relative
Lord Kelvin
Measure
Everything
Mark Twain
Measure
“Just Enough”
Invest No More Than You Need
Kelvin
$$$
Twain
MMP
Complexity
Scale
Precision
$
MMP: Measurement Focal Points
Operational Costs
Infrastructure
Player Actions
Economy
Similar Use Case: Casinos
(Harrah’s “Total Reward”)
Unified Player Action DB
Casino
Casino
Casino
Table
Table
Machine
Table///Machine
Machine
Track every Player Action
Player
Player
Player
Highly Profitable, Highly Popular
Unified Player Action DB
Analyze:
Profit (per Casino, per Player)
Patterns of Play
Modify:
Casino Operations
Player Awards Program
“This is one of the best investments that we have ever made as a
corporation and will prove to forge key new business strategies and
opportunities in the future.“ John Boushy (Harrah's CIO, 2000)
TSO: Live Monitors, Summary Views
Designers
Community
Managers
Engineers
Player Actions & Persistent Data
Operations
Server Reactions
Metrics System
Embedded Profiler (Server Side)
Automated Report Generators
Outline
Background: Metrics & MMP
Implementation Overview
Metrics in TSO
Applications & Sample Charts
Wrapup
Lessons Learned
Conclusions
Questions
Implementation:
Driving Requirements
Low overhead
Common Infrastructure
Ease of use
Esper Architecture
User
User
User
Live Server CPUs
Process
Process
Process
esperProbe
esperProbe
esperProbe
esperLog
esperLog
esperLog
Event-level Sampling,
Aggregated Reporting
esperProbes
esperView
esperView
esperView
esperStore
t1
t2
t3
Min, Max, Av, Count
esperLog
esperLog
esperLog
t4
t5
DBImporter
esperFetch
Esper Probes
• Self-organizing: “class” hierarchy
• Data driven: new probes and/or new game
content immediately visible on web
• Example: ESPER_PROBE
– (“Object.interaction.%s”, chair->picked)
– (“Object.interaction.puppet.%s”, self->picked)
• Human-readable intermediate files
EsperView: Web-Driven Presentation
Daily Reports
Report Generator
Graph Caching & Archiving
Filtering & Meta Data
EsperView:
Hierarchical Presentation
Process-Level Collection
Server Cluster
Process Class
Process Instance
Outline
Background: Metrics & MMP
Implementation Overview
Metrics in TSO
Applications & Sample Charts
Wrapup
Lessons Learned
Conclusions
Questions
Applications of Metrics
Load Testing
(Realistic Inputs)
Beta Testing & Live Operations
(Game Tuning, Community Management)
Load Testing & Live Operations
(Server Performance)
Load Testing:
“Monkey See / Monkey Do”
Sim Actions (Player Controlled)
Sim Actions (Script Controlled)
Live Beta Testers
AlphaVille Servers
Test Servers
Applications of Metrics
Load Testing:
Realistic Inputs
Beta Testing & Live Operations:
Tuning/Mngmnt
Load Testing & Live Operations:
Server Performance
History
Make Friends, Shake
Hands
beats out
Give Money / Get Money
Least Used: Disco Dancing
Meta
Data
Top: TD Dance, Woohoo
Bottom: Dance
Players per Lot
0 to 70 players: < 2 / lot
70 to 400 players: > 3 /
lot
Top: Metrics Bug (sorta)
Next: Garden Gnome, Toilet
Bottom: Buffet Table
Beta: numPlayers by numRMates
1. Most have one
Roommate
2. Hard-core players: 8
Highest
1. Using Skill Objects
2. Using Pizza Maker
3. Selling Objects
Lowest
1. Visitor Bonus
2. New Avatar
3. Object “Rentals”
Economy: Detailed View
1. Using Objects = $$$
2. Visitor Bonus == $
Visitor Bonus: Who Makes Money?
1. Most getting no V. Bonus
2. Hard-core players: $$
4 of top 5:
windows?
?
House Categories (Beta Test)
1. Not a well-used feature
2. Shopping least of all
Community Management
Community Actions & Trends
Influencing Player Activity
Free Content
Tracking Problem Players
Marketing
In-Game Brand Exposure
Special Events
Press Release Teasers
NYEve: Kiss Count
Esper Cities
All Cities (extrapolated)
===========================================
New Year's Kiss
32,560
271,333
Be Kissed Hotly
7,674
63,950
Be Kissed
5,658
47,150
Be Kissed Sweetly 2,967
24,725
Blow a Kiss
1,639
13,658
Be Kissed Hello
1,161
9,675
Have Hand Kissed
415
3,458
===========================================
Total
52,074
433,949
Applications of Metrics
Load Testing:
Realistic Inputs
Beta Testing & Live Operations:
Tuning/Mngmnt
Load Testing & Live Operations:
Server Performance
DB byte count
oscillates out of
control
A single DB
Request is clearly
at fault
Most-Used DB Queries (unfiltered)
11,000,000 level
Queries need
attention, and
drown out others
DB Queries (Filtered)
Filters on 11,000,000
level Queries show
patterns of 7,000
level Queries
Incoming & Outgoing Packets
Outline
Background: Metrics & MMP
Implementation Overview
Metrics in TSO
Applications & Sample Charts
Wrapup
Lessons Learned
Conclusions
Questions
Lessons Learned
•
•
•
•
•
Implement early
Ownership, senior engineers
Aggregated probes vs event-level tracking
Automation: collect / summarize / alarms
“There can be only one”…
Conclusion: Very Useful!
Game Design
Realistic Load Testing
Engine Fixes, Optimization
Data Mining
On Players:
Untapped Gold…
Server Cost, Launch Timing
Critical Feature: Accessibility
Questions
Slides available
@
www.maggotranch.com/MMP