Transcript Document 7140591

Issues and Implementations
in Multiplayer Game
Development
Online game development is difficult and riddled with expensive
risks. Whether you are adding multiplayer support to a singleplayer game or fielding a massively multiplayer persistent virtual
world, the additional complexity of connecting your players over a
network adds substantial development time and testing costs. Get it
right, and online can bring innovative game play and a sense of
community to your player base. Get it wrong, and going online
brings you a Pandora’s box of missed deadlines, frustrated
developers, and unhappy customers…
Introduction
Why Online?
Challenges
Playing with Friends
Exciting Competition
Community
Community
Game Play
Technology
Limited Media
Non-linear time
Persistence
Fairness
Style
Our Focus
Network Distortion
Unreliability / Non-Determinism
Multi-Player Inputs
Multi-Process Execution
System Complexity
Tutorial Takeaway Messages



Building online games with single-player
game techniques is painful
Early changes to your development process &
system architecture to accommodate online
play will greatly ease the pain
“Lessons Learned” from our background will
provide your project with useful “Rules of
Thumb”
“A wise man learns from his own mistakes, a
wiser man learns from someone else’s”
Network Terminology
Bandwidth
A measure of the width or capacity of a communications
channel. Greater bandwidth allows communication of more
information in a given period of time.
Important: Bandwidth refers only to the amount of data a
channel can handle. As an analogy, a dump truck has a
higher bandwidth than a sports car. This does not mean that
the dump truck is faster, just more efficient at moving large
amounts of material
Source: Ubisoft Online Terminology Guide, used with
permission
Cheat
An exception to a game’s normal rule set that is included
by the game’s developers. Usually, cheats are enabled
only in the single-player mode of the game (if it has one) or
were included for use (and forgotten about) by the
developers to aid with testing and/or the media.
Client/Server
This is any networking architecture in which one machine
(the server) is responsible for making final determinations
on what occurs in the game. This can range from game
designs in which the client merely displays information to
the user and returns user inputs to the server, to game
designs in which all machines are running nearly complete
versions of the game, and the server only serves to centralize
communications and adjudicate any disagreements.
Important: This is an architectural distinction. It is entirely
possible that the server is a consumer operated machine.
Source: Ubisoft Online Terminology Guide, used with
permission
Consumer-Hosted
A game in which all of the machines involved are run by the
consumer.
Source: Ubisoft Online Terminology Guide, used with
permission
Dedicated Server
A client-server game in which there is a machine which is
serving as a server without having any local player. This
may be hosted by a consumer, or by a game service.
Source: Ubisoft Online Terminology Guide, used with
permission
Dumb-Client
A client-server game design in which the client is largely a
dumb input/output device.
Source: Ubisoft Online Terminology Guide, used with
permission
Exploit
An unexpected or unintended game result generated by the
actions of the player that modifies the course or outcome the
game in a way not intended by the game’s designers. Usually
the result of players discovering a bug in the game.
Hack
A modification made to a game or its data by an end user.
This can take the form of modifying either code or data, in
memory while the game is running, and/or to the game’s
files on disk. The game itself or a service that it uses (such
as a driver), or the data it shares with other computers, or
hardware can be modified.
Hybrid Game
Mix between Client-Server and Peer-to-Peer, where a
server determines the outcome for some events, while each
player’s computer determine the outcome of some events
(usually local to that player).
Firewall
A security product that employs a combination of hardware
and software to prevent unauthorized users or traffic from
the Internet from gaining access to a private local area
network. Many problems in online gaming arise because
firewalls block communication between games.
Source: Ubisoft Online Terminology Guide, used with
permission
Integrated Server
A client-server game in which the machine which has a user
playing the game is also serving as the server. On some PC
applications, even though there may be a server running on
the same machine as a player, they are separate
applications.
Source: Ubisoft Online Terminology Guide, used with
permission
IP Address
This is the identifying number for any machine on the
Internet or for any LAN using the IP protocol. The original,
and most widely used form, is IPv4. Under IPv4, each
address is made up of four octets. If expressed as text, it
will take the form of "www.xxx.yyy.zzz" (sometimes called
a dotted IP string). The broader Internet is slowly switching
over to IPv6 to increase the number of available addresses.
As the name implies, IPv6 addresses are made up of six
octets.
Source: Ubisoft Online Terminology Guide, used with
permission
IP Address
Additionally, under IPv4, the following rules apply to
network addresses: The local machine is always reachable
by the "loopback" address of 127.0.0.1, and the following
ranges of addresses are reserved for use on local LANs:
10.0.0.0 - 10.255.255.255
172.16.0.0 - 172.31.255.255
192.168.0.0 - 192.168.255.255
Source: Ubisoft Online Terminology Guide, used with
permission
Lag
A slowing of the execution of a game. Lag can be caused by
two main factors: network issues or computer performance.
Network lag cause an online game to miss
information bits necessary for the execution of it. For
instance, the last message indicating the position of another
player can be lost or is late, but the game has to display
something on the screen. Depending on how this situation
has been taken into account by the developers, the display of
the other player will freeze and the computer will wait for the
next message to arrive before continuing the execution of the
game.
Source: Ubisoft Online Terminology Guide, used with
permission
Lag
Performance lag is caused by the overload of the main CPU
(too much information to process at a time) or the graphical
CPU (too many polygons or lightning to process).
Source: Ubisoft Online Terminology Guide, used with
permission
Latency
A measure of how long it takes a single bit to propagate
from one end of a link or channel to the other. Latency is
measured strictly in terms of time.
Source: Ubisoft Online Terminology Guide, used with
permission
MAC Address
Media Access Control Address, a hardware address that has
been embedded into the network interface card by the
manufacturer to uniquely identify each node or point of
connection of a network.
Source: Ubisoft Online Terminology Guide, used with
permission
NAT (Network Address
Translation)
Enables a local area network to use one set of IP addresses
for internal traffic and a second set of IP addresses for
external traffic.
Source: Ubisoft Online Terminology Guide, used with
permission
Octet
An 8 bit value. Because a byte is the smallest addressable
amount of memory on the machine (6 bit, 7 bit, 9 bit, and
16 bit machines have all been on networks, along with the
now standard 8 bit bytes), network engineers like to use
octet to specify that we really mean 8 bits.
Source: Ubisoft Online Terminology Guide, used with
permission
Peer-to-Peer
In this networking architecture, all games are running full
versions of the game engine. Although one machine may be
used to host the setup of the game, once the game is running,
all machines are effectively equal. Disputes may be resolved
algorithmically (i.e. “claims about your own damage are
always valid”) which lends itself to cheating, or via a voting
mechanism.
Important: This is an architecture. It does not mean “any
game where all the machines involved are consumer
owned”.
Source: Ubisoft Online Terminology Guide, used with
permission
Server-Validation
A client-server game design in which the clients are running
relatively complete versions of the simulation, and the
server is validating claims, rather than determining all
results.
Source: Ubisoft Online Terminology Guide, used with
permission
Service Hosted
A game in which one or more of the machines involved is
provided by a game service (often the game’s publisher).
Source: Ubisoft Online Terminology Guide, used with
permission
TCP
Transmission Control Protocol. TCP is one of the main
protocols in TCP/IP networks. Whereas the IP protocol
deals only with packets, TCP enables two hosts to establish
a connection and exchange streams of data. TCP guarantees
delivery of data and also guarantees that packets will be
delivered in the same order in which they were sent.
Source: Ubisoft Online Terminology Guide, used with
permission
UDP
A connectionless protocol that, like TCP, runs on top of IP
networks. Unlike TCP/IP, UDP/IP provides no error
recovery services. UDP/IP packets are CRC checked, but if
there is a CRC failure or part of a UDP/IP packet fails to
arrive, the packet is lost entirely. Additionally, where
TCP/IP is stream based, UDP/IP is datagram based, with all
the data in a UDP/IP send arriving at once, if it arrives at
all. It can be used for broadcasting messages over a local
network to find games, and is used by many games as the
primary game data channel.
Source: Ubisoft Online Terminology Guide, used with
permission
What is a WAN?



Wide Area Network (WAN) is a “network of
networks”
Allows different networks to talk to each other
Size and distance doesn’t matter
LAN Environment




Local Area Network (LAN) shares common
address space
Actually independent of media; usually
Ethernet
Also independent of protocol; usually TCP/IP
Can “broadcast”
WAN Environment





Multiple networks
Can have overlapping address spaces
Very media dependent
Uses edge device to hide the details
Different types based on connection details
WAN Connection Types



Point-to-point – a single leased line between
two sites
Circuit Switching – like a phone call,
connection made on demand
Packet switched – Virtual circuits are created
Point to Point






A leased line from the provider
Always on
Fixed bandwidth
Works like a pipe, whatever goes in at one end
comes out the other
Can set up an extended LAN environment
More expensive
Circuit Switching




Similar to a leased line in that there is a
physical connection between the two sites
Gets set up and torn down so its only on when
needed
Similar to making a phone call
ISDN is an example
Packet Switched





Uses provider infrastructure
No real connection between sites, uses a virtual
circuit
Routes data packets to final destination by whatever
path is available
Two types of circuits – Switched Virtual Circuit
(SVC) and Permanent Virtual Circuit (PVC)
Examples are ATM, Frame Relay, X.25, SMDS
Virtual Circuits


PVC’s are always on – “permanently
established”
SVC’s are brought up and down
Call Setup
 Idle
 In use
 Tear down

Quality of Service




Quality of Service (QOS) requires a queuing
mechanism
Differs between media used
Prioritizes data according to packet
information
Line purchased may or may not support Make sure you buy the right thing!
Edge Device





Any device that hides the complexity of what
you are talking to from your LAN
Does the protocol disassembly and reassembly
Handles signaling
Handles media conversion
Router is an example
ATM






Asynchronous Transfer Mode
Used by Telco’s to handle voice
Breaks everything up into small, fixed size
packets
Very common, cheap
T1’s, T3’s, OC3’s etc.
Need to test for equipment compatibility!
Typical ATM Usage




Local Exchange Carrier (LEC) provides
physical line
Line is located in the carrier’s Point of
Presence (POP) on your facility
You connect your router to your internal
Ethernet LAN
LEC connects line to your carrier
Typical ATM Usage cont.




Your carrier (Sprint, ATT, Global Crossing,
etc.) has a small, fixed number of hops to get
you onto main trunk
Main trunk will get your data to destination
LEC site
LEC to POP to Router to Destination LAN
Magic is in how it knows where to go!
Break until 11:15
11:15am-12:30pm
Moving from Single Player to
Multiplayer
NETWORKED GAMES ARE HARDER THAN
SINGLE PLAYER GAMES
Use Case: Steal Ball Being Dribbled By Another Player
Network Distortion
Player B (New York)
Player A (San Francisco)
?
Ball Position:
State Updates
?
?
Local machine always has an accurate
representation of ball position
Remote machine always has
an approximation of ball position
Issue: Kicking a Shared Soccer Ball





Your opponent has the _exact_ position of the ball
Your computer has only an _approximate_ position of
the ball
When you (the player or the coder) make a decision,
it may be on incorrect data
Level of distortion impacts quality of gameplay
Inaccuracies of data lead to serious coding defects
The more computers involved, the greater the problem
Shared Worldview: Each Node Has A Potentially
Different Representation Of Every Value
Client A (Local Memory)
Clear Views
Game
Logic
Player
Views
True Values
Player
Position
Damage
Levels
Client B,C
Distorted Views
AI
State
Shared View
Network
Latency / Unreliablity
At any slice in time, fuzzy, loosely coupled views
produce an inconsistent shared worldview
Machine A, time t
Machine B , time t
Network
Shared View @ (t) Is
ALWAYS DIFFERENT
B
Network
C
A
Machine C , time t
Network
The loss of precision




Compression saves bandwidth
What to compress – and how much?
Possibly compress then decompress before you
use it in game to minimise divergence
Examples:
Car position – vibration
 Car position – aggressive driving

What is an atomic operation?




Each message becomes an atomic operation
Interframe and intraframe timing is not
preserved across the network
Troublesome case: Multiple frame actions such
as shell transitions
Troublesome case: Multi-step operations in
which the interim cases are invalid at frame
start
Event Ordering And Atomicity Across Transactions
Further Impact GamePlay And Coding Errors
Near-Endless (and illogical)
Ordering Are Possible
Machine A
Machine B
A1,A2,B1,B2
A1, B1,B2,A2
B1,B2,A1,A2
A1,A2,B1,B2
A1,A2
A2,A1,B1,B2
B2,A2,B1,A1
…
Machine C
B1,B2
Indeterminate timing and
sequencing of information


Player A sends a stream of packets describing
game state
Timing:
Player A picks up invulnerability pickup (10s)
 Player A gets shot 9.9s later
 If the second packet is delayed before arriving at
B…

Indeterminate timing and
sequencing of information

Sequencing:
Player A picks up rocket-launcher
 Player A fires rocket
 If these arrive in the wrong order at B…

Atomicity Across Transactions
(Sharing Internal State)
Client Process
Logical Thread (A)
1.
2.
Logical Thread (B)
Generate House Info Request (11,11)
Set Local State: waiting for reply on house (11,11)
Local House State (11,11)
1.
2.
Generate House Info Request (22,22)
Set Local State: waiting for reply on house (22,22)
Local House State (22,22)
3. EnterHouse (LocalHouseState)
Thread (A) acts off of incorrect data
TSO Solution: Encapsulate State &
Logic in a State Machine
Regulator A
Client Process
Logical Thread (A)
1.
2.
Regulator B
Logical Thread (B)
Generate House Info Request (11,11)
Set Local State: waiting for reply on house (11,11)
Local House State (11,11)
1.
2.
Generate House Info Request (22,22)
Set Local State: waiting for reply on house (22,22)
Local House State (22,22)
3. EnterHouse (LocalHouseState)
Thread (A) now acts off of correct data
Latency (and why it isn't constant)
The internet:
Latency (and why it isn't constant)
Router buffers:
Latency (and why it isn't constant)
Different routes:
Latency (and why it isn't constant)
Packet loss:
API options: State Replication
Versus Event Driven
Client A
State
Changes
Client B
State
Changes
“Best Available”
representation of state
Shared State
DB
Partial Updates
Pros:
Intuitive Model
Hides Network Optimizations
“Best Available”
representation of state
Shared State
DB
Cons:
API can create new problems
Hidden network costs
Recommendation:
Support Shared State & Events
Client A
State
Changes
Client B
Immediate
Events
State
Changes
“Best Available”
representation of state
Shared State
DB
Partial Updates
Shared State:
Instrument for volume & latency
Optimize to network budget
“Best Available”
representation of state
Shared State
DB
Events:
Use for stateless transactions,
Immedidate actions
Shared WorldView: Summary
(Impact On Players & Programmers)


Player immersion: jerky visuals, occasional
discontinuities
Fairplay: who killed who?



Well, who shot first? How can you tell?
Was Player A actually located where Player B shot him?
Coding issues



Race conditions
Event ordering & loss of precision can change future
outcomes per machine
Bugs of this sort are hard to reproduce …
Shared WorldView: Summary
(Mitigating Factors)




High degrees of precision are rarely required
Predictable motion patterns simplify hiding the
latency (dead reckoning, predictive contracts, et al)
Rule of Thumb: design game play to minimize the
required accuracy (and volume) of shared data
Useful tactic: handle all inter-process communication
via state machines that handle each transaction from
initiation to conclusion (TSO’s Regulator template
worked well)
Scale and Scaleability

Scale: the N2 problem

Data Transmission Optimizations: minimize the network
cost of sharing the required amount of Shared State




Interest management, predictive contracts (dead reckoning),
variable resolution, compression, …
Application Scalability: minimize the amount of required
Shared State
References: www.maggotranch.com/ADS
Bandwidth: these days, who cares?


Caveat: see “Scale”!
Best bet: metrics, metrics, metrics



Set budget per player, optimize to that point, then STOP
Danger: pre-mature optimization
Danger: game play changes that break optimizations
Why “every” and “all” are
dangerous


SP: all information is always available
MP: shared information must be packaged,
transmitted and unpackaged



Each step costs CPU & bandwidth, and can happen 10’s to
100’s of times per minute
May also cause additional overhead (e.g. DB calls)
Scalability is key: many shared data structures grow
with the number of players, AI, objects, terrain, …

Caution: early prototypes may be cheap enough, but as
game progresses, costs may explode
Why “every” and “all” are
dangerous



Example: TSO’s Data Service
Initial design: Transmit entire ReservedLotList to all
connected clients, every 30 seconds
Initial fielding: no problem


Complete disaster as clients & DB scaled


Development testing: < 1,000 Lots, < 10 clients
Shipping requirements: 100,000 Lots, 4,000 clients
DO THE MATH BEFORE CODING



LotElementSize * LotListSize * NumClients
20 Bytes * 100,000 * 4,000
8,000,000,000 Bytes, TWICE per minute!!
Why “every” and “all” are
dangerous
22,000,000 DS
Queries! 7,000
next highest
Costs of QA

6 player game – you need at least 6 testers at
the same time
Testers
 Consoles, TVs and disks
 Network connections




End-user grade network connections (ADSL)
More players - more likely to see hard-to-find
bugs so they are marked as online bugs
Automated testing
Debugging: Trials & Tribulations

MP games require MP inputs to test



Non-determinism is a serious issue




Often, sufficient machines & QA testers not available
Developers significantly handicapped
Running the same test twice does NOT necessarily give the same result
twice
Offline code changes frequently breaks online code
functionality
Run-time debugging of networked games often becomes postmortem debugging
What helps



Automated testing
Architectural support for forcing causality (but $$)
Strong isolation of online / offline code
Debugging: Trials & Tribulations
Simple TSO Test:
One avatar holds the
Lot open, while 3
Avatars jump in & out
Debugging: Trials & Tribulations
Debugging: Trials & Tribulations

TSO EnterLot: 30 test runs, 4 behaviours
Successful entry
 Hang or Crash
 Owner evicted, all possessions stolen



Random results observed in all major features:
very difficult to track
What worked
Massive repetition of tests to establish true
pass/fail conditions
 Continual repetition of unit tests (monkeys rock!)
 Initial calibration tests: 400x runs per unit test


7% to 30% failure of any given unit test
Debugging: Trials & Tribulations
Monkeys: Continual Repetition of Critical Path Unit Tests
Code Repository
Compilers
Reference Servers
Impact On The Development Team




Changes to non-networked gamplay often break
networked play (shared code & implicit assumptions
& timing changes)
Changes to online code can inhibit content
development (“server down” often equals “nobody
working”)
Non-determinisim produces frustrating defects
Mitigating factors



Make sure people can work in isolation: no “critical path”
failures that bring down the team
Strong testing in the dev team
Online/offline code sharing via modules, not #ifdef
Stability Analysis:
What Brings Down The Team?
Test Case: Can an Avatar Sit in a Chair?
use_object ()
buy_object ()
enter_house ()
buy_house ()
create_avatar ()
login ()
Failures on the
Critical Path block
access to much of
the game.
Worse, unreliable
failures…
Regression
Impact
On
Others
Smoke
Build
days
Bug
Introduced
Checkin
Time to Fix
Development
Cost of Detection
Process Shift: Comb Filter Testing
Sniff Test, Monkey Tests
- Fast to run
- Catch major errors
- Keeps coders working
Smoke Test, Server Sniff Full Feature Regression, Full Load Test
- Is the game playable? - Do all test suites pass?
- Are the servers stable
- Are the servers stable
under a light load?
under peak load conditions?
- Do all key features work?
$
New code ready
For checkin
Full system build
$$$
$$
Promotable to
full testing
Promotable to
paying customers
• Cheap tests to catch gross errors early in the pipeline
• More expensive tests only run on known functional builds
Lunch Break until 2:00
2:00pm – 4:00pm
Case Studies
Break until 4:30
4:30pm – 5:30pm
Post Ship Issues
Cheating: The Big Picture

A few serious questions every developer
should ask:
Why do we care if anyone cheats at our game?
 Is it really a problem?
 What is our relationship with our customers?
 What are we willing to do about it?
 What about our publisher?

Who is cheating?

Anyone who can get away with it




Profiling is valid here – Young males are at
greatest risk.
We have a win at all costs culture that does not
value honesty
The Anonymous nature of the net allows
people to step outside their normal behavior
limits
Single player cheats have not helped
Why are they cheating?


To Win
To cause others not to win


To exercise control


Crashing others is considered a ‘win’
Almost pathological need for some
To show off (attention)
Matt’s Rules about Cheating




Rule #1: If you build it, they will come -- to hack and
cheat.
Rule #2: hacking attempts increase with the success
of your game.
Rule #3: Cheaters actively try to keep developers
from learning their cheats.
Rule #4: Your game, along with everything on the
cheater's computer, is not secure. The files are not
secure. Memory is not secure. Services and drivers
are not secure.
Rules, cont.



Rule #5: Obscurity is not security.
Rule #6: Any communication over an open line
is vulnerable to interception, analysis, and
modification.
Rule #7: There is no such thing as a harmless
cheat or exploit. Cheaters are incredibly
inventive at figuring out how to get the most
out of any loophole or exploit.
More Rules…



Rule #8: Trust in the server is everything in a
client-server game. Actually its true for all
game types.
Rule #9: Honest players would love for a game
to tip them off to possible cheating. Cheaters
want the opposite.
Rule #10: No one likes to have their ass
handed to them on a platter every time
Rules, cont.



Rule #11: Once a cheat has been perpetrated,
a patch, sequel, or similar game will suffer
similar hacking attempts sooner than later.
Rule #12: There is more than one way to
perpetrate a cheat.
Rule #13: It only takes one person to break an
online community.
The Last Rule…

Rule #99: There is no “Silver Bullet” to make
your game’s vulnerabilities go away. It is an
ongoing, never ending war. What a developer
can do is to choose which battles are worth
fighting.
Reflecting on the rules…

What is the purpose of a computer game?
To entertain & create enjoyable experiences
 To make the developer money
 To make users desire your future products


Who owns the game experience?
What exactly did the player buy?
 What are the obligations and expectations?

General Categories of “Cheats”







Reflex augmentation
Authoritative clients
Information exposure
Compromised servers
Bugs and design loopholes (exploits)
Environmental weaknesses
Social Cheats
Reflex Augmentation





Anything that replaces human skill & response
with automated input.
FPS Aiming proxies or bots.
Situation analyzers or AI players.
Timing assistants.
Changes the game from a contest of two
humans to one of man vs. machine.
Authoritative Clients

When a player’s client is the final arbitrator of
any aspect of the game
Less of an issue for server based games
 Obviously someone has to make some definitive
decisions in a peer to peer game.
 Can all players make the same decision and then
compare their conclusions?

Information Exposure

The revealing of any game information hidden
from the player
More of a problem for synchronous simulations
 Can be totally passive




RTS Map & Fog of War hacks
Skin & model hacks
View of other player’s status
Compromised Servers





Related to general PC security issues
More likely done by the server operator, but
could be done by someone else
A perceived problem with clans
How to detect?
Ties into matchmaking issues.
Exploits




Coding bugs and design loopholes
Catchall category of problems
Often discovered by accident
Question of who makes the call
Environmental Weaknesses






When the game’s environment is subverted
Corrupted Communication data
Hacked & Modified Drivers
OS bugs
Hacked System Clocks
Bandwidth and latency calculations
Social Cheats






Social Engineering that takes place inside of a
game
Online forms of issues and scams we’ve had in
person for thousands of years.
Player Collusion and betrayal.
Player in the same physical location.
Player controls pace of game abusively
Player text/voice chats others abusively
Other problems: Identity Theft



The impersonation or hijacking of a player’s
online identity and legacy.
More of problem as we push MMO games,
persistent games and meta-games.
Players often have a lot invested in their online avatars and identities.
Other: Reverse Engineering

Reverse engineering of Game Rules and Stats
This is a design issue
 Non published Statistics


Random Number Generators
Some Grey Areas

Engine & Game Configurations


Connection Type


Separates the ‘hardcore’ player from the causal
gamer.
Not much we can do.
Equipment Configuration

Ditto for PC
Grey Areas cont.


Cheaters vs. Impossibly Good Players
What happens if someone can’t tell the
difference?
The skill curve of your game
 Can be prohibitive to new players

What the developer can do







Plan for anti-cheating efforts in the schedule.
Audit your (final) communication formats.
Audit your data & program layout.
Attempt to make cheats using your insider
knowledge.
Plan for a patch.
Provide auditing tools to your players.
Create persistent player identities.
What developers can do, cont.

Know in advance the public stance you and
your publisher will take if a cheat is found.
Denial can work, but is risky
 Honesty is appreciated by players



But it could be an admission of liability?
Communicate with your players

Active community management.

Cultivate contacts in the user community
How long do you support a game
after it ships?




Problem – no available staff
Problem – It costs $$$
One solution – metric based on sales
Make a decision, be willing to set an EOL date
for support?
Dealing with Authoritative
Clients


Run the full simulation on all machines
Communicate only requests, not actions
Harder than you think
 Assume nothing regarding game rules


Verify the full state of the simulation

Great debugging benefits
Combating Information
Exposure


Add detection processes to client.
Share information on the engine’s display
operation.
Good for complex data
 Simple data can still be passively extracted



Audit previous gameplay for information
assumptions.
Use Honeypot designs.
Dealing with Compromised
Servers


Give the users tools to determine the server’s
state & configuration.
Link Identities to operators.


Allow users to filter operators
Use third party products like Punkbuster or
SecurePlay.
Dealing with Exploits

Patch, baby, patch….


Most effective patch case
Have a consistent policy regarding changes
And document the changes to your users
 changes in patches not communicated can be seen
as new cheats or bugs.

Other things developers can do

Develop an internal “cheater” team from your
QA team during the development process.


Provide them with assistance to crack your own
game.
Provide Tools to combat Social Cheating

Recorded games and auditing tools
Programming Recommendations

Assuming that your game will be hacked for
the purpose of cheating.
Make an effort to determine the likely forms that
the cheat will take, and put code in your game
whose sole purpose is to detect evidence that a
cheat has taken place.
 For very likely cheats, place detection code at
multiple levels in the game if possible.
 Once cheating has been detected, delay responding
if possible.

More recommendations


Single player cheats need to go in their own
separate game system.
Have your game validate the integrity of its
executables, dynamic link libraries, and nonvariant data files
Check your output!

Don’t leave debug symbols or developer functionality
compiled into the game’s executable files.

Always scan your game’s executable files for strings and other
revealing data before releasing the files to anyone outside of
the development team.

Make sure that all debugging and developer code is actually
not compiled in external release versions of your game.
Don’t forget to check data files
too!

Make sure that all developer and debugging
information and functionality is removed from the
data files that accompany the game’s executables.

If your game uses a scripting language or other data
system that exposes human readable function or
command names, implement a tokenizer or compiler
that will change that data to a non-obvious form, and
don’t release the uncompiled data files.
More thoughts about cheating

Never underestimate the techniques hackers will use
to rip apart a game.

Once a code-related hack has been created, fixing it
with a patch can be a no-win situation.



Get it right the first time when you patch.
Consider performing radical changes to the game’s code to
fix a hack.
Previous versions of a game provide a source of difference
data for a hacker.
Patching Multiplayer Games



Patching on consoles ranges from impossible, to very
difficult, to “only” requiring a certification pass.
Who pays for the patch to be built?
How do you get it to your players?




Network budget
Impact on the player experience
Differential copies: game changes to simplify the problem
What conditions require a patch




How bad do things have to be to require a bug patch?
Or do you patch to deliver new content?
Remember that any patch has a customer service cost.
Remember that the nature of a multiplayer game requires
the players to patch if they want to play with others.
Long-term customer support




Making support calls not happen must be a priority.
Especially for games without a recurring income
stream, any CS call may put you cash negative on the
sale.
Poor customer support can drive customers away
However, many customers may have unreasonable
expectations and they will never be satisified,
regardless of the CS quality.
Official forums tend to be extremely negative. Happy
players usually don’t post, and unhappy players want
everyone to know they are unhappy, and will post in
the only forum they know the developer/publisher
reads.
Cross platform support



Risks: All platforms that are interoperable
must be able to be patched roughly
simultaneously
Risks: Small differences in floating point
processing can give an unintentional advantage
to some platforms
Risks: Depending on the differences between
platforms, input devices may give a significant
advantage
New Content

MMP: New content must be added, constantly




Strong impact on old content
Gameplay balances, long term additive properties (N2’ed?
Worse?)
Code rot, tools, and long term maintainability
Free vs user created vs commercial expansions




Legal issues
Distribution & installation with N prior expansions
Gameplay imbalances
Providing hooks for player generated content is not cheap!
Open Q&A
5:30pm - 6:00pm
References

References available as part of the final slide
set

Final Slides available online @
www.maggotranch.com/MMP