Transcript Supported by Novartis The Scientific Community Game: Education and Innovation Through Survival in a Virtual World of Claims Karl Lieberherr Northeastern University College of Computer and.

Supported by Novartis
The Scientific Community Game:
Education and Innovation Through
Survival in a Virtual World of Claims
Karl Lieberherr
Northeastern University
College of Computer and Information Science
Boston, MA
joint work with
Ahmed Abdelmeged and Bryan Chadwick
Why Scientific Community Game
(SCG)
• … motives in academic publishing:
– desire for recognition and respect from the people
one regards as peers,
– desire to have impact (on conclusions being
reached, on the development of the discipline,
etc.), and
– desire to participate in significant knowledgebuilding discourse.
• e.g., Scardamalia, M., & Bereiter, C. (1994)
Bionetics 2010
2
SCG is Bio-inspired
• Virtual world of scholars based on natural
selection
– propose, oppose (refute and strengthen) claims
– maximize reputation, weak scholars are removed.
• Turn problem-solving software into virtual
organisms that fend for themselves and
survive in a virtual world inhabited by virtual
organisms created by your peers.
Bionetics 2010
3
SCG is a web-based implementation
of Karl Popper’s science ideas
• One of the greatest philosophers of science of
the 20th century.
• Falsifiability or refutability is the logical
possibility that an assertion could be shown
false by a particular observation or physical
experiment.
• Error elimination (refutation), performs a
similar function for science that natural
selection performs for biological evolution.
from Wikipedia
Bionetics 2010
4
Comparison
• Karl Popper: Conjectures and Refutations,
1963
• Scientific Community Game: Claims and
Refutations, 2007
Bionetics 2010
5
Recognition in SCG
• Scholars build their reputation by proposing
and opposing claims, by defending their own
claims and refuting or strengthening the
claims of others.
• The higher their reputation, the more
recognition.
Bionetics 2010
6
Impact in SCG
• Second-order environment
– what one scholar does in adapting, changes the
environment so that others must readapt.
• Developing novel techniques to find superior
solutions, challenges others to catch up.
Bionetics 2010
7
Knowledge-Building Discourse
in SCG
• Communication or debate.
• Refutation protocol defines the structure of
the debate and who wins. Claims are defined
through a refutation protocol.
• Knowledge-building:
– claims that have been defended predominantly
are candidates for truth
– claims that have been refuted predominantly are
probably false.
Bionetics 2010
8
Goals of SCG
• Put knowledge-building discourse on the web
giving participants the option to gain
recognition and to have impact.
• Focus the discourse through precise definition
of claims with refutation protocols.
• Make knowledge building discourse fun and
educational from the high school to the
advanced research level.
SCG = Scientific Community Game = Specker Challenge Game
Bionetics 2010
9
What do we mean by science?
• Science consists of the formulation and testing
of hypotheses based on observational
evidence.
• Ours: Science consists of the formulation and
testing of constructive claims based on
observational evidence. Construction is
computable.
Bionetics 2010
10
What do we mean by Scientific
Method
• Hypothetico-deductive method: Formulate a
hypothesis in a form that could conceivably be
falsified by a test on observable data.
• Ours: Formulate a constructive claim in a form
that could conceivably be falsified by a test
using a protocol. The refutation protocol is
part of the claim to make very explicit when
refutation is successful.
Bionetics 2010
11
Tartaglia against Fior
1535
Tartaglia was famed for his
algebraic solution of cubic
equations which was
published in
Cardan's Ars Magna.
Bionetics 2010
12
Outline
• Introduction
– Popper Science, Renaissance History: Tartaglia and Fior
• Definition of SCG
– Example (Highest safe rung)
•
•
•
•
•
Applications: Teaching, Software Development, Research
Claims with secrets and other protocol variants
Output of SCG, Equilibrium
Advantages and Disadvantages
Conclusions
Bionetics 2010
13
Definition of SCG: Domain
•
•
•
•
Problem: Set
Solution: Set
valid: relation(Problem, Solution)
quality: function(Problem, Solution)->[0..1]
Bionetics 2010
14
Claim(Domain)
makes predictions
about the future
• Problems: Powerset(Domain.Problem)
• q: Quality = [0,1]
• r: Resource = N+ = positive integer
Alice claims to have a technique to solve problems in Problems
with at least quality q and using at most resources r.
Bionetics 2010
15
Implied Protocol of Claim(Domain)
• Alice claims (problems,q,r), Bob refutes
• Bob provides problem prob in Claim.Problems.
• Alice solves problem prob providing sol in
Domain.Solution.
• check: valid(prob,sol) and quality(prob,sol)>=q and
sol.resource<=r.
• sol.resource returns Alice’ resource consumption to
solve problem prob.
Karl Popper: Only hypotheses capable of clashing with observation
reports are allowed to count as scientific.
Bionetics 2010
16
Claim
• Problems: subset of problems
• quality in [0,1]
1
quality
(how well
problems in
Problems can be
solved)
0
Bionetics 2010
17
Claim
over strengthening
1
correct valuation
quality
strengthening
0
Bionetics 2010
18
18
Bio-inspired computing:
Virtual World of SCG-Avatar
• SCG-Avatar (Claim(Domain))
– State: Reputation = positive rational number
– Activity
• propose new claims
• oppose claims of others
– refute claim(Problems, q, r)
– strengthen claim(Problems, q’, r’), q’>q or r’<r
• Reputation gain: refute others’ claims and defend own
claims (counter refutation attempts)
• Reputation loss: unsuccessful refutation of other’s
claim and refutation of own claims
Bionetics 2010
19
Tournament
Bionetics 2010
1. round-robin
2. Swiss-style
3. elimination
1. single
2. double
20
Summary of SCG Definitions
Domain
Problem
Solution
valid(Problem, Solution)
quality(Problem, Solution) →[0,1]
Claim(Domain)
Problems: PowerSet(Domain.Problem)
q: Quality = [0,1]
r: Resource = N+
Rules of the Scientific Community: propose and oppose,
be an active scholar, rules for reputation accumulation.
Tournaments
Bionetics 2010
21
Highest Safe Rung
• You are doing stress-testing on various models
of glass jars to determine the height from
which they can be dropped and still not break.
The setup for this experiment, on a particular
type of jar, is as follows.
Bionetics 2010
22
Highest Safe Rung
Bob
Alice
You have a ladder with n rungs, and
you want to find the highest rung
from which you can drop a copy of
the jar and not have it break. We call
this the highest safe rung. You have
a fixed ``budget'' of k > 0 jars.
Only two identical bottles to determine
highest safe rung
Bionetics 2010
23
Highest Safe Rung
Bob
Alice
HSR(9,2)
≤4
Only two identical bottles to determine
highest safe rung
Bionetics 2010
I doubt it: refutation attempt!
Alice constructs
decision tree T of
depth 4 and gives it
to Bob. He checks
whether T is valid.
Bob wins if he finds
a flaw.
24
x
Highest Safe Rung Decision Tree
HSR(9,2)=5
3
no
yes
y
1
z
6
u
0
2
highest safe rung
4
9
1
2
3
5
7
4
9
5
8
6
7
Bionetics 2010
8
25
Finding solution for HSR(n,2)
• Approximate
Find minimum x, s.t. MaxRungs(x,2) > n
min x in [0,n] (n/x) + x
• Exact
breaks at root
does not break at root
– MaxRungs(x,y) =MaxRungs(x-1,y-1)+MaxRungs(x-1,y)
– MaxRungs(x, 2) = x + MaxRungs(x – 1, 2)
Keith Levin CS 4800 Fall 2010
– MaxRungs(0, 2) = 1
MaxRungs(x,y) = the largest number
– Applied to HSR(9,2)
• MaxRungs(3,2) = 7 < 9
• MaxRungs(4,2) = 11 > 9
Bionetics 2010
of rungs we can test with y jars and
x experiments.
26
MaxRungs
• MaxRungs(x,y) = sum [k=0 .. y] binomial(x,k)
• All paths are of length x. At most k branches
may be left branches.
• Note: y = x implies MaxRungs(x,y) = 2x
meaning a complete binary tree of depth x.
• Example: binomial(3,2)+binomial(3,1)+
binomial(3,0) = 7
Bionetics 2010
27
Formal: HSR
• Domain:
– Problem: (n,k), k <= n.
– Solution: Decision tree to determine highest safe
rung.
– quality(problem, solution): depth of decision tree /
number of rungs
– valid(problem, solution): at most k left branches, ...
Bionetics 2010
28
Formal: HSR
• Claim(Domain):
– Alice claims ({(25,2)},9/25,5 seconds)
• {(25,2)}: set of problems (singleton)
• 9/25: quality
• 5 seconds: resource
• Refutation Protocol:
– Bob refutes: only one problem: (25,2)
– Alice: solves problem by providing decision tree t.
– predicate: t is a valid decision tree for (25,2) of depth 9
Bionetics 2010
29
Claim involving Experiment
Claim ExperimentalTechnique(X,Y,q,r)
I claim, given raw materials x in X,
I can produce product y in Y of quality q
and using resources at most r.
Bionetics 2010
30
Outline
• Introduction
– Popper Science, Renaissance History: Tartaglia and Fior
• Definition of SCG
– Example (Highest safe rung)
•
•
•
•
•
Applications: Teaching, Software Development, Research
Claims with secrets and other protocol variants
Output of SCG, Equilibrium
Advantages and Disadvantages
Conclusions
Bionetics 2010
31
Applications:
Software Development
• Software Development
• Teaching Constructive Domains
Bionetics 2010
32
Gamification of Software
Development etc.
• Want reliable software to solve a
computational problem? Design a game
where the winning team will create the
Doesn’t TopCoder already do this?
software you want.
• Want to teach a STEM domain? Design a game
where the winning students demonstrate
superior domain knowledge.
STEM = Science, Technology, Engineering, and Mathematics
Bionetics 2010
33
SCG and TopCoder
• SCG is an abstraction and generalization of
what TopCoder does.
Bionetics 2010
34
The Traditional Approach
Team A
Solver A
Team B
Solver B
Team C
Solver C
Parameterized
by the domain.
Static
Benchmark
HSR(9,2)=4
HSR(25,2)=7
Ranking
measure how close
to minimum
Software: Solving HSR Problem:
construct decision tree of min. depth
Bionetics 2010
36
The Bio-Inspired Approach
Dynamic
Benchmark
Solver A
Team A
prop-opp A
Solver B
Team B
prop-opp B
Solver C
Team C
Avatar
A
Avatar
B
Virtual
World
(Game)
Ranking
Avatar
C
prop-opp C
Parameterized
by the domain.
Bionetics 2010
37
A Virtual World
Avatar’s View
Avatar
Claims,
Problems,
Solutions
Opponents’ communication,
Feedback
Administrator
• Problems: Benchmark output
• Solutions: Software output
• Claims: statements about algorithms
Results
Bionetics 2010
38
What Scholars think about!
• If I propose claim C, what is the probability
that
– C is successfully refuted
– C is successfully strengthened
• If I try to refute claim C, what is the probability
that I will fail.
• If I try to strengthen claim C, what is the
probability that I will fail?
Bionetics 2010
39
SCG = Scientific Community Game
• Make software development more scientific.
• Software developers build reputation
– propose and defend claims about their software
– oppose claims made by others
• refute claims
• strengthen claims
• claim includes refutation protocol
Bionetics 2010
40
Why a web application with avatars? Fair Evaluation.
Who are Alice and Bob?
• They are avatars developed by real Alice and
real Bob.
• Alice and Bob compete with 10 other avatars
in a full-round robin tournament.
• Who is the winner: The avatar with the
highest reputation, i.e., the avatar who has
the strongest, not successfully opposed claims
(like in a real scientific community).
Bionetics 2010
41
our focus
What is SCG(X)
avatar Bob
Alice
degree of automation used by scholar
1
0
no automation
human plays
some automation
human plays
full automation
avatar plays
transfer to reliable,
efficient software
more applications:
test constructive knowledge
Bionetics 2010
42
Real Scholars and Avatars:
Same rules
• Are encouraged to
1. propose claims that are not easily strengthened.
2. offer claims that they can successfully support.
3. strengthen others’ claims, if possible.
4. stay active and propose new strong claims or
oppose others’ claims.
5. become famous!
Bionetics 2010
43
Clear Feedback
Sense of Progress
What we want
Authenticity (Facebook)
• Engage software developers
– let them produce software that models an
organism that fends for itself in a real virtual world
while producing the software we want. Have fun.
Focus them.
– let them propose claims about the software they
produce. Reward them when they
• defend their claims successfully or
• oppose the claims of others successfully.
Possibility of Success
Bionetics 2010
44
SCG
• Gamification of software development for
computational problems
• A Sociotechnical System for knowledge
dissemination, innovation, and integration
Bionetics 2010
45
Software Engineering Properties
fostered by SCG
• Reliable (otherwise the avatar is removed
from the game)
• Flexible, modular (otherwise the avatar
cannot be easily updated between
tournaments) Adaptive and Aspect-Oriented Software is relevant!
• Efficient (otherwise you cannot defend your
claims and oppose the claims of others)
Bionetics 2010
47
State of Avatar SCG-Avatar: Our
Vision
• Companies come to SCG website and define a
competition by defining a claim domain X.
• Participating teams get baby avatars
generated from X that participate in daily
competitions.
• Competition generates a wealth of
information: educated employees, good
(undefeated) software, good algorithms, good
potential employees. Reward is paid to the
winner.
Bionetics 2010
48
State of SCG-Avatar: Our Vision
• Not only companies but faculty members who
want to give their students a rich learning
experience for computational problem X.
• Or editors of special issues in journals who
want to use a competition to get a real world
comparison of all approaches to solve
computational problem X.
Bionetics 2010
49
Life of an avatar: (propose+ oppose+ provide* solve*)*
Avatars propose and oppose
proposed claims
egoistic
Alice
CA1
CA2
egoistic
Bob
social welfare
opposes (1)
CB1
CB2
CA3
provides problem (2)
CA4
LOSES
solves problem
WINS!
not as well as she expected based on CA2 (3)
reputation 10
reputation 1000
transfer 200
Bionetics 2010
50
What is SCG(X)?
Team
Alice
I am the best
Teams
Design Problem Solver
Develop Software
Deliver Avatar
Avatar Alice
Team
Bob
No!!
Avatar Bob
Administrator
SCG police
Bionetics 2010
Let’s play
constructively
51
competitive / collaborative
Avatar Alice: claim C
loses reputation r
wins knowledge k
Avatar Bob: opposes C, refutes: provides
evidence for !C
wins reputation r
makes public knowledge k
Bionetics 2010
52
Outline
• Introduction
– Popper Science, Renaissance History: Tartaglia and Fior
• Definition of SCG
– Example (Highest safe rung)
•
•
•
•
•
Applications: Teaching, Software Development, Research
Claims with secrets and other protocol variants
Output of SCG, Equilibrium
Advantages and Disadvantages
Conclusions
Bionetics 2010
53
Protocol Variants
• secrets: approximation problems
• involving trusted third party
– renaissance: exchange of problems
Bionetics 2010
54
Example: Triple HSR
Highest Safe Rung
• Alice claims ({(25,2,0), (25,2,1), (25,2,2),
(25,2,3), … ,(25,2,25)},9/25, 5 seconds)
• Refutation Protocol:
– Bob refutes (25,2,17)
– Alice solves problems (25,2,*) by providing
decision tree to trusted third party which reveals
path p from root to 17.
– predicate: p is valid and length(p) <= 9
Bionetics 2010
55
Protocol Variation Secrets
• problem has public and private part, private
part is a secret solution
• predicate has secret as argument
Bionetics 2010
56
Protocol Variation Secret Program
for SCG-Avatar
• problem has public and private part, private
part is a secret solution and goes to
administrator
• Alice gives her algorithm to administrator who
applies it to public part of problem
• predicate has secret as argument
Bionetics 2010
57
Example Claims involving secrets
• My algorithm can solve more problems using
resources r than your algorithm using r.
• If I create problems for you for which I have a
solution, you cannot recreate or approximate
the solution with quality q using resources r.
Bionetics 2010
58
Output and Equilibrium
• Rich tournament history
• What is an equilibrium in SCG?
Bionetics 2010
59
Soundness Theorem
• SCG is sound: The avatar with the best
algorithms / knowledge wins (there is no way
to cheat)
– best: within the group of participating avatars
– issues:
• Does an avatar win because she is good at solving? Or
good at proposing, opposing and providing? Answer:
proposing, opposing and providing all reduce to solving.
11/7/2015
Games for SD
60
SCG Equilibrium
• reputations of scholars are stable
• the ranking of the scholars is invariant from
tournament to tournament
• the science does not progress; bugs are not
fixed, no new ideas are introduced
• extreme example: All scholars are perfect:
they propose optimal claims C(ps,q) that can
neither be strengthened nor refuted.
Bionetics 2010
61
second-order environment!
Survival in SCG(X)
• [Scientific Innovation in X] Avatars get skills
programmed into them by clever scientists in
domain X. Scientists use data mining to learn
from competitions and manually improve the
avatars.
• [Machine Learning Innovation in X] Avatars get
skills programmed into them by an avatar
caregiver programmed with learning skills and
data mining skills for domain X. Avatar gets
updated automatically.
Bionetics 2010
62
Blame assignment
• Where is the proposer to blame?
– Bad claim that is refuted.
– Bug in problem finding algorithm?
– Bug in problem solving algorithm?
Bionetics 2010
63
How to use SCG(X)
• Company AB needs new ideas about how to
solve optimization problems in domain X.
• Define claims language for X
– X-problems
– claims, includes protocol
• Submit claims language definition to SCG
server.
Bionetics 2010
64
How to use SCG(X)
• Offer prize money for winner with conditions,
e.g., performance must be at least 10% higher as
performance of avatar XY that AB provides.
• 10 teams from 6 countries sign up, committing
to 6 competitions. Player executables become
known to other players after each competition.
One team from company AB.
• The SCG server sends them the basic avatar and
the administrator for testing.
Bionetics 2010
65
How to use SCG(X)
• Game histories known to all. Data mining!
• First competition is at 23.59 on day 1.
Registration starts at 18.00 on same day. The
competition lasts 2.5 hours.
• Repeat on days 7, 14, … 42.
• The final winner is: Team Mumbai, winning
10000 Euro. Delivers source code and design
document describing winning algorithm to AB.
Bionetics 2010
66
Benefits for company AB
of using SCG(X)
• Teams perform know-how retrieval and
integration and maybe some research.
– Participating teams try to find the best knowledge in
the area.
– Claims language gives control!
• The non-refuted claims give hints about new Xspecific knowledge.
• A well-tested solver for X-problems that
integrates the current algorithmic knowledge in
field X.
Bionetics 2010
67
Outline
• Introduction
– Popper Science, Renaissance History: Tartaglia and Fior
• Definition of SCG
– Example (Highest safe rung)
•
•
•
•
•
Applications: Teaching, Software Development, Research
Claims with secrets and other protocol variants
Output of SCG, Equilibrium
Advantages and Disadvantages
Conclusions
Bionetics 2010
68
Benefits/Disadvantages
• Benefits
– competitive / collaborative
– structured feedback, game history
– Teaching
– Research
– Software Development
• Dynamic testing and evaluation
• Disadvantages
– addictive
Bionetics 2010
69
Disadvantages of SCG
• The game is addictive. After Bob having spent 4
hours to fix his avatar and still losing against Alice,
Bob really wants to know why!
• Overhead to learn to define and participate in
competitions.
• The administrator for SCG(X) must perfectly
supervise the game. Includes checking the legality
of X-problems.
– if admin does not, cheap play is possible
– watching over the admin
Bionetics 2010
70
How to compensate
for those disadvantages
• Warn the scholars.
• Use a gentleman’s security policy: report
administrator problems, don’t exploit them to
win.
• Occasionally have a non-counting “attack the
administrator” competitions to find
vulnerabilities in administrator.
– both generic as well as X-specific vulnerabilities.
Bionetics 2010
71
Benefits of SCG
• Social Welfare
– Supported knowledge
• Claims are refuted and strengthened.
• Better supported knowledge comes from better
algorithms and software.
Bionetics 2010
72
Advantage: Democratic
• Problem to be solved: Develop the best practical
algorithms for solving computational problems in
domain X.
• Issue: There are probably hundreds of papers on
the topic with isolated implementations. What
are the best practical algorithms?
• Our solution: Use the scientific community game
SCG(X) with a suitably designed claims language
to compare the software. The winning avatar has
the best practical algorithms/software.
Bionetics 2010
73
Experience with MAX-CSP
• MAX-CSP Problem Decompositions
• T-Ball (one relation), Softball (several
relations, one implication tree), Baseball
(several relations).
• ALL, SECRET
Bionetics 2010
74
Stages for SECRET T-Ball
• MAXCUT
– R(x,y)= x!=y
– fair coin ½
– maximally biased coin ½
– semi-definite programming / eigenvalue
minimization 0.878
Bionetics 2010
75
Stages for SECRET T-Ball
• One-in-three
– R(x,y,z) = (x+y+z=1)
– fair coin: 0.375
– optimally biased coin: 0.444
Bionetics 2010
76
Stages for ALL Baseball
• Propose/Oppose/Provide/Solve
– based on fair coin
– optimally biased coin
• correctly optimize polynomials
– correctly eliminate noise relations
– correctly implement weights
– …
Bionetics 2010
77
References
• Karl Popper, Conjectures and Refutations,
London: Routledge (1963).
• Scardamalia, M., & Bereiter, C. (1994).
Computer support for knowledge-building
communities. The Journal of the Learning
Sciences, 3(3), 265-283.
• Renaissance: Tartaglia and Fior challenge
(1535).
Bionetics 2010
78
Conclusions
• To address a computational problem domain X:
– “map it to second life”: define a scientific community game
for X on the web: SCG(X)
– let the game SCG(X) run a few times and choose the
winner; use strongest unrefuted claims.
• Benefits
– Evaluates fairly, frequently, constructively and dynamically.
Encourages retrieval of state-of-the-art know-how,
integration and discovery.
– Challenges humans, drives innovation, both competitive
and collaborative.
– Avatars point humans to what needs attention in problem
solution / software.
Bionetics 2010
79
Conclusions
• Broad applicability, e.g.,
• SCG(X) provides a learning process for any
constructive domain.
• Benefits
– Social Engineering: makes it fun through game.
– Fair: Only hard work makes you win.
– Engage a large community on one domain X.
Bionetics 2010
80
Thank You
Bionetics 2010
81
Title
• The Scientific Community Game: Education
and Innovation Through Survival in a Virtual
World of Claims
Bionetics 2010
82
Abstract
•
•
The Scientific Community Game (SCG) is a generic game for constructive domains
where claims are defined by interactive protocols. As a starter, this includes
mathematical claims containing alternating quantifiers but also non-mathematical
claims involving the interaction between two parties. Scholars in SCG propose and
oppose claims. Opposition means refutation or, strengthening followed by
refutation. The winning scholars are good at proposing strong, un-refutable claims
and at spotting refutable claims of other scholars. Scholars collaborate through
competition. Applications include teaching constructive topics such as calculus
and algorithms.
An especially interesting version of SCG is Avatar SCG where the scholars are
implemented in software. Avatar SCG is a web application implementing
competitions between hundreds of avatars spread over the web.
– Applications of Avatar SCG include:
•
•
•
distributed software development for computational problems
distributed knowledge maintenance and integration for computational problems
teaching software development skills
Bionetics 2010
83
Extra
Bionetics 2010
84
• 1st International ICST Conference on Theory
and Practice of Algorithms in (Computer)
Systems,
• 18-20 April 2011 - Rome, Italy
• experimental analysis of algorithms
Bionetics 2010
85
Formal: Pair HSR
•
•
•
•
Alice claims ({(25,2)},9/25)
{(25,2)}: set of problems
9/25: quality
Refutation Protocol:
– Bob refutes: only one problem: (25,2)
– Alice: solves problem by providing decision tree t.
– predicate: t is a correct decision tree for (25,2) of
depth 9.
Bionetics 2010
86
Domain
•
•
•
•
Problem: Set
Solution: Set
valid: relation(Problem, Solution)
quality: function(Problem, Solution)->[0..1]
Bionetics 2010
87
Claim(Domain)
• Problems: Powerset(Domain.Problem)
• q: Quality = [0,1]
• r: Resource = N+ = positive integer (optional)
Alice claims to have a technique to solve problems in Problems
with at least quality q and using at most resources r.
Bionetics 2010
88
Implied Protocol of Claim(Domain)
• Alice claims (problems,q,r), Bob refutes
• Bob provides problem prob in Claim.Problems.
• Alice solves problem prob providing sol in
Domain.Solution.
• check: valid(prob,sol) and quality(prob,sol)>=q and
sol.resource<=r.
• sol.resource returns Alice’ resource consumption to
solve problem prob.
Bionetics 2010
89
Benefit of a game
SCG-Avatar(Claim(Domain))
• Knowledge discovery and evaluation
– If a claim was attacked 1000 times and refuted
only twice, it is a reasonable candidate for truth.
– Depends on strength of avatars: Can they defend
true claims? Can they refute wrong claims?
Bionetics 2010
90
Game kinds
• 2-player game
– used in tournament (full round robin or Swiss
style)
• n-player game
Bionetics 2010
91
Data produced by a game
SCG-Avatar(Claim(Domain))
• Ranking of avatars
• History: data exchanged through refutation
protocols, claims proposed and strengthened
• A claim strengthened must be opposed by
original proposer.
• Data mining of history
Bionetics 2010
92
Two modes
• Teaching
– give warning if
• true claim is refuted
• false claim is supported
• Research
– don’t know the true and false claims
Bionetics 2010
93
Reflection 2
• Claim: For all NPO optimization problem P,
there exists an SCG-Avatar game G:
– better than putting 20 people into a room
– learn more (give them a test)
– have more fun
• Refutation protocol:
Bionetics 2010
94
Reflection 1
• For all constructive domains X, there exists a
SCG game G=SCG(X):
– if there is at least one good scholar, the
participating scholars compared to when they
cooperate in a non-structured way:
• create more knowledge about X using G, given the
same amount of time, because they stay focused on X.
• have more fun thanks to the competition and
collaboration.
Bionetics 2010
95
Reflection 3
• For all challenging constructive domain X
teaching tasks, there exists an SCG family
game G for X
– students learn more
Bionetics 2010
96
Reflection Software Development
• Software Development Process based on SCG
Bionetics 2010
97
The Traditional Approach
Team A
Solver A
Team B
Solver B
Team C
Solver C
Parameterized
by the domain.
Static
Benchmark
HSR(9,2)=4
HSR(25,2)=7
Ranking
measure how close
to minimum
Software: Solving HSR Problem:
construct decision tree of min. depth
Bionetics 2010
98
The Bio-Inspired Approach
Dynamic
Benchmark
Solver A
Team A
prop-opp A
Solver B
Team B
prop-opp B
Solver C
Team C
Avatar
A
Avatar
B
Virtual
World
(Game)
Ranking
Avatar
C
prop-opp C
Parameterized
by the domain.
Bionetics 2010
99
A Virtual World
Avatar’s View
Avatar
Claims,
Problems,
Solutions
Opponents’ communication,
Feedback
Administrator
• Problems: Benchmark output
• Solutions: Software output
• Claims: statements about algorithms
Results
Bionetics 2010
100
What Scholars think about!
• If I propose claim C, what is the probability
that
– C is successfully refuted
– C is successfully strengthened
• If I try to refute claim C, what is the probability
that I will fail.
• If I try to strengthen claim C, what is the
probability that I will fail?
Bionetics 2010
101
Best response dynamics
Nash Equilibria
Bionetics 2010
102
Highest Safe Rung
• You are doing stress-testing on various models
of glass jars to determine the height from
which they can be dropped and still not break.
The setup for this experiment, on a particular
type of jar, is as follows.
Bionetics 2010
103
Highest Safe Rung
Bob
Alice
You have a ladder with n rungs, and
you want to find the highest rung
from which you can drop a copy of
the jar and not have it break. We call
this the highest safe rung. You have
a fixed ``budget'' of k > 0 jars.
Only two identical bottles to determine
highest safe rung
Bionetics 2010
104
Highest Safe Rung
Bob
Alice
HSR(9,2)
≤4
Only two identical bottles to determine
highest safe rung
Bionetics 2010
I doubt it: refutation attempt!
Alice constructs
decision tree T of
depth 4 and gives it
to Bob. He checks
whether T is correct.
Bob wins if he finds
a flaw.
105
x
Highest Safe Rung Decision Tree
HSR(9,2)=5
3
no
yes
y
1
z
6
u
0
2
highest safe rung
4
9
1
2
3
5
7
4
9
5
8
6
7
Bionetics 2010
8
106
Finding solution for HSR(n,2)
• Approximate
min x in [0,n] (n/x) + x
• Exact
breaks at root
does not break at root
– MaxRungs(x,y) =MaxRungs(x-1,y-1)+MaxRungs(x-1,y)
– MaxRungs(x, 2) = x + MaxRungs(x – 1, 2)
Keith Levin CS 4800 Fall 2010
– MaxRungs(0, 2) = 1
MaxRungs(x,y) = the largest number
– Applied to HSR(9,2)
• MaxRungs(3,2) = 7 < 9
• MaxRungs(4,2) = 11 > 9
Bionetics 2010
of rungs we can test with y jars and
at most x experiments.
107
x
Highest Safe Rung Decision Tree
HSR(4,2)=3
2
root
no
yes
y
1
z
4
u
0
1
highest safe rung,
leaf
3
4
2
3
(2 y (1 y h 0 h 1) n (4 y (3 y h 2 n h 3) n h 4))
Properties of decision tree:
1. at most two yes from root to any leaf.
2. longest root-leaf path has 3 edges.
3. each rung 1..n appears exactly once as internal node of the tree.
4. each rung 0..n appears exactly once as a leaf.
Bionetics 2010
108
x
2
root
no
yes
y
1
z
4
u
0
1
3
4
2
3
Bionetics 2010
109
x
no
yes
y
z
u
Bionetics 2010
110
Swiss Tournament
Bionetics 2010
1. round-robin
2. Swiss-style
3. elimination
1. single
2. double
111
The small
player can
win unexpectedly.
Bionetics 2010
112
Elimination Tournament
Bionetics 2010
113
Claim involving Experiment
Claim ExperimentalTechnique(X,Y,q,r)
I claim, given raw materials x in X,
I can produce product y in Y of quality q
and using resources r.
Bionetics 2010
114
Why Scientific Community Game
• … motives in academic publishing:
– desire for recognition and respect from the people
one regards as peers,
– desire to have impact (on conclusions being
reached, on the development of the discipline,
etc.), and
– desire to participate in significant knowledgebuilding discourse .
• e.g., Scardamalia, M., & Bereiter, C. (1994)
Bionetics 2010
115
Recognition in SCG
• Scholars build their reputation by proposing
and opposing claims, by defending their own
claims and refuting or strengthening the
claims of others.
• The higher their reputation, the more
recognition.
Bionetics 2010
116
Impact in SCG
• second-order environment
– what one scholar does in adapting changes the
environment so that others must readapt.
• Developing novel techniques to find superior
solutions challenges others to catch up.
Bionetics 2010
117
Knowledge-Building Discourse
in SCG
• communication or debate.
• Refutation protocol defines the structure of
the debate and who wins. Claims are defined
through a refutation protocol.
• Knowledge-building:
– claims that have been defended predominantly
are candidates for truth
Bionetics 2010
118
Bereiter & Scardamaglia
• Scardamalia, M., & Bereiter, C. (1994).
Computer support for knowledge-building
communities. The Journal of the Learning
Sciences, 3(3), 265-283.
Bionetics 2010
119
Discourse
• We have roughly divided characteristics for
knowledge-building discourse into three
categories:
– focus on problems and depth of understanding;
– decentralized, open knowledge environments for
collective understanding; and
– productive interaction within broadly conceived
knowledge-building communities.
• Scardamalia, M., & Bereiter, C. (1994)
Bionetics 2010
120
SCG Equilibrium
• reputations of scholars are stable
• the ranking of the scholars is invariant from
tournament to tournament
• the science does not progress; bugs are not
fixed, no new ideas are introduced
• example: All scholars are perfect: they
propose optimal claims C(ps,q) that can
neither be strengthened nor refuted.
Bionetics 2010
121
Claims and Refutation Protocol
• Alice claims: I have a program that solves inputs
in domain X with quality Q and resources R.
– AliceClaim(X,Q,R)
• Bob is critical. He prepares an input in X and
gives it to Alice who applies her program. Bob
refutes AliceClaim(X,Q,R) iff Alice achieves < Q
or uses > R.
– Refutation protocol
Bionetics 2010
122
State of Avatar SCG
• Domain is hard-wired to Constraint
Satisfaction Problems
• One Master student worked on making it
generic but work is not complete.
Bionetics 2010
123