Transcript Chess Tactical Problem

A COMPUTATIONAL MODEL FOR ESTIMATING
THE DIFFICULTY OF CHESS PROBLEMS
automated difficulty estimation,
human problem solving
Simon Stoiljkovikj, Ivan Bratko, Matej Guid
Faculty of Computer and Information Science
University of Ljubljana, Slovenia
The Third Annual Conference on Advances in Cognitive Systems
ACS 2015: Atlanta, Georgia, USA, May 28-31, 2015
RESEARCH GOAL
Given a problem, can we automatically
predict how difficult the problem will be
to solve by humans?
THE GOAL OF OUR RESEARCH
• find a formal measure of difficulty of mental problems for humans
• implement such a measure, possibly as an algorithm
• enable automated difficulty estimates by computers
POSSIBLE APPLICATIONS
• intelligent tutoring systems
• estimating problem solving performance
• preparation of student exams
• automated commenting
• game playing
• better understanding of what makes problems easy or difficult for a human
• ...
A SLIP BY THE WORLD CHAMPION...
Deep Fritz – Kramnik
Man vs Machine, Bonn 27.11.2006
position after Kramnik‘s 34... Qa7-e3, allowing checkmate in one with 35.Qe4-h7#
We are NOT trying to model the difficulty of a
particular problem for a particular individual!
Chessbase.com: How could Kramnik overlook the mate? http://www.chessbase.com/newsdetail.asp?newsid=3512
ESTIMATING PERFORMANCE IN PROBLEM SOLVING
the scores of computer analysis,
reflecting the champions‘ performance
based on their quality of play
How difficult were chess positions that
each one of them faced over the board?
Guid M., Bratko I. Computer Analysis of World Chess Champions. ICGA Journal, Vol. 29, No. 2, pp. 65-73, 2006.
SEARCH-BASED ESTIMATION OF PROBLEM DIFFICULTY
This approach is not appropriate
for estimating the difficulty of
chess tactical problems...
Guid M., Bratko I. Search-Based Estimation of Problem Difficulty for Humans. AIED 2013, LNCS 7926, pp. 860-863, 2013.
CHESS TACTICAL PROBLEM
Black to move wins. How difficult is this problem for a human?
„THE TREE OF ANALYSIS“ BY GM KOTOV
Chess grandmaster Kotov:
• first identify candidate moves
• methodically examine them to build up an “analysis tree”
Kotov A. Think Like a Grandmaster. Batsford, 1975.
STATE-OF-THE-ART CHESS ENGINES...
... are terribly strong at solving tactical problems.
How to use chess engines to determine what is difficult for humans?
CHESSTEMPO.COM
Chess Tempo website offers statistic-based difficulty ratings (Glicko Rating System).
Such ratings can provide a basis from which we estimate the difficulty of a problem.
ADVERSARIAL PROBLEM SOLVING
odd levels: player‘s turns
even levels: opponent‘s turns
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
THE PROPOSED APPROACH
• use computer heuristic search to estimate
the values of particular nodes in the game tree
• only nodes and actions that meet certain
criteria are kept in the „meaningful search tree“
THE HYPOTHESIS
• By analyzing properties of such a tree, we should be able to infer certain
information about the difficulty of the problem for a human.
THE CONCEPT OF „MEANINGFUL SEARCH TREES“
a
level 1: player‘s decisions
b
d
c
...
e
level 2: opponent‘s decisions
f
g
i
h
...
level 3: player‘s decisions
j
k
l
m
winning positions
relatively good positions for the opponent
not winning positions / too bad for the opponent
n
o
p
r
...
BUILDING THE MEANINGFUL SEARCH TREE
Domain specific parameters:
• heuristic-search engine searches up to depth d
• minimal heuristic value w: indicates a won position for player
• margin m: how much can opponent‘s response differ from his best alternative
• max(L): the maximum depth of the meaningful search tree
EXAMPLE 1
Black to move wins.
• the player has a wide range of winning moves at Level 5
• the opponent has very limited options
=> indicators that the problem is not difficult
EXAMPLE 1I
Black to move wins.
• the opponent has a wide range of winning moves at Level 4
• the player has to find a unique winning move after each one of them
=> indicators that the problem is not easy
reflect properties
of the DOMAIN
reflect properties
of the TREE
ATTRIBUTE DESCRIPTIONS
EXAMPLE II: MULTI-LEVEL ATTRIBUTE VALUES
indicators that
the problem is
difficult
EXPERIMENT 1: EVALUATION OF THE COMPUTATIONAL MODEL
• the aim: assess the utility of the meaningful search trees for estimating the difficulty
• 900 chess tactical problems with well established ChessTempo ratings:
difficulty
problems
avg. ChessTempo rating
st. dev
„hard“
300
2088.8
74.3
„medium“
300
1669.3
79.6
„easy“
300
1254.6
96.4
• chess engine STOCKFISH:
• d = 10 plies (the engine always found a solution to the problem)
• w = 200 centipawns, m = 50 centipawns, max(L) = 5
• 5 standard ML classifiers, 10-fold cross validation
• observed separately: all attributes, tree attributes, domain attributes
EXPERIMENT 1: RESULTS
• All classifiers were able to differentiate between easy, medium, and difficult
problems with a high level of accuracy.
• The performance remained almost the same when only TREE attributes (computable
from the structure of meaningful search trees, contain no domain-specific knowledge).
EXPERIMENT 2: CHESS EXPERTS’ DIFFICULTY ASSESSMENT
• How good are chess experts at estimating difficulty of chess tactical problems?
• 10 participants, chess experts (average FIDE ELO rating: 2098.5, σ = 93.6) were
presented with 14 chess tactical problems (classes „easy“, „medium“, and „difficult“).
examples
problem positions
The questionnaire is available at http://www.ailab.si/matej.
input form
solutions of examples
EXPERIMENT 2: RESULTS
• The average classification accuracy (CA): 0.53 (σ = 0.12).
• The highest CA was 0.71.
=> less than the expected accuracy by any of the 5 ML classifiers
• Several disagreements between the participants’ difficulty estimations were observed.
(e.g., in 9 out of 14 cases the problem was classified as any of the three class values)
CONCLUSIONS
• Our goal was to construct an automatic difficulty estimator of chess problems.
• The concept of a meaningful search tree was introduced.
(experienced player only has to search
a small part of the complete search tree!)
• The meaningful search trees can be constructed
automatically. (e.g. by a heuristic-search engine)
• Properties of meaningful search trees may contain certain information about the
difficulty of the problem for a human.
• In the experiments, the accuracy of classifications by computer models compared
very favorably with the accuracy of human experts.
Next step: estimating the difficulty of problems that require single-agent search.
Thank you
Matej Guid
[email protected]
http://www.ailab.si/matej