Transcript Document

Evaluation of IR systems
By Barbara Otchere
7/16/2015
Evaluation
1
Presentation Outline
 The
importance of evaluation to
IR.
 Relevance profiling within
documents.
 Retrieval performance
measurement
 Why Google is much preferred.
 Conclusion.
7/16/2015
Evaluation
2
How does evaluation works in IR?

Evaluation means assessing performance
or a value of a system, process,
technique, procedure, product or policy.
 Evaluation can be seen as an important as
well as difficult part of information
retrieval.
 Relevance within documents is very
important. Relevance in this sense means
how well a document satisfies a user's
information need.

Measurement of relevance.
7/16/2015
Evaluation
3
Evaluation and Retrieval Strategy
Methods in IR evaluation.
 The elements of evaluation
 How to measure instruments in IR
evaluation.

7/16/2015
Evaluation
4
The Fundamental Measures





Recall and Precision are the standard
retrieval performance measures.
Recall: Proportion of relevant items that
are retrieved. Recall can hardly be
measured in the Web.
|A  R| / |R|
Precision: Proportion of items retrieved
that are relevant.
|A  R| / |A|
7/16/2015
Evaluation
5
Diagrammatical view of
Precision/Recall

Below is a diagrammatic view of the documentation collection which represents recall and
precision. This document collection is partitioned by each answer.
B = Relevant/not retrieved
A = Relevant retrieved
D =Non-relevant / not retrieved
C = Non-relevant / retrieved
7/16/2015
Evaluation
6
Contingency Table for
Recall/Precision.
Relevant
R
Not Relevant
~R
Retrieved A
AR
A  ~R
Not retrieved
~A
~A  R
~A  ~R
7/16/2015
Evaluation
7
What makes Google unique from the
other Search Engines?



: What is different about Google?
Google is distinguished by its ranking algorithm
based on how many good sites link to each site,
along with other factors like the proximity of the
search keywords or phrases in the documents. It
claims not only to use the number of other links,
but also the importance of the other links (where
they are linked to, qualitatively -- based on
directories, it seems).
PageRanking has been one of the best technique
for Google.
7/16/2015
Evaluation
8
Conclusion

Evaluation is an important part of IR.
 Relevance as being criteria for
Precision/Recall has become the preferred
pair of measures of IR evaluation studies
on the processing level.
 The most successful engines follows the
simple algorithms of Precision/Recall
7/16/2015
Evaluation
9
Questions for discussion 1
 Why
do we evaluate? There are
different reasons why we evaluate.
We evaluate for both economic and
social reasons. We also evaluate for
users’ satisfaction.
7/16/2015
Evaluation
10
Questions for discussion 2
How do we evaluate?
•By measurement of relevance.
Relevance is a very important issue as
evaluation of IR is concern.
7/16/2015
Evaluation
11