Transcript L643: Evaluation of Information Systems

S519: Evaluation of
Information Systems
Analyzing data:
value and importance
Ch6+7
Last week


How to infer causation: 8 strategies?
How to put them together?
„Values“ in evaluation (DCh6)


Adding value to descriptive data to make our
evaluation explicit
Our goal



Using quantitative value to evaluate the quality or
value of the evaluand in a particular context.
Build up our conclusions based on a level of
certainty
What are values:

„good“, „valuable“, „worthwhile“
„Values“ in evaluation

Adding „values“ to descriptive data collected
about


Process, outcomes, costs, comparisons,
exportabilities
Weighting all the strengthens or weaknesses
of these values to draw overall conclusion
about the evaluand. How



Importance weighting
Merit determination
synthesis
methodology
Subjective

Before we go for methodology, we have to
answer the question:

Whether our data are subjective
Three types of subjectives
(Scriven, 1991)

Subjective 1: Inappropriate application of personal or
cultural preferences/biases:


Subjective 2: assessment or interpretation by a person,
rather than a machine, of something external to the
person



arbitrary, idiosyncratic, unreliable, highly personal (i.e., based
purely on personal preference, cultural biases, gender biases)
Using well-founded expert judgements
Robust evidence
Subjective 3: about a person‘s inner life or experiences
(e.g., headaches, fears, beliefs, emotions, stress)

Usually not independently verifiable
Avoid Subjective



We provide our conclusion baesed on
certainty in the relevant decision-making
context
Keep the whole evaluation well documented
and justified
All evaluations, especially high-stakes ones,
should be meta-evaluated (i.e., evaluation
itself should be evaluated)
Exercise

Suppose that a client does not like the
findings of your evaluation and says:




„Well, that is just your opinion“
„evaluations are always just so subjective“
How could you response?
Form a group and discuss
Exercise

Do you agree or disagree:



All evaluative claims (about the value of certain outcomes or
attributes, their relative importance, and what mixed results
indicate about overall value) are arrivated at „subjectively“; more
specifically, the values that are applied to descriptive facts (data)
to arrive at evaluative conclusions are personal values.
Allowing stakeholders to make up their own minds, either
individually or collectively, is the only valid way in which
evaluative conclusions can be drawn at all.
Form a group with the same opinion and debates with
another group holding opposite opinion.
Determining importance (Dch7)

Importance determiniation is the process of
assigning labels to dimensions or components to
indicate their importance.

Importance weighting



Prioritize improvements
Identify whether identified strengths or weakness are serious
or minor
Work out whether an evaluand with mixed results is doing
fairly well, quite poorly, or somewhere in between.
Determining importance (Dch7)

Different evaluations



Dimensional evaluation
 Looking at multiple dimensions of merit that pertain to the evaluand
as a whole rather than separately to its parts.
Component evaluation
 Looking at each of the evaluand‘s components (or parts) separately
and then synthesizing these findings to draw conclusion about the
evaluand as a whole.
 Each component can be evaluated on several dimensions that
pertain to this component only rather than to the evaluand as a
whole.
Holistic evaluation

Looking evaluation as a whole without division into dimensions or
components
Determining the importance of
dimensions
Determining the importance of
componentss
Determining importance


Weak performance on minor criteria (e.g.
dimensions, components) may be no big
deal,
But weak performance on important criteria
can be very serious issues.
When to use what

Component analysis

Evaluating policies, programs, or interventions
that have several quite distinct parts



An international program consisting of projects
implemented in different locations (e.g. „WIC“ in IU)
A government policy includes multiple policy
measurements (e.g. Juvenile delinquency)
An organizational transformation includes several
distinct interventions (e.g. Career support)
When to use what

Dimensional evaluation

Entities whose quality or value is experienced by
consumers on multiple dimensions that pertain to
the evaluand as a whole


Single-component program or intervention
Product evaluation (i.e. Car evaluation)
When to use what

Holistic evaluation


Unusual in the evaluation of programs, policies
and other large complex evaluands.
More common in personnel, product and service
evaluation (expertise-oriented evaluation)





Judging the overall quality of a sample of writing
Grading essays
Classroom teaching
Athletic performance
cosmetics
Determining importance: 6
strategies

1. having stakeholders or consumers „vote“
on importance



Commonly used in both participatory and
nonparticipatory evaluations
Collecting opinions from everybody
Assumptions




Each person is well informed
Stakeholder‘s belief what (s)he chooses is important
Stakeholder‘s important should be treated equally
Pros and cons?
Determining importance: 6
strategies

2. Drawing on the knowledge of selected
stakeholders



Using selelcted stakeholder input to guide the
assignment of importance weightings
Collecting opinions from selected experts
Setting up the Bars

A bar is a defined minimum level of critierion
performance below which the evaluand is considered
completely unacceptable, regardless of performance
on other criteria.
Determining importance: 6
strategies

2. Drawing on the knowledge of selected
stakeholders

Assumptions:



The stakeholders should be sufficiently well informed
to provide valuable relevant information
The combination of stakeholder input wil provide
sufficient certainty about importance for the given
decision-making context
Prons and cons?
Determining importance: 6
strategies

3. Using evidence from the literature




Literature review
Evaluations of similar evaluations in similar contexts
Research documenting the key drivers (or strongest predicators)
of success or failure with this type of evaluand.
Assumptions



The volume and quality of the available research is sufficient to judge the
importance
The context of other research is sufficiently similiar to yours and therefore
that the findings can be reasonably applied to your setting
Prons and cons?
Determining importance: 6
strategies

4. Using specialist judgment




When you have tight timeline, no time for
gathering stakeholders and looking for literature
Identify one or two (or two or more) well-known
specialists in the domain
Better be supplemented with other evidence
Prons and cons?
Determining importance: 6
strategies

5. Using evidence from the needs and values
assessments

Determining the importance of criteria
(dimensions)


Still remember needs and value assessment (see
Table 3.3)?
How to get importance from Table 3.3?



Any frequently mentioned characteristics?
Looking for poor-performing evaluators that cause serious
problem
Looking for top-notch evaluators that have dramatic impacts
on success
Determining importance: 6
strategies

5. Using evidence from the needs and values
assessments

Determining the importance of components





Severity of dysfunction addressed (primary consideration)
Scarcity of alternatives: no other options for addressing the
need.
Intent to use alternatives: if the evaluand component in
question did not exist.
Rubics to measure (Table7.4, 7.5 and 7.6, 7.7, 7.8
(combined))
Prons and cons?
Exercise

Join your project group


Discuss Table 7.8
Think about how to draw similar table for your
group project
Determining importance: 6
strategies

6. Using program theory and evidence of
causal linkages

When criteria or components are linked to needs through a
complex logic chain.



Such as „soft“ skills or attributes (e.g., inspirational leadership, selfesteem, stress management, a kind of instrumental needs)
More upstream variables (see Exhibit 7.5)
How to estimate the strengths of the links



Interview
Anaylse your previous data
...
Strengths and weaknesses of
the six strategies


Always think whether they are applicable
Choose mulitple of them
Exercise


Table 7.10 (class dissusion)
Form a group


Discuss which strategies you will choose to
determine the importance for the “student
services in the school health program” (see Table
7.8)
Discuss which strategies you will choose to
determine the importance for your group project