Transcript Program Design & Evaluation Ethics and Values Every profession has distinct ethical obligations to the public.

Program Design & Evaluation
Ethics and Values
Every profession has distinct ethical obligations to the public. These
obligations include professional competency, integrity, honesty,
confidentiality, objectivity, public safety, and fairness.
All of these obligations are intended to preserve
and safeguard public confidence
Stem cell field rocked by scam of star scientist
The star scientist at the heart of that deception –
Hwang Woo Suk -- resigned Friday from his
university post, his meteoric rise to fame coming
to a wrenching end.
12/24/05
Unfortunately, all too often we hear reports in the
media of moral dilemmas and unethical behavior
by professionals. These reports naturally receive
considerable attention by the public, whose
confidence in the profession is undermined with
each report.
Educators are not immune to unethical practices
Houston, Texas
Testing Scandal
in Texas Schools
NPR News
March 21, 2005
by Claudia Sanchez
Existing Ethical Standards
• Concerns about ethical practices in assessment
are not new.
• As early as 1972, the National Council on
Measurement in Education (NCME), the
Association for Measurement and Evaluation in
Guidance (AMEG), and the American
Counseling Association (ACA) developed a
position paper on the responsible use of tests
that was intended to ensure that tests are given,
and examinees are treated, fairly and wisely
(AMEG, 1972).
Testing & Evaluation Standards
• Later in the 1970s, ACA developed a
statement on the responsibilities of the
users of standardized tests and evaluation
• That document was revised as recently as
1989 (ACA, 1989).
Assessment Code of Ethics
• Both of these early documents recognized
the need to positively influence the
practices of those who use tests in ways
that promote responsible use.
• These statements have been followed by
the development of ethical standards by a
number of other organizations having an
interest, or directly involved, in testing
assessment, and evaluation.
What About Evaluation Ethics?
Principles & Standards
for
Program Evaluators
American National Standards Institute
ANSI
The American National Standards Institute (ANSI) is
a private, non-profit organization (501(c)3) that
administers and coordinates the U.S. voluntary
standardization and conformity assessment system.
The Institute's mission is to enhance both the global
competitiveness of U.S. business and the U.S. quality
of life by promoting and facilitating voluntary
consensus standards and conformity assessment
systems, and safeguarding their integrity.
Founded in 1918
Evaluation Standards
JOINT COMMITTEE ON STANDARDS FOR
EDUCATIONAL EVALUATION
Created in 1975, the Joint Committee is a coalition of major professional
associations concerned with the quality of evaluation. The Joint Committee
is housed at The Evaluation Center, Western Michigan University.
The Joint Committee has published three sets of standards for evaluations:
1. The Personnel Evaluation Standards published in 1988
2. The Program Evaluation Standards published in 1994
3. The Student Evaluations Standards was published in 2003
The Joint Committee is accredited by the American National Standards
Institute (ANSI). Standards approved by ANSI become American National
Standards.
Four Program Evaluation Standards
“…as it is written, so it shall be done!




Utility Standards
Feasibility Standards
Propriety Standards
Accuracy Standards
Utility Standards (7)


To ensure that the evaluation will serve the information
needs of intended users
U1 Stakeholder Identification Persons involved in or
affected by the evaluation should be identified, so that
their needs can be addressed.
U2 Evaluator Credibility The persons conducting the
evaluation should be both trustworthy and competent to
perform the evaluation, so that the evaluation findings
achieve maximum credibility and acceptance.
Utility Standards


U3 Information Scope and Selection
Information collected should be broadly
selected to address pertinent questions about
the program and be responsive to the needs
and interests of clients and other specified
stakeholders
U4 Values Identification The perspectives,
procedures, and rationale used to interpret
the findings should be carefully described, so
that the bases for value judgments are clear.
Utility Standards


U5 Report Clarity Evaluation reports should
clearly describe the program being evaluated,
including its context, and the purposes,
procedures, and findings of the evaluation, so
that essential information is provided and
easily understood
U6 Report Timeliness and Dissemination
Significant interim findings and evaluation
reports should be disseminated in a timely
manner.
Utility Standards

U7 Evaluation Impact Evaluations should be
planned, conducted, and reported in ways that
encourage follow-through by stakeholders, so
that the likelihood that the evaluation will be
used is increased, and to ensure that the
evaluation will serve the information needs of
intended users
Feasibility Standards (3)
To ensure that an evaluation will be realistic, prudent,
diplomatic, and frugal
F1 Practical Procedures The evaluation procedures should
be practical, to keep disruption to a minimum while needed
information is obtained.
Feasibility Standards

F2 Political Viability The evaluation should be
planned and conducted with anticipation of the
different positions of various interest groups, so that
their cooperation may be obtained, and so that
possible attempts by any of these groups to curtail
evaluation operations or to bias or misapply the
results can be averted or counteracted.

F3 Cost Effectiveness The evaluation should be
efficient and produce information of sufficient value,
so that the resources expended can be justified
Propriety Standards (8)
Intended to ensure that an evaluation will be conducted
legally, ethically, and with due regard for the welfare of
those involved in the evaluation, as well as those affected
by its results.
P1 Service Orientation Evaluations should be designed to assist
organizations to address and effectively serve the needs of the
full range of targeted participants.
P2 Formal Agreements Obligations of the formal parties to an
evaluation (what is to be done, how, by whom, when) should
be agreed to in writing, so that these parties are obligated to
adhere to all conditions of the agreement or formally to
renegotiate it.
Propriety Standards



P3 Rights of Human Subjects Evaluations should be
designed and conducted to respect and protect the rights and
welfare of human subjects.
P4 Human Interactions Evaluators should respect human
dignity and worth in their interactions with other persons
associated with an evaluation, so that participants are not
threatened or harmed.
P5 Complete and Fair Assessment The evaluation should be
complete and fair in its examination and recording of strengths
and weaknesses of the program being evaluated, so that
strengths can be built upon and problem areas addressed.
Propriety Standards



P6 Disclosure of Findings The formal parties to an evaluation
should ensure that the full set of evaluation findings along with
pertinent limitations are made accessible to the persons
affected by the evaluation and any others with expressed legal
rights to receive the results.
P7 Conflict of Interest Conflict of interest should be dealt
with openly and honestly, so that it does not compromise the
evaluation processes and results.
P8 Fiscal Responsibility The evaluator's allocation and
expenditure of resources should reflect sound accountability
procedures and otherwise be prudent and ethically responsible,
so that expenditures are accounted for and appropriate
Accuracy Standards
Intended to ensure that an evaluation will reveal and convey
technically adequate information about the features that
determine worth or merit of the program being evaluated.
A1 Program Documentation The program being evaluated
should be described and documented clearly and accurately, so
that the program is clearly identified.
A2 Context Analysis The context in which the program exists
should be examined in enough detail, so that its likely
influences on the program can be identified.
A3 Described Purposes and Procedures The purposes and
procedures of the evaluation should be monitored and
described in enough detail, so that they can be identified and
assessed
Accuracy Standards
A4 Defensible Information Sources The sources of information
used in a program evaluation should be described in enough
detail, so that the adequacy of the information can be assessed.
A5 Valid Information The information-gathering procedures
should be chosen or developed and then implemented so that
they will assure that the interpretation arrived at is valid for the
intended use.
A6 Reliable Information The information-gathering procedures
should be chosen or developed and then implemented so that
they will assure that the information obtained is sufficiently
reliable for the intended use.
Accuracy Standards
A7 Systematic Information The information collected,
processed, and reported in an evaluation should be
systematically reviewed, and any errors found should be
corrected.
A8 Analysis of Quantitative Information Quantitative
information in an evaluation should be appropriately and
systematically analyzed so that evaluation questions are
effectively answered.
A9 Analysis of Qualitative Information Qualitative information
in an evaluation should be appropriately and systematically
analyzed so that evaluation questions are effectively answered.
Accuracy Standards
A10 Justified Conclusions The conclusions reached in an
evaluation should be explicitly justified, so that stakeholders
can assess them.
A11 Impartial Reporting procedures should guard against
distortion caused by personal feelings and biases of any party
to the evaluation, so that evaluation reports fairly reflect the
evaluation findings.
A12 Meta-evaluation The evaluation itself should be
formatively and summatively evaluated against these and other
pertinent standards, so that its conduct is appropriately guided
and, on completion, stakeholders can closely examine its
strengths and weaknesses.
What is Meta-Evaluation?
Simply stated, meta-evaluation is the evaluation of an
evaluation, evaluation system or evaluation device to
enhance the validity of the evaluation results.
Operationally, metaevaluation is also defined as the
process of delineating, obtaining, and applying descriptive
information and judgmental information - about the utility,
feasibility, propriety, and accuracy of an evaluation and its
systematic nature, competent conduct, integrity/honesty,
respectfulness, and social responsibility - to guide the
evaluation and/or report its strengths and weaknesses.
Four Program Evaluation Standards




Utility Standards – serve the information needs
of the intended users
Feasibility Standards – realistic, prudent,
diplomatic, and frugal
Propriety Standards – legal, ethical, and due
regard for welfare of participants
Accuracy Standards –reveal and convey
information to determine the worth, value, and
merit of the program evaluated
American Evaluation Association
Five Guiding Principles for Evaluators

Ratified by the AEA membership, July 2004

Supports Four Program Evaluation Standards
Principle 1 Systematic Inquiry
Evaluators conduct systematic, data-based inquiries
1. To ensure the accuracy and credibility of the evaluative
information they produce, evaluators should adhere to the
highest technical standards appropriate to the methods they
use.
2. Evaluators should explore with the client the shortcomings
and strengths both of the various evaluation questions and the
various approaches that might be used for answering those
questions.
3. Evaluators should communicate their methods and
approaches accurately and in sufficient detail to allow others
to understand, interpret and critique their work..
Principle 2 Competence
Evaluators provide competent performance to stakeholders



1. Evaluators should possess the education, abilities, skills and
experience appropriate to undertake the tasks proposed in the
evaluation.
2. To ensure recognition, accurate interpretation and respect
for diversity, evaluators should ensure that the members of the
evaluation team collectively demonstrate cultural competence.
3. Evaluators should practice within the limits of their
professional training and competence, and should decline to
conduct evaluations that fall substantially outside those limits.
4. Evaluators should continually seek to maintain and improve
their competencies, in order to provide the highest level of
performance in their evaluations.
Principle 3 Integrity/Honesty: Evaluators must display
honesty and integrity in their own behavior and in the
evaluation process.




1. Evaluators should negotiate honestly with clients and relevant
stakeholders concerning the costs, tasks to be undertaken,
limitations of methodology, scope of results likely to be obtained,
and uses of data resulting from a specific evaluation.
2. Before accepting an evaluation assignment, evaluators should
disclose any roles or relationships they have that might pose a
conflict of interest with their role as an evaluator.
3. Evaluators should record all changes made in the originally
negotiated project plans, and the reasons why the changes were
made.
4. Evaluators should be explicit about their own, their clients',
and other stakeholders' interests and values concerning the
conduct and outcomes of an evaluation.
Principle 3 Integrity/Honesty (continued)
 5. Evaluators should not misrepresent their
procedures, data or findings.
 6. If evaluators determine that certain
procedures or activities are likely to produce
misleading evaluative information or
conclusions, they have the responsibility to
communicate their concerns and the reasons for
them. If discussions with the client do not
resolve these concerns, the evaluator should
decline to conduct the evaluation.
 7. Evaluators should disclose all sources of
financial support for an evaluation, and the
source of the request for the evaluation.
Principle 4 Respect for People: Evaluators respect the
security, dignity and self-worth of respondents, program
participants, clients, and other evaluation stakeholders.

1. Evaluators should seek a comprehensive understanding of the important
contextual elements of the evaluation.

2. Evaluators should abide by current professional ethics, standards, and
regulations regarding risks, harms, and burdens that might befall those
participating in the evaluation; regarding informed consent for participation in
evaluation; and regarding informing participants and clients about the scope
and limits of confidentiality.

3. Because justified negative or critical conclusions from an evaluation must
be explicitly stated, evaluations sometimes produce results that harm client
or stakeholder interests. Under this circumstance, evaluators should seek to
maximize the benefits and reduce any unnecessary harms that might occur,
provided this will not compromise the integrity of the evaluation findings.
Principle 4 Respect for People (continued)



4. Knowing that evaluations may negatively affect the interests
of some stakeholders, evaluators should conduct the evaluation
and communicate its results in a way that clearly respects the
stakeholders' dignity and self-worth.
5. Where feasible, evaluators should attempt to foster social
equity in evaluation, so that those who give to the evaluation may
benefit in return. For example, evaluators should seek to ensure
that those who bear the burdens of contributing data and
incurring any risks do so willingly. Program participants should be
informed that their eligibility to receive services does not hinge on
their participation in the evaluation.
6. Evaluators have the responsibility to understand and respect
differences among participants, such as differences in their
culture, religion, gender, disability, age, sexual orientation and
ethnicity, and to account for potential implications of these
differences when planning, conducting, analyzing, and reporting
evaluations.
Principle 5
Responsibilities for General and Public Welfare
Evaluators articulate and take into account the diversity of
general and public interests and values that may be related to the
evaluation.
 1. When planning and reporting evaluations, evaluators
should include relevant perspectives and interests of the full
range of stakeholders.
 2. Evaluators should consider not only the immediate
operations and outcomes of whatever is being evaluated, but
also its broad assumptions, implications and potential side
effects
Principle 5
Responsibilities for General and Public Welfare
(continued)


3. Freedom of information is essential in a democracy. Evaluators
should allow all relevant stakeholders access to evaluative information
in forms that respect people and honor promises of confidentiality. In
all cases, evaluators should strive to present results clearly and simply
so that clients and other stakeholders can easily understand the
evaluation process and results.
4. Evaluators should maintain a balance between client needs and other
needs. Evaluators necessarily have a special relationship with the client
who funds or requests the evaluation. By virtue of that relationship,
evaluators must strive to meet legitimate client needs whenever it is
feasible and appropriate to do so. However, that relationship can also
place evaluators in difficult dilemmas when client interests conflict
with other interests, or when client interests conflict with the obligation
of evaluators for systematic inquiry, competence, integrity, and respect
for people.
Principle 5
Responsibilities for General and Public Welfare
(continued)

5. Evaluators have obligations that encompass the public
interest and good. These obligations are especially important
when evaluators are supported by publicly-generated funds;
but clear threats to the public good should never be ignored in
any evaluation. Because the public interest and good are rarely
the same as the interests of any particular group (including
those of the client or funder), evaluators will usually have to
go beyond analysis of particular stakeholder interests and
consider the welfare of society as a whole.
Summary of 5 Guiding Evaluation Principles





1. Systematic Inquiry: Evaluators conduct systematic, databased inquiries about whatever is being evaluated.
2. Competence: Evaluators provide competent performance to
stakeholders.
3. Integrity/Honesty: Evaluators ensure the honesty and
integrity of the entire evaluation process.
4. Respect for People: Evaluators respect the security, dignity
and self-worth of the respondents, program participants, clients,
and other stakeholders with whom they interact.
5. Responsibilities for General and Public Welfare:
Evaluators articulate and take into account the diversity of
interests and values that may be related to the general and
public welfare.
Program Design & Evaluation
Standards and Principles

Four Program Evaluation Standards
Utility, Feasibility, Propriety, & Accuracy

Five Guiding Principles for Evaluators
Systematic Inquiry, Competence, Integrity/Honesty,
Respect for People, Responsibilities for General &
Public Welfare
Program Design & Evaluation
Standards
Four Program
Evaluation Standards
UFPA
Utility,
Feasibility,
Propriety,
Accuracy
Principles
Five Guiding Principles
for Evaluators
SCIRR
Systematic Inquiry,
Competence,
Integrity/Honesty,
Respect for People,
Responsibilities for
General & Public
Welfare
PROGRAM
DESIGN
&
EVALUATION
PROCESS
Define Purpose
Engage
Stakeholders
5 Guiding Principles
4 Standards
Report
Evaluation
Ensure Use &
Share Lessons
Describe
Program
Evaluation
5 Principles
6 Steps
Analyze Data
Justify
Conclusions
5 Guiding Principles
Collect the
Data & Gather
Evidence
Focus Design,
Develop
Methodology,
Data Collection
Plan
When conducting program evaluations, we must always
remember the words of wisdom coined by a famous scientific
researcher and evaluator:
“Not everything that can be counted counts…and
not everything that counts can be counted.”
Einstein