Transcript 360 Degree Academic Performance Assessment Model

Incorporating Authentic
Assessment in the Classroom
Narrowing the
Gulf Conference
March/April 2011
St. Petersburg College
Incorporating Authentic Assessment in the Classroom
2011
Presenters
 Dr. James Coraggio, Director, Academic
Effectiveness and Assessment, St. Petersburg
College
 Dr. Carol Weideman, Mathematics Professor,
Gibbs Campus, St. Petersburg College
March/April 2011
Narrowing the Gulf Conference
2
Assessment Basics
Incorporating Authentic Assessment in the Classroom
2011
Why do we assess?





To see how well we are doing
To confirm what we already know
To share our progress with others
To see where we can improve and change
In some cases to demonstrate what does not work
J
March/April 2011
Narrowing the Gulf Conference
3
Assessment Basics
Incorporating Authentic Assessment in the Classroom
2011
Why do we assess?
Source: http://www.c-pal.net/course/module2/pdf/Week1_Lesson5.pdf
March/April 2011
Narrowing the Gulf Conference
J
4
Purpose of an Assessment
Incorporating Authentic Assessment in the Classroom
2011
 “Clearly delineate between those that know the
content and those that do not.”
 To determine whether the student knows the
content, not whether the student is a good testtaker.
 Likewise, confusing and tricky questions
should be avoided to prevent incorrect
responses from students who know the
material.
March/April 2011
Narrowing the Gulf Conference
5
J
Types of Assessments
Incorporating Authentic Assessment in the Classroom
2011
 Objective assessments
 Authentic assessment
J
March/April 2011
Narrowing the Gulf Conference
6
Objective Assessments
Incorporating Authentic Assessment in the Classroom





2011
Measure several types of learning (also levels)
Wide content, short period of time
Variations for flexibility
Easy to administer, score, and analyze
Scored more reliability and quickly
C
March/April 2011
Narrowing the Gulf Conference
7
Types of Objective Tests
Incorporating Authentic Assessment in the Classroom
2011
 Written-response
 Completion (fill-in-the-blank)
 Short answer
 Selected-response




Alternative response (two options)
Matching
Keyed (like matching)
Multiple choice
C
March/April 2011
Narrowing the Gulf Conference
8
Assignment Example I
Incorporating Authentic Assessment in the Classroom
2011
 Multiple Choice Assessment
 Assignment Objectives
 Solving equations using addition and
multiplication principles
 Solving applied problems by translating to
equations
C
March/April 2011
Narrowing the Gulf Conference
9
Assignment Example I
Incorporating Authentic Assessment in the Classroom
2011
 Multiple Choice Assessment
1. Solve for x:
3(x + 8) = 4 (x – 4)
a. -8
b. 40
c. 8
d. -40
2.
Write the sentence as an algebraic equation:
The sum of 18 and a number is 5
a. x -18 = 5
c. 18 = x + 5
b. 5 + 18 = x
d. 18 + x = 5
C
March/April 2011
Narrowing the Gulf Conference
10
Issues with Objective Assessments
Incorporating Authentic Assessment in the Classroom




2011
Limited depth of content
Not able to reveal student misconceptions
Limited ability to test critical thinking skills
Students are ‘Test-wise’
C
March/April 2011
Narrowing the Gulf Conference
11
Test-wise Students
Incorporating Authentic Assessment in the Classroom






2011
Are familiar with item formats
Use informed and educated guessing
Avoid common mistakes
Have testing experience
Use time effectively
Apply various strategies to solve different
problem types
C
March/April 2011
Narrowing the Gulf Conference
12
Authentic Experiences
Incorporating Authentic Assessment in the Classroom
2011
 Is the course aligned with the expectations
for the student in the ‘real-world’?
 Authentic Learning
 Authentic Assessment
J
March/April 2011
Narrowing the Gulf Conference
13
Class Without Authentic Experiences
Incorporating Authentic Assessment in the Classroom
2011
 Didactic instruction where students are
presented with factual information from a
text book
 Assessment is primarily multiple choice
items where students are expected to
regurgitate factual information
J
March/April 2011
Narrowing the Gulf Conference
14
Class With Authentic Experiences
Incorporating Authentic Assessment in the Classroom
2011
 Interactive learning environment where
students not only learn facts but the
relationship between the facts and the
application of that information
 Authentic assessment where students are
able to model the applications of the
discipline through simulations, projects,
etc.
J
March/April 2011
Narrowing the Gulf Conference
15
Authentic Assessments
Incorporating Authentic Assessment in the Classroom
2011
 Authentic assessments serve dual purposes of
 encouraging students to think critically and
 providing assessment data for measuring improved
student learning.
 These assessment techniques fall into three
general categories:
 criterion-referenced rubrics,
 student reports (reflection or self-assessments), and
 student portfolios.
J
March/April 2011
Narrowing the Gulf Conference
16
Authentic Assessments
Incorporating Authentic Assessment in the Classroom
2011
Authentic assessments include…



Criterion-referenced rubrics. Complex, higher-order
objectives can be measured only by having students create a unique
product, whether written or oral [in-class essays, speeches, term
papers, videos, computer programs, blueprints, or artwork] (Carey,
2000).
Student Reflection. Written reflection is espoused to have
several important benefits: it can deepen the quality of critical
thinking, increase active involvement in learning, and increase
personal ownership of the new learning by the student (Moon,
1999).
Student Portfolios. Collections of students’ work over a course or
a program and can be an effective method of demonstrating student
progress in the area of critical thinking (Carey, 2000).
J
March/April 2011
Narrowing the Gulf Conference
17
Assignment Example II
Incorporating Authentic Assessment in the Classroom
2011
 Authentic Assessment: Painting a Room
 Assignment Objectives
 Solving Equations using addition and
multiplication principles
 Solving applied problems by translating to
equations
C
March/April 2011
Narrowing the Gulf Conference
18
Assignment Example II
Incorporating Authentic Assessment in the Classroom
2011
You have decided to paint your family room. Assume that one coat will be
sufficient. Paint is currently on sale at Home Depot for $15.99 per gallon
and one gallon will cover 400 square feet.
Your family room is 10 ft by 14 ft and the ceiling is 9 ft above the floor.
There are 3 windows, 30 in by 50 in and 2 doorways, 36 in by 7 ft high.
Answer the following questions:
How many square feet of walls
do you need to cover?
How much paint is needed?
How much will it cost?
Is there any additional information
that would be helpful?
C
March/April 2011
Narrowing the Gulf Conference
19
Assignment Example II
Incorporating Authentic Assessment in the Classroom
2011
A. Write an equation for the square feet of wall.
Let
x = square feet of wall Room is 10 ft wide x 12 ft long x 9 ft high.
2 walls that are 10 ft x 9 ft = 90 ft x 2 = 180 sq ft
y = Total cost for paint 2 walls that are 14 ft x 9 ft = 126 x 2 = 252 sq ft
Total Square feet = 432 sq. ft
Calculate the square footage.
BUT: we need to adjust for windows and doors
B. Using the given information and the equation from Part A, how much
paint do you need?
One gallon covers 400 sq ft; we need one gallon of paint
C. How much will the paint cost?
Are we painting the ceiling?
Do
wecost
needissupplies
Paint
$15.99
(brushes, drop clothes, etc)?
D. What other information would be helpful? Do we need to add sales tax
to paint cost? If so, what isC
rate?
March/April 2011
Narrowing the Gulf Conference
20
Assignment Example II
Incorporating Authentic Assessment in the Classroom
2011
A. Write an equation for the square feet of wall.
Let
x = square feet of wall 3 windows: 30 in x 50 in x 3= 4500/144 sq in
per sq ft = 31.25 sq ft
y = Total cost for paint 2 doorways: 3 ft by 7 ft = 21 sq ft x 2 = 42 sq ft
Calculate the square footage.
Total Square ft = 432 – 31.25 – 42 = 369.5 sq ft
B. Using the given information and the equation from Part A, how much
paint do you need?
One gallon covers 400 sq ft; we need one gallon of paint
C. How much will the paint cost?
Paint cost is $15.99
D. What other information would be helpful?
March/April 2011
Are we painting the ceiling?
Do we need supplies (brushes,
drop clothes, etc)?
Do we need to add sales tax to
paint cost? If so, what is rate?
C
Narrowing the Gulf Conference
21
Rubrics
Incorporating Authentic Assessment in the Classroom
2011
What is a rubric?
 Scoring guidelines, consisting of specific
pre-established performance criteria, used
in evaluating student work on
performance assessments
C
March/April 2011
Narrowing the Gulf Conference
22
Rubrics
Incorporating Authentic Assessment in the Classroom
2011
SPC currently uses rubrics in such
programs as…
 College of Education
 College of Nursing
 Paralegal Studies Program
C
March/April 2011
Narrowing the Gulf Conference
23
Rubric Development Process
Incorporating Authentic Assessment in the Classroom
1.
2.
3.
4.
5.
6.
7.
2011
Re-examine the learning objectives to be addressed by
the task
Identify specific observable attributes your students
should demonstrate
Describe characteristics at each attribute
Write narrative descriptions for each level of
continuum
Collect samples of student work
Repeat as
Score student work and identify samples that
Needed
exemplify various levels
Revise the rubric as needed
J
March/April 2011
Narrowing the Gulf Conference
24
Assignment Profile
Incorporating Authentic Assessment in the Classroom




2011
Designed to provide consistency and accuracy
as well as provide guidelines for the use
Rubric is an evaluation ‘tool’, but for a tool to be
effective it must be in the correct situation or
‘job.’ It would be inefficient to use a machete to
conduct heart surgery.
Rubric must be aligned to the most appropriate
course assignment
The instructor is the assessment instrument not
the rubric
J
March/April 2011
Narrowing the Gulf Conference
25
Assignment Example III
Incorporating Authentic Assessment in the Classroom
2011
 Rubric: STA2023 Sampling Project
 Assignment Objectives
 Identify the sampling strategies commonly
employed to collect data
 Describe potential biases encountered with
sampling strategies used in various statistical
applications
 Suggest strategies to avoid potential biases when
using sampling to collect data for a statistical
application
March/April 2011
Narrowing the Gulf Conference
J/C
26
Assignment Example III
Incorporating Authentic Assessment in the Classroom
2011
Management at a retail store is concerned about the possibility of drug
abuse by people who work there. They decide to check on the extent
of the problem by having a random sample of the employees undergo a
drug test. The lawyers for the retail store have assured the
management that there are no legal issues with the proposed drug
testing as long as the individual test results are not identified to a
specific employee. Depending the extent of illegal drugs identified in
the drug testing, drug counseling may be offered to all employees
under the promise of complete confidentiality.
You have been hired as the statistician who will design the sampling
plan. The budget for the drug testing will cover the cost of 40 drug
tests.
March/April 2011
Narrowing the Gulf Conference
J/C
27
Assignment Example III
Incorporating Authentic Assessment in the Classroom
2011
Management has proposed several different ideas about the best way
to obtain the random sample of 40 employees who will be drug tested.
There are currently 500 employees at this retail store. There are four
classifications of employees: supervisors, full-time sales clerks, parttime sales clerks and maintenance staff. These sampling possibilities
are listed below:
 Select one of the employee classifications and sample all
employees in that classification.
 Choose every fourth person who clocks in for each shift.
 Randomly select 10 employees from each classification.
 Each employee has a three-digit employee number. Randomly
select 40 employees.
J/C
March/April 2011
Narrowing the Gulf Conference
28
Assignment Example III
Incorporating Authentic Assessment in the Classroom
2011
Answer the following questions regarding this scenario:
 Define this problem in your own words.
 Compare and contrast the four proposed sampling plans.
 Select one of the proposed sampling plan that you feel is most
appropriate for this situation and defend your choice.
 Describe any weaknesses in your selected sampling plan.
 Make suggestions on ways to improve/strengthen the sampling plan.
You may include information not described in the scenario above.
 Reflect on your own thought process after completing the
assignment.
“What did you learn from this process?”
J/C
“What would you do differently next time to improve?”
March/April 2011
Narrowing the Gulf Conference
29
Assessment Rubric for CT
Performance
Exemplary
Proficient
Developing
Incorporating
Authentic
Assessment
in the Classroom
Element
(4)
(3)
(2)
Emerging
(1)
Not Present
(0)
2011
I. Communication
Define problem in
your own words.
Identifies the main idea or
problem with numerous
supporting details and
examples which are
organized logically and
coherently.
Identifies the main
idea or problem with
some supporting
details and examples
in an organized
manner.
Identifies the main
idea or problem with
few details or
examples in a
somewhat organized
manner.
Identifies the main idea
or problem poorly with
few or no details or
states the main idea or
problem verbatim from
the text.
Does not
identify the
main idea or
problem.
II. Analysis
Compare & contrast
the available
solutions.
Uses specific inductive or
deductive reasoning to
make inferences
regarding premises;
addresses implications
and consequences;
identifies facts and
relevant information
correctly.
Uses logical
reasoning to make
inferences regarding
solutions; addresses
implications and
consequences;
Identifies facts and
relevant information
correctly.
Uses superficial
reasoning to make
inferences regarding
solutions; Shows
some confusion
regarding facts,
opinions, and
relevant, evidence,
data, or information.
Makes unexplained,
unsupported, or
unreasonable inferences
regarding solutions;
makes multiple errors in
distinguishing fact from
fiction or in selecting
relevant evidence.
Does not
analyze
multiple
solutions.
III. Problem Solving
Select & defend your
final solution.
Thoroughly identifies and
addresses key aspects of
the problem and
insightfully uses facts and
relevant evidence from
analysis to support and
defend potentially valid
solutions.
Identifies and
addresses key aspects
of the problem and
uses facts and
relevant evidence
from analysis to
develop potentially
valid conclusions or
solutions.
Identifies and
addresses some
aspects of the
problem; develops
possible conclusions
or solutions using
some inappropriate
opinions and
irrelevant information
from analysis.
Identifies and addresses
only one aspect of the
problem but develops
untestable hypothesis;
or develops invalid
conclusions or solutions
based on opinion or
irrelevant information.
Does not select
and defend a
solution.
March/April 2011
Narrowing the Gulf Conference
30
Assessment Rubric for CT
Performance
Exemplary
Proficient
Developing
Incorporating
Authentic
Assessment
in the Classroom
Element
(4)
(3)
(2)
Emerging
(1)
Not Present
(0)
2011
IV. Evaluation
Identify weaknesses in
your final solution.
Insightfully interprets
data or information;
identifies obvious as well
as hidden assumptions,
establishes credibility of
sources on points other
than authority alone,
avoids fallacies in
reasoning; distinguishes
appropriate arguments
from extraneous
elements; provides
sufficient logical support.
Accurately interprets
data or information;
identifies obvious
assumptions,
establishes credibility
of sources on points
other than authority
alone, avoids fallacies
in reasoning;
distinguishes
appropriate arguments
from extraneous
elements; provides
sufficient logical
support.
Makes some errors in
data or information
interpretation; makes
arguments using weak
evidence; provides
superficial support for
conclusions or
solutions.
Interprets data or
information
incorrectly;
Supports conclusions
or solutions without
evidence or logic;
uses data,
information, or
evidence skewed by
invalid assumptions;
uses poor sources of
information; uses
fallacious arguments.
Does not evaluate
data, information,
or evidence
related to final
solution.
V. Synthesis
Suggest ways to
improve/strengthen
your final solution.
Insightfully relates
concepts and ideas from
multiple sources; uses
new information to
enhance final solution;
recognizes missing
information; correctly
identifies potential effects
of new information.
Accurately relates
concepts and ideas
from multiple
sources; uses new
information to
enhance final
solution; correctly
identifies potential
effects of new
information.
Inaccurately or
incompletely relates
concepts and ideas
from multiple
sources; shallow
determination of
effect of new
information on final
solution.
Poorly integrates
information from
more than one source
to support final
solution; Incorrectly
predicts the effect of
new information on
final solution.
Does not identify
new information
for final solution.
March/April 2011
Narrowing the Gulf Conference
31
Assessment Rubric for CT
Performance
Exemplary
Proficient
Developing
Incorporating
Authentic
Assessment
in the Classroom
Element
(4)
(3)
(2)
VI. Reflection
Reflect on your own
thought process.
“What did you learn
from this process?”
“What would you do
differently next time to
improve?”
Identifies strengths and
weaknesses in own
thinking: recognizes
personal assumptions,
values and perspectives,
compares to others’, and
evaluates them in the
context of alternate points
of view.
March/April 2011
Identifies strengths
and weaknesses in
own thinking:
recognizes personal
assumptions, values
and perspectives,
compares to others’,
with some
comparisons of
alternate points of
view.
Identifies some
personal assumptions,
values, and
perspectives;
recognizes some
assumptions, values
and perspectives of
others; shallow
comparisons of
alternate points of
view.
Narrowing the Gulf Conference
Emerging
(1)
Identifies some
personal assumptions,
values, and
perspectives;
does not consider
alternate points of
view.
Not Present
(0)
2011
Does not reflect
on own thinking.
32
Questions
Incorporating Authentic Assessment in the Classroom
March/April 2011
Narrowing the Gulf Conference
2011
33
Assessment Basics
Incorporating Authentic Assessment in the Classroom
2011
 Alignment of course objectives
 Competency, clarity, bias, level of difficulty
 Validity and Reliability
C
March/April 2011
Narrowing the Gulf Conference
34
Assessment Basics
Incorporating Authentic Assessment in the Classroom
2011
Alignment
 Everything needs to align (objectives
through assessment)
C
March/April 2011
Narrowing the Gulf Conference
35
Competency
Incorporating Authentic Assessment in the Classroom
2011
 Items should test for the appropriate or
adequate level of knowledge, skill, or ability
(KSA) for the students.
 Assessing lower division students on graduate
level material is an ‘unfair’ expectation.
 The competent student should do well on an
assessment, items should not be written for only
the top students in the class.
C
March/April 2011
Narrowing the Gulf Conference
36
Clarity
Incorporating Authentic Assessment in the Classroom






2011
Clear, precise item and instruction
Correct grammar, punctuation, spelling
Address one single issue
Avoid extraneous material (teaching)
One correct or clearly best answer
Legible copies of exam
C
March/April 2011
Narrowing the Gulf Conference
37
Bias
Incorporating Authentic Assessment in the Classroom
2011
Tests should be free from bias…
No stereotyping
No gender bias
No racial bias
No cultural bias
No religious bias
No political bias
C
March/April 2011
Narrowing the Gulf Conference
38
Level of Difficulty
Incorporating Authentic Assessment in the Classroom
2011
 Ideally, test difficulty should be aimed a
middle level of difficulty. This can not
always be achieved when the subject
matter is based on specific expectations
(i.e, workforce area).
J
March/April 2011
Narrowing the Gulf Conference
39
Trivial and Trick Questions
Incorporating Authentic Assessment in the Classroom
2011
 Avoid trivia and tricks
 Avoid humorous or ludicrous responses
 Items should be straight forward, they
should cleanly delineate those that know
the material from those that do not
 Make sure every item has value and that it
is contributing to the final score
J
March/April 2011
Narrowing the Gulf Conference
40
Assessment Basics
Incorporating Authentic Assessment in the Classroom
2011
Does one size fit all?
 Assessments need to be valid
 Assessments need to be reliable
J
March/April 2011
Narrowing the Gulf Conference
41
Validity
Incorporating Authentic Assessment in the Classroom
2011
Does the assessment measure what it
is suppose to measure?
 “Validation is the process of accumulating
evidence that supports the
appropriateness of inferences that are
made of student responses…” (AERA,
APA, & NCME, 1999)
J
March/April 2011
Narrowing the Gulf Conference
42
Types of Validity Evidence
Incorporating Authentic Assessment in the Classroom




2011
Content Related - the extent to which a student’s
responses to a given assessment reflect that student’s
knowledge of the content area
Construct Related - the extent to which the responses
being evaluated are appropriate indicators of the
underlying construct
Criterion Related - the extent to which the results of
the assessment correlate with a current or future event
Consequential – the consequences or use of the
assessment results
J
March/April 2011
Narrowing the Gulf Conference
43
Reliability
Incorporating Authentic Assessment in the Classroom
2011
Consistency of the assessment scores
 Types of reliability…



Interrater Reliability – scores vary from
instructor to instructor.
Intrarater Reliability – scores vary from a
single instructor from paper to paper
A test can be reliable and not valid, but
never valid and not reliable
March/April 2011
Narrowing the Gulf Conference
J
44
Reliability Concerns
Incorporating Authentic Assessment in the Classroom
2011
Reliability
1. Are the score categories well defined?
2. Are the differences between the score
categories clear?
3. Would two independent raters arrive at
the same score for a given student
response based on the scoring rubric?
J
March/April 2011
Narrowing the Gulf Conference
45
Improving Scoring Consistency
Incorporating Authentic Assessment in the Classroom





2011
Provide grading rubrics or scoring
criteria to students prior to assessment
Grade papers anonymously
Use anchor papers to define levels of
proficiency for reference
Use multiple scorers
Calculate reliability statistics during
training and grading
J
March/April 2011
Narrowing the Gulf Conference
46
Assessment Basics
Incorporating Authentic Assessment in the Classroom



2011
Multiple Measures
Always to good to implement multiple
measures when possible
Ideally direct and indirect measures of
competency
C
March/April 2011
Narrowing the Gulf Conference
47
Indirect Methods
Incorporating Authentic Assessment in the Classroom
2011
 “indirect measures …help deepen the
interpretation of student learning” (Maki,
2004).
 SSI is a good example of an indirect
measure.
C
March/April 2011
Narrowing the Gulf Conference
48
Incorporating Authentic
Assessment in the Classroom
Narrowing the
Gulf Conference
March/April 2011