Transcript Assessment Matrix Workshop - Florida International University

Katherine Perez
Jacqueline Peña
Office of Planning and Institutional Effectiveness
January 2009
ASSESSMENT MATRIX WORKSHOP
OVERVIEW OF WORKSHOP

SLO matrix

SLOs and Corresponding Assessment Methods
 Workshop:

Reporting Results and Use of Results
 Workshop:

Revising SLOs and Assessment Methods
Writing the Results and Use of Results
PO matrix
POs and Corresponding Assessment Methods
 Reporting Results and Use of Results

 Workshop:
Revising POs and Assessment Methods + Writing
Results and Use of Results

Additional individualized assistance
WHY FOCUS ON ASSESSMENT?

SACS Accreditation

Key SACS Deadlines:
 September
10, 2009—Report due
 March 8-12, 2010—Onsite Review

Key IE Deadlines:
 February
1, 2009—Fall 2008 SLO matrices due
 June 1, 2009—Spring 2009 SLO and PO matrices due
Continuous improvement
 Institutional Effectiveness
 Sound, research-based and assessment-based
decisions

BLANK SLO MATRIX
08-09
Link to the Mission:
Student Learning Outcome
Assessment Method
(Stated in Measurable Terms)
Use of Results for Improving Student Learning
Results
(Data Summary and Analysis)
WRITING THE SLO

Criteria:
Can be observed and measured
 Relates to student learning towards the end of the
program
 Reflects an important concept


Formula: Who + Action Verb + What
Students will identify and apply the key components of
a learning system.
 Students will identify and apply the five key
components of a learning system.

TECHNOLOGY SLO

Principle 3.4.12
 The
institution’s use of technology enhances
student learning and is appropriate for meeting the
objectives of its programs. Students have access to
and training in the use of technology. (Technology
use)
Required for all undergraduate programs
 Optional for 2007-2008 SLO matrices
 Required for 2008-2009 SLO matrices

SAMPLE TECHNOLOGY OUTCOMES

General (not appropriate for SACS):


More Specific:


Students will use information technology to gather and disseminate information.
Graduates of the program will demonstrate the ability to use appropriate
computer technology, software, and the Internet to complete their capstone
research papers.
Much more Specific (appropriate for SACS):


Students will be able to effectively demonstrate information technology skills by
locating and retrieving information on economic topics and issues, published
research in Economics and related fields, and/or by finding information about
the generation, construction, and meaning of economic data for their final
research projects.
Students will write and present a capstone project that requires the use of Word,
Excel, PowerPoint, and information technology.
Action Verbs
[Who + Action Verb + What]
Knowledge
Comprehension
Application
Analysis
Synthesis
Evaluation
List
Summarize
Apply
Analyze
Combine
Assess
Define
Describe
Demonstrate Separate
Integrate
Rank
Describe
Interpret
Illustrate
Explain
Modify
Test
Identify
Predict
Solve
Connect
Create
Measure
Show
Distinguish
Examine
Classify
Design
Judge
Examine
Estimate
Modify
Arrange
Compose
Critique
State
Paraphrase
Classify
Compare
Prepare
Discriminate
Find
Compare
Calculate
Contrast
Write
Support
Label
Contrast
Compute
Infer
Formulate
Hypothesize
Infer
Construct
Order
Construct
Implement
Note: This table reflects the original Bloom’s Taxonomy Table. The modified version places
evaluation as the fifth column and synthesis (creating) as the sixth column.
ASSESSING THE SLO
1.
2.
Artifact
Collection of the artifact
 Where,
when, and how
 Census versus sampling
3.
Criteria
 Minimum
standards on a rubric/scale or the
percentage of correct items
4.
Evaluation of the artifact
 Faculty
panel or external evaluators (reliability)
 Rubrics, embedded questions, etc.
DIRECT AND INDIRECT MEASURES
Common Direct Measures
•Standardized exams
•Exit examinations
•Portfolios
•Pre-tests and post-tests
•Locally developed exams
•Papers
•Oral presentations
•Behavioral observations
•Thesis/dissertation
Common Indirect Measures
•Surveys or questionnaires
•Student perception
•Alumni perception
•Employer perception
•Focus groups
•Interviews
•Student records
SAMPLE: SLO + ASSESSMENT
Student Learning Outcome
Assessment Method
(Stated in Measurable Terms)
Doctoral students will identify, define, and
apply key literary theories (including formalism,
historicism, mimesis, post-colonialism,
structuralism, post- structuralism,
psychoanalysis, modernism, and
postmodernism) to their specialization areas in
the field of English literature.
After completing all their courses, English
literature doctoral students will take the theory
comprehensive exam, which will be divided into
five key literary theories (one theory per exam
question) that the students learn throughout
their coursework.
All comprehensive exams will be evaluated by
a faculty panel of 3 members using a
department rubric.
For each of the five literary theories, students
will achieve a 3 or better on a 4-point rubric:
1 = poor
2 = acceptable
3 = good
4 = excellent
Use of Results for Improving Student Learning
Results
(Data Summary and Analysis)
REPORTING THE RESULTS

Summary of Results

Direct measures



Indirect measures



Surveys and questionnaires
Interview and focus group data
Format



Test items
Performance as determined by rubrics
Narrative
Tables or charts
Analysis of results

If pertinent, explain results in a narrative form by interpreting results or
using qualitative analysis of the data.

Every student learning outcome must have at least:


One set of results
One student learning improvement strategy (use of results)
RESULTS ACTIVITY
All the students passed the assessment.
 75% of the students met the criteria for
success.
 Our students passed the dissertation defense
on the first attempt.
 75% of the students (n=15) achieved a 3 or
better on all the rubric categories for the
capstone course research paper.

MASTER’S THESIS – RESULTS TABLE
Frequency of Student Results for all Four Categories of the Research Paper
(n=20)
1
NOVICE
2
3
APPRENTICE PRACTITIONER
4
EXPERT
PERCENTAGE MEETING
CRITERIA
Grammar
2 (10%)
0 (10%)
8 (40%)
8 (40%)
80% (n=16) met the
criteria.
Essay
Structure
4 (20%)
0 (15%)
11 (55%)
2 (10%)
65% (n=13) met the
criteria.
Coherence of
Argument
2 (10%)
0 (35%)
10 (50%)
1 (5%)
55% (n=11) met the
criteria.
Application of
MLA Rules
3 (15%)
0 (25%)
12 (60%)
0 (0%)
60% (n=12) met the
criteria.
Criteria: Students will achieve a 3 or better on a 4-point rubric
on all five sections of the master’s level thesis.
SAMPLE: SLO + ASSESSMENT + RESULTS
Student Learning Outcome
Assessment Method
(Stated in Measurable Terms)
Doctoral students will identify, define, and
apply key literary theories (including formalism,
historicism, mimesis, post-colonialism,
structuralism, post- structuralism,
psychoanalysis, modernism, and
postmodernism) to their specialization areas in
the field of English literature.
After completing all their courses, English
literature doctoral students will take the theory
comprehensive exam, which will be divided into
five key literary theories (one theory per exam
question) that the students learn throughout
their coursework.
All comprehensive exams will be evaluated by
a faculty panel of 3 members using a
department rubric.
For each of the five literary theories, students
will achieve a 3 or better on a 4-point rubric:
1 = poor
2 = acceptable
3 = good
4 = excellent
Use of Results for Improving Student Learning
Results
(Data Summary and Analysis)
20 students took the theory comprehensive
exam during the 2008-2009 academic year.
70% of the students achieved the minimum
criterion (i.e. scoring a 3 or better on all five
theory components).
20% of the students scored less than 3 on at
least 1 theory component.
10% of the students (n=2) scored less than a 3
on all five theory components and had to be
rescheduled to retake the test during the
summer during the summer B session.
EXPLAINING YOUR USE OF RESULTS
DO
DON’T
•DO focus on making specific
•DON’T focus on simply planning for
improvements based on faculty consensus. improvements or making improvements
without faculty feedback.
•DO address specific program
improvements that will impact student
learning.
•DON’T address assessment improvement
plans such as revising the rubric.
•DO use concrete ideas.
•DON’T write vague ideas.
•DO state strategies that are sustainable
and feasible.
•DON’T use strategies that are impossible
to complete in one year considering your
resources.
•DO use strategies that can improve the
•DON’T focus on only improving the
curriculum and help students learn outside curriculum.
of courses.
USE OF RESULTS ACTIVITY






Target met. Will continue to monitor.
The faculty will meet and revise the three introductory
courses.
A larger sample will be obtained.
We will revise the rubric and have a calibration session
with the faculty prior to evaluating the student papers
each semester.
A capstone course will be created that emphasizes
research and thesis writing methods.
A student resource center will be created with computer
equipment, writing tutors, and statisticians who can
assist our students with course- and thesis-related
writing and research.
SAMPLE: COMPLETED SLO MATRIX
Student Learning Outcome
(Stated in Measurable Terms)
Assessment Method
Doctoral students will identify, define, and apply
key literary theories (including formalism,
historicism, mimesis, post-colonialism,
structuralism, post- structuralism,
psychoanalysis, modernism, and postmodernism)
to their specialization areas in the field of English
literature.
After completing all their courses, English
literature doctoral students will take the theory
comprehensive exam, which will be divided into
five key literary theories (one theory per exam
question) that the students learn throughout their
coursework.
All comprehensive exams will be evaluated by a
faculty panel of 3 members using a department
rubric.
For each of the five literary theories, students will
achieve a 3 or better on a 4-point rubric:
1 = poor
2 = acceptable
3 = good
4 = excellent
Results
(Data Summary and Analysis)
20 students took the theory comprehensive exam
during the 2008-2009 academic year.
70% of the students achieved the minimum
criterion (i.e. scoring a 3 or better on all five
theory components).
20% of the students scored less than 3 on at
least 1 theory component.
10% of the students (n=2) scored less than a 3
on all five theory components and had to be
rescheduled to retake the test during the summer
during the summer B session.
Use of Results for Improving Student Learning
1. The department will assist faculty in organizing student-led comprehensive exam study groups a year prior to the expected examination
deadline.
2. A theory unit will be added to each literature course so that students are constantly exposed to multiple literary theories throughout their
coursework.
3. In addition, the department will create and distribute a study list that includes major topics covered on the comprehensive exams,
useful websites for tutorials and reviews, and a bibliography of texts that are critical for success with the five theoretical concepts.
SAMPLE: COMPLETED SLO MATRIX
Student Learning Outcome
Assessment Methods
(Stated in Measurable Terms)
Results
(Data Summary and Analysis)
Undergraduate English students will
communicate effectively in written format.
100 students took the seminar in 07-08.
20 papers were sampled.
During their final semester, in their senior
seminar course (PPP 4677), students will write
a 10-page research paper on a critical topic as
applied to an English, Caribbean, or American
novel.
20% of the students’ papers will be randomly
selected and evaluated by a faculty panel of 3
members using a rubric that measures:
(1) Grammar, (2) Essay Structure, (3)
Coherence of Argument, (4) Application of MLA
rules.
Students will achieve a 3 or better on a 4-point
rubric:
1 = novice
2 = apprentice
3 = practitioner
4 = expert
100% met the minimum criterion for success
(i.e. scoring a 3 or better on all four
components).
1
2
3
4
G
0
0
12
8
E
0
0
10
10
C
0
0
15
5
M
0
0
20
0
71%
29%
Use of Results for Improving Student Learning
1. Undergraduate English students will be required to visit the Center for Academic Excellence once each semester to work with a writing tutoring
for at least one 60-minute session.
2. Undergraduate English students who have difficulty with grammar, essay structure, coherence of argument, or application of MLA rules at any
point during their academic program will be referred to a writing tutor at the Center for Academic Excellence.
3. Every undergraduate English course will require at least one 5-page paper that requires the use of MLA rules and principles.
THE SUMMARY PAGE

Overview and Comparisons:



Give an overview or summary of all the outcomes together
Discuss trends that you have seen over the years
Explanations


Provide qualitative explanations for poor results or
exceptionally high results
Notes and documentation affecting results
Response rate (e.g. Only 50% of the students completed the project.)
 Inter-rater reliability (e.g. 2 faculty members reviewed the artifacts
and the inter-rater reliability was only 60%.)


Assessment Commentary


Revise or create instrument (e.g. artifact, rubric)
Modify assessment methods (e.g. data collection, sampling,
criteria, evaluation process)
ACTIVITY: RESULTS AND USE OF RESULTS

Write or revise your Results and Use of Results.

Results:


Use of Results:


Clearly state the results of each PO assessment (table or narrative).
Explicitly state how you will use the results to improve the program
during the following assessment cycle/year.
Summary Page:
1.
2.
3.
4.
5.
6.
Overview of results and use of results (looking at the big picture)
Trends over the last year or several years
Comparison to national or state trends or data
Explanations for poor or unexpected results
Notes or documentation that could explain the results
Assessment improvement plans
END OF THE SLO PORTION
What Is Next?
10-minute overview of Program Outcomes
 PO writing assistance
 Continued SLO writing assistance

WRITING THE PO
Program-level outcomes
 Focus on student success (not student learning)


Formula = Who + Action Verb + What

Examples:
Graduates seeking employment in the field will find such
employment within 6 months of graduation.
Graduates will be satisfied with advising and mentoring
services.
Faculty will publish at least one journal article or book
chapter every two years.
SAMPLE: COMPLETED PO MATRIX
Program Outcome
(Stated in Measurable Terms)
Assessment Method
Results
(Data Summary and Analysis)
Full-time students will graduate from the English
doctoral program within 7 years of program
admission.
The program administrative assistant will keep
track of students’ progress in the program from
date of admission, making updates for each
student at the end of each academic semester.
10 students graduated from the English doctoral
program during the 2008-2009 academic year.
Time of Completion for the 10 Students
5 students
6 years
2 students
7 years
2 students
8 years
1 student
9 years
The program assistant analyzed the completion
timeframe for each individual student and
discovered that all of the graduating students
spent over half of their graduate time working on
the dissertation (3-6 years).
Use of Results for Improving Student Learning
1. Doctoral advisors and dissertation chairs will be required to meet with their students twice a semester to go over the program of study, the students’
progress, and the students future goals and timelines.
2. The college will support the graduate student organization in developing dissertation work/study groups, for all incoming students in the program.
The dissertation groups should meet once a month to share drafts, establish new goals, create completion timelines, and do any other activities that
the group members find helpful for successful completion of the program.
3. Students will be mailed and emailed warning letters after 4 years in the program reminding them that they need to graduate within 7 years of program
admission.
ACTIVITY: PROGRAM OUTCOMES

Take 15 minutes to review or write at least one program outcome
with corresponding assessment methods, results, and use of results.

PO Formula:
Who + Action Verb + What

Assessment Methods Formula:
1. State the instrument/method/tool (e.g. survey, national exam results).
2. Explain the data collection steps (e.g. when, where, how).
3. State the success criteria (minimum standards for success).

Results:


Clearly state the results of each PO assessment (table or narrative).
Use of Results:

Explicitly state how you will use the results to improve the program
during the following assessment cycle/year.
THE IE TEAM

Marta Perez


















Coordinator
[email protected].
305-348-1367
Computer Specialist
[email protected]
305-348-0115
Amanda Berhaupt




Computer Specialist
[email protected]
305-348-0115
Karla Felipe

Coordinator
[email protected]
305-348-1418
Jacqueline Peña
Mayelin Felipe

Coordinator
[email protected]
305-348-0459
Katherine Perez


Director
[email protected]
305-348-2733
Maria Corrales


Graduate Assistant
[email protected]
305-348-2731
Randhir Kaur



Graduate Assistant
[email protected]
305-348-2731