Transcript Assessment Matrix Workshop - Florida International University

Katherine Perez
Jacqueline Peña
Office of Planning and Institutional Effectiveness
January 2009
ASSESSMENT WORKSHOP
COLLEGE OF EDUCATION
WHY FOCUS ON ASSESSMENT?

SACS Accreditation (http://www.sacs.org)

Key SACS Deadlines:



September 10, 2009—Report due
March 8-12, 2010—Onsite Review
Key IE Deadlines:

February 1, 2009


June 1, 2009






Fall 2008 SLO matrices (with results and use of results)
Spring 2009 SLO matrices (with results and use of results)
08-09 PO matrices (with results and use of results)
08-09 Operational Objectives (with results and use of results)
Continuous improvement
Institutional Effectiveness
Sound, research-based and assessment-based decisions
BLANK SLO MATRIX
08-09
Link to the Mission:
Student Learning Outcome
Assessment Method
(Stated in Measurable Terms)
Results
(Data Summary and Analysis)
Use of Results for Improving Student Learning
http://w3.fiu.edu/irdata/portal/inst_effectiveness
CLEAR SACS GUIDELINES

Student learning outcomes (SLO) matrix
Undergraduate programs = 3 SLOs minimum
 Graduate programs = 3 SLOs minimum


Program outcomes (PO) matrix

All program = 3 POs
1 outcome per page
 Align each outcome with an assessment, set of
results, and a clear use of results
 Reliable and valid assessment of each outcome
 Appropriate use of results to enhance student
learning for the outcome

SLO MATRICES – MORE SACS GUIDELINES

There must be some difference among degree
programs.


The BA and BS in chemistry cannot have the same SLOS
and the same assessment procedures (i.e. artifacts).
Similar Programs?
Outcomes: Show different levels (Bloom’s Taxonomy)
 Assessments: Use different artifacts, rubrics, or criteria


Principle 3.6.1

The institution’s post-baccalaureate professional degree
programs, master’s and doctoral degree programs, are
progressively more advanced in academic content than its
undergraduate programs.
WRITING THE SLO

Criteria:




Can be observed and measured
Relates to student learning towards the end of the program
Reflects an important concept
Formula: Who + Action Verb + What




As Stewards of the Discipline, students will apply reading education pedagogical
and content knowledge and skills in a K-12 learning environment.
As Stewards of the Discipline, students will apply reading education pedagogical
and content knowledge and skills in a K-12 learning environment.
Students will analyze and reflect on students’ language abilities and develop
appropriate lesson plans to address their specific language needs.
Students will analyze and reflect on students’ language abilities and develop
appropriate lesson plans to address their specific language needs.
TECHNOLOGY SLO

Principle 3.4.12
 The
institution’s use of technology enhances
student learning and is appropriate for meeting the
objectives of its programs. Students have access to
and training in the use of technology. (Technology
use)
Required for all undergraduate programs
 Optional for 2007-2008 SLO matrices
 Required for 2008-2009 SLO matrices

SAMPLE TECHNOLOGY OUTCOMES

Too General:


Students will use information technology to gather and disseminate information.
More Specific

Students will be able to effectively demonstrate information technology skills by
locating and retrieving information on education topics and issues and
published research in education and related fields.

Students will write and present a capstone project that requires the use of Word,
Excel, PowerPoint, and information technology.
Action Verbs
[Who + Action Verb + What]
Knowledge
Comprehension
Application
Analysis
Evaluation
Synthesis
List
Summarize
Apply
Analyze
Assess
Combine
Define
Describe
Demonstrate Separate
Rank
Integrate
Describe
Interpret
Illustrate
Explain
Test
Modify
Identify
Predict
Solve
Connect
Measure
Create
Show
Distinguish
Examine
Classify
Judge
Design
Examine
Estimate
Modify
Arrange
Critique
Compose
State
Paraphrase
Classify
Compare
Discriminate
Prepare
Find
Compare
Calculate
Contrast
Support
Write
Label
Contrast
Compute
Infer
Hypothesize
Formulate
Infer
Construct
Order
Implement
Construct
SLO ACTIVITY
Graduate students will learn basic concepts in
their field.
 Students will do an oral presentation.
 Students will communicate effectively in an oral
format.
 Students will take all the courses in order.
 Graduate students will apply and analyze
various statistical concepts in an appropriate
quantitative study.

ASSESSING THE SLO
1.
Artifact


2.
Data Collection / Collection of the Artifact



3.
Where, when, how
Census versus sampling
Sampling technique (if applicable)
Criteria

4.
Paper or project
Presentation or behavioral observation
Minimum standards on a rubric/scale or the percentage of
correct items
Evaluation of the Artifact


Faculty panel or external evaluators (reliability)
Rubrics or embedded questions
DIRECT AND INDIRECT MEASURES
Common Direct Measures
•Standardized exams
•Exit examinations
•National or regional exams
•Portfolios
•Pre-tests and post-tests
•Locally developed exams
•Papers
•Oral presentations
•Behavioral observations
•Thesis/dissertation
Common Indirect Measures
•Surveys or questionnaires
•Student perception
•Alumni perception
•Employer perception
•Focus groups
•Interviews
•Student records
SAMPLE: SLO + ASSESSMENT
Student Learning Outcome
Assessment Methods
(Stated in Measurable Terms)
Undergraduate English students will
communicate effectively in written format.
During their final semester, in their senior
seminar course (PPP 4677), students will write
a 10-page research paper on a critical topic as
applied to an English, Caribbean, or American
novel.
20% of the students’ papers will be randomly
selected and evaluated by a faculty panel of 3
members using a rubric that measures:
(1) Grammar, (2) Essay Structure, (3)
Coherence of Argument, (4) Application of
MLA.
Students will achieve a 3 or better on a 4-point
rubric:
1 = novice
2 = apprentice
3 = practitioner
4 = expert
Use of Results for Improving Student Learning
Results
(Data Summary and Analysis)
REPORTING THE RESULTS

Summary of Results

Direct measures
 Test
items
 Performance as determined by rubrics

Indirect measures
 Surveys
and questionnaires
 Interview and focus group data

Format
Narrative
 Tables or charts


Every student learning outcome must have at least:
 One
set of results
 One student learning improvement strategy (use of results)
RESULTS ACTIVITY
75% of the students met the criteria for
success.
 Our students passed the dissertation defense
on the first attempt.
 All the students passed the national exam.
 75% of the students (n=15) achieved a 3 or
better on the 5 rubric categories for the
capstone course research paper.

SAMPLE: SLO + ASSESSMENT + RESULTS
Student Learning Outcome
Assessment Methods
(Stated in Measurable Terms)
Undergraduate English students will
communicate effectively in written format.
During their final semester, in their senior
seminar course (PPP 4677), students will write
a 10-page research paper on a critical topic as
applied to an English, Caribbean, or American
novel.
20% of the students’ papers will be randomly
selected and evaluated by a faculty panel of 3
members using a rubric that measures:
(1) Grammar, (2) Essay Structure, (3)
Coherence of Argument, (4) Application of
MLA.
Students will achieve a 3 or better on a 4-point
rubric:
1 = novice
2 = apprentice
3 = practitioner
4 = expert
Use of Results for Improving Student Learning
Results
(Data Summary and Analysis)
100 students took the seminar in 07-08.
20 papers were sampled.
100% met the minimum criterion for success
(i.e. scoring a 3 or better on all four
components).
1
2
3
4
G
0
0
12
8
E
0
0
10
10
C
0
0
15
5
M
0
0
20
0
71%
29%
MASTER’S THESIS – RESULTS TABLE
Frequency of Student Results for all Four Categories of the Research Paper
(n=20)
1
NOVICE
2
3
APPRENTICE PRACTITIONER
4
EXPERT
PERCENTAGE MEETING
CRITERIA
Grammar
2 (10%)
0 (10%)
8 (40%)
8 (40%)
80% (n=16) met the
criteria.
Essay
Structure
4 (20%)
0 (15%)
11 (55%)
2 (10%)
65% (n=13) met the
criteria.
Coherence of
Argument
2 (10%)
0 (35%)
10 (50%)
1 (5%)
55% (n=11) met the
criteria.
Application of
MLA Rules
3 (15%)
0 (25%)
12 (60%)
0 (0%)
60% (n=12) met the
criteria.
Criteria: Students will achieve a 3 or better on a 4-point rubric
on all five sections of the master’s level thesis.
EXPLAINING YOUR USE OF RESULTS
DO
DON’T
•DO focus on making specific
•DON’T focus on simply planning for
improvements based on faculty consensus. improvements or making improvements
without faculty feedback.
•DO address specific program
improvements that will impact student
learning.
•DON’T address assessment improvement
plans such as revising the rubric.
•DO use concrete ideas.
•DON’T write vague ideas.
•DO state strategies that are sustainable
and feasible.
•DON’T use strategies that are impossible
to complete in one year considering your
resources.
•DO use strategies that can improve the
•DON’T focus on only improving the
curriculum and help students learn outside curriculum.
of courses.
USE OF RESULTS ACTIVITY
Target met. Will continue to monitor.
 The faculty will meet and revise the three
introductory courses.
 Students demonstrating difficulty writing research
papers will be referred to the Academic Center for
Excellence.
 A larger sample will be obtained.
 We will revise the rubric and have a calibration
session with the faculty prior to evaluating the
student papers each semester.
 A capstone course will be created that emphasizes
research and thesis writing methods.

SAMPLE: COMPLETED SLO MATRIX
Student Learning Outcome
Assessment Methods
(Stated in Measurable Terms)
Results
(Data Summary and Analysis)
Undergraduate English students will
communicate effectively in written format.
100 students took the seminar in 07-08.
20 papers were sampled.
During their final semester, in their senior
seminar course (PPP 4677), students will write
a 10-page research paper on a critical topic as
applied to an English, Caribbean, or American
novel.
20% of the students’ papers will be randomly
selected and evaluated by a faculty panel of 3
members using a rubric that measures:
(1) Grammar, (2) Essay Structure, (3)
Coherence of Argument, (4) Application of MLA
rules.
Students will achieve a 3 or better on a 4-point
rubric:
1 = novice
2 = apprentice
3 = practitioner
4 = expert
100% met the minimum criterion for success
(i.e. scoring a 3 or better on all four
components).
1
2
3
4
G
0
0
12
8
E
0
0
10
10
C
0
0
15
5
M
0
0
20
0
71%
29%
Use of Results for Improving Student Learning
1. Undergraduate English students will be required to visit the Center for Academic Excellence once each semester to work with a writing tutoring
for at least one 60-minute session.
2. Undergraduate English students who have difficulty with grammar, essay structure, coherence of argument, or application of MLA rules at any
point during their academic program will be referred to a writing tutor at the Center for Academic Excellence.
3. Every undergraduate English course will require at least one 5-page paper that requires the use of MLA rules and principles.
THE SUMMARY PAGE

Overview and Comparisons:



Give an overview or summary of all the outcomes together
Discuss trends that you have seen over the years
Explanations


Provide qualitative explanations for poor results or
exceptionally high results
Notes and documentation affecting results
Response rate (e.g. Only 50% of the students completed the project.)
 Inter-rater reliability (e.g. 2 faculty members reviewed the artifacts
and the inter-rater reliability was only 60%.)


Assessment Improvement Plans


Revise or create instrument (e.g. artifact, rubric)
Modify assessment methods (e.g. data collection, sampling,
criteria, evaluation process)
PROGRAM OUTCOMES
Program-level outcomes
 Focus on student success (not student learning)


Formula = Who + Action Verb + What

Examples:
Graduates seeking employment in the field will find such
employment within 6 months of graduation.
Candidates will pass the FTCE and score higher than the
state average.
Program graduates will be satisfied with advising and
mentoring services.
SAMPLE: COMPLETED PO MATRIX
Program Outcome
(Stated in Measurable Terms)
Assessment Methods
Results
(Data Summary and Analysis)
Full-time students will graduate from the English
doctoral program within 7 years of program
admission.
The program administrative assistant will keep
track of students’ progress in the program from
date of admission, making updates for each
student at the end of each academic semester.
10 students graduated from the English doctoral
program during the 2008-2009 academic year.
Time of Completion for the 10 Students
5 students
6 years
2 students
7 years
2 students
8 years
1 student
9 years
The program assistant analyzed the completion
timeframe for each individual student and
discovered that all of the graduating students
spent over half of their graduate time working on
the dissertation (3-6 years).
Use of Results for Improving Student Learning
1. Doctoral advisors and dissertation chairs will be required to meet with their students twice a semester to go over the program of study, the students’
progress, and the students future goals and timelines.
2. The college will support the graduate student organization in developing dissertation work/study groups, for all incoming students in the program.
The dissertation groups should meet once a month to share drafts, establish new goals, create completion timelines, and do any other activities that
the group members find helpful for successful completion of the program.
3. Students will be mailed and emailed warning letters after 4 years in the program reminding them that they need to graduate within 7 years of program
admission.
THE IE TEAM (HTTP://W3.FIU.EDU/IRDATA/PORTAL/INST_EFFECTIVENESS)

Marta Perez


















Coordinator
[email protected].
305-348-1367
Computer Specialist
[email protected]
305-348-0115
Amanda Berhaupt




Computer Specialist
[email protected]
305-348-0115
Karla Felipe

Coordinator
[email protected]
305-348-1418
Jacqueline Peña
Mayelin Felipe

Coordinator
[email protected]
305-348-0459
Katherine Perez


Director
[email protected]
305-348-2733
Maria Corrales


Graduate Assistant
[email protected]
305-348-2731
Randhir Kaur



Graduate Assistant
[email protected]
305-348-2731
OUR ROLE AT IE AND IN FIU

Institutional Effectiveness

Improve student learning
 Student,
program, and operational levels
Improve efficiency and effectiveness of the university
 Assist university with the assessment process


Assessment Coordinators

Assist with the assessment process, including:
 Education
concerning assessment
 Articulation of outcomes and assessments
 Institutionalization of assessment practices
 Translation of successful assessment work for the SACS world
 Dissemination of assessment and accreditation information