Transcript Document
1
Closing the Loop: Assessing LMU’s Undergraduate Learning Outcomes
Laura J. Massa, Ph.D.
Director of Assessment
2
Please Note:
Please note:
The data provided in the following presentation are provided for the purposes of internal assessment and improvement. Findings should not be communicated to an external audience (e.g., website, bulletin or other publicity materials) without the expressed authorization of the Office of Assessment.
3 • Goals of the Presentation • Undergraduate Learning Outcomes • Closing the Loop • Sources of Evidence • Focus on 3 Learning Outcomes • Additional topics of interest from NSSE • What’s next?
Outline
4
Goals of the Presentation
LMU places a high value on teaching and learning, on providing a transformative educational experience, and on inquiry processes that focus on understanding and improving student learning.
–Guiding Principles for Assessment at LMU 1. Provide evidence of student achievement of selected outcomes 2. Engage you in process of closing the loop for the assessment of selected outcomes
5
Undergraduate Learning Outcomes
• Adopted in February 2010 • 2 ½ year development process • Endorsed by the Faculty Senate • Outcomes can be found at www.lmu.edu/assessment • Focusing on selected outcomes: • Creative and Critical Thinking • Respect for Others • Written and Oral Communication
6
Closing the Loop
Assessment: A systematic, ongoing process aimed at understanding and improving student learning
IDENTIFY SPECIFIC OUTCOMES DETERMINE PRACTICES USED TO ACHIEVE OUTCOMES GATHER EVIDENCE ARTICULATE MISSION/ GOALS RECOMMEND ACTIONS REVIEW & INTERPRET RESULTS
7
Sources of Evidence
•
Direct Measure:
Looks at student work products or performances that demonstrate learning • Rubric to assess Written Communication – Papers collected in Spring 2010 •
Indirect Measure:
Captures students’ perceptions of their learning and the educational environment that supports learning • National Survey of Student Engagement (NSSE) – Participated in Spring 2010
8
NSSE
• Assesses the extent to which students engage in educational practices associated with high levels of learning and development • NSSE items + Jesuit Consortium items align with our University Undergraduate Learning Outcomes • All LMU Freshmen and Seniors invited to participate # Institutions # Respondents Response Rate
LMU
1 856 35%
Jesuit
11 8,935 37%
Masters
122 96,847 31%
NSSE Total
563 360,611 32%
9
Creative and Critical Thinking
Students will be able to ask questions, solve problems and produce works through the innovation of ideas and concepts and by developing and justifying solutions through critical evaluation and analysis
10
Creative and Critical Thinking
100% To what extent has your experience at this institution contributed to your thinking critically and analytically?
1= very little, 4 = very much First Year Seniors
92% 92% 90% 90% 89% 87% 87% 82% 83% 80% 70% 60% 50% 40% 30% LMU Jesuit Masters NSSE Total
Based on NSSE data. Values in red refer to significant differences between mean responses of LMU students and corresponding students (p<.05).
11
Creative and Critical Thinking
100% To what extent has your experience at this institution contributed to your ability to solve complex real-world problems?
1= very little, 4 = very much First Year Seniors
90% 80% 70% 60% 50% 40% 30% 58% 72% 69% 71% 59% 63% 59% 64% LMU Jesuit Masters NSSE Total
Based on NSSE data. Values in red refer to significant differences between mean responses of LMU students and corresponding students (p<.05).
12
Creative and Critical Thinking
How much has your coursework emphasized:
Mean Response (1-4 scale; 1 = very little, 4 = very much) LMU Jesuit Masters NSSE Total
Analyzing
the basic elements of an idea, experience, or theory, such as examining a particular case or situation in depth and considering its components FY SR 3.37
3.46
3.33
3.43
3.10
3.26
3.15
3.29
Making judgments
about the value of information, arguments, or methods, such as examining how others gathered and interpreted data and assessing the soundness of their conclusions
Applying
theories or concepts to practical problems or in new situations FY SR FY 3.02
3.11
3.17
3.08
3.17
3.21
2.95
3.04
3.05
2.94
3.05
3.08
SR 3.34
3.38
3.24
3.25
Based on NSSE data. Means in red are significantly different from the mean responses of LMU students (p<.05)
13
Respect for Others
Students will demonstrate respect for individual and group difference in their interactions with others
14
3,5
Respect for Others
4 To what extent does your institution emphasize encouraging contact among students from different economic, social, and racial or ethnic backgrounds?
1= very little, 4 = very much First Year Seniors
3 2,85 2,83 2,93 2,74 2,74 2,72 2,57 2,57 2,5 2 1,5 1 LMU Jesuit Masters NSSE Total
Based on NSSE data. Means in red are significantly different from the mean responses of LMU students (p<.05)
15
Respect for Others
During the current school year, about how often have you done each of the following:
Mean Response (1-4 scale; 1 = never, 4 = very often) LMU Jesuit Masters NSSE Total Had serious conversations with students of a different race or ethnicity than your own FY SR Had serious conversations with students who are very different from you in terms of their religious beliefs, political opinions, or personal values FY SR 2.86
3.06
2.87
2.73
2.79
2.84
2.57
2.66
2.65
2.61
2.68
2.69
2.99
2.87
2.69
2.72
Based on NSSE data. Means in red are significantly different from the mean responses of LMU students (p<.05)
16
Respect for Others
To what extent has your experience at this institution contributed to your development in each of the following areas:
Demonstrating respect for others’ differences FY Mean Response (1-4 scale; 1 = very little, 4 = very much) LMU 3.16
Jesuit 3.22
SR 3.28
3.19
Actively working toward a more inclusive community FY 2.92
2.92
SR 2.92
2.82
Based on NSSE data. Means in red are significantly different from the mean responses of LMU students (p<.05)
17
Oral Communication
Students will effectively express information both in writing and orally using conventions and forms appropriate to the intended audience
18
Oral Communication
100% To what extent has your experience at this institution contributed to your ability to speak clearly and effectively?
1= very little, 4 = very much First Year Seniors
90% 77% 77% 80% 70% 69% 68% 69% 75% 67% 73% 60% 50% 40% 30% LMU Jesuit Masters NSSE Total
Based on NSSE data. Values in red refer to significant differences between mean responses of LMU students and corresponding students (p<.05).
19
During the current school year, how often have you done the following:
Made a class presentation FY
Oral Communication
Mean Response (1-4 scale; 1 = never, 4 = very often) LMU Jesuit Masters NSSE Total 2.35
2.26
2.32
2.29
2.89
2.86
2.79
SR Tutored or taught other students (paid or voluntary) FY SR 3.07
1.81
2.18
1.79
1.92
1.65
1.81
1.70
1.86
Based on NSSE data. Means in red are significantly different from the mean responses of LMU students (p<.05)
20
Written Communication
Students will effectively express information both in writing and orally using conventions and forms appropriate to the intended audience
21
Rubric for Written Communication
• Applied to a random sample of 75 final papers from multiple sections of Phil 320: Ethics • Teaching Fellows from the English Department hired and trained to apply rubric to papers • Each paper independently read by 2 Teaching Fellows BCLA 28%
Student authors from:
CBA CFA CSE 30.7% 14.7% 14.7% SFTV 12%
22 Context & Purpose Thesis/ Central Idea Organization & Coherence Support & Development Style Accomplished 4 Demonstrates a clear and effective understanding of context, audience, and purpose that is responsive to all elements of the assignment and focuses all elements of the work.
Thesis/central idea is clearly communicated, worth developing, and engaging.
Proficient 3 Demonstrates adequate consideration of context, audience, and purpose and a clear focus on the assignment.
Presents a thesis/central idea that can be developed. Developing 2 Demonstrates inconsistent awareness of context, audience, purpose. May not address all elements of the assignment.
States thesis/central idea that is weak, or too broad to be developed. Demonstrates minimal attention to context, audience, and purpose. Does not address assignment.
Attempted thesis/central idea is unclear Novice 1 Uses a logical structure appropriate to paper’s subject, purpose, and audience Sophisticated transitional sentences often develop one idea from the previous one or identify their logical relations. It guides the reader through the chain of reasoning or progression of ideas.
Uses appropriate, relevant, and compelling content to support ideas, convey understanding of the topic and shape the whole work. Uses words with precise meaning and an appropriate level of specificity. Sentences are varied, yet clearly structured and carefully focused, not long and rambling.
Shows a progression of ideas and uses fairly sophisticated transitional devices (e.g., may move from least to more important idea). Content is appropriate and relevant so that ideas are supported sufficiently. Work is generally shaped through support. Primarily uses words accurately and effectively. Sentences are primarily clear, well-structured, and focused, though some may be awkward or ineffective.
May list ideas or arrange them randomly rather than using any evident structure. May use transitions, but they are likely to be sequential (first, second, third) rather than logic based. Demonstrates use of supportive content but assumes that supportive content speaks for itself and needs no application to the point being discussed, or inconsistently supports ideas with content. Word choice is sometimes vague, imprecise, or inappropriate. Sentence structure is generally correct, but sentences may be wordy, unfocused, repetitive, or confusing.
May have random organization, lacking internal paragraph coherence and using few or inappropriate transitions. Often uses ineffective or inappropriate content (e.g., opinions, examples, or clichés) to support points, or offers little evidence of any kind. Misuses words; employs inappropriate language. Contains many awkward sentences; sentence structure is simple or monotonous. Mechanics Almost entirely free of spelling, punctuation and grammatical errors.
May contain some errors, which may annoy the reader but not impede understanding.
Contains several mechanical errors, which may temporarily confuse the reader but not impede overall understanding Contains either many mechanical errors or a few important errors that block the reader’s understanding and ability to see connection between thoughts.
23
Written Communication
Written Communication Rubric Scores
1= novice, 4 = accomplished Juniors (N=30) Seniors (N= 45) 4,00 3,50
3,18 3,22
3,00
3,05 3,04 3,15 2,93 3,13 3,14 3,10 3,01 3,32 3,18
2,50 2,00 1,50 1,00
Context & Purpose Thesis/Central Idea Organization & Coherence Support & Development Style Mechanics
24
15 43
Written Communication
Written Communication Rubric Scores by Frequency 6 9 5 12 17 47 43 51
4 3 2 1
38 47 22 21 18 17 0
Context & Purpose
0
Thesis/Central Idea
2
Organization & Coherence
2
Support & Development
17 2
Style
15 1
Mechanics
25
3,5 4
Written Communication
To what extent has your experience at this institution contributed to your ability to write clearly and effectively?
1 = very little, 4 = very much First Year Seniors
3,23 3,21 3,19 3,27 3.08
3,13 3.04
3,13 3 2,5 2 1,5 1 LMU Jesuit Masters NSSE Total
Based on NSSE data. Means in red are significantly different from the mean responses of LMU students (p<.05)
26
Written Communication
100% During the current school year, how often did you prepare two or more drafts of a paper before turning it in?
1= never, 4 = very often First Year Seniors
90% 80% 70% 60% 50% 40% 30% 76% 41% 56% 43% 61% 50% 58% 48% LMU Jesuit Masters NSSE Total
Based on NSSE data. Values in red refer to significant differences between mean responses of LMU students and corresponding students (p<.05).
27
During the current school year, how much reading and writing have you done?
Written Communication
1
= None,
2
= 1-4,
3
= 5-10,
4
= 11-20,
5
= More than 20 LMU Jesuit Masters NSSE Total Number of written papers or reports of 20 pages or more FY 1.45
1.24
1.33
1.32
1.79
1.71
1.64
1.65
SR Number of written papers or reports between 5 and 19 pages FY SR Number of written papers or reports of fewer than 5 pages FY SR 2.93
2.82
3.50
3.28
2.53
2.82
3.31
3.20
2.23
2.52
3.02
2.96
2.27
2.55
3.03
3.00
Based on NSSE data. Means in red are significantly different from the mean responses of LMU students (p<.05)
28 Academic Challenge
Additional Topics
29
Academic Challenge
How often have you worked harder than you thought you could to meet an instructors expectations?
1 = never, 4 = very often First Year
4,00
Seniors
3,50 3,00 2,79 2,89 2,79 2.80
2,75 2,82 2,73 2.79
2,50 2,00 1,50 1,00 LMU Jesuit Masters NSSE Total
Based on NSSE data. Means in red are significantly different from the mean responses of LMU students (p<.05)
30
Survey Question
About how often have you come to class without completing readings or assignments?
Mean Response (1-4 scale; 1 = never, 4 = very often) FY
Academic Challenge
LMU Jesuit Masters NSSE Total SR 2.07
2.27
2.01
2.11
1.95
2.03
1.99
2.09
About how many hours do you spend in a typical 7-day week preparing for class (studying, reading, writing, doing homework, rehearsing, and other academic activities)?
Percent stating 11 or more hours To what extent does your institution emphasize spending significant amounts of time studying and on academic work?
Mean Response (1-4 scale; 1 = very little, 4 = very much) FY SR FY SR 75% 61% 3.25
3.13
75% 67% 3.28
3.23
56% 57% 3.12
3.13
60% 61% 3.16
3.16
Based on NSSE data. Means in red are significantly different from the mean responses of LMU students (p<.05)
31
Academic Challenge
To what extent have your examinations during the current school year have challenged you to do your best
1 = very little, 7 = very much First Year Seniors
7,00 5,66 6,00 5,60 5,56 5,60 5,41 5,52 5,48 5,51 5,00 4,00 3,00 2,00 1,00 LMU Jesuit Masters NSSE Total
Based on NSSE data. Means in red are significantly different from the mean responses of LMU students (p<.05)
32 Student Satisfaction
Additional Topics
33
Student Satisfaction
4,00 How would you evaluate your entire experience at this institution?
1 = poor, 4 = excellent First Year Seniors
3,45 3,44 3,50 3,33 3,38 3,19 3,19 3,23 3,24 3,00 2,50 2,00 1,50 1,00 LMU Jesuit Masters NSSE Total
Based on NSSE data. Means in red are significantly different from the mean responses of LMU students (p<.05)
• • • • 34
Learning Outcomes Summary
• • Creative and Critical Thinking Clear improvement from 1 st to senior year LMU contributes to ability and coursework emphasizes this • • Respect for Others Clear improvement from 1 st to senior year LMU emphasizes this and behaviors reflect learning • • Oral Communication Clear improvement from 1 st to senior year LMU contributes to ability and behaviors reflect skill • • Written Communication 1 st year students report writing more and being engaged in writing more drafts of their papers than seniors Rubric scores indicate that more students are ‘developing’ than ‘accomplished’ in each of the 6 elements examined
35
What’s next?
1. Discuss evidence with faculty in your program: • www.lmu.edu/assessment : University Assessment Reports • Program assessment data on related learning outcomes 2. Recommend actions for improvement within your program • Possible types of actions: – Making improvements to pedagogy, assignments or curriculum – Planning for professional development – Allocation of resources 3. Make record of actions taken 4. Report actions to Office of Assessment
36