Sarah Murnen, Kenyon College Barbara Andereck, Ohio Wesleyan University (Nancy Grace, College of Wooster) F ACULTY U SE OF A NALYSIS TO P RIMARY A SSESS C T RAIT REATIVE T HINKING

F UNDED BY THE T EAGLE F OUNDATION

   

Grant to the “Ohio-5” Colleges administered by the College of Wooster 2006-2009 “Creativity and Critical Thinking: Assessing the Foundations of a Liberal Arts Education” Organized by Lori Bettison-Varga, former Associate Provost at the College of Wooster, and then Nancy Grace, Professor of English at the College of Wooster http://www.wooster.edu/teagle

How Do You Define Creativity?

S TARTING O UR P ROJECT

 Year 1:   Assembled groups of faculty across four colleges in the Ohio-5 consortium of colleges to talk about evaluating creative and critical thinking among students Developed survey for students and faculty, asked about traits defining critical and creative thinking

Creative Thinking Critical Thinking Openness to Novelty Idea Generation Curiosity Invention Imagination Elaboration Synthesis, integration & combination Complexity Evaluation & assessment Categorization & classification Reasoning through logic Reasoning by metaphor & analogy Abstraction & simplification Identification Interpretation Divergent thinking Awareness of Environment Analysis Description Playfulness

How Can We Best Assess Creative Thinking in Students?

P RIMARY T RAIT A NALYSIS

• • • Identify the “primary traits” – essential or central components of the discipline (both knowledge and skills) to be learned by the student Build a scale for scoring the student’s performance on the trait Evaluate the student’s performance against those criteria

P RIMARY T RAIT A NALYSIS Resources

: Presentation by Dr. Douglas Eder on “Primary Trait Analysis” Presentation by Barbara Walvoord on the use of rubrics for effective grading Walvoord, B. E., P& Anderson, J. A. (1998). Effective Grading: A tool for learning and assessment. Jossey Bass: San Francisco, CA.

S TAGES IN C ONSTRUCTING A R UBRIC From: Stevens & Levi (2005). Introduction

to rubrics: An assessment tool to save grading time, convey effective feedback,

and promote student learning. Stylus Publishing: Sterling, Virginia.

Reflecting –

• • • • What do we want for our students and from our students? (Learning objectives) Why did we create the assignment?

What happened the last time we used the assignment? What are our expectations for the assignment?

S TAGES IN C ONSTRUCTING A R UBRIC

Listing – Focus on particular details of the assignment and what specific learning objectives we hope to see completed (Sometimes helps to imagine the best and the worst performance on the assignment) Grouping and Labeling the goals together Organize the results of our reflections in first two stages, grouping similar expectations together in what will probably become the dimensions of the rubric Application – Transfer groupings to a rubric grid; define assessment categories

G ROUP D ISCUSSION R UBRIC

HTTP :// WWW .

MASHELL .

COM /~ PARR 5/ TECHNO / GROUP .

HTML Measures Emerging (1), Competent (2), and Exemplary (3) Performance on Each Trait LISTENING 1. Recognizes and responds to others speaking.

2. Uses and practices listening processes regularly.

3. Habitually uses listening processes.

NON-VERBAL COMMUNICATION Eye contact, gestures, posture, facial expression, voice.

1. Comprehends some information from non-verbal cues.

2. Draws accurate conclusions from body language and facial expressions.

3. Able to recognize and use subtle non-verbal communication cues.

CO-OPERATION 1. Sometimes shows ability to wait to give appropriate verbal / non-verbal responses.

2. Usually shows ability to wait to give appropriate verbal / non-verbal responses.

3. Habitually shows ability to wait with openness and awareness to give appropriate verbal / non-verbal responses.

PARTICIPATION Tells thoughts, feelings, ideas so others understand.

1. Rarely talks during the discussion or talk is off the subject. Offers few ideas to the discussion.

2. Shares freely and explains with details. Makes connections to what others say.

3. Talk inspires others. Supports and leads others in discussion.

D EVELOPING T RAIT A R UBRICS THROUGH NALYSIS P RIMARY



Year 1:

 Faculty developed rubrics to measure creative and critical thinking for their courses or departments 

Year 2:

  Taught other faculty to develop and use rubrics, and examined the rubrics they created along with student performance data from the rubrics Developed a “generic” rubric using PTA designed to measure creative and critical thinking

“GENERIC RUBRIC”

CRITICAL

AND

CREATIVE

THINKING TRAITS

1. Elements of Argumentation a. Explanation b. Analysis c. Evaluation d. Interpretation e. Logic 2. Domain and Disciplinary Knowledge 3. Synthesis and Connections 4. Abstract Thinking 5. Complexity of Thought 6. Ideas Generated a. Fluency b. Flexibility

“GENERIC RUBRIC”

CRITICAL

AND

CREATIVE

THINKING TRAITS

7. Completeness/Coherence 8. Elegance 9. Divergent Thinking 10. Novelty or Uniqueness a. Germinal b. Original c. Transformational 11. Engagement 12. Risk Taking

“GENERIC RUBRIC”

CRITICAL AND CREATIVE THINKING RATING SCALE DESCRIPTIONS

6 = Trait is evident to an exceptionally high degree 5 = Trait is evident to a high degree 4 = Trait is evident to an intermediate degree 3 = Trait is somewhat evident 2 = Trait is barely evident 1 = Trait is not evident

What Assignments/ Pedagogies Promote Creative Thinking?

F ACULTY U SE G ENERIC R UBRIC



Year 3:

 Faculty across many disciplines measured student performance using the generic rubric in 31 classes (N = 444 students) across three of the participating colleges (College of Wooster, COW; Kenyon College, KC; and Ohio Wesleyan University, OWU)

YEAR 3: KEY FINDINGS

  Creative and critical thinking can be assessed in students, and we found improvement of performance on many creative and critical thinking traits among the students we studied (N = 444 students across 31 courses). The traits measured most frequently across all disciplines were

complete

and

original

measured in 90% of students, followed by

knowledge

measured in 85% of students. The traits least likely to be measured included abstract,

logic ,

and

novel

. There were some differences across disciplines in the traits that were emphasized; for example, faculty teaching in interdisciplinary courses were more likely to examine complexity, and they shared with fine arts faculty a greater concern for engagement and

risk .

YEAR 3: KEY FINDINGS

   In the longitudinal data measuring change in student performance across a semester, the traits where the most change was evident included

complete , elegant , knowledge

interdisciplinary courses. , and engaged; and the most change occurred among students in In the cross-sectional data comparing less expert students to more expert, the most reliable differences were on the traits of

analyze, logic, knowledge abstract,

, and

explain

. There was no significant difference between the groups’ performance on

fluent, flexible, elegant, divergent, novel ,

and

risk .

Participating faculty who filled out an on-line survey indicated that they had discussed the use of the rubric with colleagues, and that the rubric helped them learn more about creative and critical thinking among their students

T

RAITS

M

EASURED IN

D

ISCIPLINE

– A

LL

D C

LASSES BY ATA

ALL FA HUM NSCI SOSC INT Explain Analyze Evaluate 0.63

0.54

0.44

Integrate Logic Abstract Complex 0.43

0.40

Knowledge

0.85

Synthesis 0.53

0.38

0.62

Fluent Flexible Complete Elegant Divergent 0.63

0.45

0.90

0.58

0.52

0.64

0.44

0.00

0.75

0.44

1.00

0.79

0.00

0.00

0.45

0.21

0.89

0.89

0.21

0.89

0.89

0.54

0.74

0.12

0.60

0.48

0.12

0.63

0.74

0.89

0.63

0.12

0.48

0.87

0.87

0.73

0.56

0.71

0.84

0.58

0.71

0.58

0.58

0.28

0.86

0.28

0.14

1.00

0.72

1.00

0.28

0.72

0.72

1.00

0.72

0.45

1.00

0.56

0.72

0.12

0.12

0.12

0.12

0.12

0.99

0.12

1.00

0.12

0.55 1.00 0.65

0.47

1.00

0.81

0.81

Novel Original Transform Engaged

0.40

0.90

0.74

0.72

0.25

0.79

0.68

0.89

0.74

0.89

0.88

0.12

0.69

0.73

0.56

0.56

0.45

1.00

0.72

0.72

0.12

1.00

0.81

1.00

Risk 0.59

0.75

0.48

0.14

0.28

1.00

_____________________________________________________________________ N Students 444 73 65 85 82 139 N Classes 31 5 6 6 8 _____________________________________________________________________ 6

A VERAGE A CROSS T P ERFORMANCE ON IME – L T ONGITUDINAL RAITS D ATA

Trait Explain Analyze Evaluate Integrate Logic Knowledge Synthesis Abstract Complex Fluent Flexible Complete Elegant Divergent Novel M time 1

3.85

3.76

3.28

3.67

3.76

3.40

3.77

3.53

3.27

3.56

2.85

3.56

3.03

2.53

3.19

M time 2 4.52

4.73

4.27

4.51

4.78

4.35

4.63

4.58

4.14

4.39

3.82

4.41

4.05

3.61

4.22

tvalue 7.26*** 7.84*** 8.86*** 5.86*** 6.85*** 10.63*** 8.50*** 9.76*** 9.60*** 9.16*** 8.56*** 10.98*** 10.83*** 9.00*** 5.78*** *** indicates t value is significant at p<.001 revealing change in mean across time Note: High responses are noted in bold, low in italics.

Original Transform 3.19

2.23

4.01

3.31

9.40*** 9.22*** Engaged Risk 3.56

2.80

4.48

3.55

10.58*** 7.60*** _____________________________________________________________

C ROSS -S ECTIONAL DATA

Scores on the traits could range from 1 to 6, with a higher score indicating better performance. Differences between groups were analyzed with independent sample t-tests.

Trait Explain Analyze Evaluate Integrate Logic Knowledge Synthesis Abstract Complex Fluent Flexible Complete Elegant Divergent Novel Original Transformation Engaged Risk M Less Exp M More Exp Difference?

3.99

3.64

4.71

4.65

4.35*** 5.86*** 4.18

3.47

3.87

3.37

4.70

4.19

4.80

4.01

2.51* 3.72*** 5.28*** 4.39*** 3.67

4.26

3.15

4.45

3.92

4.61

4.56

3.71

3.93

3.59

4.32*** ns 3.04** ns ns 3.60

3.04

2.87

3.38

2.91

2.30

3.81

2.34

4.16

3.23

3.21

3.67

3.58

3.02

4.29

2.53

3.52** ns ns ns 3.30*** 3.34*** 2.59** ns Note: ns indicates the t value is not significant; indicates significant at p<.05, **p<.01, ***p<.001

Y EAR 3 I NTERVIEWS F ACULTY P ARTICIPATING

  Twenty-two faculty who employed the “generic” rubric to measure creative and critical thinking among the students were interviewed about their experience Interviewers asked questions about  the specific assignment used and whether it promoted creative thinking,    other assignments and pedagogies that might promote (or inhibit) creative thinking, aspects of the discipline that might promote (or inhibit) creative thinking, aspects of the college environment that might be important

Y EAR 3 I NTERVIEWS F ACULTY P ARTICIPATING

   Most of the faculty who participated chose an assignment they had used previously, and did not develop a new assignment for the grant. Many of these assignments seemed designed to stimulate critical thinking more than creative thinking, but using a rubric that emphasized creative thinking increased the focus on these traits and likely increased the amount of creativity students showed. In general, most people thought their assignments did promote creative thinking in students (and this was validated by the quantitative data gathered from the rubrics). However, the assignments people used for their data collection were not necessarily the most creative assignments the faculty had used or could imagine using.

FACULTY INTERVIEWS YEAR 3:

VALUE OF “ACTIVE LEARNING TECHNIQUES”  Take devil’s advocate position  Use problem solving  Students debate ideas  Use small group discussion  “Hands-on” science activities

FACULTY INTERVIEWS YEAR 3:

VALUE OF “ACTIVE LEARNING TECHNIQUES” One natural scientist spoke about student experience with creating their own experiments versus using “canned” labs. She said, “I didn’t know when I

started doing this in my upper-level comparative class whether they would like it.... The experiments don’t usually work, because experiments usually don’t work, period. And, so, I would always say next year I am going to write labs. By the end of the class students come to my office to tell me emphatically, they write it on their evaluations, do not change the lab. This is what we want to do. This is what scientists do. I am always surprised by that… they always like it that way. Even if they have to put in more time. Evenings, weekends to feed their animals, take measurements…. the students like it better, it is better for them… thinking of something that has not been done but is doable is hard and you do have to be creative to do it.”

FACULTY INTERVIEWS YEAR 3:

VALUE OF “TAKING THE PERSPECTIVE OF OTHERS”  Similarly, another historian said, “I ask them a

question that is surprising. It might be something like, ‘What does it smell like in 14 th century Florence?’ A question we have not talked about at all, but they have done a great deal of reading from primary sources about markets in Florence, or about demography, or about domestic spaces and ask them to think about that evidence in some new sort of way and to extrapolate about the lived realities of 14 th century people about things they know from records.”

FACULTY INTERVIEWS YEAR 3:

VALUE OF “TAKING THE PERSPECTIVE OF OTHERS”  A couple of other professors also mentioned types of “role playing.” One humanities professor said, “In my

journalism class they do mock press conferences. They are given a topic – political topic - that is not a scandal. They have to draw on real-world knowledge, using realistic people who would appear at a press conference. Have to generate the story, present the information, and the rest of the class asks questions as reporters and they write the news story as journalists. Then the student presenting has to field the questions. They think outside, but within the box as well. They have to be active. I go back to the definition of [creativity as] ‘new but useful.’ “

FACULTY INTERVIEWS YEAR 3: VALUE OF “STUDENT CHOICE”

 One professor of an interdisciplinary course talked about the creation of a student-led senior colloquium. She said, “...second semester senior year giving seniors

a term paper is just a death sentence. So I designed the senior colloquium as a way to get them to be more creative and to flow a little bit more with the material, rather than a research paper. The students design some of the colloquium itself, they get to pick the topic, they get to pick the readings and they get to determine the project that they will do collectively…. So for instance this year they did a magazine and I think they liked the opportunity to explore the genre of this sort of women’s magazine at the same time trying to make some sorts of statements about women and feminist issues that are important to them.”

FACULTY INTERVIEWS YEAR 3:

VALUE OF “PEER EVALUATION, INFLUENCE”  Many faculty talked about the importance of students engaging with their work, of “owning” their work. Faculty thought that to the extent that their peers would see their work, students would care more about it. The professor who had her students use “Moodle” to discuss their ideas publicly found that there was healthy competition among the students to present their unique voice which led to very creative expression.

FACULTY INTERVIEWS YEAR 3:

OF STUDENTS MAKING CONNECTIONS OUTSIDE THE CLASSROOM VALUE  A social science professor who has students do work outside the classroom said, “If they are out in the community, if they

are working with a group of people, I think it gives them a way to think about an issue in a much bigger way… I mean you can look at the school violence issue, you can look at a bunch of material about school violence and have good book knowledge of it and the way that boys and girls interact in high school, but when you go to a football game and you watch it happen, you see these little kids pushing each other around and you see the bullies… it makes them come back with different questions, different ways to think about what they read.”

FACULTY INTEVIEWS, YEAR 3: VALUE OF “INTERDISCIPLINARITY”

 At Ohio Wesleyan the “National Colloquium” is a way to promote interdisciplinary thinking and discussion of issues outside of the classroom. One of the professors brought this up and said,  “There’s a different topic every year, a major national or

international topic, and they bring in speakers and we try to bring tie in other events on campus and stage a play that focuses on that topic. Students will receive partial credit for attending the lectures or the performances or participating in activities which have something to do with the topic of the National Colloquium and one year it was about the arts. One of my colleagues raised the topic of having the National Colloquium for the arts and students could get other class credits for attending art openings or jazz band performance…. Making it part of the curriculum kind of forces the students to take advantage of things that are available to them… but that kind of thing is what I think educational institutions could do.”

Where does Creativity Happen on Campus?

S UMMARY Y EAR 2 S URVEY D ATA

Surveys were administered to 260 first-year students, 375 senior students, and 147 faculty across the four colleges during the 2007-2008 academic year to assess perceptions and experiences. Out of a wealth of information yielded by this research, a few interesting items are described below.

1. There seem to be more opportunities for critical than creative thinking in classes.

2. Class projects are evaluated by students as a means to engage in both critical and creative thinking.

3. Our campus environments were rated favorably with respect to the potential for creative thinking, although faculty rated “idea time” and “risk taking” as somewhat limited in their experience.

S UMMARY Y EAR 2 S URVEY D ATA

4. Women students reported more experience with both creative and critical thinking and more positive attitudes about creativity compared to men students.

5. When asked to write about their most creative experiences, 61% of faculty mentioned their own research. Teaching and collaboration with students were each mentioned by 32% of the faculty.

6. When asked about barriers to creative thinking on campus, the most common response of senior students was that class pedagogy was a barrier (40% of students mentioned this); while faculty most often wrote about time as a barrier (18% of faculty). 7. A diverse campus environment (in terms of class, race, sex, sexual orientation, and national origin) is perceived to facilitate creative thinking.

YEAR 2 SURVEY DATA:

FACULTY AND STUDENT PERCEPTIONS OF CREATIVE ENVIRONMENT CHARACTERISTICS

Participants rated the extent to which each characteristic existed on their campus: 1 = not at all, 7 = extensively

Faculty 1 st Yr Seniors Item 5.02

5.08

4.69

4.45

5.50

5.50

5.30

5.28

5.31* 4.94* 4.90* 4.88* Challenge Freedom Idea Support Trust/Openness *indicates significant difference between groups 4.65

4.50

4.72

3.83

3.73

4.14

5.11

5.28

5.23

4.79

4.80

4.28

4.89* 5.05* 5.32* 4.36* 4.45* 4.56* Dynamism/Liveliness Playfulness/Humor Debate Risk Taking Idea Time Conflict 4.67

4.86

4.58

4.88

5.40

5.00

5.14

5.22

4.99* 5.04

4.98* 5.10

Supportive Environment Working in groups Active models of creative thinking Assignments encourage independent problem- solving & risk-taking

YEAR 2 SURVEY:

TO STUDENTS?

WHERE DOES CREATIVE THINKING HAPPEN ON CAMPUS, ACCORDING

Students rated how much each event involved creative thinking, rating 1 = low to 5 = high

CREATIVE THINKING Engaging in a hobby Completing projects for class Writing papers for classes Extra-curricular or co-curricular activity Interacting w/ students outside of class Cultural events on campus Interacting w/ faculty outside class Interacting w/ students, faculty in class Listening to speakers on campus Communicating w/ peers internet Reading material for class “Surfing” the internet First Year Seniors 4.18

3.74

3.74

3.45

3.51

3.19

3.02

3.02

2.78

3.03

2.83

2.63

4.13

3.78

3.68

3.55

3.52

3.18

3.14

3.04

2.74

2.73* 2.61* 2.59

*indicates significant difference between groups

S TUDENT C REATIVE I DEAS ON THE E NCOURAGEMENT OF T HINKING , F OCUS G ROUPS Y EAR 2

• • • • • • • • • • • • • • • Professors allow students to give input (open-minded professors); discussion-based classes Bringing in outside artists, bands, speakers, poets to “stir the pot” Open-ended assignments in classes Opportunities to connect work in different courses Classes with real-world applications Need more time Student-themed housing Classes where professors challenge students “to the edge” Comfortable spaces where students can come in and “do their own thing” Lunch tables with faculty Diversification and diversity requirements encourage students to learn something new Classes that require internships, service learning (first-year seminars with these components were praised) Study abroad Senior research project (projects in general); Fewer required courses would mean more freedom for students; Any way for students to propose their own classes?

•

F ACULTY C REATIVE I DEAS ON THE T HINKING , F E NCOURAGEMENT OF OCUS G ROUPS Y EAR 2

General Comments.

• • • • • • • • Spaces for faculty to get together and discuss pedagogy Fifty minute time periods are not long enough Student-controlled spaces, events, organizations Need to find time to live “the life of the mind” Service learning that involves many members of the community Senior seminars; Independent research project, summer science programs, senior projects; Teaching abroad with students from your institution Encourage diversity, appreciation for internationalism • • • • Taking students to conferences, on field trips, etc. Provide faculty with opportunities to do things differently – team-teach, time to do projects Make interdisciplinary work easier, e.g., break down departmental autonomy Allow faculty to remain “learners” as much as possible

•

F ACULTY C REATIVE I DEAS ON THE T HINKING , F E NCOURAGEMENT OF OCUS G ROUPS Y EAR 2

• • • • • • In the classroom.

Guest speakers Use of Moodle and other forms of technology (that encourage engagement) A final project with several options to the traditional paper (assignments encouraging active engagement) Provide opportunities for students to interact with the “real world” Setting up projects, assignments where outcome is unclear Assignments that require “vulnerability” – students have to show something about themselves

FACULTY INTERVIEWS, YEAR 3: VALUE OF ASSESSMENT

“Doing the rubrics – I didn’t know how that would work, I had my doubts two years ago. I actually started using a rubric and I find it easier to grade consistently …. I really like it and use it now for all my labs, it helps my grading. If students would look at them they could realize exactly what they aren’t doing well. I give this back to students with written comments, and the ones who look at it could improve their labs which is helpful…. I found it helpful and I was totally shocked. Someone said you can’t do it with lab reports – I think you can – I showed that you could.”

FACULTY INTERVIEW YEAR 3: QUOTE

“The challenge is to encourage faculty members to be experimental in the ways they imagine student work, without abandoning key commitments to core educational goals having to do with having students express themselves clearly in argument to also say that the ways in which you might go about cultivating a sense of the discipline should go beyond, at least experiment with going beyond, writing 5 or 25 page papers. How you encourage faculty members to do that I am not sure. I suppose one way of doing that is giving them inspirational models. Make sure people within their departments and across disciplines talk to each other about whatever works, whatever seems to gain traction in this area.”

FACULTY INTERVIEWS YEAR 3: Changing Classroom

“Expectations of what a classroom is supposed to be like are changing. Using the classroom as a place to do presentations, as a place to do small group work, as a place to do debates… use the classroom as a theatre some days, making a place where we do lecture some days… making it a place where when they are getting ready to go to my class they are wondering what are we going to do today? Mixing it up, building some excitement. It can ‘t be like that everyday, but…. “

AAC&U VALUE AND O HIO 5 T EAGLE

VALUE

15 general rubrics Developmental scale Wide range of institutions Adaptable Useful for assessment www.aacu.org/value/rubrics

TEAGLE

1 general rubric Evaluative scale Liberal arts colleges Adaptable Useful for assessment www.wooster.edu/teagle