Transcript Using the Intercultural Development Inventory as an

Assessing Impact: Study Abroad
at Two Institutions
Elaine Meyer-Lee
Saint Mary’s College
Carol Warfield
Auburn University
2006 AIEA Annual Conference
February, 2006
Why Assess Outcomes?

1,022 US institutions reported information on
their study abroad programs

174,629 U.S. students studied abroad
Data from Open Doors Report on International Educational Exchange 2003-2004

Inform continuous quality improvement

Advocate for international education

Satisfy regional accrediting bodies

Add to basic understanding of student
growth in this area

Young, wide-open field

Much need for better methods and more
data on many different dimensions
This presentation is not a step-by-step
overview, but two examples
(See Forum’s upcoming Guide to Outcomes
Assessment in Education Abroad)

Multi-method Longitudinal Assessment of
Intercultural Engagement, IC Sensitivity,
Identity, and Goals: Preliminary Results
Elaine Meyer-Lee
Director of International & Intercultural Learning
Joy Evans
Research Analyst-Center for Women’s InterCultural Leadership
Saint Mary’s College
Notre Dame, Indiana
206 AIEA Annual Conference
February 25, 2006
Presentation Overview

Context

Assessment Methods

Review of Instruments

Preliminary Results

Contributions to Study Abroad Assessment
Saint Mary’s College

Catholic Women’s Liberal Art


Established 1844
About 1600 enrollment
•
•
•
•
98% full-time
82% on-campus
85% Catholic
9% students of color
20% of students study abroad
Mission of the
Center for Women’s InterCultural Leadership
Saint Mary's College's mission includes
preparing women to make a difference
in an increasingly complex and
interdependent world. The role of the
Center for Women's InterCultural
Leadership (CWIL) within this mission
is to foster the intercultural knowledge
and competence critical to educating
the next generation of women leaders.
CWIL at Saint Mary’s



Encompasses much more than study
abroad (see brochures)
Is adopting Byram’s definition of
intercultural competence
Is launching a portfolio-based
certificate program in Women’s
Intercultural Leadership
Study Abroad Programs

Year/ Semester
• Ireland
(Year only)
• India (Semester only)
• Rome
• Spain
• France
• Australia
• Austria
• South Africa
Study Abroad Programs


4-2 Weeks
• European Studies
(4)
• Internships in Europe (4+)
• Honduras
(3)
• Ecuador
(3-2)
• Greece
(2 ½)
• S. Korea (2)
1 Week
• Prague/Poland
• Jamaica
• Mexico
Assessing Study Abroad…
Going Beyond
Student Satisfaction
Evaluation & Assessment Goals


Evaluate programs’ effectiveness
Measure Students’:
• Changes in sensitivity to cultural differences
• American identity development
• Own sense of growth toward goals

Explore correlations of change with:
• Intercultural engagement while abroad
(interaction and reflection)
• Program or demographic characteristics
Assessment Methods/Design

Longitudinal Research Design
• Pre-test/Post-test (some grad follow-up)
• Procedures – 70% response rate so far

Control Group
• 30 First-year students

Focus Group
• Study Abroad Re-entry Course
• Written reflection and Kolb’s LSI
Assessment Instruments: Pre-Test
Control Group
Study Abroad Participants



Intercultural Development
Inventory (IDI)
American Identity
Questionnaire
Goals/motivations from
Saint Mary’s College Study
Abroad Survey


Intercultural
Development Inventory
(IDI)
American Identity
Questionnaire
Assessment Instruments: Post-Test
Study Abroad Participants


Intercultural Development
Inventory (IDI)
Control Group

American Identity
Questionnaire (AIQ)


SMC Study Abroad Survey
Program Impact &
Measure of Intercultural
Engagement (MIE)
Intercultural
Development
Inventory (IDI)
American Identity
Questionnaire
Data Collection Timeline
2004-2005
Fall
Spring
Pre-test
N=16---- ----------
Pre-test
N=44---- ----------
Pre-test
N=83----
Control
N=32----
2005-2006
Fall
Spring
2006-2007
Fall
Post-test
N=(16)
Post-test
N=(44)
Post-test
N=(6)---------
----------------
Pre-test
N=(?)---------
Spring Semester
Post-test
N=(48?)--
Fall
Semester
-----------
Post-test
N=(29?)
Year
-----------
Post-test
N=(?)
Pre-test?
N=(?)----
Summer
Control
N=(32)
Post-test?
N=(?)
Spring
Review of Instruments
Intercultural Development Inventory
(IDI)

50-item Self-Assessment

Valid & Reliable Psychometric Measure

Culture general in focus

Quantifies the subjective experience of cultural difference

Most widely used currently

Based on Milton Bennett’s Developmental Model of
Intercultural Sensitivity (constructivist stage model)
Slide Available Online:http://www.coe.int/T/E/North-South_Centre/Programmes/3_Global_Education/c_GENE/GENE%20Webpage%20-%20(3)%20Hesse%20article.asp
IDI Measurement Scales

Denial/Defense

Reversal

Minimization

Acceptance/Adaptation

Encapsulated Marginality
Overall Profile
American Identity Questionnaire

10-item Likert-style Self Assessment

Adaptation of Jean Phinney’s Multigroup Ethnic Identity Measure (good
track record)

Based on Eriksonian identity development theory. Measures 2 Factors:


•
identity search (e.g. I have often talked to other people about what it means to
be an American.)
•
affirmation, belonging, and commitment (e.g. Being American plays an
important part in my life.)
One goal of study abroad is more complexity in understanding own
culture, and students in pilot focus group felt it captured an important
dimension
Currently refining adaptation’s psychometric properties
Saint Mary’s College
Study Abroad Survey
Semester/Year-Long
Programs


Core Survey
• 111 Items
• 4 open-ended
questions
Customized Addendum
Summer/Short-term
Programs


Core Survey
• 58 Items
• 1 open-ended
questions
Customized Addendum
Saint Mary’s College
Study Abroad Survey
Semester/Year-Long Programs

Summer/Short-term
Programs
Question Sections Include:
•
•
•
•
•
•
•
•
•
•
Demographics/Background
Goals
Pre-Departure
Academic Program
Cultural Tours & Excursions
On-Site Student Services
Intercultural Engagement
Overview of Program
Program Impact (Goals)
Volunteer Interests

Question Sections
Include:
•
•
•
•
Demographics
Pre-Departure
Living Arrangements
Intercultural
Engagement
• Overview of Program
• Volunteer Interests
Self-reported Growth in:

Understanding of different cultures

Different perspective on American culture

Greater understanding of the US in world affairs and history

Different perspective on your ethnic heritage

Understanding of my faith

Self-awareness

Maturity and self-confidence

Interpersonal skills and the ability to adapt

Intellectual development in the liberal arts

Ability to communicate in a foreign language
Measure of Intercultural
Engagement
Asks students to quantify the
frequency of their engagement
through:



Interaction with cultural resources
Interaction with local people
Explicit reflection on these
interactions
Interaction with cultural resources

Theater/opera/symphony/concerts

Museums/ historical sites

Forests/parks/beaches

Traveling outside of the host country

Traveling within the host country
Interaction with local people










Shopping/visiting local markets
Movies/entertainment catering to locals
Restaurants/pubs/cafes catering to locals
Visiting local families/friends in their homes
Participating in worship services
On-campus informal conversations
Events sponsored by the study abroad program
Extra-curricular activities (student clubs,
intramurals, etc.)
Volunteering, service-learning, internships,
shadowing
Reading or listening to local news
Explicit reflection on these
interactions

Journaling/Blogging

Photography

Other artistic expression (painting, drawing, collages, etc.)

Class discussions

Informal conversations with locals

Informal conversations with students studying abroad

Calling/e-mailing family/friends/faculty in America about
experience
Pre-test Results
IDI Worldview Profile: Pre-Abroad
IDI Worldview Profile “Pre”:
Controls
Pre-test Comparison to Control:
American Identity Questionnaire


Affirmation, belonging and
commitment
 no significant difference
Identity search
 statistical significance
(at the .05 level)
Summary of Pre-test Findings


IDI: Students’ scores do not support
developmental model
IDI: Low Minimization Score
• Catholic beliefs ….We’re all God’s children

Control on both: Heritage Seeker Factor
• Are study abroad participants a different
population than other SMC students?
Preliminary Analysis of Change
Data for Matched Set


No significant change in means for
IDI or AIQ yet (on paired T-test on
pre-post mean scores, n=42 and 27)
BUT, high self-reported growth (1.7
when 1 is high) and wide range in
change scores and program means
What about those who do change?


Correlation of AIQ with IDI scales
(bivariate non-parametric correlations)
Correlations with Intercultural
Engagement:
• Cultural resource scale with AIQ change
• Overall and reflection with program impact
(especially interpersonal items)

50% did grow on at least one scale
(defined as 2 standard deviations of
positive change, n=21)
How do the “growers” compare?
Characteristic
From N.America
White
Latina
Never been abroad
Abroad < 3 months
Whole year program
Semester program
Summer program
Rome program
Seville program
Mexico
France
Ireland
Growers
91%
67%
24%
53%
24%
10%
76%
14%
24%
24%
14%
14%
10%
Non-Growers
95%
86%
4.5%
63%
26%
46%
46%
8%
32%
5%
9%
9%
46%
Methodological Findings



Importance and Challenges of
Control Group
Procedural effects (e.g. on-line vs.
paper)
Signs of internal consistency in
measures (inter-item correlations)

Intercultural
Development
Inventory:
?
Intercultural
Development
Inventory:
 Standardized
instrument
? Requires qualifying
training to administer
 Ability to compare
study abroad program
outcomes with
domestic intercultural
programs
? Highly rigorous and
proprietary
? Labor- and costintensive for large
scale use

Program Impact &
MEIM
?
 Comparable & program
specific data on students’
intercultural interactions

AIQ
 Examine identity
dimension, connect to
literature on this
Program Impact &
MEIM
? Labor-intensive
development and
administration
?
AIQ
? Reliability and Validity of
adaptation
? Needs refining
Next Steps

Analysis
• More on change and changers with full dataset
• More on correlations with intercultural engagement,
program variables (including Engle & Engle’s category),
and student variables (including original motivation)
• Qualitative
• Psychometrics on AIQ, Program Impact, MIE

Data collection
• Add 1 year follow-up
• Draw on Women’s Intercultural Leadership Portfolios?

Applications
• Many improvements to (and rationales for) programs
already. Overhauling re-entry course.
For more handouts or bibliography on
assessing intercultural outcomes of
study abroad programs, go to
http://www.saintmarys.edu/~cwil/php/intercultural.learning/
IILOutcomes.php
or email me: [email protected]