Transcript Design Issues in Assessment: Quantitative and Qualitative

Assessing International
Education at our
Institutions: An Overview
Darla Deardorff, Duke University
David Comp, University of Chicago
Elaine Meyer-Lee, Saint Mary’s College
Goals

Gain overview of assessment in international
education including brief history

Explore ways of thinking about
internationalization outcomes assessment

Discuss some examples of current assessment
Important Note: This presentation is not a
methods, how-to, or findings, but overview and
examples to inspire
Why assess?
We’d love to hear from you…!
Why assess international
education?

Inform continuous quality improvement

Advocate for international education

Satisfy regional accrediting bodies

Add to basic understanding of student
growth in this area
Why Assess? Measure success
Moving beyond
OUTPUTS to OUTCOMES
What are meaningful outcomes (results) of
internationalization efforts?
__________________________________________________________________________
INTERNATIONALIZATION
at institutions of higher education
Inputs/Resources
needed for implementation of components of internationalization
|
Activities/Components of Internationalization
(college leadership, faculty involvement, curriculum, study abroad, international
students/scholars/faculty, international co-curricular units)
|
Outputs of Internationalization
(i.e., number of international students, number of study abroad programs, number of students
studying foreign languages, etc.)
|
Outcomes of Internationalization
Intercultural competence – what is it?
How do higher education administrators define it? intercultural experts?
How can it be assessed?
= Long-Term Impact of Internationalization
_______________________________________________________________
Figure 2. General program logic model applied to internationalization. (Deardorff, 2004, Page 58)
Brief Historical Overview
Where has the international ed
field been in regard to
assessment?
Recent History of Assessment in
International Education
 NAFSA: Association of International Educators
– New Teaching, Learning and Scholarship - Knowledge
Community.
– NAFSA Task Force on Research, 2004.
– SECUSSA Research Committee, 2003-2005.
 Forum on Education Abroad
–
–
–
–
–
Focus on organizational goals.
Outcomes Assessment Committee.
Guide to Assessment in International Education.
Partnership with Frontiers Journal.
Standards of Good Practice.
Recent History of Assessment in
International Education
 Association of International Educators (AIEA)
– Burn, B., & Smuckler, R.H. A Research Agenda for the
Internationalization of Higher Education in the United
States: Recommendations and Report Based on August
10-11, 1995 Meeting.
– International Education Forum, Association Journal
during the 1990’s
– 2004 conference session with significant discussion on
research in international education (transcripts
distributed).
 American Council on Education (ACE)
– 2004 Roundtable on Research in International Education,
Baltimore, Maryland
Recent History of Assessment in
International Education

Association for Studies in International Education (ASIE)
– “The Association is an inter-organizational body whose mission is to encourage
serious research and publications dealing with international education and
academic mobility, to stimulate interest in such work (both in the international
education community and in academic circles in general), and to develop and
promote ways to disseminate this work in cost-effective and accessible formats.
The initial objective and primary focus of the Association in fulfillment of this
mission is the publication and distribution of the Journal of Studies in
International Education. “ (ASIE)
ASIE organizations include: AIEA: Association of International Education
Administrators; CBIE: Canadian Bureau for International Education; The CIEE:
Council on International Educational Exchange; EAIE: European Association for
International Education; IEAA: International Association of Australia; IEASA:
International Education Association of South Africa; NAFSA: Association of
International Educators; NUFFIC: Netherlands Organization for International
Cooperation in Higher Education; UKCOSA: The Council for International
Education; and, WES: World Education Services.
Ways of approaching
assessment
Some key points…
LEVELS OF INTERNATIONALIZATION
OUTCOME ASSESSMENT
 Institutional
 Program (inc. faculty)
 Student (inc. alumni)
Student outcome assessment is not a
measurement of learning but an integral part of
learning. Can include language/disciplinary
assessment, personal growth, intercultural
competence, etc.
Involve students in negotiating outcomes
ASSESSING OUTCOMES
Starting point:
Mission Statement
into
Goals
into
Measurable Objectives
Assessment Methods?
Methods determined by clearly articulated
assessment questions – what do we want
to measure?
Include:


DIRECT METHODS – authentic assessment, portfolios,
observation

INDIRECT METHODS – surveys, focus groups, interviews

COMBINATION OF QUANTITATIVE AND QUALITATIVE
Internationalization
Assessment-Contexts

“At Home” – curriculum, extra-curricular
activities, community involvement/impact,
domestic/internat’l student interaction, policy,
financial, campus climate

Abroad (Cross-border) – involves cross-
border delivery of education through exchange,
distance and e-learning, branch campuses,
partnerships, host community impact
Assessment Cycle
Define outcomes (based on mission/goals)
and establish measurable criteria
 Identify appropriate assessment methods
 Collect data
 Analyze data
 Reflect on needed changes
 Design and apply changes

INTERNATIONALIZATION
ASSESSMENT
 Multi-method,
 Ongoing
 Integrated
multi-perspective
Assessment Examples:
Saint Mary’s
Chicago and beyond
ICC Study
Saint Mary’s College Context

Catholic Women’s Liberal Art
(about 1600 students)

20% of students study abroad

8 semester options

9 short-term options

Encompasses much more than study abroad
(see brochures), and assessing as much of it as
possible

Is adopting Byram’s definition of intercultural
competence

Is launching a portfolio-based honors program in
Intercultural Leadership
Intercultural Leadership
Honors Program

Capture Saint Mary’s strengths and CWIL’s unique location at the
intersection of intercultural (local and global) and leadership education in
the context of a women’s college

Guide students through a developmental process into a deeper capacity for
intercultural leadership

Bring coherence and provide an organizing framework for every effort
(curricular and co-curricular) being undertaken at Saint Mary’s College to
encourage intercultural leadership

Portfolio assessment of students’ proficiency as an intercultural leader in
6 proficiency areas






Recognize the Leader Within
Articulate Your Ethical/Spiritual Center
Engage With & Value Diversity
Dialogue on Power & Privilege
Create Inclusive & Equitable Community
Make Your Difference in the World
Education Abroad
Evaluation & Assessment Goals

Evaluate programs’ effectiveness

Not satisfaction, measure Students’:
– Changes in sensitivity to cultural differences
– American identity development
– Own sense of growth toward goals

Explore correlations of change with:
– Intercultural engagement while abroad
(interaction and reflection)
– Program or demographic characteristics
Education Abroad
Assessment Methods/Design

Longitudinal Research Design
– Pre-test/Post-test (some grad follow-up)
– Procedures – 70% response rate so far

Control Group
– 30 First-year students

Focus Group
– Study Abroad Re-entry Course
– Written reflection and Kolb’s LSI
Education Abroad
Assessment Instruments
Study Abroad Participants
Control Group
 Intercultural Development Inventory  Intercultural Development
(IDI)
Inventory (IDI)
 American Identity Questionnaire
 Pre: Goals/motivations from SMC
Study Abroad Survey
 Post: Program Impact & Measure of
Intercultural Engagement from SMC
Study Abroad Survey
 American Identity
Questionnaire
Data Collection Timeline
2004-2005
Fall
Spring
2005-2006
Fall
Pre-test
N=16----
----------
Post-test
N=(16)
Pre-test
N=44----
----------
Post-test
N=(44)
Pre-test
N=83----
Control
N=32----
Spring
Post-test
N=(6)---------
Post-test
N=(51)--
Fall Semester
-----------
----------------
Control
N=(20)
Pre-test
N=(43)---------
Spring Semester
-----------
Pre-test
N=(29)----
Summer
2006-2007
Fall
Post-test
N=(?)
Year
Post-test
N=(?)
Post-test
N=(?)
Spring
On-campus International and
Intercultural Education Assessment

Qualitative Methods
*Student Intercultural Learning Showcase
 Community Education Program during International
Education Week
 Reports up on CWIL web-site
– Student & faculty intercultural learning grants: reports
– International Fellows: 6-mo. reports & exit interviews

Quantitative Methods
– Institutional Research: College Student Survey, HERI
Faculty Survey, etc.
On-campus International and
Intercultural Education Assessment

Mixed Methods
*International Student Services Survey
 Comprehensive survey to assess students’ ability
to fully participate in the college
 Comparable with Study Abroad Survey ‘s Measure
of Intercultural Engagement
– Individual programs for faculty development, study
abroad pre-departure & re-entry, or international
education week: surveys
– Intercultural Learning/Living Floor: survey &
Intercultural Development Inventory
– Catalyst Trip: survey & interviews
Assessing Internationalization at
the University of Chicago

Ad Hoc Committee on International and
Regional Programs at the University of
Chicago (April 15, 1995)

Ad Hoc Committee on International
Programs at The University of Chicago
(June 20, 2001)
Assessment Highlights in the Field

Assessment Tools
– The IES MAP (Model Assessment Practice) for Study
Abroad: Charting a Course for Quality. (Institute for
the International Education of Students (IES), 2003).
 The IES MAP focuses on four academic areas:
– Student learning environment
– Student learning: assessment and intercultural
development
– Resources for academic and student support
– Program administration and development
Assessment Highlights in the Field

Assessment Tools
– Standards of Good Practice for Education Abroad.
(The Forum on Education Abroad, 2005).
 The Standards of Good Practice for Education Abroad use a
queries based approach under the following eight standards:
–
–
–
–
–
–
–
–
Ethics and integrity
Mission, policies and procedures
Academic framework
Organizational and program resources
Student selection and code of conduct
Learning environment
Student learning and development
Health, safety and security
Institutions currently assessing
students’ ICC?
Don't know/
Not sure
9%
No
50%
Yes
38%
Frequency of students’ ICC
assessment (ideal)
Pre/post study
abroad
8%
Beg/End Coll.
22%
On-going
70%
ICC assessment methods
currently being used
Student interviews
Student paper/presentation
Observation
Student portfolios
Professor evaluation
Pre/post test
Custom/Adapted Self-Rpt. Inst.
Commercial Self-Rpt. Inst.
0
1
2
3
4
5
6
# of Institutions
7
8
9
SUMMARY

Young, wide-open field

Great need for better methods and more data on many
different dimensions of international education

Assessment data need to be utilized to provide feedback
to students, improve the program and so on

Assessment needs to be multi-method, multiperspective, ongoing, intentional, integrated, part of
strategic plan
CONTACT INFORMATION
Darla Deardorff: [email protected]
David Comp: [email protected]
Elaine Meyer-Lee: [email protected]
LINKS: For more handouts or bibliography on
assessing intercultural outcomes of study abroad
programs, go to
http://www.saintmarys.edu/~cwil/php/intercultural.learning/IILOutcomes.php
http://www.forumea.org/related_links.html