Transcript MPI Mission Perception Inventory Institutional Characteristics and Student Perception of Mission: What Makes a Difference? Ellen Boylan, Ph.D.

MPI
Mission Perception Inventory
Institutional Characteristics and Student Perception of Mission:
What Makes a Difference?
Ellen Boylan, Ph.D. ● Marywood University
NEAIR 36th Annual Conference ● Baltimore, Maryland ● Nov. 7 – 10 , 2009
What’s in a mission?
2
scope of research
Purpose
Develop an instrument to measure
student perception
of institutional
mission.
Test instrument reliability.
Uncover constructs (factors).
Observe constructs longitudinally.
3
GRANTS IN HIGHER EDUCATION FOR IMPROVED ASSESSMENT METHODS
scope of research
Purpose
Develop
an instrument
Test instrument
reliability.
Research Questions
1.
Is the Mission Perception
Inventory (MPI) a valid and reliable
reliable measure of student
perception of institutional mission?
2.
What are the factors in the MPI?
3.
Do the factors recur in repeated
administrations of the revised MPI?
4.
Are the factors equally reliable over time?
Uncover constructs.
Observe constructs
longitudinally.
Develop a prediction
5. Can a school’s performance be predicted?
equation.
4

leaders of public and private institutions alike are
thinking about spirituality
these days, as the
data suggest that's what their students are thinking about, too
(Inside Higher Ed, 2009).
background
connection between institutional
programs and student learning environment
(Pascarella, 2001).

There is strong

…institutions influence levels of
engagement
on
campus as a result of structural features, programs, policies,
and
organizational culture (Kuh et al., 2005).
5
design
Select the National Survey of
Student Engagement (NSSE) as
the vehicle for inserting research
questions.
Assemble a NSSE consortium to
jointly engage in research to explore
student perception of mission.
Develop question items;
administer them to the consortium
as a NSSE attachment.
Test the questions: reliability, factor
analysis, and correlation analysis. Repeat annually.
6
GRANTS IN HIGHER EDUCATION FOR IMPROVED ASSESSMENT METHODS
derivation of the MPI and subscales
Sense
Of
Mission
Administer
Consortium
mission
questions
(20 items)
Mission
Perception
Inventory
Respect
for
Diversity
(MPI)
(~19 items)
Individual
Values
Spiritual
Practice
7
•
The mission of this institution is widely understood by
students.
•
Social and personal development is an
important part of the mission.
•
Ethical and spiritual development of students is
important.
This institution offers opportunities for volunteering and
•
This institution offers opportunities for developing
•
There are opportunities for students to
their religious commitment.
•
•
•
•
•
sense of mission
(10 items  = .90)
community service.
leadership skills.
strengthen
This institution’s religious heritage is evident.
Professors here discuss the ethical implications of what
is being studied.
As a result of my experience here, I am more aware of my
own personal values.
The mission of this institution is reflected in course
offerings.
8
respect for diversity
(5 items  = .878)
The faculty, staff, and students here…
•
respect different religions
respect different races and
cultures
………………………………………………………………………………
•
•
Students feel free to express individual spirituality.
•
Different sexual orientations are accepted.
•
The environment encourages appreciation of diversity.
9
administration
The NSSE survey with
attached Mission
Perception Inventory (MPI)
questions has been
administered 182 times to
close to 50,000 first-year
and senior students at 112
unique institutions across
the United States
every year since 2004.
10
GRANTS IN HIGHER EDUCATION FOR IMPROVED ASSESSMENT METHODS
Consortia institutions and respondents
by year
Respondents
Institutions
First Year
Senior
2004
15
2,000
1,827
2005
16
1,279
1,332
2006
24
2,684
2,854
2007
36
4,533
4,331
2008
35
4520
5063
Indepen’t 2008
19
6805
6970
145*
21,821
22,377
Catholic
*(112 unique institutions)
11
2004
2005
2006
2007
2008
Number of items in factors
Sense of Mission
8
10
10
10
10
Respect for Diversity
5
5
5
5
5
Individual Actions
2
0
0
0
0
Religious Practice/Spirituality
2
3
3
2
2
Mission Perception Inventory
17
18
18
19
19
12
2004
2005
2006
2007
2008
Catholic
2008
Indept
cronbach 
cronbach 
cronbach 
cronbach 
cronbach 
cronbach 
Sense of Mission
.87
.88
.88
.90
.91
.88
Respect for Diversity
.85
.84
.87
.86
.88
.87
Individual Actions
.67
n/a
n/a
n/a
n/a
n/a
Religious
Practice/Spirituality
.62
.55
.54
.62
.62
.64
MPI scale
.88
.89
.90
.91
.91
.90
Subscales
13
(MPI)
NSSE 2008
Mission Perception Inventory Report
Mission Engagement Consortium for Independent Colleges
My University- Sample
Sense of Mission
Reliability and factor analysis of the instrument based on results from administrations of consortium
questions have consistently produced a highly reliable scale measure called the Mission Perception Inventory
(MPI)*. Further analysis reveals three highly correlated and reliable subscales within the MPI:
First Year
Mean 0.00
Median 0.00
25%ile 0.00
75%ile 0.00
 Sense of Mission ( 11 items)
 Respect for Diversity (5 items)
 Spirituality (2 items)
(Cronbach’s alpha=__)
4.6
Also presented are mean comparisons on another scale, NSSE Beliefs and Values (NSSE/BV).
Senior YearThe
Mean a priori,
0.00 the
Consortium administrator created the NSSE/BV scale for this study by a process of selecting,
Median 0.00
question items from the NSSE instrument that seemed most related to mission and then grouping
them.
25%ile 0.00
Scale mean comparisons between My University and Consortium schools are presented
in the0.00
bar graphs.
75%ile
The graphs compare My University first year and senior means compared to Consortium group means on the
(out of to
x) the
study’s five scales or benchmarks. The mean scores are shown on the vertical or “y” axisRank
according
First
Year:were rated on
Likert scale range used for items in that scale. Sense of Mission and Respect for Diversity
items
Senior Year:
a scale from 1 – 5 (1=Strongly disagree, 5=Strongly agree).
In the column to the left of each graph, the rank (high to low) and semi-interquartile range are given for both
first year and senior scale means. Note carefully that the y-axis interval is 0.5 for some scales, and 0.1 for
others. In the columns under and/or to the right of the graph, the items contained in that scale are listed.
When comparing first year and senior mean scores, recall that the results of the NSSE 2008 produce a
snapshot in time of student responses. Results do not indicate growth or increase in scale means from first to
senior years, but simply a comparison of those cohort classes at one moment.
Sense of Mission Score
Mission
Perception
Inventory
Report
In spring 2008, My University participated in the National Survey of Student Engagement (NSSE) Mission
Engagement Consortium for Independent Colleges, and administered additional questions
on mission
the
2008
MissiontoPerception
Inventory Report
survey’s first year and senior student respondents. Of ___ total survey respondents at My
University,
My
Universitythere
- Sample
were ___ first year students and ___ seniors.
4.4
This institution offers opportunities
developing leadership skills.
for
As a result of your experience here, you are
more aware of your own personal values.
4.0
The heritage of the founding religious
community of this institution is evident here.
3.8
First Year
Senior
My University
4.69
4.56
Consortium
4.02
3.97
The mission of this institution is widely
understood by students.
The mission of this institution is reflected in its
course offerings
Social and personal development of students is an
important part of the mission at this institution.
At this institution, there are opportunities for
students to strengthen their religious
commitment.
Ethical and spiritual development of students is an
important part of the mission at this institution.
The faculty at this institution discusses the
ethical implications of what is being studied.
As a result of my experience here, I am more
aware of my own personal values.
2.0
(Cronbach’s alpha=__)
Within the past week, have you participated in
a religious service?
Within the past week, have you spent time in
private prayer or meditation?
Respect for Diversity
1.5
First Year
Mean 0.00
Median 0.00
25%ile 0.00
75%ile 0.00
1.0
0.5
0.0
First Year
Senior
My University
1.96
1.92
Consortium
2.07
2.08
2008 Mission Perception Inventory Report
Senior Year
Mean 0.00
Median 0.00
25%ile 0.00
75%ile 0.00
Rank (out of x)
First Year:
Senior Year:
(Cronbach’s alpha=__)
The faculty, staff, and students here are
respectful of people of different religions.
4.1
Respect for Diversity Score
Rank (out of “x” schools)
First Year: ___
Senior Year:
Religious Practice Score
Senior Year
Mean 0.00
Median 0.00
25%ile 0.00
75%ile 0.00
for
4.2
Spirituality
First Year
Mean 0.00
Median 0.00
25%ile 0.00
75%ile 0.00
This institution offers opportunities
volunteering and community service.
The faculty, staff, and students here are
respectful of people of different races and
cultures.
4.0
3.9
Students feel free to express their individual
spirituality here.
3.8
3.7
People of different sexual orientations are
accepted socially here.
3.6
3.5
First Year
Senior
My University
3.92
3.60
Consortium
3.98
3.87
The environment here encourages students to
develop an appreciation of diversity.
14
GRANTS IN HIGHER EDUCATION FOR IMPROVED ASSESSMENT METHODS
Teagle Foundation Research Continues
•
Develop the Mission Engagement
Index.
•
•
•
•
•
•
Obtain NSSE 2009 consortia data
Test using reliability analysis to
produce factors
Distribute MPI reports
Compare consortia results
Conduct regression analysis to create
the new index
Use the Mission Engagement Index
(MEI) to assess performance.
15
GRANTS IN HIGHER EDUCATION FOR IMPROVED ASSESSMENT METHODS
NSSE 2008 Benchmark Statistics
by Consortia
16
•
Dependent variable (Institution score)
Mission Perception Inventory
• Sense of Mission scale
• Respect for Diversity scale
•
•
Independent variables (2008 data;
need 15 cases per variable*)
• Selectivity
15 institutions
• Enrollment
30 institutions
• Urbanicity
45 institutions
• Resident %
60 institutions
• Female %
75 institutions
• Another?
90 institutions
*…a recommended ratio of subjects to IVs of at least 15 to 1 will provide a reliable regression equation (Stevens,17
1992).
Selecting IVs for regression analysis
Institution
region
Carnegie
Class
%
resident
%
female
%
ethnic/
cau
UG
enroll
%
Parttime
Setting
%
Accept
a
b
c
d
e
North
U Master's
39
61
95
2119
25
Suburb
74
Midwest
U Master's
33
67
70
928
40
Suburb
57
South
National U
32
68
39
4130
42
Suburb
54
South
Bacc.. Coll
43
57
76
1011
21
Rural
66
Midwest
U Master's
43
57
70
1692
23
Suburb
85
f
g
h
Midwest
Bacc. Coll
42
58
89
978
83
Urban
70
Midwest
U Master's
31
69
78
2985
37
Suburb
93
West
U Master's
32
68
23
1096
60
Urban
94
i
j
North
Bacc. Coll
92
58
62
8568
16
Suburb
42
Midwest
U Master's
28
72
91
1502
37
Urban
79
l
North
U Master's
24
76
79
1320
815
Suburb
60
18
participating institutions by region
West
9%
Midwest
31%
South
17%
Is there
sufficient
variability?
North
43%
2008 = 2 consortia, 54 institutions
19
“urbanicity” of participating institutions 2008
Urban
31%
Is there
sufficient
variability?
Rural
15%
Suburban
54%
20
MPI 2008 Scales:
Comparison of means by institution type
Scale
Sense of Mission
Respect for Diversity
Spiritual Practice
MPI
NSSE Beliefs & Values
Mean Score
Independent
Catholic
3.77
4.04
4.03
4.00
1.37
1.46
3.46
3.63
2.57
2.62
t(df)
-2.637(14.356)
0.330(40.000)
-1.310(14.856)
-2.296(14.756)
-0.882(15.876)
2-tailed sig
.019*
NS
NS
.037*
NS
*p≤.05
21
Correlation of regression variables
with MPI mean
Correlations
religious_affliation
usn_setting
usn_setting
pct_residence
mpi_mean
-.203
-.258
.429
Sig. (2-tailed)
NS
NS
.005
N
42
42
42
Pearson Correlation
.590
Sig. (2-tailed)
.000
.001
42
42
Pearson Correlation
Sig. (2-tailed)
N
-.484
**
Pearson Correlation
N
pct_residence
**
**
-.456
**
.002
42
**. Correlation is significant at the 0.01 level (2-tailed).
22
Deriving coefficients
for the Mission Engagement Index (MEI)
Model
R
Model Summary
Adjusted R Std. Error of
R Square
Square
the Estimate
1
.484a
.234
.215
b
2
.591
.349
.315
a. Predictors: usn_setting
b. Predictors: usn_setting, religious_affliation
.163
.153
23
Correlation of regression variables
with Mission scale mean
Correlations
usn_setting
religious_affliation
usn_setting
Pearson Correlation
mission_mean
-.258
Sig. (2-tailed)
NS
NS
.001
N
42
42
42
Pearson Correlation
N
Pearson Correlation
Sig. (2-tailed)
N
.590
**
.486
**
-.203
Sig. (2-tailed)
pct_residence
pct_residence
-.429
**
.000
.005
42
42
-.268
.086
42
**. Correlation is significant at the 0.01 level (2-tailed).
24
Predictive Equation*
Institution Predicted MPI Score =
(Beta1)*(Value of “setting”)
(-0.414)*(1, 2, or 3)
(Beta2)*(Value
of
“institution type”)+
(0.345)*(1 or 2)
Constant
(3.687)
*Mortenson, T. (1997). Actual Vs Predicted institutional graduation rates for 1100 Colleges and universities. Opportunity, 58.
25
Proposed
Mission Engagement Index


The MEI will compare actual versus predicted scores
on mission constructs.
Progress on mission effectiveness can be assessed
by comparing MEI outcomes to institutional goals.
26
Sample
Mission Engagement Index (MEI)
by Institution Type and Setting
27

For statistical regression, cross validation with a
second sample is highly recommended (Tabachnick, p. 153).
No rest for the weary…
28
References
Inside Higher Education (2009). Spiritual accountability. Retrieved from
http://www.insidehighered.com/layout/set/pri...assessment/01/02/2007/News
Kuh, D. G., Kinzie, J., Schuh, J. H., and Whitt, E. J. (2005). Never let it rest:
lessons about student success from high-performing colleges and universities.
Change, 37(4), 44-51.
Mortenson, T. (1997). Actual Vs predicted institutional graduation rates for 1100
colleges and universities. Opportunity, 58.
Pacarella, E. T. (2001). Identifying excellence in undergraduate education.
Change, 33(3), 18-27.
Stevens, J. P. (1992). Applied multivariate statistics for the social sciences (2nd
edition). Hillsdale, New Jersey: Erlbaum.
Tabachnick, B. G. & Fidel, L. S. (1996). Using multivariate statics, third edition.
New York, NY: HarperCollins.
29
Chestnut Hill College
Philadelphia
NorthPA
Bennington College
Christ the King Seminary
East Aurora
SouthNY
Berry College
Walsh University
Our Lady of Corpus Christi
Christendom College
Front
Royal
SouthVA
Birmingham-Southern College
Our Lady of Holy Cross College
Bennington College Christian Brothers University
Memphis
NorthTN
Clark University
Our Lady of the Lake College
Clarke College
Dubuque The United
NorthIA College
Clarkson University
of San Antonio
Berry College
of the Lake University
College of Mount Saint Joseph
Cincinnati Thomas
OH Our Lady
NorthAquinas
Daemen College
College
Birmingham-Southern College
Collegeof Mount Saint Vincent
Riverdale
NY Presentation College
SouthMore
Faulkner University
Thomas
College
Clark University
College of New Rochelle
New Rochelle
NY Providence College
SouthMore
Florida Hosp College of Health
SciThomas
College
ofRosary
Liberal
Queen
of the Holy
CollegeAr
Our
Lady of University
Corpus Christi
College of Notre Dame of MarylandManchester College
Baltimore
MD
Midwest
Clarkson
Quincy University
Trinity University
Our Lady of Holy Cross College
College of Saint Benedict
NorthMN Regis College
Manhattanville College St. Joseph Trocaire
Daemen
College
College
College of SaintDonnelly
Catherine
St. Paul Kansas CityNorthMN
Our Lady of the Lake College
College
KS University
Marlboro College
University of Regis
Dallas
Faulkner
University
College of SaintDuquesne
ElizabethUniversityMonmouth
of the Holy Spirit
PA College
Our Lady of the Lake University
University MorristownPittsburgh NorthNJ
University
of Rivier
Dayton
College
of
Saint
Joseph
in
Vermont
Rutland
VT
D'Youville
College
Buffalo
NY
Florida
Hosp
College
of
Health
Sci
Midwest Rockhurst
Mountain State University
Presentation College
University
University
of Detroit
Mercy
College of SaintEdgewood
Mary
Omaha
NE
College
Madison
WI
West
Rosemont
College of the Holy Child
Naropa
University
Providence
College
Manchester College College of Saint Rose
University
of
Great
Falls
Albany
NY
Elms College
Chicopee
MA
Sacred Heart School of Theology
North
Paul Smiths College
Queen
of the Holy College
RosaryCollege
Collegeof Saint Scholastica
Manhattanville
University
of
Mary
Duluth
MN
Emmanuel College Springfield College
Boston
MA Heart University
Sacred
North
Quincy University
College of SaintFairfield
ThomasUniversity
More
Fort
Worth
TX
Fairfield
CT Ambrose
University
of
Notre
Dame
du Lac
Marlboro College
Saint
University
South
Stetson University
Regis College
College
NJ Anselm College
College of SantaFelician
Fe
Santa Fe Lodi
NM
University
of
Portland
Saint
South
Webster University Worldwide
Monmouth
University College of the Holy
Fontbonne
Louis
MO Anthony College of Nursing
Regis
University
Cross University
WorcesterSt.University
MA Sacramento
North of Saint
Yeshiva University
Mountain
State
University
Fordham
University
Bronx
NY Bernard's
Creighton
University
Omaha
NE
School of Theology a
Rivier College
University of Saint
Saint
Francis
Franciscan
School
of
Theology
Berkeley
CA Bonaventure University
De Paul University
Chicago University
IL of Saint
Naropa University
Rockhurst
University
Saint Francis
Franciscan University of Steubenville
Steubenville
OH Catharine College
University
Center Valley
PA Saint
Rosemont
College
of the DeSales
Holy Child
Jesus
University
of
Saint
Mary
Paul Smiths
College
Gannon
University
Erie
PA Charles
Saint
Borromeo Seminary
Divine Word College
Epworth
IA
Sacred
Heart School
University of Saint
Saint
Mary
of the Lake
Springfield
Collegeof Theology
Georgetown University
Washington
DC Edward's
University
Dominican College
Orangeburg
NY
University
of
Saint
Thomas
Sacred Heart University Dominican House
Georgian
Court University
Lakewood
NJ Elizabeth College of Nursing
of Studies
Washington
DC Saint
Stetson University
Our Lady of Corpus Christi
University
of
Saint
Thomas
Saint Ambrose University Dominican School
Gonzaga
University
Spokane
WA Elizabeth
Saint
School of Nursing
of Philosophy and Theology
Berkeley
CA
Webster
University
Our Lady of Holy Cross College Gwynedd
Gwynedd-Mercy
College
Valley
PA
University
of
San
Diego
Saint
Francis
College
Saint
Anselm
College Worldwide
Dominican University
River Forest
IL
Our Lady of the Lake College
Hilbertof
College
Hamburg
NY Francis
Medical Center College
Yeshiva
University
University
San
Francisco
Saint
Anthony
College of Nursing
Dominican University
California
San Rafael
CAof Saint
Our Lady of the Lake University Cromwell
Holy Apostles College & Seminary
CT
Saint
Francis
University
University of Scranton
Saint Bernard's School of Theology and Ministry
Presentation College
Holy Cross College
Notre
Dame
IN
Saint
Gregory's
University
University of the Incarnate
Word
Saint Bonaventure University
Providence College
Holy Family University
Philadelphia
PA
Ursuline College
Saint Catharine College
Queen of the Holy Rosary College
Holy Names University
Oakland
CA
Villa Maria College
of Buffalo
Quincy University
Saint Charles Borromeo Seminary
Immaculata University
Immaculata
PA
Villanova
University
Regis College
Iona College
New
Rochelle
NY
Saint Edward's University
Regis
University
Viterbo
University
Jesuit
School
of
Theology
Berkeley
CA
Saint Elizabeth College of Nursing
Rivier College
Walsh University
John Carroll University
Cleveland
OH
Saint Elizabeth School of Nursing
John Paul the Great CatholicRockhurst
University University
San
Diego
CA
Washington
Theological
Union
Saint Francis College
Rosemont College of the Holy Child
Jesus
King's College
Wilkes-Barre
PA
Weston Jesuit School of Theology
Sacred Heart School of TheologyPittsburgh
Saint Francis Medical Center College of La
Nursing
Roche College
PA University
Wheeling Jesuit
Sacred Heart University
Saint Francis University
Wyoming Catholic College
Saint Ambrose University
Saint Gregory's University
Xavier University
Saint Anselm College
Saint Anthony College of Nursing Xavier University of Louisiana
Discussion:
30
Institutional Characteristics & Student Perception of Mission:
What Makes a Difference?
Ellen Boylan, Ph.D.
Director of Institutional Research and Assessment
[email protected]
Office of Planning and Institutional Research
http://cwis.marywood.edu/instresearch/activity.stm
Marywood University
Supported by a grant from
http://www.teaglefoundation.org/grantmaking/grantees/assessmentmethods.aspx
31
32
MPI
Mission Perception Inventory
Institutional Characteristics and Student Perception of Mission:
What Makes a Difference?
Ellen Boylan, Ph.D. ● Marywood University
NEAIR 36th Annual Conference ● Baltimore, Maryland ● Nov. 7 – 10 , 2009