Regional Accreditation and Learning Outcomes Assessment
Download
Report
Transcript Regional Accreditation and Learning Outcomes Assessment
VOLUNTARY SYSTEM OF
ACCOUNTABILITY AND LEARNING
OUTCOMES: AN UPDATE
Teri Hinds
Voluntary System of Accountability
Natasha Jankowski
National Institute for Learning Outcomes Assessment
October 30, 2012
Assessment Institute
Overview
NILOA’s work with Transparency
Internal and External Communication
Voluntary System of Accountability
Evaluation of VSA College Portrait Pilot
Future of Student Learning Outcomes on College
Portrait
VSA as a communication tool
The Heat is On
Unprecedented public challenges for higher
education to be more accountable for results.
Accompanying external demands for information
about student and institutional performance are
growing calls for institutions and accreditors to
become more transparent about what they do
and the results they achieve.
(Ewell, 2010)
Overview of NILOA
NILOA’s mission is to document student learning
outcomes assessment work, identify and disseminate
best practices, and support institutions in their
assessment efforts.
SURVEYS ● WEB SCANS ● CASE STUDIES ● FOCUS GROUPS ● OCCASIONAL PAPERS ● WEBSITE
● RESOURCES ● NEWSLETTER ● LISTSERV ● PRESENTATIONS ● TRANSPARENCY FRAMEWORK ●
FEATURED WEBSITES ● ACCREDITATION RESOURCES ● ASSESSMENT EVENT CALENDAR ●
ASSESSMENT NEWS ● MEASURING QUALITY INVENTORY ● POLICY ANALYSIS ● ENVIRONMENTAL
SCAN ● Degree Qualifications Profile
www.learningoutcomesassessment.org
www.learningoutcomeassessment.o
Reports on Transparency
Question
What information does your institution share
publically on its website?
How are results of student learning shared
internally?
Transparency Framework
http://www.learningoutcomesassessment.org/TransparencyFramework.htm
Transparent Communication
Internal communication
External communication
Telling an institutional story about student learning
Questions
What types of stories does your institution tell about
its students?
What types of stories does your institution tell about
how it assures quality?
Background
VSA launched in 2007
Partnership between APLU and AASCU
320 participating institutions
Three original objectives
Demonstrate
accountability and transparency
College search tool
Support innovation in the measurement of
student learning outcomes
Student Learning Outcomes Pilot
Student learning outcomes page includes link to
institution specific learning outcomes data;
Selection of the CLA, CAAP or ETS Proficiency
Profile to measure value-added learning gains
Report results no more than four years after
joining
Link to information on test administration, sample
and response rate
Pilot period ends 12/31/12
What was learned from the pilot?
Evaluate the Effectiveness of SLO Pilot
Focus groups
Interviews
Surveys
Google Analytics
College Portrait statistics
Findings related to Transparency
Both VSA-participating and nonparticipating institutions agree
that VSA in 2007 was a wise, timely, useful, and necessary
response to the accountability and transparency demands of the
time
Yet, nearly half of the participants have not yet met
expectations of College Portrait student learning outcomes pilot
45% of participating institutions have yet to post student
learning outcomes information
Information that was provided on student learning outcomes
lacked explanatory or contextual material
82% of participating institutions had traffic on their
student learning pages but almost 1 in 5 did not have
a single visitor to their student learning outcomes page
since 2009.
College Cost Estimator
College Cost and Financial Aid
Students
Admissions
Student Success and Progress
Classes & Campus Life
Majors, Graduation, & Next
Student Learning Outcomes
Student Experiences
0
20
40
Percentage of Total Views
60
Information posted on the College Portrait may
not reflect the needs of prospective students and
families or provide the information they actually
seek to make decisions about where to attend
college.
The authorized standardized test measures of
student learning outcomes lack broad credibility
and acceptance in the higher education
community – it is thought the tests do not
accurately reflect student performance within
the institution.
Concerns about displaying results include:
Inability to understand what scores represent
Questions regarding what tests measure
Issues of reliability and validity
Difficulty in using scores to improve student learning
Motivation of students & difficulty to obtain a sample
Questions regarding quality of results
81% of survey respondents stated that the pilot requirements did not
align with their assessment practices and that the results were not
usable for campus improvement efforts
Participating institutions agree - expanding the
number & nature of student learning measures will:
produce more accurate portrayals of student attainment,
provide more useful information for campuses,
make information more meaningful for all audiences.
Recommendations
Focus on specific audiences and communicating
meaningful information to them
Contextualize information
Expand range of assessment tools and approaches
Increase traffic to site
Include examples of use of evidence to improve learning
Consider creating template which could be used by all
postsecondary institutions
Working Group Deliberations
Technical work group considered alternative
measures of learning outcomes to recommend to
VSA Board
Confirmed importance of student learning
outcomes reporting within the VSA
No perfect of measure of students learning exists
for all audiences
External
accountability
Institutional improvement
College selection
Working Group Deliberations
Make value-add measurement using existing
measures (CAP, CLA, ETS Proficiency Profiler)
optional
Introduce option to use VALUE Rubrics based on
AAC&U essential learning outcomes (written
communication and critical thinking)
Introduce option to use aggregate scores from
professional and graduate admissions exams
such as the GRE, GMAT, LSAT, and MCAT
Work Group Ratings of Recommended VSA Instruments in Key Areas
Instrument
1= lowest rating
10=highest rating
CLA
CAAP
ETS PP
(short
form)
Allow comparisons, benchmarking across
institutions?
6
8
9.5
9.5
6
7
Offer a representative sample?
7
7
7
7
7
2
inst-2
class-7
5
2
2
9
3
Transparency of method, results
6
8
8
8
9
8
External accountability
8
7
7
7
5
5.5
Ease of administration on campus
5
6
7
6
4
9.9
Reasonable costs (time, resources)
5
7
7
6
startup -3
Motivation for students to do well?
3
2
3
2
3
2
3
2
5.5
6
6
6
Usefulness to faculty/students for learning
improvement?
Likely interest to consumers?
Valid learning outcomes instrument for
average student?
ETS PP
(long
form)
VALUE
Rubrics
GRE
marginal- 8
9
9
9.9
5
6
portfolios -9
assign -7
3
VSA Board Decisions
College Portrait re-focus from “college selection tool for
prospective students” to “consumer information tool for
institutions to demonstrate effectiveness of educational
programs.”
Primary audiences: state officials, policy makers, consumers
Instrument options:
CAAP, CLA, ETS Proficiency Profile
AAC&U VALUE rubrics – critical thinking, written
communication
GRE General Test
Reporting options: value-added, benchmarking
Implementation/Next Steps
All VSA institutions report results of pilot by end of 2012
to fulfill reporting requirement
3 part report with tabbed display
Did
you participate (or try to)? If so, how’d it go? If not, why
not?
Give an example of another institutional-level evaluation
Which of the new options are you considering for the future?
Reporting
Format Preview:
http://www.collegeportraits.org/slo_preview.html
Implementation/Next Steps
Communications Task Force working with College Portrait
Support Team on display of new options
Emphasis on more graphical displays
Additional guidance on what the scores and results mean
Click-through links for additional details, e.g., subscores
New options released for 2012-13 data cycle (Jan 2013)
Develop outreach strategy to recruit new participants and
connect with key audiences – state officials, accreditors
Websites
www.voluntarysystem.org
www.collegeportraits.org
www.learningoutcomesassessment.org
Teri Hinds: [email protected]
Natasha Jankowski: [email protected]
Let’s Discuss.