Christine Keller Voluntary System of Accountability Paul Gore University of Utah Orlando, Florida ▪ Session #19 Southern Association for Institutional Research September 22-25, 2012

• • • • Review of original goals of VSA and student learning outcomes Selected findings and recommendations from NILOA evaluation of VSA Pilot Application of evaluation results: technical work group, communications advisory group Next steps for VSA: wrap up pilot, new reporting options, refocus/rebranding

Initiative by public universities to supply straightforward, comparable information on the undergraduate student experience through a common web report – the

College Portrait

.

www.collegeportraits.org

• • • •

VSA developed, launched in 2007 Sponsored by APLU and AASCU 320 participating public universities Three original objectives

– Demonstrate accountability, transparency – College search tool – Support measurement and reporting of student learning outcomes

• Goal: Directly measure, publicly report student learning gains (value-added) at institution level using a common method – Skills: critical thinking, analytic reasoning, problem solving, written communication – Tests: CAAP, CLA, ETS Proficiency Profile • Reporting includes link to institution specific learning outcomes data • Pilot period ends December 2012

• Conducted by National Institute for Learning Outcomes Assessment (NILOA) – Focus groups – Interviews – Surveys – Google Analytics – College Portrait statistics

NILOA’s mission is to document student learning outcomes assessment work, identify and disseminate best practices, and support institutions in their assessment efforts.

S URVEYS P ● R M APERS T ● W EB ● W EBSITE RANSPARENCY ESOURCES EASURING Q ● S A UALITY CANS ● R ESOURCES SSESSMENT I F ● C ASE RAMEWORK NVENTORY ● S TUDIES F ● F ● N EWSLETTER EATURED OCUS E VENT ● P OLICY C ALENDAR A NALYSIS W ● G ROUPS ● L ISTSERV EBSITES ● ● O CCASIONAL ● P RESENTATIONS A CCREDITATION A SSESSMENT ● E NVIRONMENTAL S N EWS CAN ●

www.learningoutcomesassessment.org

• Both participating and nonparticipating institutions agree that VSA in 2007 was a wise, timely, useful, necessary response to the accountability , transparency demands of the time • Many eligible institutions participate in the VSA about 1/3 - do not • Nearly half of the participants have not yet met expectations of student learning outcomes pilot – 50% of participating institutions have yet to post student learning outcomes information (Sept 2012)

College Cost Estimator College Cost and Financial Aid Students Admissions Student Success and Progress Classes & Campus Life Majors, Graduation, & Next Student Learning Outcomes Student Experiences 0 20 40

Percentage of Total Views

60

• College Portrait: “ Information posted may not reflect the needs of prospective students, families or provide the information they seek to make decisions about where to attend college.” • Student Learning Outcomes: “ The standardized test measures of student learning outcomes lack broad credibility and acceptance in the higher education community, undermining institutional participation and engagement with the VSA and campus faculty and staff support of the VSA initiative.”

“Mend it, don’t end it”

•

Expand range of assessment tools and approaches

•

Focus on specific audiences and communicating meaningful information

• Technical work group considered alternative measures of learning outcomes to recommend to VSA Board • Confirmed importance of student learning outcomes reporting within the VSA • No perfect of measure of student learning exists for all audiences – External accountability – Institutional improvement – College selection

• Continue use of value-added measurement using CAAP, CLA, ETS Proficiency Profile • Introduce option to use VALUE Rubrics based on AAC&U essential learning outcomes – written communication – critical thinking • Introduce option to use aggregate scores from professional and graduate admissions exams such as the GRE, GMAT, LSAT, and MCAT

Work Group Ratings of Recommended VSA Instruments in Key Areas Instrument

1= lowest rating 10=highest rating

CLA CAAP ETS PP (short form) ETS PP (long form)

Allow comparisons, benchmarking across institutions? Offer a representative sample?

Usefulness to faculty/students for learning improvement?

Transparency of method, results External accountability Ease of administration on campus

6 7

inst-

class-7 6

8

5

2

8 7 5 8 7 6

9.5

7

2

8 7 7

9.5

7

2

8 7 6

Reasonable costs (time, resources) Motivation for students to do well?

Likely interest to consumers?

Valid learning outcomes instrument for average student?

5

3 2

5.5 7

3 2

6 7

3 2

6 6

3 2

6

VALUE Rubrics

6 7

9 9

5

4

startup -

3

marginal- 8

9

5

portfolios 9

assign -7

GRE

7

2

3

3

8 5.5

9.9

9 9.9

6

• • • College Portrait re-focus from “ college selection tool ” to “ consumer information tool to demonstrate effectiveness of educational programs ” Instrument options: – CAAP, CLA, ETS Proficiency Profile – AAC&U VALUE rubrics – critical thinking, written communication – GRE General Test Reporting options: value-added, benchmarking

• • VSA participants describe, publish pilot project experiences using three part form – – – Required for institutions with no SLO results published Strongly encouraged for all institutions Data entry opens week of Oct 1, closes Jan 7 New SLO reporting options released for 2012 13 data cycle (week of Jan 18, 2013) – – Deadline: Spring of 2015 No data more than 3 years old

• • • Report overall experiences participating in the pilot – whether 1 of the 3 value-added tests was administered or not. Provide information about an additional institutional assessment initiative on campus, including results from a recent administration and/or cycle. Disclose which of the newly approved SLO reporting options your campus is likely to consider for future VSA reporting.

http://www.collegeportraits.org/slo_preview.html

• Includes individuals from public affairs, communications, government affairs – Develop outreach strategy to recruit new participants and connect with key audiences – Identify and promote key elements on College Portrait: success and progress rate, cost of attendance, financial aid, future plans – Create streamlined view targeted at policy makers, government affairs professional

• • Unprecedented public challenges for higher education to be more accountable for results.

Accompanying external demands for information about student and institutional performance are growing calls for institutions and accreditors to become more transparent about what they do and the results they achieve. (Ewell, 2010)

Reports & Presentations www.voluntarysystem.org

Christine Keller, VSA Executive Director [email protected]

Teri Hinds, VSA Associate Director [email protected]

Paul Gore, U of Utah [email protected]