Got Benchmarking Data? Now what?

NIRSA Pre-conference April 1, 2009

Overview

• Introductions: Take a note of colleges/universities represented that might be most like your campus. • General overview of benchmarking/intended outcomes • Definition of and types of benchmarking • Considerations and challenges of benchmarking • NIRSA/StudentVoice Benchmarking Study • Making sense of benchmarking data • Assessment reporting • Panel of campus recreation professionals • Discussion/questions throughout!

Intended Outcomes

• Participants will leave with an increased understanding of the value of benchmarking • Participants will get hands-on experience viewing and understanding benchmarking data • Participants will learn how to utilize benchmarking data for public relations, marketing, and importantly, they will leave with ideas on how to report data in meaningful ways to their various audiences • Armed with new confidence around understanding and interpreting benchmarking data, participants will leave with an increased love of assessment and data!

Session Warning Labels

• Three hours is a long time… especially if you are talking assessment! (We have one planned break around 9:45, but take breaks as needed.) • Warning: High doses of assessment are likely.

• Some of the information is not going to make sense until your college/university actually participates in benchmarking. • Getting the most out of the group exercise (of interpreting benchmarking data) is going to be based on what emotions ‘data’ and assessment invoke in you! • Set some of your own goals/outcomes for the session: Do you want to show others on your campus how to use the data? Do you want to learn best practices from other campuses? Do you want to be able to go back to your campus and produce a report? What else?

• It’s ok if you don’t get ‘it’ all! StudentVoice benchmarking webinars are available each May and December for

free

. Or I can do a webinar for you and others on your campus specific to your data.

Benchmarking

• Comparing performance and other measures across organizations/units.

• Benchmarking is done primarily to benefit from wisdom and success of others (best practices) so that you don’t have to “reinvent the wheel.” • Allows institutions or departments to document what they are doing well (in comparison to national or peer data), as well as document opportunities for improvement.

Uses of Benchmarking

• Improvement; learning of best practices • Short and long term planning • Forecasting and predicting trends • Generating new ideas • Provide data for analyzing and determining affordability and cost effectiveness • Exposes areas that are in need of follow-up/more in-depth assessments

Types of Benchmarking

Types (typical in higher education)

1) Internal: departmental comparisons 2) External: comparisons to peer institutions • Operational benchmarking (comparing staffing, facilities, programs, budgets) – may occur both internally and externally • Student data benchmarking (comparing student responses) – may occur both internally and externally

Benchmarking Considerations

• Common questions asked across institutions • Timing of the data collection • Sampling methods • Selecting comparable institutions for comparisons • Many benchmarking opportunities have very strict guidelines for participating campuses (rightly so – to ensure that the data is valid and reliable)

History of the NIRSA/StudentVoice Benchmarking Project

• Inspired by assessment efforts of Juliette Moore, Director of Recreational Sports at the University of Arizona and NIRSA Past President, the

Campus Recreation Student Outcomes Benchmarking Project

provides a benchmarking mechanism to capture ways in which campus recreation enhances the lives of students.

• Online assessment to random sample of students (sometimes faculty, staff, other stakeholders).

• Over 85 colleges and universities have participated in the last five years.

• Data from over 70,000+ students, faculty, staff, and community members.

Campus Use of the Data

• Data on the impact of campus recreation programs and services on recruitment and retention • Document student learning outcomes • Justify new facilities, expansion, new programs • Change and enhance existing programs and services • Determine support for fee increases • Gauge interest in new programs and services • Implications for staff training and professional development • Spotlight areas of achievement and areas for improvement • Guide future/follow-up assessments

Benchmarking Topics

• Importance of recreation on recruitment and retention • Usage of facilities, services, programs, and offerings • Barriers to participation • Satisfaction • Outcomes • Needs assessment • Marketing/Promotions • Feasibility questions (if appropriate) • Demographics

Are we really measuring outcomes?

• •

Program Outcomes

examine what a program or process is to do, achieve or accomplish for its own improvement; Generally needs/satisfaction driven.

Learning Outcomes

examine cognitive skills that students develop through department interactions; Measurable, transferable skill development; what

students

are expected to

demonstrate

in terms of knowledge, skills, and attitudes upon completion of a program, course, interaction, or activity.

Direct versus Indirect Assessment Methods

• •

Direct Methods -

Any process employed to gather data which requires subjects to display their knowledge, behavior, or thought processes.

Indirect Methods -

Any process employed to gather data which asks subjects to reflect upon their knowledge, behaviors, or thought processes.

Process

• Sign-up for the NIRSA/StudentVoice benchmarking project. • Customize instrument as appropriate – keeping in mind that any changes to the instrument may impact the ability to benchmark.

• Obtain institutional approval/IRB – since your data will become part of an aggregate dataset. • Launch the project; All data is collected online preferably with a random sample of students.

• Institution-specific data is available in real-time for immediate use.

• Benchmarking data and reports are available once all campuses are done collecting data.

• No college or university names are revealed alongside any college or university data.

Viewing Benchmarking Data

Data Reporting Tools

Benchmarking Wizard

Making sense of multiple layers of data

• Institutional data including filtered views to understand sub-populations or respondents with certain characteristics • Longitudinal data • National results • Peer comparison data (who do you want to be able to compare with?) • Filtered views by sub-populations across national and peer comparison data

What’s in a StudentVoice benchmarking report?

• • Frequency tables For scaled items: • Means and standard deviations • Whether the difference in means is statistically significant (green = ‘your’ mean is statistically higher at the p<.05; red = ‘your’ mean is statistically lower).

NOTE: Statistical significance occurs with large sample sizes and the significance indicators need to be considered with this in mind. Advanced statistical analysis should be conducted in a stats package such as SAS or SPSS.

• Total N (number of responses), if this number is not ‘private’ • Top 2 and Bottom 2 percentages (Top 2 = Strongly agree plus somewhat agree) • Ranking of ‘your’ mean score • No basis to judge / Not applicable are excluded from all mean calculations. All benchmarking questions utilize Likert scales of 1-5, 1-4, or 1-3.

Stats 101

What basic stats knowledge do you need (or at least need to be aware of) to interpret a benchmarking report?

• Sample size and response rates (http://www.touchpoll.com/calculator.htm

) • Generalizability • Descriptive stats: Basic frequency tables • Likert scales versus categorical data • Means and standard deviations • T-tests: Do two mean scores differ?

• Statistical significance More advanced uses of assessment data: • Factor analysis • Correlation studies • Regression

Small Group Work

• Find a work group with similar types of campuses or similar types of recreation facilities and programs • Maximum of four-five people per group will be best • Knowing that each small group might approach the exercise differently, if you are attending with another colleague from the same institution, it might make sense to split up and then share experiences afterwards.

Group Exercise Scenario

Let’s look at page 1 of the handout labeled Benchmarking Data Questions 2 and 3 together in a large group to make sure all is clear about how the data is presented and what is in the report.

Let’s also look at page 1 of the handout labeled Benchmarking Data Question 4 together.

Reporting out your findings

• Create manageable sections/chunks of data.

• Assign/delegate to staff or teams and come up with an appropriate strategy to make the data analysis feasible and manageable.

• Explore possibility of hiring a graduate assistant to assist with data analysis or find other campus partners.

Assessment Reporting

1) Determine the ‘client,’ purpose, and audiences for the report 2) Determine what data and information are available or must be gathered 3) Select the type of report and report format 4) Decide how to depict data and information 5) Produce report 6) Dissemination

Findings vs. Results vs. Implications

• Findings: Data represented as value-free facts. What is the data?

• Results: Interpretations of the findings. What do the findings mean?

• Implications: Uses of the results. Who should care about these results and what should they do with the results?

Knowing your Audience

• What do my audiences

need

to know about this subject?

• What do my audiences

want

to know about this subject?

• What do

I want them to know

about this subject?

• What decisions will or might my audience make based on this report?

• What other audiences might my primary audiences send this report to, even if I haven’t intended for them to receive the material. • What other audiences might be interested in the same subject?

(Bers & Seybert, 1999)

Report Components

1) Meaningful title 2) Executive summary 3) Table of contents 4) Introduction and purpose 5) Methodology 6) Findings 7) Summary, conclusion, implications, and recommendations 8) References 9) Glossary 10) Appendices

Graphical Representations of Data

Graphs are depictions of numerical relationships to demonstrate the following: 1) Size (e.g., largest to smallest) 2) How things change over time 3) What is typical or exceptional 4) How one thing predicts or is related to another Avoid chart clutter/chart junk….

Huh?

Working - Studying & Grades Transfers - Fall 2006

80% 70% 60% 50% 40% 30% 20% 10% 0% 3.40

3.20

3.00

2.80

2.60

2.40

2.20

0 1-5 6-10 11-15 16-20 21-25 26-30 31-35 36-40 >40

Work Hour Category

2.00

Studying More than 10 Hours Per Week Average First Semester GPA

Presenting Clear Visual Representations of Data

Panelists

• Amy Davenport: University of Oklahoma • Mary Chappell: University of Kansas • Tina Clawson: Oregon State University

Opportunities to share best practices after today:

https://www.studentvoice.com/app/views/community/

Online Resource Center

Additional Resources

Bender, B. E., & Schuh, J. H. (2002). Using benchmarking to inform practice in higher education. New Directions for Higher Education no. 118. San Francisco: Jossey-Bass Taylor, B. E., & Massey, W. F. (1996). Strategic indicators for higher education. Princeton, NJ: Peterson’s.

Upcraft & Schuh, (1996). Assessment in Student Affairs. San Francisco: Jossey-Bass.

Contact Information

Kim VanDerLinden www.studentvoice.com

Email: [email protected]

716.652.9400 and then press 1