Institutional Research and the Campus Commitment to a Culture of Assessment

AIRPO Conference June 20, 2006 James A. Anderson, Ph.D.

Vice President for Student Success Vice Provost for Institutional Assessment and Diversity University at Albany [email protected]

Integrative Assessment Reporting

Institutional Goals or Objectives College/Division/Accreditation Goals and Objectives Department/Program Goals/Objectives/Outcomes Course-Bases Objectives/Outcomes

Integrated Assessment Process Means Integrating

Institutional Effectiveness Program Review Outcomes Assessment Institutional Research Enrollment Management Survey Research

Student-Centered Learning Environment

Refers to a student success effort which is characterized by: – Clearly defined and measurable outcomes in the areas of teaching, learning, retention, and student social-psychological development – Front-loaded activities which engage the student, as soon as possible, to the academic and social expectations of the institution – An ongoing assessment effort associated with student and program improvement – Faculty and staff who are well-trained to deliver the appropriate services

Profiling the Student Support Clientele

1.

2.

3.

4.

5.

What institutionally based information on student needs and characteristics is utilized by student support personnel?

What specific information about student subpopulations (women, students of color, nontraditional students, etc.) is utilized by student support personnel? What information about student culture, Faculty concerns and campus climate is utilized by student support personnel?

Describe a profile of the students who utilize student support services the most/least?

Who benefits the most/least?

Generate Longitudinal Database to Monitor Key Indicators of Student Success

Examples Overall course completion ratios Successful completion of gatekeeper courses Successful completion of developmental educational courses Semester-to-semester retention Year-to-Year retention Degree and certificate attainment rates From: “Achieving the Dream: Community Colleges Count” Lumina Foundation & Pell Institute

Retention Scorecard

Indicators of Institutional Effectiveness Indicators of Program Effectiveness Indicators of Student Academic/Social Integration Indicators of Student Success/Achievement Indicators of Student Learning Indicators of Impact of/on Diversity

Questions That May Help You Recast Your Services or Delivery Systems What are the thinking tasks, intellectual experiences, and/or cocurricula experiences that need to be designed relative to the preparation level and diversity of the students at your institution?

What structures need to evolve to assure that students have the opportunity to enhance their academic self-concept and understand their role in the culture of learning at your institution?

What data sets do I need to access to generate the appropriate questions and answers about student engagement and student learning?

Academic success |

Developmental/ general curriculum courses

|

Academic supports

|

Study skill assessment

|

Test-taking skills

|

Active learning strategies

Integrated Model of Undergraduate Education Transition to First Year of College |

Intrusive advising

Assessment and Self Discovery |

Learning styles

|

College course analysis and time management

|

Information and computer literacy

|

Co-curricular experiences Adapting to Culture of College

|

Non-cognitive factors

| |

Career exploration Active learning strategies

ACADEMIC OUTCOMES

Grades (General) Grades (By Course Type) Retention (Semester to Semester) 1 st to 2 nd Year Graduation Rates Time to Degree (Calendar vs. Seat Time) Changes in Academic Interests Transfers to Four-Year Institutions

STUDENT OUTCOMES

Academic Adjustments/Integrations Social Adjustments/Integration Attitudes Towards Learning Expectations for Success Academic Self-Esteem Utilization of Academic Supports Non-Cognitive Factors

Questions to Ask Yourself

Which outcome and assessment methods will help me understand what it is that I am doing that is leading to the outcome?

Which outcome and assessment methods help me understand why I am doing what I am doing?

Will this kind of evidence help me make the decisions I need to make?

Components of An Assessment Plan

Mission Objectives Outcomes Evaluation Methods – By Outcomes Implementation of Assessment – Who is Responsible for What?

– Timeline Results Decisions and Recommendations

Clark College (Allied Health Program)

Category A – SAT – HS Transcript – Personal Interviews – Recommendations Category B – California Study Test – Coop Eng. Test – Watson-Glaser Critical Thinking – Appraisal Category C – Basic Information Sheet (Personal Information & Personal/Social Adjustment)

Program Outcomes

Students in your program will develop a positive attitudes towards learning and increase their commitment and responsibility for their own learning – Pre-post gains or comparison with matched pair cohort on dimensions like academic skills, critical thinking, values toward learning, etc.

– Develop a plan of study – Develop a learning portfolio – Learning environment preferences Survey (Perry Scheme)

Reasons for Learning Style Assessments that Incorporate Diversity

Self-assessment feedback Cohort comparisons Cluster analysis of behaviors Development of effective cooperative clusters Matching of learning styles/teaching styles Correlation with other dimensions Identification of critical dimensions

Meaningful Use of Data

from Peggy Maki, Ph.D.

Collect data from different sources to make a meaningful point (

for example, program samples and other samples of student work).

Collect data you believe will be useful to answering the important questions you have raised.

Collect data that will help you make decisions for continuous improvement.

Organize reports around issues, not solely data.

Interpret your data so that it informs program improvement, budgeting, planning, decision making, or policies.

Some Methods That Provide Indirect Evidence

from Peggy Maki, Ph.D.

Alumni, Employer, Student Surveys Focus groups Exit Interviews with Graduates Graduate Follow-up Studies Percentage of students who go on to graduate school Retention and Transfer Studies Job Placement Statistics

Indirect Evidence

Cont.

Courses selected or elected by students Faculty/Student ratios Percentage of students who study abroad Enrollment trends Percentage of students who graduate within five-six years Diversity of student body CAS Standards

Some Methods That Provide Direct Evidence

Student work samples Collections of student work (e.g. Portfolios) Capstone projects Course-embedded assessment Observations of student behavior Internal juried review of student projects Evaluations of performance

Direct Evidence

from Peggy Maki, Ph.D.

Cont.

External juried review of student projects Externally reviewed internship Performance on a case study/problem Performance on problem and analysis (Student explains how he or she solved a problem) Performance on national licensure examinations Locally developed tests Standardized tests Pre-and post-tests Essay tests blind scored across units

Building an Assessment Website

Easy to navigate Definitions Principles of Student Learning Resources and Tool Kits Projects and Portfolios Presentations and Papers Contact Information

Community College Assessment Challenges

Academic programs Administrative services Student development programs Integrate/connect proposed and ongoing assessment efforts across units and levels Developing a culture of evidence Emphasize link between MACRO-LENS and MICRO LENS of assessment Helping faculty see link between institutional effectiveness, aggregate data gathering, and outcomes – based assessment

Community College Assessment Challenges

Questions pertaining to the creation and improvement of an “Effective Academic Learning Environment” cannot be answered by – Admissions/enrollment data – Persistence/attrition data – Survey data – Demographic comparisons

Community College Assessment Challenges

– Global teaching evaluations – Department comparisons – Teaching loads and credit hours – Entering first year student statistics – Transfer rates

How Can We Support Faculty Work and Impact Academic and Student Development?

Lead the discussion that illuminates critical questions/concerns/issues Provide initial sources of data and documentation directly to departments and programs Keep the focus on measurable objectives and outcomes Neutralize political concerns Do all necessary programming Facilitate the development and implementation of an assessment web site “Champion” the development of assessment plans Partner in assessment with your other providers of the service