Transcript No Slide Title
A Longitudinal Survey Program to Meet Institutional and Academic Department Assessment Needs
AIRPO Annual Conference West Point, New York June 16-18, 2004
Bruce P. Szelest
1
Albany History and Forward-looking Needs:
Long tradition in assessing general student outcomes
•W.K. Kellogg Foundation support •Albany outcomes assessment model •Ad hoc studies •Albany assessment report series
http://www.albany.edu/ir/reports.htm
Future-oriented developments
•Middle States accreditation requirements •Campus assessment plan •Building a campus culture that embraces assessment at the
unit level as well as at the institutional level
2
The Albany Outcomes Assessment Model
Personal Traits Pre
-
College Characteristics
Age, Ethnicity, & Gender Parents’ Education Aptitude Test Scores High School Achievement College Preparation Parental Influence Involvement Receptivity to Services Educational Aspirations Acad Motivation: Intrinsic, Extrinsic Self-regulation
College Experiences Acad Integration
Acad Conscientiousness Classroom Experiences Faculty Contact Field of Study
Social Integration
Peer Relations Employment Extra-Curr. Activities Residential Experience
Inst’l Integration
Inst’l Commitment Advisement Financial Aid Affinity of Values
Educational Outcomes Academic
Arts & Letters Disciplinary Study Scientific Method Academic Achievement Persistence/Graduation
Personal
Interpersonal Skills Openness & Tolerance Functioning Independently
Alumni Outcomes
Graduate Degrees Earned Occupational Status Income Level Satisfaction with Job and Career Leadership & Service Awards & Recognition Alumni Giving 3
The Albany Longitudinal Survey Program
Year 1 Entering Student Survey Freshmen Student Experience Survey Year 2 Year 3 Year 4 + Senior Experience Survey Surveys administered every year
4
Entering Student Survey
•Focus on student pre-college characteristics to
identify students at risk prior to first semester of study
•Concentrate institutional resources on students in-
need of attention - individual student profiles
•Communicate general findings back to new
freshmen to raise awareness of successful student profiles
•One leg of the Advisement Services Center’s
tripartite assessment program
•http://web.albany.edu/ir/samplesurveys/ess.html 5
6
7
8
9
Pre-college Scales
Parental Influence
Composite of mother's and father's education My parents encourage me to do the best in whatever I do I can go to my parents when I have a problem Educational Aspirations (scale: importance) To gain a broad, well-rounded education (in the sciences, humanities, and arts) To learn to think creatively and analytically To learn more about myself, my values, and my life's goals To learn more about other cultures and groups of people.
Involvement (scale: degree of chance) Participate in intercollegiate or intramural sports Belong to a fraternity or sorority Join a student club (academic, political, or cultural) Participate in student government, student newspaper, and/or campus radio station Receptivity to Services (scale: degree of chance) Visit professors during their office hours Attend a study group from one of your classes Contact my academic advisor more than once per semester Visit the Career Development Center College Preparation (scale: yes or no) Pre-calculus or calculs Foreign language Science Advanced placement (AP) courses 10
Pre-college Scales – cont’d
Academic Motivation - Intrinsic (scale: not true to true) I enjoy the classes that challenge my mind.
I will feel successful if I have the highest test scores.
Thinking about my future motivates me to work hard in school.
Solving difficult problems in my classes gives me a sense of satisfaction.
I feel successful when I learn something interesting.
Learning new things gives me a sense of accomplishment.
Schoolwork is interesting because you see yourself making progress.
I read books not assigned in classes.
Academic Motivation - Extrinsic (scale: not true to true) I study hard because I want to prove myself as capable as anyone else in class.
I worry about not doing as well academically as others in college.
I don't feel I can gain much in college other than obtaining a degree.
I feel that as long as my teachers are satisfied with my academic performance, that is enough.
I don't really care how well I will do in college, as long as I can pass.
Getting a high paying job is the main reason I study hard in college.
I feel ashamed if I don't do as well as other students do in college.
I worry about failing exams and getting into academic trouble.
Self-Regulation (scale: not true to true) When I get something wrong in school, I always stop and try to find out what went wrong rather than simply asking my instructor the correct answer.
I take notes while reading textbooks.
When I run into difficulties in schoolwork, I feel if I put in extra effort, I will eventually figure them out.
When I am reading my textbooks, I stop and check whether I really understand what I have just read.
If I study harder my academic abilities will improve.
I keep a regular study schedule.
When learning a new concept, I never check if I really understand its meaning.
11
Entering Student Survey
•Focus on student pre-college characteristics to
identify students at risk prior to first semester of study
•Concentrate institutional resources on students in-
need of attention -individual student profiles
•Communicate general findings back to new
freshmen to raise awareness of successful student profiles
•One leg of the Advisement Services Center’s
tripartite assessment program
•http://web.albany.edu/ir/samplesurveys/ess.html 12
Freshmen Student Experience Survey
•Examine changes in students’ psychological make-up
after the first college year
•Gauge institutional engagement and relationships with
educational outcomes
•First pass at gauging centralized student advisement •Communicate general findings back to students and
faculty to support the development of a campus culture of assessment
•http://web.albany.edu/ir/samplesurveys/froshses.html 13
14
15
16
17
18
19
20
21
22
23
24
College Experience Scales & Items Academic Integration: Academic Conscientiousness (scale: various 5 pt scales)
How frequently have you worked harder than you thought you could to meet an instructor's expectations?
Developed your study skills.
Utilized your study skills.
I do very little studying on weekends.
I put a good deal of effort into being well prepared for examinations. My primary goal at UAlbany is to take advantage of academic opportunities.
Doing homework and studying outside of class.
Classroom Experiences (scale: never to almost always) Been satisfied with your academic experiences?
Had out-of-class assignments that were good learning experiences?
Worked with other students on class assignments?
Received feedback (written or oral) from instructors on the quality of your work?
Had to critically appraise the value of information, arguments, or methods in your classes or class assignments?
Faculty Contact Items
How frequently have you had discussions, meetings, or conversations with instructors outside of class?
Visit professors during their office hours?
Seek out a faculty mentor?
25
College Experience Scales & Items– cont’d Social Integration:
Peer Relations (scale: disagreement to agreement) I have developed strong friendships with other students. My interpersonal relationships with other students have had a positive influence on my personal growth.
The student friendships I have developed have been personally satisfying.
It has been difficult for me to make friends with other students.
I know several UAlbany students who would be willing to help me if I had a personal problem.
My interpersonal relationships with other students have had a positive influence on my intellectual growth.
I am confident in my ability to make friendships.
Extra-curricular activities (scale: yes or no) Participate in intercollegiate or intramural sports?.
Belong to a fraternity or sorority?
Employment
Working on-campus (volunteer or paid).
Working off-campus (volunteer or paid).
26
College Experience Scales & Items– cont’d Institutional Integration:
Institutional Commitment (scale: yes or no) Visit the Writing Center?
Visit the Career Development Center?
Contact your academic advisor more than once per semester?
Financial Aid Advisement Items (scale: disagreement to agreement)
Advisor was knowledgeable about course requirements for the major(s). Advisor was knowledgeable about course requirements for the minor(s). Advisor was helpful in assisting with scheduling/ registration procedures. Advisor showed genuine interest in my academic progress.
Advisor went over my degree audit with me to inform me of course requirements I have not yet met.
Advisor was knowledgeable about academic policies (e.g., graduation requirements, GPA requirements, General Education guidelines, residence requirements, etc.). Advisor was available when I needed him/her.
Advisor showed concern for my personal growth and development.
Advisor helped me to identify career areas (or graduate school opportunities) which fit my skills, abilities, and interests Advisor encouraged me to talk about myself and my college experience.
Advisor was easy to talk with. Advisor promptly returned my phone calls, e-mails, and/or messages.
27
Educational Outcomes
Academic (scales: none to very large college contribution)
Arts & Letters
Writing effectively.
Developing intellectual curiosity.
Developing the ability to formulate creative ideas and/or solutions.
Appreciating artistic and creative expression.
Placing current problems in historical perspective.
Scientific Method
Evaluating ideas, materials, and methods critically.
Developing problem solving skills.
Thinking analytically and logically.
Understanding scientific findings.
Understanding mathematical concepts.
Disciplinary Study
Gaining factual knowledge.
Learning how to learn.
Synthesizing a body of information.
Understanding a particular discipline's research methods.
Understanding a particular discipline's various schools of thought.
Understanding the inter-relatedness of different fields of study.
Using information and/or computer technology in your academic discipline 28
Educational Outcomes – cont’d
Social (scales: none to very large college contribution)
Interpersonal Skills
Speaking effectively. Developing interpersonal and social skills.
Functioning effectively as a member of a team.
Preparing for active participation in a democratic society.
Coping with conflict.
Adapting to different social situations.
Openness & Tolerance
Understanding cultural differences.
Gaining exposure to a variety of new intellectual areas.
Developing an openness to new ideas.
Coping with moral and ethical issues.
Developing a better understanding of myself (e.g., interests, talents, values, limitations).
Understanding cultural differences.
Relating well to people of different races
Functioning Independently
Functioning independently (self-reliance).
Exercising personal responsibility.
Developing a better understanding of myself.
29
Freshmen Student Experience Survey
•Examine changes in students’ psychological make-up
after the first college year
•Gauge institutional engagement and relationships with
educational outcomes
•First pass at gauging centralized student advisement •Communicate general findings back to students and
faculty to support the development of a campus culture of assessment
•http://web.albany.edu/ir/samplesurveys/froshses.html 30
Senior Survey
•Examine changes in students’ psychological make-up
over their UAlbany career
•Gauge institutional engagement and relationships with
educational outcomes
•Connect engagement and educational outcomes with
student experiences in the major
•Focus on academic advisement in the major •Communicate general findings back to students, faculty,
and academic units to support the development of a campus culture of assessment
•http://web.albany.edu/ir/samplesurveys/seniorses.html 31
32
33
34
35
Entering Student Survey ID at-risk students General profiles of successful students Personal Traits Pre
-
College Characteristics
Age, Ethnicity, & Gender Parents’ Education Aptitude Test Scores High School Achievement College Preparation Parental Influence Involvement Receptivity to Services Educational Aspirations Acad Motivation: Intrinsic, Extrinsic Self-regulation
College Experiences Acad Integration
Acad Conscientiousness Classroom Experiences Faculty Contact Field of Study
Social Integration
Peer Relations Employment Extra-Curr. Activities Residential Experience
Inst’l Integration
Inst’l Commitment Advisement Financial Aid Affinity of Values
Educational Outcomes Academic
Arts & Letters Disciplinary Study Scientific Method Academic Achievement Persistence/Graduation
Personal
Interpersonal Skills Openness & Tolerance Functioning Independently
Alumni Outcomes
Graduate Degrees Earned Occupational Status Income Level Satisfaction with Job and Career Leadership & Service Awards & Recognition Alumni Giving
Freshmen & Senior Surveys Expansion of the Albany Model Gauge institutional engagement Evaluate advisement (centralized & in the major) Build culture of assessment (generally & in acad units)
Pros
•Longitudinal survey program
tailored to institutional needs
•Demonstrates institutional
commitment to self examination
•Amenable to post-hoc
analyses of special student populations
•Immediate impact in
enhancing student retention
•Promotes academic unit use
and participation
•Cost effective approach
Cons
•No benchmarking •Subject to validation and
reliability criticisms
•Considerable start-up
investment by IR staff and collaborators
•Requires IT infrastructure
and support
37
END
38