Transcript Document

Developing a Comprehensive Learning
Outcomes Assessment Strategy
for Adult Learners
Cindy MacMillan&
Susan Davenport
Agenda
• Introductions
• Characteristics of Thomas Edison State College
• Challenges in an adult learning environment
2
Introductions
• Susan Davenport, Dean of Heavin School of Arts
and Sciences
• Cynthia MacMillan, Director of Outcomes
Assessment and Accreditation
• Audience
– Who’s here?
– Where you are from?
– Why this session?
3
Why Learning Assessment?
• Is it really all about accountability?
•
•
•
•
Accreditors
Students
Mission
You
4
What are the challenges assessing learning
in an adult environment?
• Heterogeneous population with different
needs and skills
• Significant learning that the student
brings to the learning environment
• Access to students for assessment
5
Where to Start
• Phase One – Define Learning Outcomes at Institutional and
Programmatic Levels
• Phase Two-- Operationalize the learning outcomes
• Phase Three – Implementation of assessment (finalize
assessment matrix)
• Phase Four – Data collection and Analysis
• Phase Five -- Recommendations and transformations made
in response to Phase Four data
6
Where to start! (cont.)
7
Phase One
Define Learning Outcomes at Institutional and
Programmatic Levels
• Buy In
• Strength in diversity
• Define and Describe Goals and Process
– Institutional, Programmatic, Course
8
Thomas Edison State College Institutional
Outcomes
1.
Information Literacy
The Information literate graduate will demonstrate the ability required to recognize when
information is needed and has the ability to identify, locate, evaluate and use effectively
and appropriately the information needed.
2. Technological Competency
Graduates will be able to solve problems and perform tasks using appropriate
technological tools.
3. Responsible Ethical Leadership
Graduates will be able to articulate in their own words the ethical principles they have
discovered and encountered in their academic discipline or profession. Graduates will
recognize and articulate actual ethical dilemmas witnessed or experienced.
4. Scientific Reasoning
Thomas Edison State College graduates will be able to analyze issues facing citizens by
distinguishing scientific approaches from other ways to frame problems and issues.
5. Diversity/Global Literacy
The Thomas Edison State College graduate will demonstrate an understanding of globally
diverse perspectives, cultures, values, events, trends and issues.
9
Thomas Edison State College Institutional
Outcomes (contd.)
6. Quantitative Reasoning/Literacy
Thomas Edison State College graduates will be quantitatively literate.
They will be able to understand and apply mathematical concepts and
skills to inform personal and professional decision-making.
7. Written and Oral Communication
Thomas Edison State College Graduates will be able to write and speak
with proficiency.
8. Critical Analysis & Reasoning
Thomas Edison State College graduates will be able to demonstrate
critical thinking and problem solving skills and are able to analyze
content, discover meaning or significance, draw conclusions, and make
assessments.
9. Life Long Learning
Graduates will demonstrate a commitment to lifelong learning.
10
Phase Two
Operationalize the Learning Outcomes
• Indirect versus direct measures
• Quantitative versus qualitative measures
• Triangulation
• Low versus High Stakes Assessment
• Baselines and Benchmarks
11
Phase Two Continued
Methods we are using –
• Standardized Assessment – ETS Proficiency
Profile, SAILS
• Graduate Survey, Adult Learner Inventory
• Capstones
• Embedded Analysis
• Historical Data Review
• Survey
• ETS Major Field Test
• E-Portfolios
12
Example: Assessment Matrix for
Program Level Learning Outcomes
Bachelor of Science in Human Services
outcomes:
Program Element
which addresses this
outcome
Measures (indicate direct or indirect)
Benchmark/
Target
(acceptable and
ideal if appropriate)
Interpret and critically analyze the research in
the professional track
Course in Professional
track
final project/paper
A 70% earn a grade of B+
I 80% earn a grade of B+
SOS 492 Research
Methods
final research paper
A 70% earn grade of B+
I 80% earn grade of B+
ETS Proficiency
Profile
Std. Test Critical Thinking
A 60% achieve proficient or marginally
proficient
I 80% achieve proficient or marginally
proficient
Apply theory to professional practice
Capstone
final paper
A 70% earn grade of B+
I 80% earn grade of B+
Apply knowledge of the specific skills,
techniques, and agencies necessary to serve
client populations
PSY 331 Intro
Counseling
final exam
A 70% earn grade of B+
I 80% earn grade of B+
Capstone
final paper
A 70% earn grade of B+
I 80% earn grade of B+
Graduate Survey
Item 7-a
A 80% respond positively
I 90% respond positively
SOC 322 Cultural
Diversity in the US
final exam
A 70% earn grade of B+
I 80% earn grade of B+
Capstone
final paper
A 70% earn grade of B+
I 80% earn grade of B+
MAN 301 Principles
of Management
final exam
A 70% earn grade of B+
I 80% earn grade of B+
Capstone
final paper
A 70% earn grade of B+
I 80% earn grade of B+
Apply knowledge of cultural diversity as it
relates to the field of Human Services.
Apply theories of management as it relates to
Human Services
13
Phase Three
Implementation of Assessment
•
•
•
•
Be realistic in scope
Not everything needs to be assessed each year
Be pragmatic in tying plan to funds
Benchmarking
– Internal
– External
• Rubric elements
•
14
Information Literacy Data Collection Sample
15
Example: Data Collection for Human
Services Degree
Bachelor of Science in Human Services
outcomes:
Benchmark/
Target
2010 results
Interpret and critically analyze the research
in the professional track
70% earn a score of 3 or above on a 4 point scale on rubric by
reviewed independent consultant
Above target
Apply theory to professional practice
70% earn a score of 3 or above on a 4 point scale on rubric by
reviewed independent consultant
Below target
Apply knowledge of the specific skills,
techniques, and agencies necessary to serve
client populations
70% earn a score of 3 or above on a 4 point scale on rubric by
reviewed independent consultant
Below target
Apply knowledge of cultural diversity as it
relates to the field of Human Services.
70% earn a score of 3 or above on a 4 point scale on rubric by
reviewed independent consultant
Below target
Apply theories of management as it relates to
Human Services
70% earn a score of 3 or above on a 4 point scale on rubric by
reviewed independent consultant
Above target
16
Phase Four and Five–
Analysis and recommendations
17
Information Literacy - SAILS
Recommendations
• Continue to evaluate student learning outcomes in
the area of Information Literacy using the SAILS.
• Embedded SAILS in an upper level course for
general populations.
• Identify and implement additional methods to
assess the outcome area not assessed by SAILS.
• Include the SAILS in a few more courses to ensure
that all degree programs of the College are
included in this informational literacy assessment.
• Target the SAILS administrations to 250-275
students per year in order to facilitate longitudinal
analysis in the future.
18
•
•
•
•
•
•
•
Information Literacy – ENC-102
Recommendations
Expand the rubric elements to include one on grammar
and spelling.
Expand the data collection efforts in ENC-102 to include
one section per semester.
Expand the roll of the lead mentor to include collecting
reliability data.
Expand the use of Waypoint in other schools, courses an
assignments.
Explore viable options to communicate details of
Waypoint use with mentors.
Explore ways to minimize the use of instructional design
resources.
Seek efficiencies such as sharing rubric elements acros
schools.
19
Human Services – Analysis and
recommendations
Analysis of Human Services results led to the following
curricular changes:
1. Modify the program description to more accurately describe the
program and its objectives.
2. Modify the program outcomes to align with objectives in
program by including diversity and theories of management.
3. Modify the general education requirements to 60 credits to
align with the other degree offerings within the College.
Require a three credit Ethics course under Humanities.
20
Phase Four and Five– Analysis and
recommendations (contd.)
4. Modify the Core Requirements. Based on guidelines from national
accrediting bodies (i.e. The National Human Services Organization and
the Council for Standards in Human Services Education), the core
requirements were replaced with the following areas: Theoretical
Foundation, Intervention, Client Populations, Research and Mgmt. This
improves the quality of student learning outcomes and ensures that the
program objectives are appropriate and measurable.
5. Modify the Practicum to a capstone course and objectives to make the
practicum an effective instrument for assessing the degree outcomes.
6. Change the grading of the capstone from Credit/No Credit to a letter
grade. This will ensure a more accurate assessment of the outcomes of
the course. A benchmark of “B+” was recommended.
21
Your best practices, lesson learned and
questions
Please share your feedback . . .
•
•
•
22
Thank you for your time and
interest!