Transcript Module Four: Analyzing and Reporting Assessment Results

Module 4: Analyzing & Reporting Assessment Results
© 2013 Christie Cruise-Harper, PhD All Rights Reserved

The purpose of this module is to assist you
with analyzing the data you gathered for the
program/service you assessed in the 2013 –
2014 academic year.
Department
Office of Multicultural Programs
Personal Counseling
Health and Wellness
Program
Multicultural Scholars Program/Dean’s
Award Program
Mandated Substance Abuse Assessment
Program
HEROs Program
Campus Ministry and Community
Service
Student Involvement
KLILV Sophomore Colloquium
Athletics
SAAC and Champs
Residence Life
Resident Assistant Program
CAB/MSG/CSI
Student Life/Associate Dean of Students Habitat for Humanity




In module 1 you developed learning outcomes
for your program/service using Bloom’s
Taxonomy as a guide.
Module 2 allowed you to strengthen those
learning outcomes and guided you through the
process of choosing learning activities.
Module 3 assisted you with choosing appropriate
assessment tools/methods for the learning
outcomes and learning activities you established.
This module will help you with analyzing and
reporting the assessment data.
Learning
Outcome
Learning Activity
Assessment
Analysis
What should your
students be able to
do?
What activity will help
your students
achieve the learning
outcome?
How will you know
whether students
have achieved the
learning outcome?
What will you do with
all the information
you collected from
your assessment
plan?
Module 1
Module 2
Module 3
Module 4



Assessment results must be analyzed to
determine if student learning outcomes were
met.
Data is analyzed for context, understanding
and to draw conclusions.
Analysis of data gives the information
meaning.
Taken from Academic Program Assessment: Tools & Techniques for Program Improvement


Determining how to organize, synthesize,
interrelate, compare and present the
assessment results are all part of analyzing
the data.
Assessment data can be compared to
findings from previous assessments, baseline
data and existing criteria.
Taken from Academic Program Assessment: Tools & Techniques for Program Improvement

Quantitative
◦ Also known as “empirical research”
◦ Refers to any research based on something that
can be accurately and precisely measured.
◦ Collects numerical data in order to explain or
predict a particular phenomena.
Taken from University of Wisconsin – Madison’s Ebling Library & R. Ouyang’s Basic Inquiry of Quantitative Research
For more information visit:
http://researchguides.ebling.library.wisc.edu/content.php?pid=325126&sid=2940225

Qualitative
◦ Refers to any research based on something that
cannot be accurately and precisely measured.
◦ Collects narrative data to gain insights into a
particular phenomena.
Taken from University of Wisconsin – Madison’s Ebling Library & R. Ouyang’s Basic Inquiry of Quantitative Research
For more information visit:
http://researchguides.ebling.library.wisc.edu/content.php?pid=325126&sid=2940225
It’s all about the numbers!

There are four types of quantitative research methods:
◦ Descriptive: collecting data for hypotheses testing*
◦ Correlational: determining whether and to what degree a
relationship exists
◦ Cause-Comparative: establishing the cause-effect
relationship
◦ Experimental: establishes the cause-effect relationship, but
manipulates the cause
See notes section for more detailed information.
Taken from R. Ouyang’s Basic Inquiry of Quantitative Research

The following are common methods of data
collection in quantitative research:
◦
◦
◦
◦
◦
Surveys and Questionnaires
Structured Interviews
Observation or Interaction Analysis
Secondary Data or Content Analysis
Experiments
See notes section for more detailed information.


In quantitative research there are two ways in
which data are analyzed:
Descriptive Statistics
◦ Procedures used to describe a given collection of data.
◦ The purpose is to describe the sample at hand-the
collection of cases that we have examined.

Inferential Statistics
◦ Procedures that let us generalize our findings beyond
the particular sample at hand to the larger population
represented by that sample.
Taken from Diekhoff, G.M.(1996). Basic Statistics for the Social and Behavioral Sciences.


Most Student Life assessment projects do not
seek to generalize its findings to the entire
Maryville University student body. Because
our goal is to learn about the sample at hand,
descriptive statistics will be the focus of the
quantitative data analysis for module 4.
There are three types of descriptive statistics
to provide you with an overview of your data:
◦ Central Tendency Measures*
◦ Variability Measures
◦ Frequency and Percentages*


More on descriptive statistics later.
Before beginning any data analysis, you must
first identify the level of measurement
associated with your quantitative data. There
are four levels of measurement:
◦
◦
◦
◦
Nominal
Ordinal
Interval
Ratio (Scale)




Nominal: basic classification data; do not have
meaningful numbers attached to them, but are
broader categories
Ordinal: have numbers attached to them and the
numbers are in a certain order, but there are not
equal intervals between the numbers
Interval: have equal intervals between the numbers;
the distance between attributes have meaning
Ratio: have equal intervals between the numbers;
there is an absolute zero that is meaningful
Taken from: http://www.uni.edu/commstudies/researchmethods/chapterfour1.html



Once you have decided on your data
collection method, decided on the level of
measurement for your variables, and
collected the data, you are ready to begin
analyzing the data. There are two software
programs I recommend and they are available
on campus:
Qualtrics
SPSS (Statistical Package for the Social
Sciences)

The following quantitative data analysis
procedures are used to describe the data and can
be done in Qualtrics and SPSS:
◦
◦
◦
◦
Data Tabulation (Frequency Distributions, Percentiles)
Descriptive Data (Central Tendency)
Data Disaggregation
Moderate and Advanced Analytical Methods
For a detailed description of these analyses visit:
http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-quantitative-data/
OR
Chapters 1-3 in Diekhoff’ s Basic Statistics for the Social and Behavioral Sciences


Surveys and questionnaires can be developed
in Qualtrics and the data can be analyzed
within the software. There are tutorials to
assist you with analyzing your data.
Crosstabs:
http://qualtrics.com/university/researchsuite/reporting/crosstabs/about-cross-tabulations/

Understanding Statistics:
http://qualtrics.com/university/researchsuite/reporting/crosstabs/understanding-statistics/

SPSS
◦ Below are several links to resources to assist you
with using SPSS.
 http://www.slideshare.net/sspink/seminar-on-spss
 http://www.youtube.com/watch?v=eTHvlEzS7qQ
 http://www.youtube.com/watch?v=HT0Skh2UP1U&feat
ure=related
 http://calcnet.mth.cmich.edu/org/spss/toc.htm

In addition to a narrative about your data
analysis, quantitative results are presented in
the following ways:
◦ Charts
◦ Graphs
◦ Tables



The following websites provide examples, in
APA, of how quantitative results are presented:
https://owl.english.purdue.edu/owl/resource/560/19/
https://owl.english.purdue.edu/owl/resource/560/20/

Tables and graphs can be created in Qualtrics. For
more information visit:
◦ http://qualtrics.com/university/researchsuite/reporting/re
porting-beta/tables/#AboutTables
◦ http://qualtrics.com/university/researchsuite/reporting/re
porting-beta/graphs/

Graphs can also be created in SPSS. For directions
visit:
◦ http://julius.csscr.washington.edu/pdf/spss.pdf
◦ http://academic.udayton.edu/gregelvers/psy216/spss/gra
phs.htm
◦ http://www.ats.ucla.edu/stat/spss/seminars/SPSSGraphics
/spssgraph.htm
I want to know how you feel!




It is important to achieve empathic understanding to
comprehend the participant’s experience with a
minimum of distortion or bias;
The researcher must attempt to recognize their own
personal prejudices, stereotypes, myths, assumptions
and other thoughts or feelings that may cloud the
perception of other people’s experiences;
Knowledge of other’s experience cannot be assumed
regardless of familiarity with their subcultural
landscape; and
Do not expect participants to hold the same values as
you.
M. Ely, M. Anzul, T. Friedman, D. Garner, A.M. Steinmetz in Doing Qualitative Research: Circles within Circles

Qualitative Research Methodologies
◦
◦
◦
◦
◦
◦
◦
◦
Basic Interpretive Qualitative Study*
Grounded Theory
Phenomenology
Case Study
Ethnography
Postmodern Research
Critical Qualitative Research
Narrative Analysis
For more information visit:
http://www.fctl.ucf.edu/researchandscholarship/sotl/creatingsotlprojects/implementingmanaging/qu
alitativeresearchtypes.php







Participant Observation
Interviewing (formal & informal)
Focus Groups
Document Analysis
Logs (notes/reflections from observations
and interviews)
Audio and videotaping
On-going data analysis
M. Ely, M. Anzul, T. Friedman, D. Garner, A.M. Steinmetz in Doing Qualitative Research: Circles within
Circles


Coding is the process of organizing data into
chunks or segments before making meaning
of the information.
Coding involves taking text data or pictures
gathered during data collection, segmenting
it into categories, and labeling those
categories with a term, often a term used by
the actual participant.
Taken from: Creswell, J.W.(2009). Qualitative, Quantitative and Mixed Methods
Approach, chapter 9, Qualitative Procedures.

Begin the coding process by first reviewing
your learning outcomes as a reminder of what
you are assessing. Your coding scheme will
be based on your learning outcomes.
◦ For example: Undergraduate students who
participate in the Multicultural Scholars Program will
be able to describe their talents, strengths and
social group memberships.

Coding involves assigning a word, phrase,
number or symbol to each coding category.
◦ For example: describe talents and strengths

Codes can be pre-set or emergent. You should
have both.
◦ Pre-set: A list of codes created in advance by the
researcher based on the research question, learning
outcomes, or conceptual framework.
◦ Emergent: Ideas, concepts, actions and meanings that
come up from reading and analyzing the data that are
not in the pre-set codes.

Coding will serve as a system to help you to
organize your data.
For more information and an example of coding visit:
http://programeval.ucdavis.edu/documents/Tips_Tools_18_2012.pdf





Once you have gone through all documents
and coded them, they can now be gathered
into families of codes or categories.
Materials are sorted by these categories,
identifying similar phrases, patterns and
relationships.
As you code and categorize the data, look for
the interrelationships among categories.
Sorted materials are examined to isolate
meaningful patterns.
Identified patterns are used to create themes.



It is best to start any report of qualitative
results with an overview of how data were
processed and coded.
The results are presented as “findings”.
The findings are organized by themes with
substantial evidence that links to the themes
included in the findings (i.e. quotes from
participants).
So now what?
Closing the loop - using the assessment results for
program improvement.
Assessment
Plan
Implement
Changes
Data
Collection
Identify
Changes
Data
Analysis
Assessment
Report



The findings from your quantitative and/or
qualitative research will yield rich information
that will be included in your assessment
report.
These findings will provide you with insights
into what’s working with your
program/service and areas of improvement.
From these findings you will make
recommendations for improvement of your
program/service.
You may find it helpful to create a program improvement plan to transform the
recommendations made into actions for improvement.
Recommendation:
Action step(s)
What action steps must be completed to implement the
recommendation?
Estimated implementation date
When does the program expect to begin to implement
the action steps?
Estimated completion date
When does the program expect the recommendation to
be fully implemented and/or achieved?
Person(s) responsible
Who will take responsibility for seeing that the actions
steps are implemented?
Expected outcome
What is the expected impact/outcome the
recommendation will have on the program if it is
implemented?
Estimated cost(s)
What is the estimated cost of implementing the
recommendation?
Status update
What progress has been made towards achieving the
recommendation?











Accreditation
Planning and budgeting
Maryville University requirements
Student Life requirements
Program promotion/marketing
Recruitment/retention initiatives
Publications
Conference presentations
Student development opportunities
Professional development opportunities
Grant applications
Taken from: Academic Program Assessment: Tools & Techniques for Program Improvement




Academic Program Assessment: Tools & Techniques for
Program Improvement.(2013). SUNY.
Analyzing Assessment Data.(2013).
http://www.sunyorange.edu/assessmentapa/docs/Analy
zingandUtilizingAssessmentData.pdf. State University of
New York (SUNY) Orange County Community College.
Basic Inquiry of Quantitative Research.(n.d.).
http://ksumail.kennesaw.edu/~rouyang/EDresearch/details.htm. Kennesaw State University.
Center for Evaluation and Research.(2012).
http://programeval.ucdavis.edu/documents/Tips_Tools_
18_2012.pdf. University of California at Davis.




Creswell, J.W.(2009). Qualitative, Quantitative and Mixed
Methods Approach, chapter 9, Qualitative Procedures.
Diekhoff, G.M.(1996). Basic Statistics for the Social and
Behavioral Sciences.
Differences Between Quantitative and Qualitative
Research. (2013).
http://researchguides.ebling.library.wisc.edu/conte
nt.php?pid=325126&sid=2940225. University of
Wisconsin-Madison-Health Sciences Ebling Library
Ely, M., Anzul, M., Friedman, T., Garner, D., Steinmetz,
A.M.(1991). Doing qualitative research: Circles within
circles. RoutledgeFalmer: New York.




Faculty Center for Teaching and Learning.(2013). Types
of
qualitative research: Explained within a SOTL framework.
http://www.fctl.ucf.edu/researchandscholarship/sot
l/creatingsotlprojects/implementingmanaging/quali
tativeresearchtypes.php. University of Central Florida.
Gathering Data and Assessing Results.(2013).
http://nnlm.gov/evaluation/guide/stage 5.pdf. National
Network of Libraries of Medicine.
The Pell Institute and Pathways to College Network.(2013).
Evaluation Toolkit.
http://toolkit.pellinstitute.org/evaluationguide/analyze/analyze-quantitative-data/
University of Northern Iowa.(2013). Communication Studies –
Research Methods Chapter Four.
http://www.uni.edu/commstudies/researchmethods/chapte
rfour1.html