Transcript Click to Insert Text

Use of Research Data
to Assess and Improve Educational Quality
and Institutional Effectiveness
Presentation at California Community Colleges Chief Instructional Officers Workshop
by
Dorte Kristoffersen
Associate Vice President, ACCJC
www.accjc.org
1
October 28, 2010
Presentation Outline
• Some Definitions
• Characteristics of Good Data and Data Sources
• Data-driven Decision-making
• Purposes of Accreditation
• Types of Data and Data Requirements
• Examples of Effective Use of Data
• Concluding Remarks
www.accjc.org
2
October 28, 2010
Definitions of Evidence and Data
General definitions:
• Evidence includes everything that is used to determine or
demonstrate an assertion
• Data refers to groups of information that represent the qualitative or
quantitative attributes of a variable or set of variables.
www.accjc.org
3
October 28, 2010
Use of the Terms Evidence and
Data in Accreditation
• Evidence is all information included in the accreditation process, i.e.
Self Evaluation Reports of Educational Quality and Institutional
Effectiveness, information supporting the claims and descriptions in
the Self Evaluation Report, interviews during the site visit, and
previous External Evaluation Reports.
• Data is information deliberately collected by the institution to provide
it with clear information about the quality of its education and
services.
www.accjc.org
4
October 28, 2010
Characteristics of Good Data
• For data to be a useful and reliable source of information and for
decision-making, it needs to have the following characteristics.
It should be:
 Accurate and tested for validity and significance
 Up-to-date and complete in terms of aspects of analysis it is



www.accjc.org
I
meant to cover
Consistently used
From reliable sources
Longitudinal and disaggregated as appropriate
5
October 28, 2010
Data Sources
• There are several sources of data than an institution can draw
information from. Data sources can be internal, external or both.
Examples of data sources are:
Institutional data, such as statistics and survey results
Demographics at the local/district, regional or federal level
Data reported to State Government and/or relevant associations,
such as Cal-PASS
Other sources relevant for the institution
www.accjc.org
6
October 28, 2010
Data-driven decision-making
• It is a sound principle when making judgments to act on what we
know. Judgments should not be based on assumptions and feelings.
• Data is a necessary knowledge-base for decision-making
• The information that an institution collects, analyzes, and reflects
upon should be designed to answer questions the institution has
raised against its mission and institutional objectives, and the
information will thus determine what action the institution takes to
support the improvement of educational quality and support student
success
www.accjc.org
7
October 28, 2010
Data-driven decision-making continued
• ‘No News; is normally ‘Good News’
But ‘No data’ does not mean ‘good news’.
It means no sense of bearings and no direction for the future.
www.accjc.org
8
October 28, 2010
Purposes of Accreditation
• To provide assurance to the public that education provided by
institutions meets acceptable levels of quality.
• To promote continuous institutional improvement.
• To maintain the high quality of higher education institutions in the
region/nation.
www.accjc.org
9
October 28, 2010
Types of Data
• The principle of data-driven decision-making should apply to an
institution’s reflections on its achievement of its mission and
institutional objectives and its compliance with the Accreditation
Standards
• The Commission requires various kinds of specific data
• Most of the data is required to support institutions’ claims of
compliance with the Accreditation Standards and some of the data
relate to the U.S.D.E requirements
• The Commission also expects that institutions collect data which is
relevant for the institutions’ internal continuous quality improvement
activities
www.accjc.org
10
October 28, 2010
Commission Data Requirements
• Data about incoming students, i.e.:
 Student demographics
Student enrolment data across instructional programs
Student educational goals
Student development needs
www.accjc.org
11
October 28, 2010
Commission Data Requirements Continued
• Data about incoming students should include information about
students studying in Distance Education or Correspondence
Education mode, such as:
 Account of programs, courses, certificates where 50% or more is
offered in distance education or correspondence education mode
 Annual growth in headcount enrollment
www.accjc.org
12
October 28, 2010
Commission Data Requirements Continued
• Data about Program Review, such as:
 Number of enrolled students
 Student learning outcomes at program and course level
 Assessment results
www.accjc.org
13
October 28, 2010
Commission Data Requirements Continued
• Data on student grievances and complaints, including follow-up
• Data on Internationalization, i.e.:
 Numbers of international campuses
 Numbers of programs for non-U.S. students
 Numbers of non-U.S. students enrolled
www.accjc.org
14
October 28, 2010
Commission Data Requirements Continued
• Two types of data are particularly important as they relate directly to
student success, i.e.
 Student Achievement Data
 Student Learning Outcomes Data
• These two types of data will be explored in more detail as examples
www.accjc.org
15
October 28, 2010
Student Achievement Data
• Data on student achievement has been long required by the ACCJC.
Institutions already use some common measures to examine student
achievement, and some of these measures derive from federal
regulatory language.
www.accjc.org
16
October 28, 2010
Student Achievement Data Continued
• Required Student Achievement Data:
 Retention rates from term to term
 Student progression to the next course/level of course, including
from pre-collegiate to collegiate level
 Graduation rates (certificate or degree)
 Job placement rates
 Licensure pass rates
 Transfer rates to four-year institutions
www.accjc.org
17
October 28, 2010
Value of Student Achievement Data
• Student Achievement Data is end-point data providing an institution
with key information about the achievement of its institutional mission
and in terms of student success
• Such data will:
 If collected longitudinally and analyzed continuously provide the
institution with indications as to whether improvements in
pedagogy or services may be required to improve student
progression and completion
 Keep institutions informed about fluctuations and serve as a
warning if rates decrease and trends need to be reversed
www.accjc.org
18
October 28, 2010
Value of Student Achievement Data Continued
• Student achievement data will also:
 Provide information to the institutions about
barriers to
completion and transfer
 Present trends that identify the need for institutions to explore the
barriers through the collection of additional data and/or conduct
separate surveys
 If collected in disaggregated form inform institutions as to whether
all students progress and complete their studies or if attention
needs to be given to special groups
www.accjc.org
19
October 28, 2010
New Commission Requirement
• As of Fall 2012, Self Evaluation Reports of Educational Quality and
Institutional Effectiveness should include data on key student
achievement measures in disaggregated form by age, gender,
race/ethnicity, socio-economic status and other measures that the
institution considers relevant for its population
www.accjc.org
20
October 28, 2010
An Example of
Student Achievement Data
• Research study by Institute for Higher Education Leadership and
Policy (IHELP). Divided we Fail: Improving Completion and Closing
Racial Gaps in California’s Community Colleges, October 2010
• IHELP followed 250.000 degree seeking Californian community
college students over the course of six years
• The goal of the study was to use it to identify solutions to improve
completion rates for all students and closing the disparities across
racial/ethnic groups
• The study is conducted at a generic level, but the findings and how
the can be used can serve as an example for individual colleges
www.accjc.org
21
October 28, 2010
An Example Continued
•



Too few students reach the milestones: racial/ethnic disparities
abound:
Too many students fail to complete
Six years after enrolling 70% of degree-seeking students had not
completed a certificate or degree and had not transferred to a
university (most had dropped out)
Failure rates among black students (75%) and Latinos (80%) are
higher
www.accjc.org
22
October 28, 2010
An Example Continued
• Critical milestone is missed:
 Only 40% of degree-seeking students had earned at least 30
college level credits at a CCC.
 35% of Latino students and 28% of black students reached this
milestone
 Fewer Latinos who reach the above milestone complete a
certificate, degree or transfer (47%) as compared to white (60%),
Asian-Pacific Islander (58%) and black (53%) students
www.accjc.org
23
October 28, 2010
Student Learning Outcomes Data
• The Commission also requires institutions to demonstrate educational
effectiveness by collecting and analyzing student learning outcomes
data
• Student Learning Outcomes data:
 Is end-point data
 Consists of learning outcomes at course, program and degree level
 Is derived from summative assessment
www.accjc.org
24
October 28, 2010
Value of Student Learning Outcomes Data
• Student Learning Outcomes data:
 Provides information about how well students are able to achieve
institutional and programmatic missions and objectives, and
 Is therefore a key source of data for Program Review
 May help institutions identify barriers to student learning and
student success and how these barriers can be addressed
 May identify whether improvements in curriculum and teaching
and learning methods are warranted to increase the achievement
of learning outcomes
www.accjc.org
25
October 28, 2010
Key Questions to Ask about
Data Collection and Data Use
•
•
•
•
What is the mission of the institution (Institutional, Programmatic)?
What are the institutional objectives?
What is the institution trying to achieve in terms of its institutional
missions and educational quality, including possible goals and
targets?
What data is relevant to collect to gain information on institutional
achievements?
www.accjc.org
26
October 28, 2010
Key Questions to Ask continued
• What are the data sources, i.e. internal and/or external?
• Who should be involved in the data analysis?
• How should the outcomes of the analysis be reported?
• Who is responsible for taking action and ensuring that improvements
are made?
www.accjc.org
27
October 28, 2010
Concluding Remarks
•
•
•
•
•
Data is collected intentionally and its meaning and relevance has
been considered
It is purposeful and designed to provide information on institutional
achievements
It is collected systematically and continuously
Data is interpreted, reflected and acted upon
It is used for improvement
www.accjc.org
28
October 28, 2010
‘Learning without thought is labor lost;
thought without learning is perilous.’
(Confucius 551-470 BC)
www.accjc.org
29
October 28, 2010
Thank you
ACCJC/WASC
10 Commercial Blvd, Suite 204
Novato, CA 94949
Tel: 415-506-0234
Fax: 415-506-0238
Website: www.accjc.org
Email: [email protected]
www.accjc.org
30
October 28, 2010