Transcript Institutional Research: What problems are we trying to solve?

Institutional Research
What problems are we
trying to solve?
Mantz Yorke & Bernard Longden
Sorting out types of problems
“Reports that say that something hasn't happened are always
interesting to me, because as we know,
there are known knowns;
there are things we know we know.
We also know there are known unknowns;
that is to say we know there are some things we do not know.
But there are also unknown unknowns
the ones we don't know we don't know.”
Donald H. Rumsfeld, Department of Defence - news briefing, February 12, 2002
The challenge facing any University is to
Be smarter and better manage the known knowns;
Inquire into university life to seek out the
known unknowns
and
Reduce the risk of
unknown unknowns as they relate to:
Demographic changes
National policy drivers
International opportunities
Types of problem :
Types of institutional research
Two broad types of research
What’s going on
type of research
Descriptive
Why is it going on
type of research
Explanatory
Without theory, research is impossibly narrow.
Without research, theory is mere armchair contemplation.
(Silverman, 2001: 294)
Institutional research - four faces
Audience
Formative &
internal
for improvement
Role
Administrative,
Institutional
Academic,
Professional
Volkwein (1999) has
provided a useful analysis
of the aims of IR – The
four Faces of IR.
Summative & external
Accountability
Information
Presentation
Research for
improvement
Educational
research
Terenzini’s organisational intelligence
Culture – knowledge
of what a college is
and where it has
come from
Major issues
or decision areas
facing the university
Contextual intelligence
Issues Intelligence
Technical / Analytical Intelligence
Factual knowledge
analytical skills
Hierarchy
of
complexity
What problems are we trying to solve?
The view from inside - BL
Audience
Formative &
internal
for improvement
Role
Administrative,
Institutional
Volkwein (1999) has
provided a useful analysis
of the aims of IR – The
four Faces of IR.
Information
Summative & external
Accountability
Presentation
Administrative/Formative-known knowns
IR as Information Authority
Administrative – formative and internal
Emerging within institutions as:
Information office
Management information
Planning and development
Institutional Research
Primary functions:
Clarifying the known knowns
Describing the university in data terms
Educating the university community about itself
What problems are we trying to solve?
Information authority
‘Base data’ about the institution – profile and fact book
Statutory returns
Validating data – is this really us?
Size and shape of the institution to assist in setting targets and
plans
Terenzini’s organisational intelligence level – Technical/analytical
intelligence
In the UK these are well developed possibly over load the
university with data
The challenge is to reduce the plethora of tables and cross tabs
– to identify Key Questions that are crucial to the organisation
for success
What problems are we trying to solve?
Presentation facility
Marketing the best aspects – advocate role
Audit and external accountability
Performance measure
League tables
Terenzini’s organisational intelligence level – Technical analytical
intelligence
Primary functions:
Conform data to agreed descriptors
Performance Indicators
Satisfy public funding legislative requirements
Risks and concerns
Spin and managing data
Political pressure to maximise impact of data
Denial about the message data gives
Student Data flow
Student data
Data informs funding
- audit
Students
Aggregate
early return to
HEFCE
Data passed to HEFCE
who recreate December
aggregate return
Individual
Summative activity
data return to HESA
But
same data also used
to inform public through:
Publications,
commodification of data
Rankings
Data sources to assist with the questions
UK
Higher Education Statistics Agency
Published data sets
HEDI
UCAS
Unistat
HEFCE (?)
National Student Survey
US
National Centre for Education Statistics
IPEDS – universe institutional census
NPSAS – National Post Secondary Student aid
Study
BPS – Beginning Post Secondary - Longitudinal
Study
B&B – Baccalaureate & Beyond - Longitudinal
Study
National Science Foundation
Some propositions that we might need
to address
Do universities make good use of the data within their own
university?
Does IR constitute value for money?
Is data critical to teachers, administrators, decision makers?
Is the emphasis between data collection and analysis
appropriate?
Should the IR questions be articulated more clearly and data
collected in response, or should we continue to collect data and
then use what we have for policy decision?
What lessons can be learned from the research community’s use
of data sets in the US?’
Comparison – sometimes interesting
• UK HESA and US IPEDS – similar objectives and
operation
• US interest in scientific advancement not
reflected in the UK
– Possibly RAE provides nearest comparison
• US positively encourages Institutional Research
– Association for Institutional Research (US) in conjunction
with National Centre for Education Statistics provides
funded summer school to encourage analyses of public
data
• UK implicitly discourages Institutional Research
– HESA charge for the use of our data!
What problems are we trying to solve?
The view from outside - MY
Audience
Formative for
improvement
Summative & external
Accountability
Role
Academic,
Professional
Research for
improvement
Educational
research
Academic/Formative: known knowns and unknowns
IR as Research for Improvement
Academic – formative and internal
Analysing (afresh?) existing data, or investigating institutionallyfocused research questions, with a view to influencing practice.
Problem: such activity is often scattered within an institution, and
unco-ordinated.
What problems are we trying to solve?
Research for improvement
Examples:
Student feedback, including NSS
Outcomes of quality assurance
Assessment results, cross-tabulated with demographics,
subject disciplines, etc.
Incidence of plagiarism
Research assessment exercise
Academic/Summative: known unknowns
IR as Educational Research
Academic – summative and external, though with some
formative and internal relevance
Investigation of issues which are of wider relevance than to a
single institution, but which may be triggered by an internal
institutional need.
Evidence needs to be particularly robust.
Potential for dissemination of findings beyond the individual
institution, in which case criteria for peer review may well apply.
What problems are we trying to solve?
Educational Research
Examples:
Pedagogical innovations of various kinds
Assessment of achievement in workplace settings
Variation in honours degree classifications across subject
disciplines and institutional types
Relation of PT employment to achievement on FT
programmes
Students’ experiences of HE (FYE; PT; HE in FE, etc)
Non-completion
In the end what answer can we give
to Gnothi Sauton – ‘Know thyself’?