INSTITUTIONAL REPORTING

LEONE NURBASARI, ANU PAMELA SARLY, ACU HIEDI WILKINSON, USC

Institutional Reporting from the Australian Graduate Survey

Leone Nurbasari Planning and Performance Measurement July 2014

Presentation Overview

• • • • • • Key Performance Indicators Program (Course) Review reports Discipline reporting on CEQ/PREQ Graduate destination reporting Comments analyses Other reporting 3

Key Performance Indicators

• • ANU Strategic Plan College Operational Plans and KPIs 4

• •

KPI generation

Raw unit record data at 6 digit FOE, assigned to College scores based on taught disciplines (rolling 5 year average, minimum 10% of load taught by that College) Benchmarking against Go8 data that is weighted according to ANU discipline weightings 5

Example of some underlying 6 digit FOE data and calculations for KPI generation 6

Program (Course) Review Reports

• • • • Program reviews Data warehouse reports – CEQ, Employment & Further study Supplemented by additional detailed analyses (including item level results) Supplemented by anonymised open-ended comments 7

Discipline reporting – CEQ/PREQ

• • • • Pseudo colleges (custom groupings of 2, 4 and 6 digit FOEs) Pseudo departments (also custom groupings of FOE) Benchmarking – Go8 and national, based on ANU taught FOE (excludes non ANU disciplines) Percentage of individual student-item response in disagreement/neutral/agreement categories, aggregated to scales 8

9

Graduate destination reporting

• • • Data warehouse reports – employment rates, sector, industry, salary, employer, job search Cohort-specific analysis Ad-hoc data requests 10

• • •

Comment analyses

CEQuery with charting via Excel Various student cohorts and org. units Adopting Geoff Scott’s method of proxies for importance and quality 11

• •

Other reporting

Topic Papers – comprising data from various surveys and student feedback sources, eg. HDR Satisfaction Report Ad-hoc reporting from internal pivot tables – GDS, CEQ, PREQ 12

Questions?

• Email [email protected]

• Web http://unistats.anu.edu.au/surveys 13

AGS Data and Reporting at ACU

Pamela Sarly | Acting Manager, Statistical Analysis & Surveys 2014 Survey Manager Information Forum 17 July 2014

A bit about ACU… ONE OF THE FIRST EDUCATION PROVIDERS IN AUSTRALIA 1857

Teacher Training in NSW and VIC

1900

Good Samaritan Sisters

1963

Merger of education colleges, forming ACU

2014

Teacher Training in NSW, VIC, QLD and ACT

1991

One of the fastest growing universities in Australia

Australia's leading Catholic university which is supported by more than 2,000 years of Catholic intellectual tradition.

12 Schools across 4 Faculties 490 Higher Degree Research Students 6,263 21,934 Undergraduate Students 826 Postgraduate Students Non-Award Students

29,513

Enrolments

22,245

EFTSL

2,839 international students 12,952 commencing students 16,561 continuing students • •

Staff FTE

1,051 Academic 982 Professional Data as at 11 July 2014

AUSTRALIAN GRADUATE SURVEY

Data Governance at ACU • A central contact for surveys: o Office of Planning and Strategic Management • Publish aggregated data through internal sites: o Staff Site ( www.acu.edu.au/opsm ) o SharePoint site • Data updates communicated to relevant staff via email

AUSTRALIAN GRADUATE SURVEY

Institutional Reporting • Strategic Plan - Traffic Light Report • Quarterly Report • TEQSA Risk Indicators • New Course Application • Course Review and Renewal

TRAFFIC LIGHT REPORT

• Published in July and December • Progress against the University’s Strategic Goals and Key Result Areas

STRATEGIC SCORECARD

QUARTERLY REPORT

Strategic Plan Goal 1: Student Experience • Published in January, April, July and October • High level quantitative data that is linked to Strategic Goal 1 (Student Experience)

AUSTRALIAN GRADUATE SURVEY

For the future… • Use of SPSS TAS for qualitative data analysis and reporting • Better linkage to Student Evaluation on Unit and Teaching surveys • Future of AGS – to inform our new Strategic Plan 2015+

Thank you

Questions?

Australian Graduate Survey

COMMUNICATING RESULTS FOR EFFECTIVE DECISION MAKING

Note that the figures contained within this presentation are fictitious.

Why do we need to communicate results?

AGS results are used at USC to:  Inform strategic decision making   Assess program and institution performance Influence improvements to programs   Review learning and teaching Develop internal policies

Why use visualisations?

“Numbers have an important story to tell. They rely on you to give them a clear and convincing voice.”

Stephen Few

Choosing the right medium

Textual Visual Oral

Tapping into human nature

Over 50% of the cerebral cortex is involved with the processing of visual inputs.

-Lu & Dosher There are over 130 million sight receiving cells in the retina.

- Vries 2011

Source: Atranik 2011

Tableau

Knowing your audience

USC’s Organisational Chart

Know your goal

Enable insight

"The purpose of visualization is insight, not pictures"

S hneiderman 1999

GDS Dashboards Flowchart

CEQ Dashboards Flowchart

Levels of Analysis      Graduate Destination Survey Benchmark Group Dashboard Course Experience Questionnaire Longitudinal Dashboard Graduate Destination Survey Faculty Dashboard Graduate Destination Survey Program (Course) Dashboard Course Experience Questionnaire Program (Course) Dashboard

How dashboards are used

Are they being used?

Who is using them?

What are they using them for?

 Director  - Quick reference point  Program leaders  - Program reviews  Executive   - Marketing and recruitment - Identifying strengths and weaknesses  - Information sharing

What next?

  Reports by Field of Education Beyond Graduation Survey    University Experience Survey Internal Surveys Interactive Dashboards  Qualitative Data Visualisations

Bibliography

       Atranik.org 2011, http://antranik.org/functional areas-of-the-cerebral-cortex.

Azzam, T & Evergreen, S 2013, Data Visualization, Part 2: New Directions for Evaluation, Number 140, Google eBook.

Card, S, Mackinlay, J & Shneiderman, B 1999, “Readings in Information Visualization - Using Vision to Think”, Morgan Kaufmann, Massachusetts.

De Vries, J 2011, The Five Senses, Random House, Victoria.

Few, S 2006, Information Dashboard Design: The effective visual communication of data, O’Reilly, California.

Lu, Z and Dosher, B 2013, Visual Psychophysics: From Laboratory to Theory, MIT Press, ISBN: 9780262019453. University of Leicester (Learning Development) 2012, Presenting numerical data,

If you have any questions after today:

[email protected]

Questions?