Transcript Maximising Response rates and Q&A

MAXIMISING RESPONSE RATES
GHEETA KRISHNAN, UTAS
BRETT RYAN, GRIFFITH
ALBERTO MENDEZ, UTS
Getting the Results You Want:
A Response Rate Strategy
Student Evaluation, Review and Reporting Unit
(SERRU)
Contents
1.
SERRU’s role
2.
Survey Implementation
3.
What we achieved in response rates –2 snapshots
4.
How we improve our methods.
5.
How we increase our response rates.
6.
How we impress upon our stakeholders.
SERRU supports the University to……
 Develop a strong evaluative culture to improve the student and staff
experience at the University of Tasmania
 Establish strategic relationships with a range of stakeholders to ensure
quality processes and practices are maintained and improved across
the University
 Provide staff support on a range of data needs, priorities and projects
 Collaborate with people internally, nationally and internationally to
build partnerships in the quality of teaching and learning, standards
and review
What we do at SERRU with our data skill sets….
 Survey implementation and analysis
 Course and unit data evaluation
 Data analysis
 Research reporting
 Benchmarking
 Standards
 Policy development
 Project management
Survey Implementation
Develop a strong evaluative culture to improve the student and staff
experience at the University of Tasmania
 External Surveys
 Australian Graduate Survey (AGS)
 University Experience Survey (UES)
 International Student Barometer (ISB)
 Ad hoc national surveys
 Internal Survey
 eVALUate - the UTAS online student feedback system on units and
teaching
Australian Graduate Survey - Timeline of Response Rates
2009 – 2013
Response rate %
60
55
50
45
)
20
13
(n
=
48
00
)
20
12
(n
=
46
86
)
20
11
(n
=
44
35
)
40
28
(n
=
20
10
20
09
(n
=
40
30
)
40
How we compare nationally….
Survey comparison by year from 2011 - 2014
60
56.3
55.45
54.06
55.9
51.39
50
46.36
40
36
33
29
30
24
UTAS AGS
29
National Average UES
25
24 24
23
National Average AGS
UTAS UES
National Average ISB
20
17
UTAS ISB
16
14
10
0
2011
2012
2013
2014
Building awareness and engaging students and staff
through…
 Workshops with Schools and Faculties
 Student Advisers’ Newsletter
 Student Union Representative
 Social Media Engagement
 MyLo – Online Student Learning Environment
 Electronic Notice Boards
Building awareness and engaging students and staff
through…
 Posters around high traffic areas around campus
 Flyers distributed at Graduation Ceremonies
 Alumni Newsletter
 Bulletin on the Graduation Webpage
 Presence on SERRU External Surveys and eVALUate webpages with
links to the respective survey sites
Maximizing response rate potential through…..
 Initial email and letter invitation from relevant University management
 Telephone Reminder of 300 calls per day at commencement of survey
 Follow up telephone reminder towards the end of the survey round
 2 to 3 SMS reminders per round
 2 email reminders from Survey Manager
 Regular response rate updates to Student Advisers
 Offering an incentive
 Using the correct terminology e.g. International Student Barometer
Survey (for international students)
 Maintaining a Survey Register
Communicating results to stakeholders……
 Council
 Senior Management Team (SMT)
 Schools and Faculties
 Division of Students and Education (Divisional Management Team)
 Students
For more information on the
University of Tasmania’s
response rate strategy
Please contact:
Gheeta Chandra Krishnan
External Surveys Coordinator
Student Evaluation, Review and
Reporting Unit (SERRU)
[email protected]
http://www.utas.edu.au/studentevaluation-review-and-reportingunit/
Brett Ryan
Survey Manager - Griffith University
The Australian Graduate Survey captures the thoughts, opinions and outcomes of
recent graduates from all Australian universities.
Your valuable feedback is completely CONFIDENTIAL and it will:
-
inform future students on study, career, & employment outcomes
assist the federal government with higher education policy development
inform program improvements here at Griffith
qualify you to receive an engraved Griffith commemorative pen
and enter you into a draw to win other great prizes too!
Look out for an email with your personal survey link in November 2014.
Email Prep
• Spending time before you launch is critical
• Email hierarchy – D&A, PS update, home, business, uni
account
• Cleansing the list:
– Check priority 1s
• Is it a GU account? Are they a continuing student or a staff member?
• Is it an international student’s education agent?
– Check for duplicates
– Search for domain names
Other emails….
•
•
•
•
Heads of School
Pro Vice Chancellor (International)
Deans (Learning & Teaching)
Survey Manager
– Partial completers
– Staff non-responders
– Prize draw deadline looming
We use multiple senders, with multiple messages.
SMS & Paper
• SMS
• Postal surveys are expensive so it is important to have good data
– We use a similar process as to how we attack the email file
• International students receive two pieces of Direct Mail; one to their
Australian address and one overseas
• We use a different PURL for people who complete the survey online
as a direct result of receiving the letter (to track its effectiveness)
• And we also use this….
The Final Push
• Telephoning fieldwork
– Centralised Telephone Service via GCA
• The final, final push (read into panic)
– the next 200 responses receive a guaranteed…
• And if you are really desperate….
– Guaranteed offer extended!
or
– Try again on the phones yourself….but you can only
do the GDS.
Brett Ryan
07 3735 4394
[email protected]
AGS Collection at UTS Graduations
Alberto Mendez
Survey Coordinator
Planning and Quality Unit
University of Technology, Sydney
The typical AGS situation
In AGS 2008 only achieved 48% response rate …
 thus unable to publish data outside UTS!
In AGS 2009 and 2010 achieved 51%, 52% but …
 required huge effort to achieve target
(printing and posting 2 or more sets of paper forms)
 not able to drill down very far into data
Must be an easier way …
The rather fortuitous UTS solution
AGS commencement dates align well with Graduations:
 AGS = April 1 and October 1
 UTS Graduations = start of May and start of October
Main difficulties:
 reluctance from Graduations Office to fit us in
 possibly disrupt existing process, delay ceremonies
Trial run agreed on, successful*
Collection mode structure
Thus UTS able to increase AGS collection modes:
 hardcopy (postal + Graduations) and online
 telephone interviews also briefly adopted, since discontinued
So, current order of modes:
October
April
1. hardcopy (Graduations)
2. hardcopy (postal)
3. online
1. online
2. hardcopy (Graduations)
3. hardcopy (postal)
4. telephone
In the lead up to Graduations
Preparation work:
 request graduate list from Student Systems
 prepare email communications (push incentives)
 compile a “mark-off” spreadsheet and assign each graduate their
oAGSID
 print individualised hardcopy forms and letters and stuff into
addressed postal envelopes
 align graduates and ceremonies:

sort hardcopy forms into individual ceremony’s box

prepare list of graduates for individual ceremony and markoff online completions (April only)
At Graduations
UTS setup:
 component of pre-ceremony official process (1 hour):

registration  gowning  AGS  Alumni
 AGS physical space:

5 tables, each with 6 chairs
 AGS staff:

2 people, spruiker (promoter, welcomer) and form provider
 AGS completion time:

typically 5-10 minutes (few if any blanks, incompletes)
Graduates + AGS + chocolate = happiness!
Typical numbers
October
April
Total graduates
3,200
6,100
Graduation ceremonies
10
20
online (before and during Graduations)
0
2,000
1,300
1,500
hardcopy (after Graduations, postal)
100
100
online (after Graduations)
300
100
AGS completions …
hardcopy (at Graduations)
AGS response rate
Chocolates (regular size, 50-60g)
53%
1,000
61%
2,500
Conclusions
 Now seamlessly part of pre-ceremony process, graduates go with
the flow, many see it as a ‘must do’
 They don’t do it for chocolate but appreciate the gesture
 Small number of latecomers with no time left to complete AGS,
asked to take AGS and return it in post
 Small but annoying features of setup:

relatives/friends wanting to spend time with them
(sometimes hard to keep them out of AGS area, seats)

pen stealing
Final thoughts
 Great mode for maximising responses but requires considerable
time investment from dedicated staff – use of casual, un-invested
staff can reduce effectiveness
 Graduates overwhelmingly happy, good time to target them for
participation, but possible mode effect on CEQ scores?
Response mode effect on CEQ scales