Transcript NotSchool Australia Pilot (NAP) Evaluation

TRAVERSING THE INTERPLAY OF
POLITICS AND EVALUATION:
EDUCATION REFORM IN AUSTRALIA
Assoc Prof Janet Clinton
Dr Amy Gullickson
Ruth Aston
Edmund Misson
Pauline Ho
AES, Darwin 10th September 2014
Introduction
 Paper 1: Collaborative Evaluation
 Paper 2: Evaluation Methodology
 Paper 3: Dissemination of Evaluation Findings
Paper 1: Collaborative Evaluation
Janet Clinton & Edmund Misson
University of Melbourne & AITSL
Evaluation context - 1

Overview of the Standards
Map progression:
 Graduate
 Proficient
 Highly Accomplished
 Lead
Evaluation context - 2
Dual purpose:
 Improvement



Career progression




Performance and development
Professional learning
Accreditation – Graduate
Registration – Proficient
Certification – Highly Accomplished and Lead
Support Materials
Why this evaluation?


Evaluation embedded in policy implementation to inform and guide
implementation
How?
 Collaborative
 Formative
 Multi-method
 Multi-year
 Utilising existing infrastructures
Considerations

Stakeholder communication and engagement

Sharing information and findings

Timing

Alignment of requirements/needs

Changing political context

Evaluation is designed to add value
Paper 2: Evaluation Methodology
Amy Gullickson & Ruth Aston
University of Melbourne
Evaluation methodology
Evaluation methodology


Mixed-Methods
Informing policy development =
Evidence Based Policy Implementation
Evaluation phases
Stage 1- Develop and refine design
•
•
•
Phase 1
Build evaluation foundation
Establish stakeholder groups
and team
Lit review & program logic
Phase 2
•
•
•
National Forum
Stakeholder interviews
Collect existing documents
Stage 2- Collect evidence
Phase 3
•
•
National Survey #1
National online depository
Phase 4
•
•
Case studies
Data collection round #2
Stage 3- Finalise and make recommendations
Phase 5
•
•
•
Stakeholder interviews
Final data collection round
National Survey #2
Phase 6
•
•
•
Triangulation of findings
Revisit program logic
Draw overall conclusion
Levels of influence
ACCREDITATION
(GRADUATE)
STUDENTS
SUPPORT
MATERIALS AND
RESOURCES
TEACHERS AND SCHOOL
LEADERS
REGISTRATION
(PROFICIENT)
SCHOOLS AND
COMMUNITIES
SYSTEMS/SECTORS &
AUTHORITIES
STATES/
TERRITORIES
PROFESSIONAL
LEARNING
NATIONAL
POLICY
PERFORMANCE
AND
DEVELOPMENT
CERTIFICATION
(HA AND LEAD)
National Forum

Interviewees
= 82

Workshop
participants
Government teacher employers
Catholic teacher employers
Independent teacher employers
Regulatory Authorities
Principal Association
Union National Bodies
Deans of Education
AITSL Board Members
= 174
AITSL Board Alumni
National Survey #1
Respondents
Teacher
School leader
Teacher educator
Pre-service teacher
Combined

National Survey#2 (June 2015)
Total
4141
1427
214
219
6001
Case Studies
Paper 3: Dissemination of Preliminary Findings
Edmund Misson & Pauline Ho
AITSL
Driving a dynamic
communications strategy to
evidence change and impact for
the Evaluation
Edmund Misson, Pauline Ho
In collaboration with Sam Hussein, AITSL Communications, Online &
Social Media
Introduction
The Evaluation is a complex
and dynamic process of reform
implementation
- involves diverse stakeholders
across varying contexts, levels,
interests and needs.
AITSL’s Communications & Social Media
Strategy

Part of the whole of organisation Universal
Analytics Framework (UAF) tracks the
promulgation and engagement of AITSL’s tools
and resources

Multiple levels of analytics:

CEO & Board

Teams’ analytics to track engagement with tools and
resources

‘Deep dives’ to explore engagement of specific requests

‘Personas’ to understand demographics, behaviours and
horizon scanning of AITSL’s audience
Aim of the Evaluation’s Communications
& Social Media Strategy

Methodological Rigour: Gain awareness and participation in the key data collection
activities of the Evaluation e.g. National Forum, National Surveys, Case Studies.

Engage the Profession: Encourage professional conversations on the Evaluation’s
findings, analysed and reported in a variety of data visualisation ways.

Promulgate Findings: Share findings through various communication channels cross
stakeholders to value add to the policy implementation.
Ultimately, the key goal is to increase the effectiveness of the
implementation of Australian Professional Standards for
Teachers in schools and organisations across jurisdictions and
sectors.
Objective: Engage stakeholders of the education profession to participate
in the Evaluation and to drive implementation of the Standards
Measuring Impact
Acquisition
Measure
What are we doing to
gain stakeholders’
attention to the
Evaluation?
Engage
Behaviour
What are they doing
and how are they
accessing the
Evaluation’s activities?
Participate
Knowledge translation
Digital Marketing & Measurement Model by Avinash Kaushik
Outcomes
What is the impact and
outcomes of their
participation?
Reimaging
Data
Agents
Web analytics
Social media
analytics
Campaigns
Print
Acquisition - Using diversity of acquisition channels
Knowledge translation
Social Media
Print Collaterals
Facebook
Twitter
LinkedIn
Reports
Journal publications
Newsletters
Infographics
Fact Sheets
Website & Online
Marketing
AITSL E-News
Evaluation webpage
Interactive Infographics
Behaviour – Engaging with our content
8,143 unique visitors
% of audience accessing
the survey
70%
121
1st interaction
2140
Returning
Visitors
6233
New
visitors
Evaluation
page
50.5%
18.7%
Survey page
62.3%
51.8%
7,000
6,425
898
2nd
interaction
1,780
3,490
227
Utility of Devices
6,000
2000
5,000
4,000
0
3,000
1,133
2,000
Avg. Visit
Duration: 07:23
1,000
585
0
desktop
tablet
mobile
1,182
Social Networks
Behaviour – Engaging with our content
Tracking unique visitors
9000
National Survey
(Oct-Nov 2013)
8000
7,893
Monthly Analytics Report
7000
6000
Case Studies 2014
(Mar-May 2014)
5000
3,721
4000
3000
2,277
2000
1000
0
682
1,543
971
763
912
730
325
276
380
2,045
969
Outcomes – Impact of the campaigns
140 submissions
across jurisdictions
6,001 respondents
across jurisdictions
Case Studies
LAUNCH
Research
Aug
2013
National
Survey 1
Sept
2013
Targeted
Marketing
Research
Dec
2013
Case Studies
*closed*
SR 1 LAUNCH
Mar
2014
Apr
2014
Disseminate
findings
May
2014
Targeted
Campaign
Research
Stakeholder 1
Fact Sheet –
Infographics
SR 2
LAUNCH
August
2014
June
2014
Disseminate
findings
Ongoing
promulgation
Outcomes – Impact of the campaigns
Short-term
- Create awareness of the Survey and Case Studies
Mid-term
- Participate in ongoing data collection activities
- Inform findings and shape the Evaluation
Long-term
- Analyse and reporting of results and findings
Investigate the Impact of the implementation of the Standards on
improving teacher quality
Next Steps

Not purely an online/social media strategy

Involves other forms of stakeholder engagement including
workshops, symposiums, forums and meetings

Rethinking about how we deepen our understandings of
the Evaluation through other ways of working across
jurisdictions, sectors, schools and organisations
Paper 4: Bringing it together
Janet Clinton & Edmund Misson
University of Melbourne & AITSL
Final Comments
Adding value
Defining boundaries of collaboration
Incorporating knowledge translation
Transparency
Demonstrating worth
Understanding Collaborative Evaluation

2013- defining the approach and how it was going work
Difficult– the literature often promotes it but doesn't really tell us
how
 It about all about:

RELATIONSHIPS
UNDERSTANDING
TRUST
RIGOUR
ROLE DEFINITION
Collaborative Evaluation
How do we truly be collaborative and
maintain objectivity?
Rigorous methods that are open and
transparent
Understanding the influences
Relationship
• How it evolves
• Practically how it works
• What does it mean for methods
Understanding the Influences for each Party
Historical
Contextual
Economics
Collaboration
Cultural
Social
Political
Psychological
Have the conversations and act!





Relationships tensions
Strong personalities
Different management styles
Variable flexibilities
Extended partners
Conversation--• Open & frank
• Lose the emotion
• Process a solution
Collaboration about Methods
Forums
Conversations
Survey
Case
Studies
Model of Objectivity
•
•
•
•
•
•
•
•
•
Recognised Standard evaluation framework
Rigorous methodology
Mixed methods
Evidence of analysis at every level
Triangulation clarity
Sophisticated social science analysis
High level of dissemination
Evidence base content
Every step transparent & reproducible
DIAMOND STANDARD APPROACH
High
Re-analyse -- suggest—advise– reflect—promote - use
OBJECTIVITY
Analyse –demand—contact--micro-manage-- change content- little dissemination
Low
High-stakes Evaluation
If it doesn’t work
But….
HANDLE WITH CARE
Questions?
Contacts
Assoc Prof Janet Clinton
[email protected]
Ruth Aston
[email protected]
Pauline Ho
[email protected]