Transcript Presentation to the Ellis Center Board

Evaluation of the Quebec
Community Learning
Centres: An English
minority language initiative
Natalie Lacireno-Paquet
Learning Innovations at WestEd
[email protected]
1-800-347-4200
Evaluation Purposes
• To support the implementation of the project
• To gather data to assess the attainment of
the short and intermediate expected
outcomes and the longer terms impacts of
the project
• Examine and identify the processes that help
or hinder the attainment of expected
outcomes and impacts
Evaluation Approach
• Multi-pronged, mixed-methods, and
longitudinal, including:
– Implementation of the CLC model at 22 sites, guided
by the Project Theory of Change
– Work of the PRT in building CLC staff capacity and
the work of the PIC in supporting the CLCs through
external relations
– Evaluation of expected outcomes and impacts
– Documentation of lessons learned
Evaluation Questions
• Questions are at the implementation level,
the impact level, and lessons learned
• Questions developed in collaboration with
PRT and informed by literature
• Detailed evaluation questions and indicators
found at:
http://www.learnquebec.ca/en/content/clc/clc_
res_eval.html
Evaluation Questions: Implementation
• To what extent and in what ways do the PRT and
PIC implement activities and processes designed
to contribute to the capacity of the CLCs to
achieve short and intermediate outcomes? Some
sample indicators:
–
–
–
–
Types of support services provided by PRT
Perceived responsiveness of PRT to CLC needs
Types of collaborations established by PRT
Level of support in establishing connections with
regional and provincial resources
– Perceived use and value of trainings
– Perceived challenges and resolutions
• To what extent and in what ways do the CLCs, led by
principals and CLC coordinators, implement their Action
Plans? A small sample of indicators:
– Level of use and valuation of Framework for Action (FFA)
– Development and implementation of action steps
– Identification of stakeholders/ Inclusion of stakeholder groups
and steering committee in development
– Types of community involvement activities/strategies planned
and developed
– Number and frequency of community members’ use of services
– Number and types of community organizations involved in
partnerships with CLCs
– Teachers use of Community Based Learning
– Types of services provided and extent to which services meet
identified needs/gaps
Evaluation Questions: Impact
• To what extent and in what ways do the 22 CLCs
accomplish their goals? Sample of indicators:
– Number, types, and characteristics of collaborative
partnerships between schools, families, and communities
– Student success measures (i.e. achievement,
attendance, etc.) (collected by schools/CLCs)
– Level and extent of participation in CLC activities and
services by various stakeholders
– Similarities and differences in implementation and impact
across CLCs
• To what extent and in what ways do the 22
CLCs show evidence of sustainability after
the project funding and support end?
– Level, types, and nature of community
support, diversified funding, and formalized
policies and practices that are developed by
the CLCs
– Additional support and resources identified
Evaluation Questions: Lessons
Learned
• To what extent and in what ways do the PRT’s and
CLC Theories of Change adequately represent the
processes and outcomes of the initiative?
• What lessons does the project offer for policy and
practice, particularly concerning establishment and
support of CLCs in English-speaking communities in
Quebec?
– Factors identified that facilitate or hinder the work
– Processes associated with successful attainment of results
– Insights from project and CLC “theories of change”
Methods
• Mixed methods evaluation
–
–
–
–
–
Qualitative and quantitative data collected
Site visits to each CLC
Observations of trainings, meetings, etc.
Extensive interviews and document review
Portraits / case studies of 22 sites with cross-site
analysis
• Longitudinal data collection: 2007 - 2009
• Collaborative approach
– Frequent communication/feedback
– Formative as well as summative reporting
Evaluation Activities to Date
• Observations of multiple trainings and meetings
with focus groups of principals and coordinators
• Interviews with members of PRT
• Interim reports on observations/interviews
• Initial interviews with Phase 1 coordinators
• 2-day site visits to Phase 1 CLCs with “Findings
Memo” to sites
• Initial quantitative portrait development for Phase 1
CLCs
Early Findings: Project Level
• Project Resource Team (PRT) has been
flexible and responsive to formative feedback
– Changes to “Guidebook”
– Reduction in paperwork
• Introduction of “Concerns Based Adoption
Model”- help understand process of change
Importance of coordinator/principal in-put in
planning activities
• Importance of time for coordinators and
principals to share and learn from each other
• Tailored and one-on-support of CLCs
– Realistic expectations
– Responsive to unique situations
• PRT has both shared work and
specialized work
– Share supporting CLCs through “pairing” of
PRT member and CLC
– Specialized roles relating to support and
training in community based learning, selfevaluation, community partnerships and
networking
Early Findings: CLC level
• Coordinators are key
– Work hard at establishing partnerships and
CLC visibility
– Principals rely heavily upon them
– Some coordinators feel overworked but
satisfied. Can be overwhelming at first.
– Coordinators building a support network
– Some coordinators driving vision- rather
than developing shared vision
Early Findings: CLC level
• Leadership and Governance
– CLCs don’t advance in implementation
without principal support
– Leadership change may hurt or help
– Some confusion over role of school
Governing Board with respect to CLC
– CLCs use steering committees to varying
degrees
– Not all school boards supportive affects
implementation
• Involvement of teachers, parents, and
students
– Wide variation among the CLCs from low to
high
– Expression of sense of pride “our school is
a CLC” but some take a “wait and see” view
– Some teachers and students associate CLC
with video-conference equipment
– Some concern about time to learn VCN and
Community Based Learning techniques
– Use of VCN and CBL have taken off in
some communities
• Community and Partnerships
– Many new services are being offered
– Many new partnerships established
– Most CLCs engaging with partners and
community; one or two are not at this time
but renewed efforts appear to be underway
– CLCs not always selecting partnerships
strategically- some cast wide net, others
more selective
• Other findings and challenges
– Two models emerging: woven and
parallel
– CLCs with multiple schools involved see
varying levels of involvement and have
additional challenges for implementation
– Coordinators / principals value time to
come together in PRT meetings and
trainings
– Sustainability is a major concern among
stakeholders