Transcript What Do I Do Now? - Advanced Master Gardener

Programming Tools and
Techniques
Making Your Advanced Master
Gardener Program Work
VCE-Advanced Master Gardener Water Stewards
Master Gardener College
June 20, 2013
Pat Reilly
Name, unit
 Are you joining a program already
established or creating a new program?

Introductions
Understand why Advanced VCE-MG
programs need to address strategic
goals and how to do this
2. Provide logical and do-able steps
1.
a. Planning a new program
b. Evaluating / improving an existing program
Objectives
Value judged on relevance
 Strategic goals best represent
social needs
 Funding based on worth
 Accountability

Why Address Strategic Goals?





Government Performance & Results Act
Government Accounting Office
Office of Management and Budget
Council of Economic Advisors
Internal agency evaluators
Federal Level
Virginia, county, city budgets
 Effectiveness and efficiency
 Elimination of duplication of effort &
reduction in funding required

State and Local Level
Business partners
◦ Bottom line
◦ Build good will
 Extension & Master Gardener leadership
◦ Volunteer time
◦ Financial resources

Public, Private Level
Stewards’ Role in VCE Programming
2. VCE Programming Process
1.
◦
◦
◦
Situation Analysis
Program Design and Implementation
Program Evaluation and Reporting
Situation Analysis – Organization
4. Situation Analysis – Assess
Community Needs
3.
Programming Manual
Comprehensive Planning
6. Planning Level 3 or 4 Programs
7. Evaluation and Reporting
8. Community – Volunteer Management
9. Publicizing Your Program, Resource
Development
10. Presentation and Final Planning Meeting
5.
Programming Manual
To appraise, value, judge the worth of
 It involves collecting, analyzing,
interpreting and communicating
information about the effectiveness of
policies and social programs for the
purpose of improvement
 Evaluation easiest when designed
into the program

Program Evaluation Defined
Effectiveness and efficiency
Program outcome/impact
Process and implementation
Program design and theory
Need for the program
Evaluation Hierarchy






Assess the need for the program
Develop theories for why and how your
program will meet needs; design the
program
Create the steps or processes and
implement the program
Monitor for outcomes and impacts
Tell your story/report findings
Continuously improve the program
Steps for Program Evaluation
Identify who cares, and why
2. Define the target population
3. Identify the needs, outcomes and
solutions
4. Prioritize the needs
1.
Needs Assessment
1.
a.
b.
c.
d.
Identify who cares and why
Resources
Who wants the issue addressed
Who are all the “customers”?
Develop a needs statement
Needs Assessment – Step 1
1.
Define the target population
a. Who are these people?
b. Do the numbers support the scope?
c. Do you have the ability to reach them?
2.
Service delivery environment
a. Level of programming
b. Where will you offer your service?
Needs Assessment – Step 2
Level 1 – awareness; information distribution
(reactive)
Level 2 – awareness, personal commitment, and
increase in knowledge; limited educational
programs (reactive with some outreach)
Level 3 – increasing knowledge and skills; more
dynamic, one-time events (proactive)
Level 4 – behavioral change; on-going program;
community is beginning to assume responsibility
(long-term structured/proactive)
Level 5 – spin off the program (reduction in VCE
MG leadership of program)
Programming Levels - Goals
Identify the needs
2. Describe expected and current
outcomes
3. Generate possible solutions, plans,
activities
4. Chart your results
1.
Needs Assessment – Step 3
Set priorities for decision-makers
(discrepancy, or “gap” model)
2. Needed
3. Achievable
1.
Needs Assessment – Step 4

Assess the need for the program

Develop theories for why and how
your program will meet needs; design
the program
Create the steps or processes and implement
the program
 Monitor for outcomes and impacts
 Tell your story/report findings
 Continuously improve the program

Steps for Program Evaluation
What is the reasoning behind the cause
of the problem?
 Will program as designed address the
condition?

◦ Assumptions are made when creating
activities
◦ Examining the program theories will ensure
that the activities will have the desired effect
Program Design and Theory
1.
Methods for Arriving at the
Cause
a. Ishikawa or “fishbone” diagram
b. Social marketing technique
Program Design and Theory
Sample Cause and Effect Diagram
People
Policies
Lack
knowledge
Don’t know it can be easy
Piles stink
Don’t know how
Don’t have stuff
Don’t know it can
be easy
Don’t have time
Don’t have equipment
Won’t buy
Zoning or HOA
rules
Don’t know what’s green or brown
“How to” complicated
“How to” complicated
Equipment/
Supplies
Procedures
County doesn’t require
Trash haulers pick up trimmings
Landfill fills up
Environment
Citizens do
not compost
Cause &Effect Diagram – Root Causes
People
Policies
Lack
knowledge
Don’t know it can be easy
Piles stink
Don’t know how
Don’t have stuff
Don’t know it can
be easy
Don’t have time
Don’t have equipment
Won’t buy
Zoning or HOA
rules
Don’t know what’s green or brown
“How to” complicated
“How to” complicated
Equipment/
Supplies
Procedures
County doesn’t require
Trash haulers pick up trimmings
Landfill fills up
Environment
Citizens do
not compost
Social marketing technique
New
Behavior
Competing Competing
Behavior #1 Behavior #2
Perceived
Benefits
Perceived
Barriers
Program Design and Theory
Activities and outcomes
1. Why you think that the activities you
have picked for your program will
achieve the expected outcomes?
2. Graphically display the program from
idea through outcomes
a. Flow chart
b. Logic model
Program Design and Theory
Precautions!
1. Regulation is contingent on people’s
willingness to be regulated
2. Education alone has little or no effect
on sustainable changes in behavior
3. Economic self-interest has limited
success in changing behavior
4. Community-based social marketing
has had documented success
Program Design and Theory
Basic Flow Chart
Client
requests
composting
information
MG Steward
recruits for
program
Client
attends
classes
Client
receives
composter
Steward
conducts
survey
Detailed Flow Chart
Client requests
composting
information
MG Steward
Provides program
information
Client completes
enrollment form
A
B
Client provided
class schedule
Client
no
completes
program
6 mos
Steward
contacts
client
Steward
conducts
survey
yes
Form
processed
A
Client receives composter
B
Client is active
composter
Sample Logic Model
Long term
outcomes
Intermediate
outcomes
Initial
outcomes
Outputs
Neighborhoods are safer and healthier; surface and groundwater
cleaner
Residents implement BMPs
Fewer system failures; less Health Dept
resources are used/consumed
Residents are knowledgeable on
septic BMPs
Data on implementation of septic
BMPs
Classes are offered; residents attend
Pre-survey, post-assistance evals;
septic pump-outs
Activities Program provides residents with 2 classes two weeks apart in classroom on septic
system operation, individual site plan assistance and surveys before classes and 2
months after site plan assistance
Neighborhood provides meeting space, marketing support, residents to participate
Inputs
Health Department identifies communities needing septic program
Stewards provide program management, instruction, printed materials, site
assessments



Assess the need for the program
Develop theories for why and how your
program will meet needs; design the program
Create the steps or processes and
implement the program
Monitor for outcomes and impacts
 Tell your story/report findings
 Continuously improve the program

Steps for Program Evaluation
Has the program been implemented as
designed?
1. Reaching the target population
2. Service delivery consistent with design
3. Support functions providing assistance
…appropriate, adequate, intended,
sufficient, satisfactory, reasonable…
Process Theory
Regular review
1. Compare logic model or flow chart to actual
operation
2. Write a procedure of actual operation and
have program designers review
3. Ask program support if their role(s) and
activities are as described
Process Theory



Assess the need for the program
Develop theories for why and how your
program will meet needs; design the program
Create the steps or processes and implement
the program

Monitor for outcomes and impacts

Tell your story/report findings
Continuously improve the program

Steps for Program Evaluation
Data collection at regular intervals
2. Data analysis
a. Data relevance, reliability,
repeatability
b. Coverage of the target population
c. Bias
I. Self selection
II. Taking the cream off the top
1.
Monitoring
Outputs
2. Outcomes
3. Caution! When making claims…
1.
a. Attrition of your client base
b. Long-term trends which occur without your
program
Outcomes and Impacts
The measures everyone wants
(expects) to hear
2. Most difficult to determine
3. Requires the most information
4. Relies on the other layers of the
evaluation hierarchy
1.
Effectiveness and Efficiency
1.
Cost-benefit analysis
a. Outcomes expressed in monetary terms
b. Dollars on water quality program and
dollars on water treatment
2.
Cost-effectiveness analysis
a. Outcomes expressed in substantive terms
b. Dollars on converting polluters
Effectiveness, Efficiency Models
Assess the need for the program
 Develop theories for why and how your
program will meet needs; design the program
 Create the steps or processes and implement
the program
 Monitor for outcomes and impacts


Tell your story/report findings

Continuously improve the program
Steps for Program Evaluation
1.
2.
Appropriate to the stakeholders
Methods
a. Reports
b. Presentations
c. Personal interaction
3.
Content
a. Mix data and stories
Report Findings
Map your intended or current program
2. Determine what impact you want to
report
3. Collect and analyze data
4. Tell your story
1.
VCE Impact Statements
“Participants apply appropriate best
management practices”
 “Participants identify pathogen before
applying a chemical control”
 “Participants build successful coalitions
and networks with stakeholders”
 “Participants share their awareness of
community issues with others”

VCE Behavior Outcome
Examples

Assess the need for the program
Develop theories for why and how your
program will meet needs; design the program
Create the steps or processes and implement
the program
Monitor for outcomes and impacts
Tell your story/report findings

Continuously improve the program




Steps for Program Evaluation
Establish a schedule for program
evaluations every year or two
2. New people, new ideas and new
external forces require reviews
3. Consider external evaluators
1.
a. Different perspective
b. New audience aware of program(s)
Continuous Improvement
Questions?