Simple Survey Resources: Templates, Tabulation & Impact

Jeff Buckley, Jenna Daniel, & Casey Mull

• • • • •

Overview

Situation Need Being Met Audience (Agents & Associates) Basic Overview of the Training How Can This Be Utilized in Your State?

Situation

• • • • As a result of shrinking state budgets, the need to effectively document program impact is more critical than ever. Many states do not have a full time Evaluation Specialist.

County Agents and State Specialists must develop their own evaluation instruments.

Many County Agents and State Specialists may not feel qualified or confident enough to develop their own instruments and/or to tabulate and report data.

Need Being Met

• • • A 4-H program evaluation training was created Implemented by Georgia 4-H State Specialists & CAES Faculty Piloted at the 2012 Winter School – 4-H Agents & Associates

•

Simple Survey Training

Training Format:

– 90 minute session – Interactive – 4-H Agents and Associates – Participants created their own program evaluation

• •

Simple Survey Training

Why is it Important to Gather Data?

– Who is your audience?

– – Process Evaluation – Internal Use Outcome Evaluation – External Audience

How will you share the results?

– Impact Statement – – – News Article Journal Article Other

•

Simple Survey Training

What Can a Survey Measure?

– Knowledge – – Attitude Behavior •

Designing Measurable Objectives

– Refer to the handout “Verbs for Writing Measurable Objectives.” – Measurable objectives will translate into more effective constructs and questions.

• •

Simple Survey Training

Consider…

– – – What you want your participants to know.

How you want their attitude to change.

What sort of behavior changes you want to see.

Creating A Construct

– – – Pick two outcomes/constructs and develop three “questions” each, OR Pick three outcomes/constructs and develop two “questions” each.

(The template has space for six “questions”.)

•

Simple Survey Training

Creating A Construct

– – – – – – The food in the cafeteria was hot.

The dining area was clean and comfortable.

There were a variety of fruits and vegetables.

Beverage selections included healthy options.

Vegetables were not over-cooked.

The servers were on task and pleasant.

•

Simple Survey Training

Likert Style Questions

– These templates are designed for use with Likert style questions.

– Pick the rating scale that works best for your objectives/program.

– This is the most challenging part of developing your survey. Allow time. Ask for feedback.

•

Simple Survey Training

Likert Style Questions

– – – – – Consist of a statement and a rating scale. Need to conform to one rating scale.

Measurable verbs that fit the appropriate cognitive domain.

One data point per question.

Is what you’re measuring important to the intended audience of the evaluation?

•

Simple Survey Training

Instrument Templates

– – – – – Traditional Retrospective Postthenpre 6 quantitative items 1 qualitative item Feedback on the program

•

Simple Survey Training

Tabulation

– Excel Spreadsheet – Directions – Worksheet for both instrument templates – “Locked” format

Now You’re Ready to Gather Data!

Now You’re Ready to Gather Data!

Now You’re Ready to Gather Data!

•

Results

Presentations Conducted

– Galaxy – Georgia 4-H Winter Conference – Georgia Association of 4-H Agents Conference – Southern Region Volunteer Conference

•

Results

Impact

– The percentage of program participants who stated that they could… – – – – – – – develop measurable objectives increased from 60% before to 100% after the program.

design evaluation instruments for their programs increased from 69% before to 91% after.

are likely to develop their own evaluation tools in the future increased from 46% to 89/%.

A domain analysis of the qualitative revealed the following themes… Participants were very glad to have the templates.

Participants plan to use the tools and knowledge they gained to gather data.

Participants plan to use the data they collect to develop higher quality impact statements.

•

Results

Extension Agent Quotes & Impact Statements

– With a mean score of 3.1 (SD=.994), overall respondents from Question 5 indicated they "agreed" they are more likely to get involved in their community as a result of the GPK Leadership Adventure weekend.

– Retrospective post then pre: [In Question 2], 84.6% of respondents indicated an increase in their confidence concerning their ability to “design or construct a map of the distribution of major world languages”.

• • •

Simple Surveys in Florida 4-H

Tabulation of Data

– Dropping the data in the sheets is so simple and saves a ton of time. It is much easier than manually tabulating learning gains – Data is easy to interpret from the sheets

How Data Has Been Used

– Data has been used with stake holders to secure program support and has been used in annual reports of accomplishment.

– Data will be used in presentations at professional conferences to show outcomes and projected impacts of programming.

5.

Training Evaluation

•

Training Program Evaluation Exercise

– Confidence in ability – Knowledge gain – Training evaluation

How Can You Utilize this Program?

• • • •

Resources:

– www.georgia4h.org/evaluationresources

State Meetings Webinars New Agent Training

Questions?

For more information contact: University of Georgia, State 4-H Office

Jeff Buckley, [email protected]

Jenna Daniel, [email protected]

Casey Mull, [email protected]

State 4-H Office – 706-542-4444