Transcript Measurement and Data Collection
Outcomes and Program Improvement: Designing Effective Evaluations Understanding Measurement Tools and Data Collection
Presented by Christine A. Ameen, Ed.D.
Ameen Consulting & Associates ([email protected])
Agenda
1. Brief overview of Continuous Program Enhancement 2. Levels of measurement 3. Basic measurement methods 4. Technical characteristics of quality tools 5. Steps for finding new tools 6. The last resort: creating tools 7. Tips about using tools 8. Issues related to measuring outcomes 9. Don’t forget to ask the customer!
What is Continuous Program Enhancement (CPE)?
• A commitment to ongoing assessment of service delivery and client outcomes to achieve the best outcomes possible.
Planning for CPE
1.
2.
3.
A program model which describes the services for delivering services to participants and the outcomes participants should achieve An objectives model which defines objectives for the services and objectives for the outcomes An evaluation model which defines how data will be collected and used to monitor the attainment of process objectives and outcome objectives.
1.
A Conceptual Framework for Behavior Change Awareness: the initial consciousness, perception, or sense of a concept 2.
Knowledge/Skills: understanding, comprehension of a concept; demonstration of ability of that understanding or comprehension 3.
Behavior: performance or conduct in a specified way 4.
Modeling behavior: demonstration of specified behavior to others, most often to teach the behavior to others.
The Levels of Measurement
Nominal: labels descriptive of a variable, e.g., gender, ethnicity. These have no order and have no numerical value. These data are usually reported in percentages, e.g., percentage of shoreline restoration projects completed Ordinal: numerical values are assigned, e.g., a Likert 5-point scale. These have a logical order, but the distance between the points cannot be quantified. These data may be reported using percentages, e.g., percentage or participant whose participation level changes
The Levels of Measurement
Interval: numerical values are assigned, e.g., risk points. The distance between points is equal, but there is no zero. These data may be reported using averages, e.g., average score on a drug use attitude survey Ratio: numerical values are assigned, e.g., restitution hours. The distance between points is equal and there is a real zero. These data may be reported using averages, and comparing averages, e.g., an average of 26 correct answers on the substance abuse quiz is twice as many right as 13.
Why Does the Level of Measurement Matter?
Level of Measurement Description For Services or Activities For Outcomes Nominal Ordinal Interval Ratio Labels used to describe a variable, e.g., gender Not usually used Numerical values in a logical order, e.g., Likert scale Frequency counts Percentages Averages (scales) Numerical values at equal distance, no true 0, e.g., attitudes Not usually used Numerical values at equal distance, true 0, e.g., number of acres Not usually used Not usually used Sometimes used in calculating change scores, averages Percentages Change scores Averages Percentages Change scores Averages
Let’s Try It!
1.
A program wants to use a staff behavior checklist, with a 3 point rating scale, to assess the outcome of improved behavioral functioning. What level of measurement is this and how appropriate is it in this example?
Let’s Try It!
2.
A program team wants to use a restitution log, to assess the outcome of increased empathy. What level of measurement is this and how appropriate is it in this example? How might a log be used?
Basic Measurement Methods
Observation/activity logs: service delivery, participation, etc. EXAMPLE1 Interviews: satisfaction, risk assessment, etc.
EXAMPLE2 Surveys: self esteem, life skills, behavioral functioning, etc. EXAMPLE3 Formal testing or assessment: achievement tests, psychological measures, behavioral functioning, etc. EXAMPLE4
What Methods Fit Best for CPE?
Measurement Method Observation Examples for Services/Activities Staff behavior ratings Attendance rolls Activity logs Examples for Outcomes Not usually used, requires inter-rater reliability to be used as an outcome tool Interviews Surveys Formal testing or assessment Exit interviews Client satisfaction Participant feedback Not usually used Follow up interviews Empathy survey Environmental care survey Achievement testing Skills assessment
1.
The Technical Characteristics of a Quality Data Collection Tool
Has VALIDITY – it measures what the program is believed to be doing or to be impacting. The judgment of how valid a tool is should be made by program staff Does the content of the tool reflect what happens in the program?
If the tool is an outcome measure, will the participants’ responses to the tool change over time?
2.
Has RELIABILITY – it measures the program activity or service the same way every time it is used. The judgment of how reliable a tool is based on a goal of an alpha coefficient of .75 for scales
The Technical Characteristics of a Quality Data Collection Tool
3. Can be used relatively easily by the staff 4. Does not require any special credentials to be administered 5. Has a reading level appropriate to the participants (usually between 5 th and 7 th grade level) 6. Doesn’t take longer than 10-15 minutes to administer
The Technical Characteristics of a Quality Data Collection Tool
7. Uses language and examples that are culturally sensitive 8. Is available at no cost 9. Has a written procedure at the local level to assure appropriate use
Steps for Finding New Measurement Tools 1. Determine if the measurement is related to service or outcome 2. Determine if the program is already using a tool that might be modified to work
Steps for Finding New Measurement Tools
3. Discuss with the staff the following items: Define the outcome in your own words Describe the kind of awareness a participant would have to have to reach this outcome Describe the kind of knowledge a participant would have to know to reach this outcome Describe the kind of behavior a participant would have to show to reach this objective What kinds of questions would you ask the participants that would give you answers that would help you to know if the outcome was reached
4.
Steps for Finding New Measurement Tools
If a tool does not exist, especially an outcome tool, search resources to find something that responds to the information you are interested in (more on this is a minute…) 5.
6.
7.
If the tool is to be used for a service objective, ask the staff to pilot the tool for at least two weeks to be assured it will work If a tool is to be used for an outcome objective, ask the staff to pilot the tool with 30 participant, making sure to include participants with varying characteristics thought to impact participant response (e.g., gender, age, ethnicity?) When piloting, assure the staff follow the procedures for how data will be collected, under the same conditions, at the same time, etc.
3.
Steps for Finding New Measurement Tools
Begin your search by looking at these resources: Youth Development Community-based Programs Troubled Youth Personal change Mental Health
3.
Steps for Finding New Measurement Tools
Begin your search by looking at these resources: Youth violence Substance abuse Psychological skills Environmental programs Environmental Education Outcomes in Community Development
1.
Finding it DOESN’T mean you can use it!
Read very carefully the introductory language on the website describing the tool or tools it has. On some, permission to use them is given right there – here’s an example: http://www.ibr.tcu.edu/pubs/datacoll/intro.html
2.
3.
On other sites, you need to review the specific tool to look for evidence of copyright or permission; In general, if a tool is published in its entirety in a journal article, it is in the public domain and may be used;
5.
Finding it DOESN’T mean you can use it!
It is always a good idea, when in doubt, to contact the website or the author of the tool to confirm it is in the public domain; 6.
If you decide to contact the author, it is also a good idea to ask him/her for any information they have about the technical quality of the tool; 7.
Sometimes the site will not have an email address for the author. If the author is affiliated with a university, I go to that website and look there. If that’s not available, I usually “Google” the person, using their name and the name of the tool – very often you will find an article referencing the tool and can go from there.
1.
What if I can’t find anything from our resources? Google!
Go to: http://www.google.com/advanced_search?hl=en 2.
In the “ with all of the words” selection enter these words: outcomes measures evaluation instruments 3.
In the “with the exact phrase” enter the word or words that describe what you’re interested in measuring, e.g., decision making 4.
In the “with at least one of these words” selection enter words that might narrow the search, e.g., delinquent, youth, juvenile, offenders, etc.
More About Google!
5.
Don’t be concerned if you end up with thousands of records at first. As you review what you get, you can go back to the search criteria and enter into “without the words” those words that will eliminate things that just don’t apply; 6.
You will begin to notice patterns of references that will likely lead you to a potential tool – the same tool name will appear multiple times. This might be worth pursuing. If any of those references mention a publishing company or a book in which the tool is contained, it is likely NOT in the public domain.
The Last Resort: Creating Tools
1.
2.
3.
Avoid using “yes/no” responses in data collection for variables where a scale can be used – a scale representing choices from 1 to 4, for example, will demonstrate change more clearly and usually more quickly In building scales for data collection forms, it is often useful to use an even-numbered scale, e.g., 4 points, to avoid a middle point where the respondent can go to avoid making a choice response Most scales have between 4 and 7 points. Fewer than 4 makes it more difficult to show change; more than 7 may be confusing to the respondent
The Last Resort: Creating Tools
5.
The points on scales should be anchored with language that helps guide how they’re to be used (e.g., 1=not at all; 3 = sometimes, etc.). This should be included in written procedures for the tool 6.
Before you get too far along, ask for feedback to make sure you’re on the right track 7.
Make sure to pilot the tool 8.
Here’s a great resource that will help: Guide to Developing Tools
Some Tips About Tools Used for Measuring Services or Activities 1. The tools focus on information about the delivery of service, and often are completed by the staff rather than the participant 2. Use currently existing tools whenever possible,.
3. If an existing tool is to be used, review it to assure it contains all of the information desired about the activity 4. Don’t assume staff know how to use it; do some training 5. Write a procedure about how it is to be used
1.
2.
Some Issues to Address Related to Measuring Outcomes Will service delivery result in the impact on the participant?
Will service delivery result in the participant being able to maintain a positive impact after leaving the program?
3.
An increase or a decrease in something, e.g., empathy, requires the measure be used at the beginning and the end of the program (aka, pre-post test design) Note: It is also possible to ask the respondent to estimate their pre-test status at the time they take the post test only. This is called a post-retro pre design 4.
The achievement of a specific level of an outcome, e.g., the family will be able to name 3 community resources they can utilize, requires the measure be used at the end of the program only (aka, post test only design)
5.
Some Issues to Address Related to Measuring Outcomes Will instruments such as surveys be read to participant? If read to some, read to all. Because reading to participant requires more staff time and supervision, doing so should be discouraged wherever appropriate 6.
Instruments should be administered by staff with whom participant have limited or no interaction, to reduce the potential, unintended bias of participant responding based upon the relationship to the staff administering the instrument
Don’t Forget to Ask the Customer! 1. One of the most important tools in program evaluation is customer/client feedback 2. The typical key elements to ask about are: • • • • • • • Physical Surroundings Procedures Interactions with Staff Timeliness Understanding of client need Outcomes General Satisfaction
Some Examples of Feedback Surveys
1.
Parent Satisfaction 2.
Referral Agency Satisfaction 3.
Patient Satisfaction Survey
Let’s Try It!
My program has this objective: 85% of participant who complete the program will demonstrate increased personal self-confidence. What kind of tool would be appropriate to this objective and provide the rationale for your idea.
Let’s Try It!
My program has this objective: 65% of participant who complete the program will be able to function independently in the community. What kind of tool would be appropriate to this objective and provide the rationale for your idea.
Let’s Try It!
My program has this objective: 85% of participant will experience a positive and supportive relationship with their advocate. Create a tool appropriate to this objective and provide the rationale for the tool
Let’s Try It!
My program has this objective: 70% of participant will spend at least two hours weekly in positive extra-curricular activities. Create a tool appropriate to this objective and provide the rationale for the tool
Thank you for attending my session today!
If you have any feedback for how I can improve my training style or content, please email me your suggestions!
)