Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics (REMS) Oklahoma State University 918-284-7276 [email protected] [email protected] Prepared for the Oklahoma State Regents.
Download ReportTranscript Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics (REMS) Oklahoma State University 918-284-7276 [email protected] [email protected] Prepared for the Oklahoma State Regents.
Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics (REMS) Oklahoma State University 918-284-7276 [email protected] [email protected] Prepared for the Oklahoma State Regents for Higher Education 2012 Summer Grant Writing Institute To Evaluate or to Assess? Technically speaking…. ◦ Assessment Long-term outcomes, aggregated judgment ◦ Evaluation Short-term outcomes, “unique event” judgment In education terms…. ◦ Assessment Student learning outcomes ◦ Evaluation Research on programs/curricula to increase student learning [email protected] 918-284-7276 2 Why Evaluate? How will you know your project is progressing adequately to achieve objectives? How will funders know your project was successful? ◦ Increasing emphasis placed on evaluation, i.e., U.S. Department of Education National Science Foundation Substance Abuse and Mental Health Services Administration (SAMHSA) [email protected] 918-284-7276 3 Why Evaluate? Improve the program – ◦ “Balancing the call to prove with the need to improve.” (W.K. Kellogg Foundation Determine program effectiveness – ◦ Evaluation supports “accountability and quality control” (Kellogg Foundation) ◦ Significant influence on program’s future Generate new knowledge – ◦ Not just research knowledge ◦ Determines not just that a program works, but analyzes how and why it works With whom is the program most successful? Under what circumstances? [email protected] 918-284-7276 4 Why Evaluate? WHAT WILL BE DONE WITH THE RESULTS????? “Evaluation results will be reviewed (quarterly, semi-annually, annually) by the project advisory board and staff. Results will be used to make program adjustments as needed.” [email protected] 918-284-7276 5 Federal Emphasis on Scientifically Based Research (SBR) in Evaluation Experimental research design ◦ Random assignment Quasi-experimental research design ◦ No random assignment Program Evaluation Standards ◦ Joint Committee on Standards for Educational Evaluation 1. Utility 2. Feasibility 3. Propriety 4. Accuracy 5. Evaluation accountability [email protected] 918-284-7276 6 Types of Evaluation Process evaluation: What processes are used and how well do they work? Outcome evaluation: Did the project achieve its stated objectives? [email protected] 918-284-7276 7 Process Evaluation ◦ What was provided and to whom? services (modality, type, intensity, duration) recipients (individual demographics and characteristics) gender, age, race/ethnicity, income level, first-generation status context (institution, community, classroom) cost (did the project stay within budget?) • Do processes match the proposed project plan? ◦ What types of deviation from the plan occurred? ◦ What led to the deviations? ◦ What effect did the deviations have on the project and evaluation? [email protected] 918-284-7276 8 Outcome Evaluation ◦ What effect did the program have on participants? Activities / Objectives Achievement / Attitudes and beliefs ◦ What program/contextual factors were associated with outcomes? ◦ What individual factors were associated with outcomes? ◦ How durable were the effects? What correlations can be drawn between outcomes and program? How do you know that the program was the cause of the effect? You can’t! Unless you use an experiment. [email protected] 918-284-7276 9 Who will Evaluate? External evaluators increasingly required or strongly recommended ◦ ◦ ◦ ◦ Partners for effective and efficient programs Methodological orientations Philosophical orientations Experience and qualifications [email protected] 918-284-7276 10 How much will it cost? External evaluations cost money…period. Standard recommendation: 5% to 10% of total budget Kellogg Foundation; U.S. Dept of Ed.; NSF Check funder limits on evaluation Ensure cost is reasonable but sufficient [email protected] 918-284-7276 11 Two Types of Data Quantitative ◦ Numbers based on objectives and activities ◦ Types of data needed: Number of participants (process) Grade point averages (outcome) Retention rates (outcome) Survey data (outcome and process) Qualitative ◦ Interviews ◦ Focus groups ◦ Observation [email protected] 918-284-7276 12 Methods/Instruments How are you going to get your data? ◦ Establish baseline data ◦ Institutional Research Office (I.R.) GPA Retention Graduation ◦ ◦ ◦ ◦ Pre- and post-assessments (knowledge, skills) Pre- and post-surveys (attitudinal) Enrollment rosters Meeting minutes [email protected] 918-284-7276 13 Data Analysis Qualitative Data ◦ Data analysis programs NVivo , ATLAS.ti, etc… ◦ More than pithy anecdotes “May explain – and provide evidence of – those hard-to-measure outcomes that cannot be defined quantitatively.” – W.K. Kellogg Foundation Provides insight into how and why a program is successful ◦ Analyze for themes that support (or don’t) quantitative data [email protected] 918-284-7276 14 Data Analysis Quantitative data ◦ Data analysis programs: ◦ SPSS (Statistical Program for the Social Sciences), Stata, etc... ◦ Descriptive and inferential statistics: Descriptive: Frequencies Means Standard deviation Inferential (parametric, nonparametric) t-tests, Mann-Whitney U test (difference of means, two groups) ANOVA, Kruskal-Wallis, Friedman (difference among more than two groups) Correlation (relationship between variables) Regression (explanation, prediction) Etc…… [email protected] 918-284-7276 15 A Detour: Educational Research For education grants, two approaches to evaluation: ◦ Program Evaluation ◦ Educational Research Include Educational Researcher as co-PI ◦ Look in Educational Psychology programs ◦ OSU Center for Educational Research and Evaluation (a.k.a. REMS Center) [email protected] 918-284-7276 16 Educational Research (cont.) Educational research is NOT laboratory science research Requires Institutional Review Board (IRB) approval Oversight arm for Human Subjects Research May require informed consent How NOT to set up ed research ◦ One Biology section, split into two sections One receives “treatment”; other traditional instruction Students blindly enroll in one section WITHOUT knowledge this is an experiment [email protected] 918-284-7276 17 Educational Research (cont.) Example: NSF TUES ◦ Incorporate interdisciplinary, semester-long research projects into Introductory Biology ◦ Hypothesis: ◦ “We hypothesize that semester-long, interdisciplinary, collaborative research projects will increase student learning and interest in science more than the standard three-hour laboratory activity.” [email protected] 918-284-7276 18 Educational Research (cont.) Measurements follow from hypothesis ◦ Increases in student learning: Establish baseline data E.g., Performance on tests, course projects, course grades Pre/post tests Course projects compared to previous courses without intervention ◦ Increases in student interest: Establish baseline data (if possible) Pre/post attitudinal surveys Focus groups [email protected] 918-284-7276 19 Data Collection & Reporting: Two Types of Timeframes Formative ◦ Ongoing throughout life of grant ◦ Measures activities and objectives Summative ◦ At conclusion of grant funding NEED BOTH! [email protected] 918-284-7276 20 Timelines When will evaluation occur? ◦ ◦ ◦ ◦ ◦ ◦ Monthly? Quarterly? Semi-annually? Annually? At the end of each training session? At the end of each cycle? [email protected] 918-284-7276 21 Origin of the Evaluation: Need and Objectives Need: For 2005-06, the fall-to-fall retention rate of first-time degree-seeking students was 55% for the College’s full-time students, compared to national average retention rates of 65% for full-time students at comparable institutions (IPEDS, 2006). Objective: The fall-to-fall retention rate of full-time undergraduate students will increase by 3% each year from a baseline of 55% to 61% by Sept. 30, 2010. [email protected] 918-284-7276 22 Evaluation Data Collection and Reporting Plan Objectives Data collected and timeline Methods for data collection and timeline Instruments to be developed and timeline Reports/ outcomes timeline Increase fallto-fall retention by 3% per year to 61% Student enrollment in first fall and second fall within one month of start of second fall Enrollment entered by gender and race/ ethnicity into database within first four weeks of each semester Enrollment rosters separated by gender and race/ethnicity by Jan. 15, 2009 At midpoint of each semester [email protected] 918-284-7276 23 BEWARE THE LAYERED OBJECTIVE! By the end of year five, five (5) full-time developmental education instructors will conduct 10 workshops on student retention strategies for 200 adjunct instructors. [email protected] 918-284-7276 24 Logic Models From: University of Wisconsin-Extension, Program Development and Evaluation http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html A Logic Model is…… ◦ A depiction of a program showing what the program will do and what it is to accomplish. ◦ A series of “if-then” relationships that, if implemented as intended, lead to the desired outcomes ◦ The core of program planning and evaluation Situation Hungry Inputs Get food Outputs Eat food Outcomes Feel better [email protected] 918-284-7276 25 Evaluation Resources Evaluation Resource Center for Advanced Technological Education (NSF ATE) http://evalu-ate.org/ (directory of evaluators) Western Michigan University, The Evaluation Center ◦ http://ec.wmich.edu/evaldir/index.html (directory of evaluators) American Evaluation Association ◦ www.eval.org (directory of evaluators) Joint Committee on Standards for Educational Evaluation ◦ http://www.jcsee.org/program-evaluation-standards/program-evaluationstandards-statements W.K. Kellogg Foundation Evaluation Handbook ◦ http://www.wkkf.org/knowledge-center/resources/2010/W-K-KelloggFoundation-Evaluation-Handbook.aspx [email protected] 918-284-7276 26 Evaluation & Statistics Resources Statistics and Research Methods Resources ◦ http://statsandmethods.weebly.com/ The Research Methods Knowledge Base ◦ http://www.socialresearchmethods.net/ “Discovering Statistics Using SPSS,” by Andy Field ◦ http://www.sagepub.com/field3e/ Planning an Effective Program Evaluation short course ◦ http://www.the-aps.org/education/promote/pen.htm “Evaluation for the Unevaluated” course ◦ http://pathwayscourses.samhsa.gov/eval101/eval101_toc.htm OSRHE list of evaluators and other resources ◦ http://www.okhighered.org/grant%2Dopps/writing.shtml [email protected] [email protected] 918-284-7276 27