Transcript Document
Broader Impact Project Evaluation Chris Parsons Word Craft Webinar Agenda • • • • • • • • • Who are you? Define Evaluation Show Where Evaluation Fits with BI Plans How happy are you? Apply Evaluation to NSF Broader Impact Criteria Example: BI Evaluation Plan Sum Up How’d we do? Q&A Who am I? • Independent Educator & Evaluator (schooling in biology, education & business) • Current Projects – COSEE NOW Evaluator – Evaluator on Ross Sea Glider Project (J. Kohut) – Evaluation Advisor/Resource for NOAA’s California B-WET Grant Program – External Evaluator for Monterey Bay Aquarium Education Programs Front-end Evaluation At the Start: Who are you? Front-end Evaluation Who are you? #1 • Did you participate in any of the previous BI webinars in this series? Yes X No Who are you? #2 • Rate your experience with conducting evaluations of BI projects. A. B. C. D. No experience: a neophyte Little experience: involved in 1 evaluation Some experience: more than 1 evaluation A lot of experience: an expert Who are you? #3 • When does evaluation occur during a BI project? A. B. C. D. At the beginning At the end In the middle All of the above What is Evaluation? Copyright joe-ks http://www.joe-ks.com/archives_dec2003/SantaJobEvaluation.htm What is Evaluation? • Evaluation is the systematic collection of information about activities, characteristics, and outcomes of projects in order to make judgments about the project, improve effectiveness, and/or inform decisions about future programming. NSF, adapted from M.Q. Patton • Project evaluation helps determine a project’s merit (does it work?) & its worth (do we need it?). NOAA, Designing Ed Projects 2005 Why Should You Care? • Government Performance & Results Act (GPRA) • Helps connect you with your audience(s) and improve what you do • Can help you make a case to continue, to get some or more funding • Enables you to keep what’s good and toss what’s bad • And, wouldn’t you like to know that what you’re doing makes a difference? NSF Broader Impacts Criterion 2 • How well does the activity advance discovery and understanding while promoting teaching, training and learning? • How well does the proposed activity broaden the participation of underrepresented groups (e.g., gender, ethnicity, disability, geographic, etc.)? • To what extent will it enhance the infrastructure for research and education, such as facilities, instrumentation, networks and partnerships? • Will the results be disseminated broadly to enhance scientific and technological understanding? • What may be the benefits of the proposed activity to society? BI & Evaluation: The Fit Choose BI Plan BI Project Do BI Project (incl. Evaluation) Design/Develop BI Project BI & Evaluation: The Fit Choose BI Plan BI Project Do BI Project (incl. Evaluation) Design/Develop BI Project BI & Evaluation: The Fit Summative Eval Choose BI Plan BI Project Do BI Project (incl. Evaluation) Design/Develop BI Project BI & Evaluation: The Fit Summative Eval Choose BI Plan BI Project Do BI Project Formative Eval (incl. Evaluation) Design/Develop BI Project BI & Evaluation: The Fit Summative Eval Choose BI Plan BI Project Do BI Project Formative Eval (incl. Evaluation) Design/Develop BI Project Formative Eval BI & Evaluation: The Fit Summative Eval Choose BI Front-end Eval Plan BI Project Do BI Project Formative Eval (incl. Evaluation) Design/Develop BI Project Formative Eval Formative Evaluation http://www.funnydb.com/ In the Middle: How happy are you? Formative Evaluation http://www.funnydb.com/ How happy are you? • So far, are your information needs being met? A. B. C. D. Yes, definitely Yes, mostly Not sure yet No, not at all Formative check-in Where Next? NSF Broader Impacts Criterion 2 • How well does the activity advance discovery and understanding while promoting teaching, training and learning? • How well does the proposed activity broaden the participation of underrepresented groups (e.g., gender, ethnicity, disability, geographic, etc.)? • To what extent will it enhance the infrastructure for research and education, such as facilities, instrumentation, networks and partnerships? • Will the results be disseminated broadly to enhance scientific and technological understanding? • What may be the benefits of the proposed activity to society? NSF Broader Impacts Criterion 2 • How well does the activity ADVANCE discovery and understanding while promoting teaching, training and learning? NSF Broader Impacts Criterion 2 • How well does the proposed activity BROADEN the participation of underrepresented groups (e.g., gender, ethnicity, disability, geographic, etc.)? NSF Broader Impacts Criterion 2 • To what extent will it ENHANCE the infrastructure for research and education, such as facilities, instrumentation, networks and partnerships? NSF Broader Impacts Criterion 2 • Will the results be disseminated broadly to ENHANCE scientific and technological understanding? NSF Broader Impacts Criterion 2 • What may be the benefits of the proposed activity to society? (e.g., BROAD & USEFUL application in service to society) NSF Broader Impacts Criterion 2 • How well does the activity ADVANCE discovery and understanding while promoting teaching, training and learning? • How well does the proposed activity BROADEN the participation of underrepresented groups (e.g., gender, ethnicity, disability, geographic, etc.)? • To what extent will it ENHANCE the infrastructure for research and education, such as facilities, instrumentation, networks and partnerships? • Will the results be disseminated broadly to ENHANCE scientific and technological understanding? • What may be the benefits of the proposed activity to society? (e.g., BROAD & USEFUL application in service to society) BI Example Josh’s Antarctic Glider Project BI Plan • Online expedition journal (photographer & writer) • Museum program • Teacher professional development & support materials Josh’s Antarctic Glider Project BI Evaluation Plan • • • • The evaluation will focus on effectiveness of the instructional materials, including a teacher’s guide, implementation strategies, handouts, student worksheets and any other materials needed to implement the unit and meet the intended learning outcomes. The evaluator will use surveys and interviews, starting with the summer workshop and continuing through the schoolyear, to collect data and feedback on the effectiveness of the teacher professional development training and materials. Accompanying the web-based photo essays will be an online survey (similar to that used in the WHOI Live from the Poles project) to gather viewers' feedback on the expedition, our reporting of it and viewers’ understanding of polar science issues. At the Liberty Science Center presentation we will use surveys to measure viewers’ satisfaction and learning. Sum Up Summative Eval Choose BI Front-end Eval Plan BI Project Do BI Project Formative Eval (incl. Evaluation) Design/Develop BI Project Formative Eval Summative Evaluation At the End: How’d we do? Summative Evaluation Q&A • Let’s Talk.... Thank You! To COSEE NOW To All of You