EVALUATING NSF EPSCOR INITIATIVES: TRACKING AND ASSESSING THE DEVELOPMENT OF EPSCOR-SUPPORTED RESEARCH CAPACITY Julia Melkers, Associate Professor School of Public Policy, Georgia Institute of.
Download ReportTranscript EVALUATING NSF EPSCOR INITIATIVES: TRACKING AND ASSESSING THE DEVELOPMENT OF EPSCOR-SUPPORTED RESEARCH CAPACITY Julia Melkers, Associate Professor School of Public Policy, Georgia Institute of.
EVALUATING NSF EPSCOR INITIATIVES: TRACKING AND ASSESSING THE DEVELOPMENT OF EPSCOR-SUPPORTED RESEARCH CAPACITY Julia Melkers, Associate Professor School of Public Policy, Georgia Institute of Technology Eric Welch, Associate Professor Department of Public Administration, University of Illinois at Chicago Prepared for presentation at the National Research Council Washington, DC December 17, 2012 Evaluation frames are based on the understanding that: Building research capacity is a complex and continuous process. Tracking magnitude of changes over time is critical to capturing effects (interim effects). Given its mission and the structure of the awards, EPSCoR has the potential to affect research capacity in range of ways. Capacity is reflected in (and affected by): State politics and science objectives Institutions Infrastructure Human capital Collaborative relationships Resources Among others.... Year 1 Year 2 Year 4 Year 5 Institutionalization Year 1 Increased production & other impacts Year 2 Development of early collaborative outputs Year 4 Year 5 Movement toward sustainability Production Collaboration Interaction Institutionalization Year 1 Increased production Year 2 Development of early collaborative outputs Year 4 Year 5 Movement toward sustainability Dimensions of Research Capacity Capacity as infrastructure Faculty developme nt Institutions Scientists Postdocs Graduate Students Education Workforce /Students Capacity as integration Research Capacity as human capital Lab equipment Cyberinfrastructure Institutional processes and policy External ties/ community Integrating different expertise and perspectives on the research problem (disciplinary, translational, etc) Capacity as productivity Grant activity that extends and builds upon award work Using Theories of Change in Evaluation: Example Project Logic Model Sample Guiding Evaluation Questions: Collaboration Collaboration: What are the barriers and facilitators to collaboration in [state] EPSCoR? within and between sites/groups? interdisciplinary and cross institutional? early career engagement? mid career role? Native and rural community partners? private and non profit sector integration? international partners? Barriers to success critical success factors? Opportunities to enhance success Sample Guiding Evaluation Questions: Impacts Research Impacts: What are the knowledge impacts of [EPSCoR] research and activities? Stakeholder Impacts: How has [EPSCoR] developed interactions /tangible relationships with external partners/stakeholders over the course of the project? Faculty and Staff Impacts: What research and management capacity is increasing as a result of [EPSCoR] work? Student Impacts: What benefits and capacities do students accrue from [EPSCoR] ? Institutional Impacts: How has the research capacity of [state] institutions changed via [EPSCoR] ? Data Sources External ties/ community Institutions Interviews/Focus Groups Surveys (Quant & Qual) Grant proposals Publications (conf papers?) Other data (data access?) Education Research Faculty developme nt Workforce /Students Example: Some Observable Outcomes Science Knowledge advancement and integration Research productivity Research impacts Increased know-how Faculty/Researchers Production Advancement Enhanced or new collaborative ties Student engagement Leadership Visibility “Next generation” Student learning and other impacts STEM interests and advancement Institutional capacity New programs, processes Systems Support staff Culture Measuring Research Activities & Outcomes Collaborative Development • • • • • Knowledge networks Co-authorship ties and affiliation Co-PI/grant team ties Integration scores E-I Index Knowledge Impacts • • • • • • Publication activity Journal subject categories Journal Impact rates Citation rates Citation rapidity Citation location Collaborative Development Intermediate outcomes that precede academic production Emerging collaborative relationships Integration of students in research Knowledge transfer and Impacts Research-relevant behavioral changes Nature of linkages and Measurable interactions through network analysis Operationalized through Network Analysis Pre-publication interaction Faculty-faculty & faculty-student collaboration Changed conceptual/ theoretical/ methodological approaches Changed research outlets and communities Mechanisms/modes for boundaryspanning linkages and outcomes Example: Student Interaction All Collaborative Ties – Graduate Students Only Statistics – All Ties Number of nodes Number of ties Average degree centrality E-I Discipline 48 77 .42 -.299 Legend Square = Core institution Circle = Other institutions Green = environmental science Blue = computer science/engineering Collaboration Ties Worked with on a grant proposal Presented research to external stakeholders or external partners Presented research to an academic audience Presented research to a nonacademic audience Co-authored a working paper Co-authored an academic paper that has been published or is under review Orange = social science Black lines = within discipline Red lines = between disciplines Node size = larger nodes reflect more semesters supported by EPSCoR # ties 14 20 69 29 20 5 Knowledge Impacts 25,000 Publications 20,000 15,000 10,000 5,000 - Citing articles 800 700 600 500 400 300 200 100 - Publications Citing articles Alaska-Based Orgs (Federal, State, Private, NGO) International Other States Other EPSCoR States Top Ten Subject Categories (2007-12 EPSCoR Publications # Subject Category Rec ords 1 37 Environmental Sciences & Ecology 2 27 Ecology 3 18 Geology 4 17 Environmental Sciences 5 12 Environmental Studies 6 12 Zoology 7 11 Physical Geography 8 10 Geosciences, Multidisciplinary 9 8 Evolutionary Biology 10 7 Biochemistry & Molecular Biology Knowledge Impacts: Disciplinary Placement & Knowledge Sources Agri Sci Env Sci & Tech Ecol Sci Infectious Diseases Geosciences Clinical Med Chemistry Matls Sci Engineering Biomed Sci Cognitive Sci. Health & Social Issues Psychology Physics Business & MGT Computer Sci. Social Studies Econ. Polit. & Geography Global Map of Science, 2007 221 SCI-SSCI Subject Categories Rafols, Porter and Leydesdorff (2009) Reflections on EPSCoR Outcomes Research STEM workforce Early career affiliates – early grants, network reach, production, time to tenure, mobility and competitiveness, mentoring and advice relationships Students – research networks, apprenticeship structures, mentoring relationships, production, next career step Research – related support staff, sustained research relationships, new grant awards Institutions Integration across disciplines – new questions, approaches Added value of collaborative engagement -- generation of new ideas and new collaborative relationships Knowledge impacts -- National and global visibility and impacts EPSCoR offices – communication, administrative structures, management Smaller institutions – research activity, developed capacity and integration, statewide representation in decision making (eg. state S&T plan) Curricular issues – new courses, content Social Impacts engaged stakeholder organizations – economic development, social and economic objectives, support for education and research by stakeholders Issues Relevant to Evaluation of EPSCoR EPSCoR projects and settings vary. Disciplinary variance Partner relationships Under-represented populations Changes over time (historically) Multiple and competing programmatic goals Lack of clarity/changing evaluation needs from EPSCoR Data consistency, across states and over time. Increased attention to strategic plans Unpacking role/impacts of senior leads vs other faculty/researchers Institutional variation in data availability (esp. small schools). Institutional effects not always evident Futures are unclear -- difficulty/lack of support to track students and faculty post-award. Issue of relationships across awards and cumulative effects on a state are not captured.