User studies: contributing to developing regional decision support for climate services

Andrea J. Ray, Roger S. Pulwarty and Robert S. Webb NOAA-CIRES Climate Diagnostics Center Climate Prediction Assessments Workshop Tallahassee, FL 9-11 March 2004

“The timely production and delivery of useful climate data, information, and knowledge to decision makers.” National Research Council, 2001

• NRC recommends that a regional focus is needed to address societal needs, developed through regional decision support systems.

• Pillars of climate services include: – – – – Regional decision support networks, Research, Observing systems, and Mechanisms to enable user-centric design and improvement.

To be effective, regional decision support systems must not only provide data and forecasts, but also determine: – what problems and questions are important and critical,

and why

, to be determined by consensus between users and providers of products; – what information is needed to solve problems, on short and long-term time scales; – what transaction costs are involved in developing and maintaining long term partnerships and interactions.

• • Many “regional decision support,” networks exist, – NOAA weather forecast offices,data centers, river forecast centers, – Regional climate centers, – – – State climatologist offices, Agricultural extension services Some private companies These networks provide data and forecasts, but in order to meet the NRC user-centric goals of climate services, these organizations would also: – adapt products to meet specific user needs, – – – support major user groups, encourage dialogue with users, and develop a regional focus to address societal needs.

• These networks generally are not set up to do effective user studies

Effective user-centric design and implementation

• • To achieve user-centric design and implementation these interactions must:

–

identify entry points for information within existing decisionmaking and the space for communicating this information;

–

identify research needed, including strategic, applied, and adaptive;

–

create nodes for users to engage with researchers and providers, to help determine what products are to be developed and tested;

–

Characterize needed products: e.g., scale, units, formats, how accessible, when issued.

–

Inform the development of alternative decisionmaking strategies as needed Mechanisms for studying users need to be built in to regional networks.

Effective user-centric design and implementation

• Sustained and systematic communication is required on three fronts: – – – between users and providers (NOAA and regional); between researchers and providers, and between researchers and users. • Why user studies?

– – More than consultancy More than a simple conversation or communication (not anecdotal) – Understand the complexity of user context and differences in perspectives between users and scientists

Designing effective user studies

• • • User-study approach Based on case studies Based on

– – –

need to get past differences in perspective; the innovation-decision process; definitions of “use” • Dimensions to characterize about a user-case

Factor

Identifying a critical issue Time frame Spatial resolution

Scientist’s perspective

Based on a broad understanding of the nature of water management Variable

Differences in perspective

Water Manager’s Perspective

Based on experience of a particular system Immediate (operations) Long-term (infrastructure) Defined by data availability or funding Defined by institutional boundaries or authorities Goals Basis for Decisions Expectation Product Characteristics Frame Nature of Use Prediction Explanation Understanding of natural system Generalizing multiple facts and observations Use of scientific procedures and methods Availability of research funding Disciplinary perspective Understanding Prediction Ongoing improvement (project is never actually complete) Statistical significance of results Innovations in methods/theory Complex Scientifically defensible Physical (atmospheric, hydrologic, etc.) conditions as drivers Dependent on scientific discipline Conceptual Optimization of multiple conditions and minimization of risk Tradition; Procedure Professional judgment; Training Economics; Politics Job risks Accuracy of information Appropriate methodology Save money and time; Protect the public; Protect their jobs, agendas or institutions As simple as possible without losing accuracy Importance of context Safety and well being Profit Consistency with institutional culture, policy, etc.

Applied

Innovation-Decision process: conceptual steps in deciding whether to adopt an innovation

• Knowledge phase : learns about the existence of an innovation, perhaps thru an outreach function • Persuasion phase: forms a favorable/unfavorable opinion • Decision phase : engages in activities leading to adopting or rejecting • Implementation : puts the innovation into use • Confirmation phase : decision to adopt is reinforced, or could be reversed if experience is adverse (Rogers,1995)  Path towards adoption and use: studies and mechanisms for developing user-centric products should take advantage of this

Definitions of “use”

• • Consult : the product is consulted, e.g., looked up on a web page or received as a briefing or from other source (type1); Consider : after consulting the product, it is considered in management deliberations as a factor potentially influencing decisions, but not necessarily in operational models (type 2); • • - forecasts may be used in this way when they are not in appropriate forms for use in operational models Incorporate : some form of the forecast is incorporated into an operational model that is utilized in operational decisions (type 3).

Dialogue about risks : communication of risk, i.e. the forecast is used to communicate with other managers and stakeholders about the risk of certain conditions and about the need to take actions, or to justify actions (type 4).

User Study Approach

• One mechanism for assessing user needs • Dimensions to characterize about the user-context: – Identify critical problems which are sensitive to climate – Identify decisionmakers

and

their key stakeholders – Assess how climate variability interacts with their critical problems • Decision calendar may help organize recurring decisions and show when products are needed – Identify user groups who may be willing partners in testing and prototyping this new technology • Potential users in a rapidly evolving phase of their critical problem may be more open to interacting as partners • Social changes, legal or policy change, or a climate event such as drought or flood

Case study: Multi-purpose Reservoir Management in the Upper Colorado River Basin

User group is the reservoir managers in the Colorado headwaters and Gunnison basins who operate reservoirs originally built to provide reliable irrigation supply, hydropower, and recreation Redlands Diversion dam and fish ladder, Gunnison river near Grand Junction, CO

Dimensions for Aspinall Unit Case

• Critical problem sensitive to climate – Evolving reservoir management in which demands now are closely balanced with supply but new uses are being legally proscribed – More intensely managed water system is more sensitive to shortage because there is less buffer in the system • Decisionmakers and their key stakeholders – USBR has authority to manage, but USFWS, NPS, have interacting legal authorities, other stakeho • How climate variability interacts w/ the critical problem – Decision calendar for annual operating plan helped organize recurring decisions and ID potential climate • User groups who may be willing partners/adopters – Deadlines to write ESA recovery plans and formal filing for a federal reserve water right for a national park are requiring USBR to accommodate these in their operations – 2002 drought as a climate event (not forecasted) also resulted in finding flexibility and new ways of operating

Potential Uses in the Aspinall case

• • • • Identified a number of potential uses of seasonal climate forecasts in their operational models, as well as shorter term forecasts However, seasonal forecasts are currently not available in a form that can be incorporated into their operational models. Current use is to consult these products and consider them in management deliberations Aspinall decisionmakers use climate information (e.g. during drought) in a dialogue with their stakeholders about the risk of certain conditions and about the need to take actions, or to justify actions. Projects with Martyn Clark (WWA) and Dave Brandon (CBRFC) are working to translate both MRF forecasts and CPC seasonal forecasts

Conclusions I

• Regional decision support networks exist, but are not designed to carry on the systematic study of and interaction with users to ensure the development of user-centric products and services • • Mechanisms need to be built into the regional decision support networks These mechanisms could ensure user-centric design and implementation by:

–

identifying entry points for information within existing decisionmaking and the space for communicating this information, e.g., through basin water management groups, water availability task forces, emergency management meetings

–

identifying research needed, including strategic, applied, and adaptive;

–

enabling RDS networks to function as nodes for users, researchers and providers to engage in dialogue and to help determine what products are to be developed and tested;

–

effectively characterizing needed products and synthesize user needs from among many interactions (beyond consultancy)

Conclusions II

• • User studies are a means to create and sustain the communication and learning required for user-centric design and development – and also a cost.

–

create a dialogue about climate risks and not just provide information;

–

create a dialogue about the types of climate products desired by a broad spectrum of users by integrating knowledge about what climate products are needed.

NOAA’s mechanisms for developing user-centric products and services need to:

–

Recognize that differences in perspective exist and limit scientists ability to create user-centric products – unless we deliberately work to see other perspectives

–

“Use” of products is broader than just in operational models: recognize that there is a range of uses and create products for all types. Evaluation of “use” should consider all types.

–

Adoption is a process: develop mechanisms to draw potential users along this path and evaluate whether activities support evolution of use • Training needed across the RDSS networks in how to conduct user assessments

EXTRAS

Water Year Planning

Reservoir Management Decision Calendar

Next Water Year Planning Provide for late Summer/early Fall irrigation while maintaining target flows Next water year runoff unknown, reserve water until February snowpack observations Winter season precipitation forecast for Fall release decisions Winter releases based on January/February snowpack observations Legend: Planning Process Operational issues Potential use of forecasts Winter/Spring forecast for Winter release decisions Peak Flow Augmentation - fill curve Summer season forecast for Peak Augmentation planning Week 2 forecasts for Peak Augmentation

Figure 4-28

Peak Flow Augmentation releases Plan releases for Summer irrigation & hydropower Week 2 forecasts for Summer irrigation & hydropower release decisions Provide for Summer irrigation & hydropower needs while maintaining target flows Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun July Aug Sep Oct

• • • • • • • Identifying a critical issue Time frame Goals Basis for decisions Expectiation Product characteristics Frame

Differences in perspective

Characteristics of effective user studies

• Dimensions to characterize about a user-case

Understanding Barriers New Services

Fail to understand Desired information not available Other factors more important How to they understand climate? What do they use now?

Can we produce forecasts in a manner related to how they think about climate?

Are the problems critical to the user influenced by climate? If so, are there products available to address these?

Constraints prevent flexible response Are there circumstances that minimize constraints?

User studies seek answers to the right hand questions