Transcript Innovation, Standards, and NOAA Data Management Ted Habermann NOAA National Data Centers

Innovation, Standards, and
NOAA Data Management
Ted Habermann
NOAA National Data Centers
“There are special management challenges, and I think
that that's an area that we in agencies such as NOAA,
need to spend an extra amount of time on. We have very
talented workers and very talented employees, many of
whom have advanced degrees, and they have been
successful because of certain behaviors in their field.
As you progress through the system in any organization,
you need to develop other skills;…”
Vice Adm. Lautenbacher
Mature Organizations
The Technology S-Curve
We all know that new
technologies emerge slowly,
grow quickly (if they catch on)
and then fade away. This
common knowledge has been
described as the technology
S-curve.
Why does it exist?
TIME
The Adoption Curve
Geoffrey Moore has attributed the
S-curve to the technology adoption
life cycle where techies and
visionaries are early adopters,
pragmatists make up the bulk of
users, and luddites fill out the tail
of the distribution.
Luddites
Pragmatists
Visionaries
TIME
The Chasm
Moore has also described the “chasm”
in the adoption life cycle. He proposes
that many new technologies do not
make it across the chasm between
visionaries and pragmatists. They fall
into the chasm. The technology S-curve
with the chasm might look like:
TIME
Technology Cycle
Technological
Disruption
Selection
Era of Ferment
Dominant
Design
TIME
Disruption #2
(destroys existing
competence)
Types of Innovation - 1
Sustaining / Incremental Innovation: generally small
innovations in products and processes aimed at existing
customers.
Disruptive / Discontinuous Innovation: significant
innovations generally aimed at unknown or non-existent
customers.
Unidata Objectives (1998):
Sustaining Innovation
“These objectives either respond to users' current needs or advance
Unidata toward meeting future needs effectively. Most of the "responsive"
items are continuations of current Unidata objectives, and their importance
is well established. But only by looking beyond present needs to anticipate
future ones, and by pursuing the most promising technical advances, can
Unidata remain effective. This is true even though some of these
advances involve uncertainties, and the demand for them may not be
apparent as yet”. Unidata, 2003 Proposal.
Disruptive Innovation
Clayton Christensen, The Innovator’s Dilemma
Unidata (netCDF) Evolution
In the Unidata case we
are now seeing the
disruptive switch to Java
play out. The capabilities
of the Java version of the
netCDF libraries have now
surpassed the original C
version.
Java
Customer Metric
Disruptive Innovation:
Always includes a decrease
in metrics for current
customers so it is difficult
for mature organizations.
Sustaining
Innovation
C
Disruptive
Innovation
TIME
Types of Innovation - 2
Component
Innovation: Making
existing components
better.
Architectural
Innovation: putting
existing components
together in new
ways.
Architecture = Organization
Structure in mature
organizations tends
to evolve to match
product
architectures.
Architectural
Innovation,
therefore, many
times includes
elements of
organizational
change.
Innovation & Technology Cycle
Disruptive Innovation
Product Innovation
Design Competition
Community-driven
technology change
Component,
Architectural,
Sustaining and
Process
Innovation
What do we make?
How do we make
it (better)?
TIME
Other Differences
Research
Prototypes
Custom developments
Network building
Uncertainty
Operational Systems
Product Families
Predictability
Partnerships
Standards
Network Effects
Value = f(N2)
(non-compliance
cost increases
with time)
TIME
How Standards Change The Game
• Expanded Network Externalities (Network effect turns on)
• Reduced Uncertainty and Risk in Technology Decisions
• Reduced Consumer Lock-In to Particular Components
• Competition in the Market vs. Competition for the Market
• Competition on Value vs. Features
• Competition to Offer Proprietary Extensions
• Component vs. Systems Competition
Standards shift the locus of competition from systems development to
component development. Specialists tend to thrive in the mix-and-match
environment created by interface standards. Generalists and system
(stovepipe) developers tend to thrive in the absence of standards.
In the absence of standards:
1) there is no architectural innovation (no mix-and-match) and
2) the organization can not benefit from component innovation.
Once a standard has been agreed on (selection), the organization
benefits from component innovation and architectural innovation.
Andy Grove: Communication
Overcomes Computing
The framework is changing now. The
Internet is redefining software. The
Internet is redefining the role of computing
and communication and their interaction
with each other. I still don’t understand the
framework. I don’t think any of us really do.
But some aspects of it are pretty clear. It’s
proven not to be computing based but
communications based. In it computing is
going to be subordinated to the
communications task.
“Decisions
Don’t Wait”,
Harvard
Management
Update.
Kevin Kelley: The Web and
Sharing
The revolution launched by Netscape’s IPO
was only marginally about hypertext and
human knowledge. At its heart was a new
kind of participation that has since
developed into an emerging culture based
on sharing.
“We Are The
Web”, Wired,
August 2005
Technology Confluence
Geographic Information Systems
+ Relational Databases
+ World Wide Web
DESKTOP
MULTI-USER
GEOSPATIAL DATABASE
Computing
NETWORK
(ENTERPRISE GIS)
Communication
Infrastructural Technologies
IT is, first of all, a transport mechanism – it carries digital information just
as railroads carry goods and power grids carry electricity. And like any
infrastructural technology, it is far more valuable when shared than when
used in isolation. The history of IT in business has been a history of
increased interconnectivity and interoperability, from mainframe timesharing to minicomputer-based local area networks to broader Ethernet
networks and on to the Internet. Each stage in that progression has
involved greater standardization of the technology and, at least recently,
greater homogenization of its functionality. For most business applications
today, the benefits of customization would be overwhelmed by the costs of
isolation. Nicholas G. Carr, IT Doesn’t Matter, Harvard Business Review, May, 2003
Corollary to Metcalf’s Law: the
cost of non-compliance goes up
as the square of the number of
members of the network.
Why No Standards?
The longer the market takes to determine a standard,
the more expensive it will be for firms operating within
that market. The more expensive this competition
becomes, the greater the tendency for firms to
cooperate at the beginning.
The difficulty with this reasoning is that it is difficult
for individual firms to determine how expensive or how
long it will take the market to determine the dominant
standard. Nor are companies willing to cede control of
such an important aspect of their market early in a
competition. Booz Allen Hamilton, 2005.
The science community generally does not value sharing.
Organizational Approaches / Skills
Leadership
Discovery-based
Planning
Management
Detailed Plans w/
Accountability
Resolving Critical
Unknowns
Long-term Plans
& Requirements
PPBES
TIME
Multiple Technologies
NOAA relies on many technologies
which are changing rapidly. To do well
in long-term, NOAA must:
1) recognize phases of the
technology cycle,
2) develop and use mechanisms for
standards adoption and selection
3) realize that different
phases require different
organizational structures
/strategies,
4) support multiple
structures / strategies
simultaneously.
TIME
Innovation, Standards & NOAA
There is a considerable innovation literature that can help NOAA learn the
new skills required to innovate strategically and effectively.
Technology is evolving from a computing tool to a communication tool. It is
becoming an infrastructure technology.
Standards are critical to building value of infrastructure technologies.
Standards are critical to organizationally effective component and
architectural innovation.
NOAA must develop and use processes for selecting and applying standards.
The requirements and approaches to planning are very different in the
different phases of the technology cycle.
Understanding and explicitly recognizing the differences in phases of the
technology cycle and the differences in balance between management and
leadership skills might help NOAA.
Examples from NGDC
1. NOAA Maps, The NOAA Observing System Database
and NOSA Website
Internet mapping in NOAA is a great example of an “Era of
Ferment”. Many different tools produce maps that look and
behave differently. We are collecting capabilities for over 400
NOAA Internet Maps and comparing those capabilities to a single
map that includes information for ~100 NOAA Observing
Systems. How would a selection event change NOAA’s approach to
internet mapping?
2. The Data Processing Pipeline
An example of components/architecture for data processing and
ingest in an open source Java project initiated at NGDC.
NOAA Maps
Over 400 maps
have been
captured from
NOAA Web
pages and
described in the
NOAA Maps
Project.
http://www.nosa.noaa.gov/noaa_maps/
Maps at NGDC
NOSA Website @ NGDC
NOSA Website
WIST
LAS
WMS
Map
FYP
Spatial
SQL
ArcIMS
Stations,
QA
SQL
WFS
Desktop GIS
Spreadsheet,
E-mail, txt,
etc.
Pipeline
Geospatial
Database
Metadata
The Data Processing Pipeline
A pipeline executes a sequence of plug-able data processing
tasks. The NGDC data processing pipeline provides a set of
pipeline utilities designed around work queues that run in
parallel to sequentially process data objects. The pipeline is
an open source project hosted in the Jakarta Commons
Sandbox
(http://jakarta.apache.org/commons/sandbox/index.html).
Processing steps are specified as a series of stages in an
XML configuration file.
Files
OGC
Simple Features
in Memory
Spatial
Database
DMSP Orbit Processing
Stage 1. Find Matching Files
Stage 2. Avoid Duplicate Processing
Stage 3. Read Data / Create Spatial Objects
Stage 4. Low-Res Thinning
Stage 5. High-Res Thinning
Stage 6. Write Spatial Data to Database
Building on Experience
Louis Liebenberg, The Art of Tracking: The Origin of Science (Wired, June 2003)
Leadership Model: Positive Deviance
Positive deviance says that if you want to create change, you must
scale it down to the lowest level of granularity and look for people
within the social system who are already manifesting the desired
future state. Take only the arrows that are already pointing toward
the way you want to go, and ignore the others. Identify and
differentiate those people who are headed in the right direction.
Give them visibility and resources. Bring them together. Aggregate
them. Barbara Waugh
[email protected]
References
Booz Allen Hamilton, Geospatial Interoperability Return on Investment Study, 2005,
http://gio.gsfc.nasa.gov/docs/ROI%20Study.pdf.
Christensen, C., The Innovator’s Dilemma, Harvard Business School Press, 1997, 225p.
Clark and Wheelwright, Revolutionizing Product Development, The Free Press, New York, 1992, 364p.
Govindarajan, V. and C. Trimble, Building Breakthrough Businesses Within Established Organizations, Harvard
Business Review, May 2005, p. 58-68.
Lautenbacher, C., Business of Government Radio Interview,
http://www.businessofgovernment.org/main/interviews/bios/conrad_lautenbacher_frt.asp, 2005.
Moore, G., Crossing the Chasm, Marketing and Selling High-Tech Products to Mainstream Customers, Harper
Business, 1991, 211p.
O’Reilly, C.A. and Tushman, M.L., The Ambidextrous Organizations, Harvard Business Review, April 2004.
The Positive Deviance Initiative, http://positivedeviance.org/
Pascale, R.T. and J. Sternin, Your Company’s Secret Change Agents, Harvard Business Review, May 2005, p. 7281.
Tushman, M.L., Anderson, P., and O’Reilly, C.A., Technology Cycles, Innovation Streams, and Ambidextrous
Organizations: Organizational Renewal Through Innovation Streams and Strategic Change, in Managing
Strategic Innovation and Change, Tushman and Anderson, eds., Oxford University Press, New York, 1997, 657p.