Transcript SciDAC Status Report for ASCAC

U.S. Department of Energy
U.S. Department of Energy’s
Office of Science
Office of Science
Advanced Scientific Computing Research Program
Office of Advanced Scientific
Computing Research
- Status Petascale Simulations
SciDAC
Walter M. Polansky
[email protected]
U.S. Department of Energy
American
Competitiveness Initiative
Office of Science
Advanced Scientific Computing Research Program
President Bush in the President’s State of the Union Address on January 31, 2006
“I propose to double the federal commitment to the most critical basic research program
in the physical sciences over the next ten years. This funding will support the work of
America’s most creative minds as they explore promising areas such as nanotechnology,
supercomputing, and alternative energy sources.”
Secretary Bodman commenting on President Bush’s statement
“Developing revolutionary, science-driven technology is at the heart of the Department
of Energy’s mission. To ensure that America remains at the forefront in an increasingly
competitive world, our Department is pursuing transformational new technologies in the
cutting-edge scientific fields of the 21st century—areas like nanotechnology, material
science, biotechnology, and high-speed computing.”
CASC- September 6, 2006
2
U.S. Department of Energy
Office of Science: Mission
Office of Science
Advanced Scientific Computing Research Program
Secretary Samuel Bodman, House Committee on Science Hearing,
February 15, 2006
“The Office of Science plays a critical role in ensuring America’s scientific
leadership and economic dynamism…”
The mission of the Science program is to deliver the discoveries and scientific
tools that transform our understanding of energy and matter and advance the
national, economic, and energy security of the United States.
In support of it’s mission, the Science program has responsibilities in three main
areas: selection and management of research; operation of world-class, state-ofthe-art scientific facilities; and design and construction of new facilities.
“Investment in these facilities is much more than bricks and mortar: it is an
investment in discovery, and in the future of our nation.”
CASC- September 6, 2006
3
U.S. Department of Energy
Office of Science
Facilities for the Future of
Science: a 20-Year Outlook
U.S. Department of Energy
Office of Science
-- FY2007 Budget Request- $4.1B -Office of Science
Advanced Scientific Computing Research Program
FY2006 Appropriation- $3.6B
U.S. Department of Energy
ASCR Vision
Office of Science
Advanced Scientific Computing Research Program
First in Computational Science
“Best in class in advancing science and technology through
modeling and simulation”
Facilities
Enabling Technologies
Computational Partnerships
CASC- September 6, 2006
6
U.S. Department of Energy
ASCR Research Strategy
Office of Science
Advanced Scientific Computing Research Program
•
Applied Mathematics and Computer Science deliver the
operating systems, programming models, software tools,
and mathematical algorithms and libraries needed for
scientists to make effective use of petascale computing.
• Scientific Discovery through Advanced Computing
(SciDAC) program, will strengthen activities at the software
centers initiated in FY 2006 for petascale computing.
• SciDAC will foster additional research investments in
Applied Mathematics and Computer Science to accelerate
efforts in modeling and simulation on the petascale
computing facilities.
CASC- September 6, 2006
7
U.S. Department of Energy
ASCR Facilities
Office of Science
Advanced Scientific Computing Research Program
• Oak Ridge Leadership Computing Facility
• Argonne Leadership Computing Facility
• NERSC at Lawrence Berkeley National Lab
• ESnet at Lawrence Berkeley National Lab
CASC- September 6, 2006
8
U.S. Department of Energy
FY 2006 ASCR Budget
Distribution
Office of Science
Advanced Scientific Computing Research Program
SBIR/STTR
$6.3
Applied Mathematics
$29.4
Computer Science
$24.3
LCF
$53.7
SciDAC
$38.1
NERSC
$37.5
Research and
Evaluation Prototypes ESnet
$19.0
$13.0
CASC- September 6, 2006
Distributed Network
Environment Research
$13.6
9
U.S. Department of Energy
FY 2007 ACSR Budget
Status
Office of Science
Advanced Scientific Computing Research Program
350
$319M
$319M
$319M
300
$235M
Dollars in Millions
250
200
150
100
50
0
FY06
CASC- September 6, 2006
FY07 Request FY07 House
FY07 Senate
10
U.S. Department of Energy
FY 2007 - House
Office of Science
Advanced Scientific Computing Research Program
FY 2007 House Energy and Water
Development Appropriations Bill passed both
the Appropriations Committee and the House
Report language:
“ The Committee commends the Office of Science and the
Office of Advanced Scientific Computing Research for
their efforts to provide cutting-edge capabilities to meet
current scientific computational needs, and at the same
time to extend the boundaries of that cutting edge into
the next generation of high-performance scientific
computers and supporting software.”
CASC- September 6, 2006
11
U.S. Department of Energy
FY 2007 - Senate
Office of Science
Advanced Scientific Computing Research Program
FY 2007 Senate Energy and Water Development
Appropriations Bill passed the Appropriations Committee
Report language
“ …The
Committee is concerned with the relationship between
the Office of Science and the NNSA. As an example, the
ASCR strategic plan discusses the need to work with other
Federal agencies including several defense agencies, but only
discusses in general terms three areas of research where
NNSA and the Office of Science cooperated. In the area of
basic research, the strategic plan states that it is an area
that is `not important enough to justify ASCI investment at
this time.' The Committee is also aware that the Office of
Science has budgeted $13,000,000 for the DARPA to support
a petaflop computer deployment by 2010. The Committee
believes this funding would be better spent within the
Department to support a petaflop initiative. The Department
is directed to divide the funds equally between the Office of
Science and the NNSA Advanced Simulation and Computing
activities to support development of component architecture
for high-performance software and storage”
CASC- September 6, 2006
12
U.S. Department of Energy
Facilities Update
Office of Science
Advanced Scientific Computing Research Program
• NERSC – delivery of NERSC-5 in FY 2007
• LCF at ORNL –
– Upgrade Cray XT3 upgrade
• Now: 25Teraflop  50Teraflop
• By the end of 2007: 50Teraflop  250Teraflop
– Acquire 1 Petascale Cray Baker system by end of 2008
• LCF at Argonne –
– Acquire 100 Teraflop IBM Blue Gene/P in FY2007
– Upgrade to 250-500 Teraflop IBM Blue Gene/P in 2008
– On path to a Petascale IBM Blue Gene /Q by end of the
decade
CASC- September 6, 2006
13
U.S. Department of Energy
Office of Science
Innovative and Novel
Computational Impact on Theory
and Experiment- INCITE
Advanced Scientific Computing Research Program
• Initiated in 2004
• Provides Office of Science computing resources to a small
number of computationally intensive research projects of
large scale, that can make high-impact scientific advances
through the use of a large allocation of computer time and
data storage
• Open to national and international researchers, including
industry
• No requirement of DOE Office of Science funding
• Peer-reviewed
• 2004 Awards: 4.9 Million processor hours at NERSC awarded
to three projects
• 2005 Awards: 6.5 Million processor hours at NERSC awarded
to three projects
CASC- September 6, 2006
14
U.S. Department of Energy
INCITE 2006
Office of Science
Advanced Scientific Computing Research Program
Wired Magazine, August, 2006, pg. 42
• Expanded to include SC
high end computing
resources at PNNL, ORNL
and ANL in addition to LBNL
and multiple year requests.
• Received 43 proposals
requesting over 95 million
processor hours.
– 60% from Universities
– 40% had funding from other
federal research agencies
• 15 Awards for over 18.2
million processor hours
CASC- September 6, 2006
15
U.S. Department of Energy
INCITE 2007
Office of Science
Advanced Scientific Computing Research Program
• Expanded in 2007 to include 80% of resources
at Leadership Computing Facilities in addition
to 10% of NERSC and 5% of PNNL
• Call issued July 27, 2006
http://hpc.science.doe.gov
• Proposals due September 15, 2006
• Award announcements in mid-December
CASC- September 6, 2006
16
U.S. Department of Energy
Office of Science
Scientific Discovery
through Advanced Computing
Advanced Scientific Computing Research Program
www.SciDAC.gov
Structure
-- Scientific Application Areas - Breakthrough scientific
advances through computer modeling and simulation that
are impossible using theoretical and experimental studies or
improve experimental science.
-- Centers for Enabling Technology (CETs)- provide the essential
computing and communications infrastructure to support
SciDAC applications
-- Institutes- University-led centers of excellence; address specific
software issues; Engage a broader community of scientists in
modeling and simulations.
CASC- September 6, 2006
17
U.S. Department of Energy
Path to Petascale
Office of Science
Advanced Scientific Computing Research Program
Scientific
Discovery
Applications
SciDAC-2
- Accelerator
science and simulation
- Astrophysics
- Climate modeling and simulation
- Computational Biology
- Fusion science
- High-energy physics
- Petabyte high-energy/nuclear physics
- Materials science and chemistry
- Nuclear physics
- QCD
- Radiation transport
- Turbulence
- Groundwater reactive transport modeling and simulation
- Centers for Enabling Technology
Computing/
Networking
CASC- September 6, 2006
INCITE
NERSC Allocation
- Scientific Applications Partnerships
- Institutes (University-lead)
Leadership ComputingANL
100 TF
Leadership ComputingORNL
250 TF
Production ComputingNERSC
100-150
TF
ESnet
On path toward Dual rings 40Gbps/ 10 Gbps fault
tolerant
18
U.S. Department of Energy
SciDAC-2 Institutes…
Office of Science
Advanced Scientific Computing Research Program
… University-led centers of excellence that focus on major software issues
through a range of collaborative research interactions.
– Develop, test, maintain, and support optimal algorithms, programming environments,
systems software and tools, and applications software.
– Focus on a single general method or technique.
– Focal point for bringing together a critical mass of experts from multiple disciplines to focus
on key problems in a particular area of enabling technologies.
– Forge relationships between experts in software development, scientific application
domains, high performance computing, and industrial partners.
– Reach out to engage a broader community of scientists in the activities of scientific
discovery through advanced computation and collaboration.
– Incorporate training and outreach in high performance computing topics, including for
graduate students and postdocs.
CASC- September 6, 2006
19
U.S. Department of Energy
Centers for Enabling Technologies…
(CETs)
Office of Science
Advanced Scientific Computing Research Program
… provide the essential computing and communications infrastructure to
support SciDAC applications; multi-discipline approach with activities in:
•
– Algorithms, methods, and libraries.
– Program development environments and tools -- terascale and petascale program
development and tools provide maximum ease-of-use to scientific end users.
– Systems software that provides system stability and functionality needed by users for tera- to
peta- scale simulations.
– Visualization and data management systems.
CETs work directly with applications on:
– Development and application of computing systems software that allows scientific simulation
codes to take full advantage of the extraordinary capabilities of terascale and petascale
computers.
– Ensuring that the most critical computer science and applied mathematics issues are
addressed in a timely and comprehensive fashion.
– Addressing all aspects of the successful research software lifecycle including transition of a
research code into a robust production code and long term software evolution and
maintenance and end user support.
CASC- September 6, 2006
20
U.S. Department of Energy
SciDAC Accomplishments
Office of Science
Advanced Scientific Computing Research Program
• To better understand combustion
which provides 80% of the energy
used in the U.S., SciDAC teams
created first laboratory-scale flame
simulation in three dimensions
• Magnetic fusion scientists and
applied mathematicians simulated
techniques for re-fueling fusion
reactors
• Teams developed new methods for
simulating improvements in future
particle accelerators
• Partnerships improved effectiveness
of scientific applications codes
between 275% to over 10,000%
Hydroxyl radical in a turbulent jet flame
CASC- September 6, 2006
21
U.S. Department of Energy
Office of Science
The Power and Evolution
of Partnerships
Today and Tomorrow
Advanced Scientific Computing Research Program
BES
Methane Flames
Diesel Combustion
Ultrafast Science
BER
GCM Performance
Clouds
Simulation of Microbes
FES
Plasma Core
Full Simulation of
Burning Plasma
Increased Accuracy
HEP
ASCR
NP
NE
Weak Matrix
Elements
Performance of
Supernova codes
Accelerator Design
Neutron Transport
CASC- September 6, 2006
Industry
LCLS System Design
Advanced Fuel
Cycles
Groundwater Transport
EM
NNSA
Elements beyond Fe
Libraries &
Algorithms
Science Based Materials
Improvements in
Mesh Generation
Virtual Prototypes
22
U.S. Department of Energy
Road to Petascale Computing:
Combustion
Office of Science
Advanced Scientific Computing Research Program
Direct Numerical Simulation of Turbulent
Combustion, a collaborative effort between
Sandia National Laboratories, U. of
Wisconsin, U. of Michigan, U. of Maryland,
U.C. Davis, and Princeton U., has been
focusing on advancing science for
improved fuel efficiency of gas turbines
and technologies for transportation.
Petascale computing will enable
fundamental understanding of turbulencechemistry interactions, e.g. multi-stage
auto-ignition and flame stabilization in
compression ignition engine environments,
in regimes relevant to real devices.
A 50% increase in efficiency of
automobiles could save 21% of U.S.
oil consumption for transportation
CASC- September 6, 2006
This large turbulent CO/H2 jet flame was run
on the CrayX1E at ORNL LCF.
(Credit for the image is to Kwan-Liu Ma,
Hongfeng Yu, and Hiroshi Akiba of UC Davis.)
23
U.S. Department of Energy
Turbulence
Office of Science
Advanced Scientific Computing Research Program
Volume rendering of scalar dissipation rates in a temporal plane jet flame. High values are
shown in red, medium in yellow, and lower values in blue. (Credit: Jackie Chen/SNL)
CASC- September 6, 2006
24
U.S. Department of Energy
Road to Petascale Computing:
Fusion
Office of Science
Advanced Scientific Computing Research Program
Center for Simulation of Wave
Interaction with Magnetohydrodynamics, a collaboration of
PPPL, ORNL, MIT, Columbia U.,
Indiana U., Wisconsin U., etc., is a
project for very large plasma wave
simulation for fusion-based
research.
Using 4,096 processors of Cray
XT3 at LCF (ORNL), it is the
largest, most-detailed simulation
ever done of plasma control waves
in a tokamak, the reactor that will
eventually form the core of the
multinational ITER reactor.
CASC- September 6, 2006
25
U.S. Department of Energy
Road to Petascale Computing:
Astrophysics
Office of Science
Advanced Scientific Computing Research Program
A multi-institution project for largescale simulation of Neutrino-driven
Supernova moves forward to discover
a new instability that is important to
core collapse supernovae – the
supernova shock wave instability
(SASI) and a plausible mechanism for
generating the initial neutron star spin
required to give birth to pulsars.
CASC- September 6, 2006
26
U.S. Department of Energy
Road to Petascale Computing:
Modeling for Reactor Design
Office of Science
Advanced Scientific Computing Research Program
Researchers at Argonne National
Laboratory have recently developed a
Portable, Extensive, Toolkit for Scientific
(PETSc) numerical library used in dozen of
scientific applications world-wide.
PETSc encapsulates the complexity of
underlying parallel algorithms and presents
them in terms of numerical abstractions
familiar to scientists. It has been used to
extend fully compressible flow codes, most
notably the pressure-oriented implicit
continuous Eulerian (PCICE) scheme
developed at Idaho National Engineering
Laboratory.
CASC- September 6, 2006
27
U.S. Department of Energy
Office of Science
Road to Petascale Computing:
Modeling the Interaction b/w Fluids &
Embedded Solid Particles
Advanced Scientific Computing Research Program
Researchers at Lawrence Livermore
National Laboratory have recently
developed a suite of fast and scalable
adaptive algorithms for modeling shocks
and detonations interacting with movable
rigid bodies in a fluid. Accurate 2D
simulations tax existing computer
resources; extending to well-resolved 3D
simulations will require peta-scale
resources.
Shock hitting
several
movable rigid
bodies using a
Godunov
method with
adaptive mesh
refinement;
density and
refinement
grids are shown
The same fluid
solver can model
detonations using an
ignition and growth
model and JWL
equation of state.
This mock Schlieren
shows a detonation
interacting with solid
bodies.
This new algorithm combines efficient local
representations of the embedded bodies
with moving geometry and adaptive mesh
refinement.
CASC- September 6, 2006
(Credit for the images is to
Bill Henshaw (PI) and Kyle
Chand, LLNL)
28