Transcript NASA GSFC Landsat Data Continuity Mission (LDCM) Grid

NASA GSFC
Landsat Data Continuity Mission (LDCM)
Grid Prototype (LGP)
Beth Weinstein
NASA GSFC
May 8, 2006
LDCM Grid Prototype (LGP) Introduction
A Grid infrastructure allows scientists at resourcepoor sites access to remote resource-rich sites
• Enables greater scientific research
• Maximizes existing resources
• Limits the expense of building new facilities
The objective of the LDCM Grid Prototype (LGP) is
to assess the applicability and effectiveness of a
data grid to serve as the infrastructure for
research scientists to generate Landsat-like data
products
Sponsored by NASA LDCM, NASA/GSFC Code 580
Team: 586/585/SGT/QSS/Aerospace Corp/USGS EROS
LGP Milestones
Capability 1 (C1) (12/03 - 12/04)
• Demonstrated a basic grid infrastructure to enable a science
user to run their program on a specified resource in a virtual
organization
• Virtual organization (VO) included GSFC labs and USGS EROS
resources
• Basic Globus Toolkit 2.4 (e.g. GSI, GridFTP, GRAM)
Capability 2 (C2) (12/04 - 9/05)
• Demonstrated an expanded grid infrastructure to allow the
dynamic allocation of resources to enable a specific science
application
• VO included NASA GSFC labs, USGS EROS, University of
Maryland (UMD)
• Workflow enabled
NASA ROSES ACCESS A.26 (1/06 – 1/08)
• Land Cover Change Processing and Analysis System: LC-ComPS
Sponsored by NASA LDCM, NASA/GSFC Code 580
Team: 586/585/SGT/QSS/Aerospace Corp/USGS EROS
Capability 1 Science Scenario
LEDAPS
L7ESR
MODIS
MOD09GHK
Sponsored by NASA LDCM, NASA/GSFC Code 580
Team: 586/585/SGT/QSS/Aerospace Corp/USGS EROS
Capability 1 Summary
Prepare two heterogeneous data sets at different
remote locations for like “footprint” comparison from
a science user’s home site
• The MODIS Reprojection Tool (MRT) serves as our
“typical science application” developed at the science
users site (GSFC Building 32 in demo)
• mrtmosaic and resample (subset and reproject)
• Operates on MODIS and LEDAPS (Landsat surface
reflectance) scenes
• Data distributed at remote facilities
• NASA GSFC Building 23 (MODIS scenes)
• USGS EROS (LEDAPS scenes)
Solves a realistic scientific scenario using gridenabled resources
Sponsored by NASA LDCM, NASA/GSFC Code 580
Team: 586/585/SGT/QSS/Aerospace Corp/USGS EROS
Capability 2 Science Scenario
Landsat
Scene 1
Path/Row:
182/61
Date:
2/12/2002
Landsat
Scene 2
Path/Row:
182/61
Date:
6/4/2002
Sponsored by NASA LDCM, NASA/GSFC Code 580
Team: 586/585/SGT/QSS/Aerospace Corp/USGS EROS
2002 182/61 Composite
Capability 2 Summary
Create direct reflectance composite products using Landsat
data
Blender Task 1 scenario and modules were contributed by Jeff
Masek and Feng Gao
Modules
• lndcal - calibration
• lndcsm – cloud shadow mask
• lndsr – surface reflectance
• lndreg - registration
• lndcom - composite
Input data
• Up to 5 Landsat scenes: spatially coincident
• GSFC ancillary data:
• TOMS (ozone)
• Reanalysis (Water Vapor)
Output data: 1 LEDAPS/Blender composite scene
Sponsored by NASA LDCM, NASA/GSFC Code 580
Team: 586/585/SGT/QSS/Aerospace Corp/USGS EROS
Capability 2 Scenario
EROS Pool
2001 Landsat
Scene 1
2001 Landsat
Scene 2
lndcal
ancillary
inputs
lndcsm
lndsr
ancillary
inputs
2001 Landsat
Scene 3
2001 Landsat
Scene 4
lndcal
lndcal
lndcal
lndcsm
lndcsm
lndcsm
ancillary
inputs
lndsr
lndsr
lndsr
lndreg
lndreg
lndreg
lndcom
Sponsored by NASA LDCM, NASA/GSFC Code 580
Team: 586/585/SGT/QSS/Aerospace Corp/USGS EROS
ancillary
inputs
30m resolution 2001
composite
product (single pathrow)
Capability 2 Virtual Organization
Edclxs66 (2)
USGS EROS
Sioux Falls, SD
UMD1 (2)
UMD
College Park
LGP23 (4)
GSFC
B23/W316
1 Gbps
1 Gbps
USGS EROS
1 Gbps
Backbone
1 Gbps
1 Gbps
USGS EROS
OC12, 622 Mbps
vBNS+ (Chicago)
OC48, 2.4Gbps
Backbone
MacCl23 (12)
GSFC
B23/W316
MAX (College Park)
OC48, 2.4Gbps
Backbone
OC12, 622 Mbps
Shared with DREN
GSFC SEN
1Gbps
Backbone
1 Gbps
LGP32 (2)
Science User_1
GSFC
B32/C101
GSFC
Capability 2
Capability 3
Sponsored by NASA LDCM, NASA/GSFC Code 580
Team: 586/585/SGT/QSS/Aerospace Corp/USGS EROS
SEN: Science and Engineering Network
MAX: Mid-Atlantic Crossroads
DREN: Defense Research and Engineering Network
vBNS+: Very high Performance Network Service
Capability 2 Grid Workflow
In Capability 1, jobs were run on a specific resource
In Capability 2, workflow provided the ability to
submit a job to the “Grid” (VO)
• Leverage distributed resource sharing and
collaboration on a large-scale
• Grid resource management
• Automatic allocation of grid resources
• Sub task management
• Reliable job completion
• Leverage idle cpu cycles
Sponsored by NASA LDCM, NASA/GSFC Code 580
Team: 586/585/SGT/QSS/Aerospace Corp/USGS EROS
Capability 2 Workflow Software: Karajan
Karajan provides grid workflow
functions
• Includes task management
language and an execution engine
• Integrated with the Java
Commodity Grid (CoG) Kit
• Includes a task scheduler
• Runs gridExecute and gridTransfer
tasks on grid resources
• Manages both local and remote
resources
• Specifies workflow using XML
• Supplies command line and GUI
interfaces
Sponsored by NASA LDCM, NASA/GSFC Code 580
Team: 586/585/SGT/QSS/Aerospace Corp/USGS EROS
Java CoG 4_0_a1
Karajan
Globus Toolkit 2.4.3
Globus
Gate
keeper
GridFTP
GRAM
Karajan – Globus Grid Architecture
User Configuration File Specification
User creates product.spec configuration file
Path, row, and acquisition date provided for each input scene
# product.spec example file
host: edclxs66.cr.usgs.gov
base_directory: /data/LEDAPS
182 062 20010719 base
182 062 20030215
182 062 20040218
182 062 20040609
# default to host and base_directory specified above
182 061 20020212 base
182 061 20020604
182 061 20040101
182 061 20040218
182 061 20040711
-
Sponsored by NASA LDCM, NASA/GSFC Code 580
Team: 586/585/SGT/QSS/Aerospace Corp/USGS EROS
Capability 2 Architecture
Product.spec
driver.pl
driver.xml
Karajan
<parallel>
Host 1
Host 2
create_composite1.xml
Host …
create_composite2.xml
<sequential>
<sequential>
Scene 1
<path, row,
acqDate>
Base Scene
lndpm
lndpm
lndcal
lndcal
lndcsm
lndsr
lndcsm
lndsr
lndreg
Scene 2
<path, row,
acqDate>
…
Scene 1
<path, row,
acqDate>
Base Scene
lndpm
lndpm
lndcal
lndcal
lndcsm
lndsr
lndcsm
lndsr
lndreg
Scene 2
<path, row,
acqDate>
…
lndcom
lndcom
Copy_
output
Sponsored by NASA LDCM, NASA/GSFC Code 580
Team: 586/585/SGT/QSS/Aerospace Corp/USGS EROS
Copy_
output
…
Capability 2 Performance
Processing benchmarks:
# of composites*
Time to process
8
3 hours
16
5 hours 36 minutes
32 (2 16 parallel runs)
11 hours 46 minutes
48
12 hours 50 minutes
*Each composite had 4 input scenes
Transfer rates
• File Transfer using 8 parallel streams
• Raw Data Files (TIF) 57 Mb in 45-50 Sec. (~ 1.26 Mbps)
• Final Output File (HDF) 1.25 Gb in 5 Minutes (~ 4 Mbps)
• Conclusion: Larger files are more efficient
File Transfer
CPU Processing
Data Host
Remote Host
7%
93 %
9%
91%
Sponsored by NASA LDCM, NASA/GSFC Code 580
Team: 586/585/SGT/QSS/Aerospace Corp/USGS EROS
Performance Research and Potential Plans
Benchmarked processing rates for producing up to 50 output scenes
Completed initial analysis of transfer and processing rates obtained
using Netlogger
• Netlogger provides the ability to monitor applications within a complex
distributed environment in order to determine exactly where time is
spent
Room for Optimization
• Analyze process flows to optimize running in operational setting and
implement optimization strategies below
• Complete input file compression on data host prior to file transfer
• Increase the parallelization
• Parallel runs of multiple input scenes for a single composite
• Parallel file transfer
• Add more CPUs and maximize CPU utilization
• Look at error handling and possibility of automatic re-starting of jobs
Sponsored by NASA LDCM, NASA/GSFC Code 580
Team: 586/585/SGT/QSS/Aerospace Corp/USGS EROS
LGP Lessons Learned
The Open Source environment can be very beneficial
• Reuse, Collaboration incentive
• “Hardened software” (i.e. GSI)
A surprising amount of time was spent on basic network administration and security
• Network performance
• Firewall/ports
Maintaining configuration management across independent agencies and centers is difficult
• MapCenter - System status tool (QA/Calibration)
Understanding the processing flow and modules required for optimization
• Once size doesn’t fit all (at least not yet)
• Allow for remote processing; dynamic ancillary data
• CPU intensive vs. data intensive
Karajan is somewhat immature, but we have passed on requests to CoG developers
• Karajan does provide the basic framework for creating workflows in an operational setting.
Functionality not provided by the basic framework is being provided by external wrapper
scripts
• Developed workaround to pass environment variables across processing runs
• Provided wrapper script to pass arguments to underlying Globus executables
•
Very elementary Job Scheduler
Sponsored by NASA LDCM, NASA/GSFC Code 580
Team: 586/585/SGT/QSS/Aerospace Corp/USGS EROS
Current and Future Work
LDCM Grid Prototype work will continue
Receiving NASA ROSES ACCESS A.26 funding for
Land Cover Change Processing and Analysis System
(LC-ComPS)
Use grid technology to allow regional and
continental-scale land cover analysis at high
resolution
• Use Globus 4.0 as the underlying Grid infrastructure
• Improve error handling in the workflow scripts and
handle automatic re-starting of tasks in the event of
failures
• Expand the “pool” of machines in VO
Sponsored by NASA LDCM, NASA/GSFC Code 580
Team: 586/585/SGT/QSS/Aerospace Corp/USGS EROS
Backup Slides
Sponsored by NASA LDCM, NASA/GSFC Code 580
Team: 586/585/SGT/QSS/Aerospace Corp/USGS EROS
Acknowledgements
Sponsors
• LDCM - Bill Ochs, Matt Schwaller
• Code 500/580 - Peter Hughes,
Julie Loftis
LGP Team members
• Jeff Lubelczyk (Lead)
• Gail McConaughy (Branch Principal)
• Beth Weinstein (SW Lead)
• Ben Kobler (HW, Networks)
• Eunice Eng (SW Dev, Data)
• Valerie Ward (SW Dev, Apps)
• Ananth Rao ([SGT] SW Arch/Dev,
Grid Expert)
• Brooks Davis ([Aerospace Corp]
Grid Expert)
• Wayne Yu ([QSS] Sys Admin)
Sponsored by NASA LDCM, NASA/GSFC Code 580
Team: 586/585/SGT/QSS/Aerospace Corp/USGS EROS
GSFC Science Input
• Jeff Masek (Blender)
• Feng Gao (Blender)
USGS EROS
• Stu Doescher (Mgmt)
• Chris Doescher (POC)
• John Dwyer
• Tom Mcelroy
• Mike Neiers (Sys Support)
• Cory Ranschau (Sys Admin)
University of Maryland (UMD)
• Paul Davis
• Gary Jackson
Acronym List
ACCESS
EROS
FTP
GASS
GRAM
GSI
LC-ComPS
LDCM
LEDAPS
LGP
LP DAAC
MDS
MODIS
MRT
ROSES
Advancing Collaborative Connections for Earth-Sun
System Science
Earth Resources Observation and Science
File Transfer Protocol
Globus Access to Secondary Storage
Grid Resource Allocation & Management
Grid Security Infrastructure
Land Cover Change Processing and Analysis System
Landsat Data Continuity Mission
Landsat Ecosystem Disturbance Analysis Adaptive
Processing System
LDCM Grid Prototype
Land Processes Distributed Active Archive Center
Monitoring & Discovery System (MDS)
Moderate Resolution Imaging Spectroradiometer
MODIS Reprojection Tool
Research Opportunities in Space and Earth Sciences
Sponsored by NASA LDCM, NASA/GSFC Code 580
Team: 586/585/SGT/QSS/Aerospace Corp/USGS EROS