High-Performance Computing for Processing Earth Observation Data By Dr Ashok Kaushal Senior Divisional Director Enterprise Geospatial & Defense Solutions Rolta India Limited [email protected] Innovative Technologies for Insightful Impact.
Download ReportTranscript High-Performance Computing for Processing Earth Observation Data By Dr Ashok Kaushal Senior Divisional Director Enterprise Geospatial & Defense Solutions Rolta India Limited [email protected] Innovative Technologies for Insightful Impact.
High-Performance Computing for Processing Earth Observation Data By Dr Ashok Kaushal Senior Divisional Director Enterprise Geospatial & Defense Solutions Rolta India Limited [email protected] Innovative Technologies for Insightful Impact Agenda • Trends • Needs • Process Automation • GeoImaging Accelerators/ GXL • Job Processing Systems/ JPS • Conclusions Trends • 230 EO [versus 107 in last decade] satellites projected over next decade for use of satellite imagery – Emerging markets expected to account for 75 satellites four-fold increase over last decade – 41 Nations [currently 26] to have own satellites • Commercial sale of EO data expected to double – Commercial EO data from satellites expect CAGR of 15% over next 10 years, reaching $4 billion by 2019 – Optical data will represent 79% of overall sales – Number of high resolution satellites offering commercial data are expected to double from currently 24 ‘Satellites to be Built & Launched by 2019, World Market Survey’, Euroconsult Trends • Exponential increase of volumes of satellite EO data • Increasing value of EO data with applications in – Agriculture, Environment, Urban Development, Disaster Management, Surveillance and others • Increasing value of up-to-date info – RapidEye, GeoEye, Digital Globe, IRS/ Cartosat • Significant growth of awareness in EO data – Google Earth, Microsoft Bing Maps, Bhuvan • Increasing importance of collaboration and sharing of current data/information for Situational Awareness Needs • Satellite Programming • Timely Data Acquisition • Process Automation • Data Pre-Processing • Data Management • Data Dissemination • Information Sharing • Geo Collaboration Process Automation For Production Move from this To This Process Automation Incoming raw image Extract raw Image to native format Collect GCP Using Master Image Refine collected GCP Compute Math Model Orthorectify Raw Image Load Image to Oracle Database Oracle Database Geoimaging Accelerator (GXL) Geoimaging Accelerators are automated workflows created from linking together of any number of pluggable image processing functions Data Ingest Process 1 Process 2 Process 3 Delivery Geoimaging Accelerator (GXL) Objectives: • Need for large volume image data processing • to reduce image pre-processing bottlenecks • Demand for greater automation & less user interaction • to save money on operator time • Workflows that can scale across multiple processors • to add capacity as and when needed • Plug & Play architecture • to add new components or functions to expand • Cost Effective Solution to Remain Competitive • to run 24/7 with zero or little operator intervention Geoimaging Accelerator (GXL) • Distributed processing • Two levels • Basic • Automated CPU • Accelerated • Multi-core CPU • Optimized GPU • Ortho / Ortho XL • Satellite & Airphoto • PanSharp / PanSharp XL • Mosaic/ Mosaic XL Ingest GXL Output Job Processing System (JPS) Geoimaging Accelerator (GXL) RPC Model Calculation Orthorectification DEM WorldView-1 Level 1b e.g. Ortho Product Orthorectification GXL AP Model Calculation Orthorectification DEM Orthorectification GXL Airphoto / Airphoto XL Format & Tile UltraCam X Imagery e.g. Ingest GPS/INS Ortho / Ortho XL Ortho Product Geoimaging Accelerator (GXL) Pan Sharpening PanSharp GXL Pan and MS Imagery ColourBalance Mosaic Mosaic GXL Orthophotos Epipolar Rotation Stereo Pair Cutline Selection PanSharp Product DEM Extraction DEM Extraction GXL Mosaic Product Geocoding Raster DEM Geoimaging Accelerator (GXL) Accelerated GXL? • A hardware-based, GPU enabled, highperformance image processing system • Design to process large volumes – 40 times faster than desktop product – 2-4 TB per day for desk-side system – 10 TB + for rack mounted system Orthorectify & Mosaic India in a Day! Architecture Geoimaging Accelerator (GXL) Layer: Component: Integration: Interface Layer Job Processing System Data / Imagery Level Workflows: GXL Bindings: Python, Java C++ SDK GPU / HW PPFs Formats: BIL, TIFF, etc. Algorithms: Pansharp, Ortho, etc. Processing Layer Architecture Layer Operations / Systems Level HW / Architecture Level Geoimaging Accelerator (GXL) Highlights • Flexible orthorectification: – Support for several sensors (SPOT, QB, Ikonos, WV, …) – Optional radiometric calibration of SPOT images – Optional GCP collection from multiple reference data types • Flexible mosaicking: – – – – Mosaics from mixed-resolution raw scenes Optional tie point collection and refinement Various types of color balancing Various tiling schemes • High quality: – Sub-pixel accuracy of GCPs and orthoimages – Nicely color-balanced mosaics Processing Metrics Geoimaging Accelerator (GXL) Product Type Dataset SPOT5 - Level 1A 2.5 meter 8U Pan IKONOS - Geo Ortho Kit WorldView-1 and QuickBird Level 1B QuickBird - OrthoReady - 4 channel PS QuickBird - Level 1B 16U Pan Ikonos 16U Pan 16U Multispectral 16U Multispectral Resolution Volume [m] [TB/Day] 2.5 1.0 0.5 0.6 2.4 2.00 2.94 3.26 3.52 4.57 Area [km2/day] 13.7 Million e.g. Europe: 10.1M km2 1.62 Million e.g. Mongolia: 1.56M km2 448k e.g. Sweden: 450k km2 174k e.g. Florida: 170k km2 3.62 Million e.g. India: 3.17M km2 Processing Throughput Geoimaging Accelerator (GXL) Product Type Dataset MB/Sec GB/Min TB/Day SPOT5 - Level 1A 2.5 meter 8U Pan 24.23 1.42 2.00 IKONOS - Geo Ortho Kit 16U Pan Ikonos 35.67 2.09 2.94 WorldView-1 and QuickBird Level 1B 16U Pan 39.59 2.32 3.26 QuickBird - OrthoReady - 4 channel PS 16U Multispectral 42.67 2.50 3.52 QuickBird - Level 1B 16U Multispectral 55.47 3.25 4.57 Geoimaging Accelerator (GXL) Cost $1,000 500 GXL Rack Accelerated GXL Basic 200 Batch Processing 10 100 Orthos per day 50GB Project Scale 5000 Orthos per day Plus 100 Image Mosaic per day 5TB Project Scale 2000 Orthos per day 1TB Project Scale GXL Deskside Accelerated 20 Orthos per day 10GB Project Scale GB 1 - 5TB Performance /Day 5 - 10TB Applications GeoImaging Accelerator • Environmental • Carbon sequestration • Biomass estimation • Agricultural • Crop yield • Crop forecasting • Aerospace & Defense • Border monitoring • Disaster management • Data Supply • Product delivery • Archive re-processing Job Processing System • Distributed Processing System – Run multiple jobs concurrently on multiple servers JPS Database Computer JPS Processing Server Job Computer Computer JPS Processing Server Job JPS Processing Server Job Computer JPS Processing Server Job Job Job Job Job Processing System • Job: – An entry in the JPS-DB JPS-DB – A Process started and monitored by a Processing Server • Processing Server – Daemon managing jobs Processing Server Job Job Job Processing System • • • • Distributed Cloud Computing (Autonomous Nodes) Automatic Load Balancing Simple Web Interface Threefold Value: 1. Automation = Increased Throughput (Revenue) 2. Job Tracking = Improved QA (Operational Costs) 3. Multi-Platform, Multi-Language = Sustainability Job Job JPS-DB Job GXL1 GXL2 Job Other Nodes Job Job 22 Job Processing System Conclusions • Effective use of voluminous satellite imagery from numerous high-resolution satellites desires automated pre-processing using HPC • Distributed processing using multi-core CPU and GPU with CUDA and Open MP provides an ideal platform for faster turn-around-time during pre-processing of geoimaging Thank you !