Transcript No Slide Title

USGS/EDC
Archive Preservation
Status
An Update
Stuart Doescher, USGS
(Cheryl Greenhagen)
WGISS-18
September 2004
Beijing, China
TM and MSS Archive Conversion
System (TMACS)
• Operations: 1992 to 2002.
• Number of High Density Tapes (HDT)
transcribed: 32,300
– Landsat MSS, TM
• Number of DCRSi Cassette Tapes (DCT)s
output: 3,200
• Mostly in Telemetry stream format
(Frame sync required to read)
WBVT summary
• Operated: April 1999 - May 2002
• Number of HDTs converted: 21,300
• Number of Digital Linear Tape (DLT 7000)
generated: 143 and an additional 143 as
backup
• Format: BSQ ( MSS-X)
• "orphan" data (no DIAT / SLAT) : 172 HDTs
• Bad input tapes: 476 (even with cleaning &
baking).
SPOT summary
• Operated: late April 2000 - February 2002
• Number of HDTs converted: 6,100
• Number of DLT 7000 tapes generated: 901
• Format: MDA's FRED
Landsat Archive Conversion
System (LACS)
• Planned Operations: June 2004 – June 2006
• Input: About 6000 HDTs, 3500 DCTs,
plus current Landsat 5 data
• Output: ~ 600 tapes of "formatted" data
and ~ 700 tapes of "raw" data
on 9940B. (200G/tape)
• Format: BSQ
Landsat Archive Conversion
System (LACS)
• Ingest from DCRSi Cassette Tapes (DCTs)
–
–
–
–
Multispectral Scanner Archive format (MSS-A) data
Multispectral Scanner Product format (MSS-P) data
Thematic Mapper Archive format (TM-A) data
Thematic Mapper raw (TM-R) data
• Ingest TM-R data from High Density Tapes (HDTs)
• Archive the data to a high-density computer-compatible
digital tape: 9940B
• Generate and archive browse imagery
• Generate inventory information for the new archive.
Data to be converted
Data Set
Number of
Scenes
Number of
Tapes
Data Volume
MSS-P
65,128
118 DCTs
3.2 terabytes
MSS-A
262,088
277 DCTs
9.5 terabytes
13,733
108 DCTs
3.6 terabytes
TM-R (1982 – 1994)
513,514
2,967 DCTs
127.5 terabytes
TM-R (1995 +)
106,300
5,500 HDTs
27.5 terabytes
TM-A
TM-R (new)
~ 8 terabytes
960,763
Total
179.3 terabytes
LACS Context Diagram
Digital
Archive
LTS
HDTs, DCTs
LDBS-I
LACS Transcription
System
LACS LDS
Duplicator
System
Metadata
LDBS-II
LACS Database
System – Part II
Metadata Handler
Information
Master 9940B
LACS Database
System – Part I
Job Histories,
Label
Scene Tables
Landsat
Production
Database
Interval Tables,
Scene Tables,
9940B Tables
Updated Tables
Historic Scene Info
TLS
Tape Labeling
Software
Updated
Metadata
Metadata
LDB
Label
Working
Copy
9940B
LACS
Database
LTTAS
Information
LACS Tape Tracking
and Assessment
System
All LDB Info
Working Copy 9940B
Browse Tape
LBGS
Browse
Metadata
Transcription Operators
Assessment Operators
Production Control
LACS Browse
Generation System
LDBS-III
LACS Database
System – Part III
Updated Tables
EE Browse
Images
EE Browse
Images
Earth
Explorer
NLAPS
9940B Tapes
Order
Information
DORRAN
Landsat Archive Conversion
System (LACS)
• Started Operations: June 2004
• Progress to Date:
– Current Landsat 5 Data: 252 Intervals (5807 scenes)
– DCTs: 60 Tapes (508 Intervals, 9669 scenes)
– HDTs: 5499 Tapes (534 Intervals, 10073 scenes)
• This is 25,549 of 960,763 scenes in 2 ½ months
• Extrapolates to 94 months or 7.8 years
LACS Status
• Running 3 LTS strings: (LACS Transcription
Subsystem)
– LTS1: Current Landsat 5 data only
(acquired at EDC, or received on DLT from Australia.)
– LTS2: DCT Input.
( less than 1 shift a day - because LIS can't keep up)
– LTS3: HDT Input. (1 to 2 shifts per day.)
The HDTs are always cleaned before they are used.
Many of the HDTs are baked.
Some of the HDTs are tried many times before success.
We try both drives;
sometimes we have the tape cleaned again;
sometimes we have the engineer adjust the HDTR.
LACS Status
• Running 1 LIS string: (LACS Integration
Subsystem) on the same box as LTS1.
– Performs the Automated Cloud Cover Assessment
(ACCA) and Browse Generation for all 3 LTS strings.
– The LIS is the bottleneck. (as expected)
– LIS gives priority to current Landsat 5 data.
– Plan to port the LIS to Linux and run on 3 Linux
systems to relieve the bottleneck.
– Hope to get the LIS port done late this calendar year
– Hope approximately triple our throughput
Landsat Archive Conversion
System (LACS)
• Issues / Problems:
– Some tapes in very poor condition
– Performance of ACCA and Browse Generation needs
improvement
• Plans:
– Boost performance of ACCA, Browse Generation:
• Port to Linux
• Run on Multiple Linux Systems