Transcript LaMNA Dataset Processing Flow

LaMNA
Incoming Dataset
Processing Flow
http://www.KlamathBird.org/lamna/
Contents
Justification
What we do
How we do it
The challenges ahead
How you can help
Contents
Justification
New challenges...
Global climate change means local phenomena
affected by regional or continent-wide processes
Management challenge: solutions that make
sense at various scales (esp. regionally)
Need datasets that span geographic AND time
scales
The Avian Knowledge Network: gather and
compile avian datasets under a single format
New challenges...
Typical avian monitoring data: presence/absence
or counts...
...but we don’t know WHY populations are changing.
...expect where species may go extinct...
Bird banding data include detailed metrics:
sex, age, body condition, reproductive status
Bird banding data provide information critical to
UNDERSTANDING WHY bird populations will change
One of LaMNA’s goals
LaMNAestimate:
joined the Avian Knowledge Network
LaMNA
...weshort
aim to
compile,
preservedata:
and make
widely
Very
half-life
of banding
5-8 years
accessible
banding
datasetsmisplaced
through: files, lack
(due
to personnel
changes,
of interest,
improper databasing methods)
•Proper
documentation
(dataamount:
about data,
or “metadata”)
Loss
$5 million
(10% archiving
of annual monitoring budget)
•Safe
•Eased data accessibility and use
(“use it WE
or lose
it”)LOSING
- esp. forPOTENTIALLY
multi-scale analyses
ARE
VERY
IMPORTANT DATA VERY RAPIDLY!
Contents
What we do
LaMNA and the AKN
LaMNA curates and “advertises” datasets
via the AKN locally and globally
Data owner controls how
data are shared and used
Dataset
LaMNA
documents
& curates
AKN
curates &
advertises
Data sharing policy:
http://www.klamathbird.org/lamna
National
Biodiversity
Information
Infrastructure
Global
Biodiversity
Information
Facility
Contents
How we do it
Data flow through LaMNA
Cooperator contacts LaMNA with interest in
documenting, preserving and advertising data
LaMNA collects and maintains information about
cooperator in “metadata” database
Metadata documentation begins...
LaMNA metadata documentation
Simple but thorough process:
•LaMNA requests all or sample dataset
•LaMNA experts review data contents
•LaMNA follows up with brief questionnaire
•Only data on cooperator and essential
additional information requested
LaMNA metadata documentation
All correspondence
tracked, permits,
sampling protocols...
Station locations
Years
operated
Institution
&
contact
Data
access and sharing level
information
LaMNA data processing
Cooperator sends full dataset in any format
Each “dataset” should comprise 4 data files:
•Banding data
Dataset Dataset
Banding:
•Effort data
(dBase) (Paradox)
Species
Sex
•Location data
Age... Effort:
Dataset
Dataset
•Sampling protocol (Excel) Station
(Word)
Date
Bander...
Dataset
Dataset
(hard
(FoxPro)
copy)
Dataset
(Simple
text)
Location:
Latitude
Longitude
Datum Sampling
protocol:
Narrative
ofLaMNA
methods
documents
& curates
LaMNA data processing
LaMNA tracks
each file received
LaMNA data processing
EffortSample
data example:
protocol text:
Humboldt
Fort
Hunter
Bay
Sekercioglu
Las Cruces
Bird Observatory
Leggett
Environmental
2007
Protocol
Location data in
GIS database
(display: Google Earth)
LaMNA data processing
LaMNA experts process datasets
Output: single flat table holding cooperator data
in AKN data exchange format (515 fields!)
Cooperator dataset
Sampling
protocol
Location
data
Effort
data
Bird
data
LaMNA
database tools
web accessible
Single dataset in AKN’s
data exchange format
(515 fields!)
LaMNA data processing
LaMNA developed tools to track progress
of data processing
LaMNA data processing
Follow status per cooperator & protocol
Contents
The challenges ahead
LaMNA current status
4,800,000
“Banding universe”:
~790 stations
run avg. 18.5 years
avg. ~2,000 records/year
5,500,000
7,300,000
6,700,000
4,900,000
LaMNA current status
Data Universe
(30 x 106 banding records)
Data reviewed
by LaMNA
Data files received:
effort, birds, location
curated “as is”
(518,789 Bird records)
(296,103 Bird records)
Data processed:
formatted to fit AKN’s
data exchange format
Data processed:
all data properly linked
into single table “view”
(273,815 Bird records)
(203,621 Bird records)
Dataset fully documented,
curated and posted in
web-based access system
(180,600 Bird records)
Contents
How you can help
LaMNA current status
OUR GOAL: PROPERLY DOCUMENT
PRESERVE AND MAKE AVAILABLE EVERY
BANDING DATASET
Our work has just begun...
We need YOUR HELP
Contact LaMNA to see how you can help:
http://www.klamathbird.org/lamna/data_archiving.htm
Click here