Transcript JGI Training Series Katie Antypas User Services Group Lawrence Berkeley National Lab

JGI Training Series
Katie Antypas
User Services Group
Lawrence Berkeley National Lab
17 February 2012
Until all users are migrated to NERSC we plan to hold
weekly Friday sessions
Introduction to NIM
More on file and
data management
Thei
a
Rhea
Oceanus
Iapetus
Kronos?
Crius
Review of batch system
policies
Themis
Hyperion
Open Office Hours
On NIM you can change your password, change your shell
and set security questions
Login to nim.nersc.gov
Look under the actions menu to do the above tasks
File systems best practices
• Unfortunately disk is still
expensive
• All of the JGI’s data can
not be stored on disk
within the current budget
• Archive and delete data
you no longer need
• Disk usage will be
controlled through
quotas in some cases and
purging in others
Only the “house” file system will be available on both JGI
and NERSC systems initially
JGI Space
NERSC Space
Most web servers
Compute cluster
Some submit hosts
Netapps
“projectb”
house
•If your data needs access to both servers in JGI space and the
compute cluster, it MUST go into “house”
•In other words – move data out of Netapps
But “house” is 90% full……
House 90%
File systems above 90% are lower performing and at
higher risk of failure
We need your help deleting data from “house” and
moving data from the netapps to “house”
NERSC has set up 2 fast “data transfer nodes”
just for JGI users
Login to dtn03.nersc.gov or dtn04.nersc.gov
Type >df to see all the mounted file systems
Back up data to HPSS (you authenticated at last week’s training don’t remember?
Type hsi and then enter your NIM password)
> cd /house/path/to/your/data
> hsi put <filename>
Or archive an entire directory
> htar –cvf tarname.tar directory/
There are two areas of storage within the “project” layout
of the “projectb” file system
ssh phoebe.nersc.gov
/projectb/
projectdirs/
scratch/
PI/ RD/ fungal/ metagenome/ micro/ plant/ comparative/
• Group directories
• Not purged
• Subject to quota
Request a projectb directory for your
group through the Jira ticket system
user/
• User directories
• cd $SCRATCH
•Purged, 12 weeks
•1 TB, 500,000 inode
quota
Request a larger /scratch quota
through the Jira ticket system
Use the fast data transfer nodes to move data between file
systems
Login to dtn03.nersc.gov or dtn04.nersc.gov
Type >df to see all the mounted file systems
You can move data to 3 file systems $HOME “project” “scratch”
> mv /old/path/filename /new/path/filename
It is important for every group to come up with a data
retention policy
How long
should we keep
the raw data?
Can the data
be deleted or
should it be
archived?
Can we set up an
automated way to
archive and
delete data?
The JGI compute clusters have been consolidated into Crius
with the following shares
Theia
Rhea
Oceanus
Iapetus
Crius
Kronos?
Themis
Hyperion
Users should submit jobs to the normal queue
Jobs running longer than 12 hours or requesting large amounts of
memory could see longer wait times
Useful commands