Mass Storage

        long term archival storage access via ~/ms looks like ordinary disk file system – data is actually stored on tape “limitless” capacity data is backed up For storage only, not a work directory (i.e. don’t run jobs from here ) if you have many small files, use tar or zip to create a single file for better performance Sign up for this service on onyen.unc.edu

“To infinity … and beyond” - Buzz Lightyear

29

User Environment and Applications, Compiling Code

Modules

Modules

  The user environment is managed by modules Modules modify the user environment by modifying and adding environment variables such as PATH or LD_LIBRARY_PATH   Typically you set these once and leave them Note there are two module settings, one for your current environment and one to take affect on your next login (e.g. batch jobs running on compute nodes)

31

Common Module Commands

 module avail • module avail apps  module help Login version    module list  module initlist module add  module initadd module rm  module initrm

More on modules see

http://help.unc.edu/CCM3_006660

32

Parallel Jobs with MPI

 There are three implementations MPI standard installed: • mvapich • mvapich2 (currently only on topsail) • openmpi of the  Performance is similar for all three, all three run on the IB fabric. Mvapich is the default. Openmpi and mvapich2 have more the the MPI-2 features implemented.

33

Compiling MPI programs

 Use the MPI wrappers to compile your program • mpicc, mpiCC, mpif90, mpif77 • the wrappers will find the appropriate include files and libraries and then invoke the actual compiler • for example, mpicc will invoke either gcc or icc depending upon which module you have loaded

34

Compiling on Topsail/Kure

  Serial Programming • Intel Compiler Suite for Fortran77, Fortran90, C and C++, Recommended by Research Computing •  icc, icpc, ifort GNU  gcc, g++, gfortran Parallel Programming • • MPI (see previous page) OpenMP  Compiler tag: -openmp for Intel -fopenmp for GNU  Must set OMP_NUM_THREADS in submission script

35

Debugging - Totalview

 If you are debugging code there is a powerful commercial debugger, totalview      See http://help.unc.edu/CCM3_021717 parallel and serial code Fortran/C/C++ GUI for source level control too many features to list!

36

Job Scheduling and Management

What does a Job Scheduler and batch system do?

      Manage Resources allocate user tasks to resource monitor tasks process control manage input and output report status, availability, etc enforce usage policies

38

Job Scheduling Systems

 Allocates compute nodes to job submissions based on user priority, requested resources, execution time, etc.

 Many types of schedulers • Load Sharing Facility (LSF) – Used by Topsail/Kure • IBM LoadLeveler • Portable Batch System (PBS) • Sun Grid Engine (SGE)

39

LSF

   All Research Computing clusters use LSF scheduling and management to do job LSF (Load Sharing Facility) is a (licensed) product from Platform Computing • Fairly distribute compute nodes among users • enforce usage policies for established queues  most common queues:

int, now, week, month

• RC uses Fair Share scheduling, not first come, first served (FCFS) LSF commands typically start with the letter b in batch), e.g. bsub, bqueues, bjobs, bhosts, … (as • see man pages for much more info!

40

Simplified view of LSF

Login Node Jobs Queued job_J job_F myjob job_7 job routed to queue bsub –n 64 –a mvapich –q week mpirun myjob job dispatched to run on available host which satisfies job requirements user logged in to login node submits job

41

Running Programs on Topsail

 Upon ssh to Topsail/Kure, you are on the Login node.

  Programs SHOULD NOT be run on Login node.

Submit programs to one of the many, many compute nodes.

 Submit jobs using Load Sharing Facility (LSF) via the bsub command.

42

Common batch commands

     bsub - submit jobs bqueues – view info on defined queues • bqueues –l week bkill – stop/cancel submitted job bjobs – view submitted jobs • bjobs –u all bhist – job history • bhist –l

43

Common batch commands

   bhosts – status and resources of hosts (nodes) bpeek – display output of running job Use man pages • man bjobs to get much more info!

44

Submitting Jobs: bsub Command

   Submit Jobs - bsub • Run large jobs out of scratch space, smaller jobs can run out of your home space bsub [-bsub_opts] executable [-exec_opts] Common bsub options: • • • • • –o  –o out.%J -q  -q week -R “resource specification”  -R “span[ptile=8]” -n  used for parallel, MPI jobs -a  -a mvapich(used on MPI jobs)

45

Two methods to submit jobs:

   bsub example: submit the executable job, myexe, to the week queue and redirect output to the file out. (default is to mail output) Method 1: Command Line • bsub –q week –o out.%J myexe Method 2: Create a file (details to follow) called, for example, myexe.bsub, and then submit that file. Note the redirect symbol, < • bsub < myexe.bsub

46

Method 2 cont.

  The file you submitted will contain all the bsub options you want in it, so for this example myexe.bsub will look like this #BSUB –q week #BSUB –o out.%J myexe This is actually a shell script so the top line could be the normal #!/bin/csh, etc and you can run any commands you would like.

• if this doesn’t mean anything to you then nevermind :)

47

Parallel Job example

   Batch Command Line Method bsub –q week –o out.%J -n 64 -a mvapich mpirun myParallelExe Batch File Method bsub < myexe.bsub

where myexe.bsub will look like this #BSUB –q week #BSUB –o out.%J #BSUB –a mvapich #BSUB –n 64 mpirun myParallelExe

48

Some Topsail Queues

Queue

int

Time Limit

2 hrs

Jobs/User

128

CPU/Job

-- debug day week 512cpu 128cpu 32cpu chunk 2 hrs 24 hrs 1 week 4 days 4 days 2 days 4 days 64 512 512 512 512 512 512 -- 4 – 128 4 – 128 32 – 512 32 – 128 4 – 32 Batch Jobs • For access to the 512cpu queue the scalability must be demonstrated

49

Some Kure Queues

Queue

int debug bigmem week patrons

Time Limit

10 hrs 5 minutes 1 week 1 week none

Jobs/User

2 32 8 Most users have a 32 job slots limit unless they have been granted extra slots.

Queues are always

subject to change

and probably will change as Kure production ramps up. Use the bqueues command to find the current status

50

Common Error 1

  If job immediately dies, check err.%J file err.%J file has error: • Can't read MPIRUN_HOST   Problem: MPI enivronment settings were not correctly applied on compute node Solution: Include mpirun in bsub command

51

Common Error 2

    Job immediately dies after submission err.%J file is blank Problem: ssh passwords and keys were not correctly setup at initial login to Topsail

Solution:

• • • • •

cd ~/.ssh/ mv id_rsa id_rsa-orig mv id_rsa.pub id_rsa.pub-orig Logout of Topsail Login to Topsail and accept all defaults 52

Interactive Jobs

 To run long shell scripts on Topsail or Kure, use int queue  bsub –q int –Ip /bin/bash • This bsub command provides a prompt on compute node • Can run program or shell script interactively from compute node

53

Specialty Scripts

 There are specialty scripts provided on Kure for the user convenience.  Batch scripts • bmatlab, bsas, bstata  X-window scripts • xmatlab, xsas, xstata  Interactive scripts • imatlab, istata

54

MPI/OpenMP Training

 Courses are taught throughout year by Research Computing •

http://learnit.unc.edu/workshops

•

http://help.unc.edu/CCM3_008194

 See schedule for next course • MPI • OpenMP

55

Further Help with Topsail/Kure

   More details can be found on the Getting Started help documents: • http://help.unc.edu/?id=6214 - Topsail • •

http://help.unc.edu/ccm3_015682

http://keel.isis.unc.edu/wordpress/ - Kure - ON CAMPUS For assistance with Topsail/Kure, please contact the ITS Research Computing group • Email: [email protected]

• Phone: 919-962-HELP • Submit help ticket at http://help.unc.edu

For immediate assistance, see manual pages • man

56