Transcript Queuing Tutorial

An overview of Torque/Moab queuing
Topics
 ARC topology
 Authentication
 Architecture of the queuing system
 Workflow
 Job Scripts
 Some queuing strategies
Network Topology
ARC Authentication
Accounts
Your account is your VT PID
Your password is your VT PID password
Contact 4help to change your password
Architecture
 Resource Manager - Torque
 Scheduler - Moab
 Allocation Manager - Gold
Account Requests
 To request an account:
http://www.arc.vt.edu/arc/UserAccounts.php
 System X accounts
https://portal.arc.vt.edu/allocation/alloc_request.ht
ml
 To add users to a Hat/Project for System X
PI Email [email protected] to ask to have that person added
Queue Architecture
Resource Manager
 Torque (Tera-scale Open-source Research and
QUEue manager)
 Branch of OpenPBS
 2 Parts
 Pbs_mom
 Daemon on each compute node
 Handles job start up and keeps track of the node’s state
 Pbs_server
 Server that jobs are submitted to.
 Keeps track of all nodes and jobs
Moab Scheduler
 Takes state information from the resource
manager and then schedules jobs to run
 “The Brains”
 Implements and manages:
 Scheduling policies
 Dynamic priorities
 Reservations
 Fairshare
Allocation Manager
 Gold
 Keeps track of cpu-hours
Workflow
From the queuing system point of view
When a scheduling interval starts
Moab asks pbs_server the state of the nodes and of any jobs
Moab attempts to schedule any eligble jobs if there are enough
resources free
Moab tells pbs_server to schedule start any jobs that can be started
Pbs_server contacts the pbs_mom on the first node assigned to the
job (That pbs_mom is called the mother superior)
The mother superior executes the jobs scripts submitted by the
user
When a pbs heartbeat happens
The pbs_server will contact the pbs_mom and ask the status of its
node
Workflow
From a user’s point of view
 Submit a job script to the queuing system
 Wait for the job to be scheduled and run
 Get the results
The Queue
Queue divided into 3 subqueues:
 Active – running
 Eligible – idle, but waiting to run
 Blocked – idle, held, deferred
Blocked jobs
A job can be “blocked” for several reasons:
 Requested resources not available
 Reserved nodes offline
 User already has the maximum number of
eligible jobs in the queue
 User places intentional hold
Moab supports four distinct types of holds: user,
system, batch, and deferred
Job Scripts
 The job script has a few definitions to inform the
queuing system of your job requirements and who you
are
 Includes environment variables and commands to run
your application
Script Definitions
 Walltime request
 #PBS -lwalltime=hh:mm:ss
 CPU request
 For System X

#PBS -lnodes=X:ppn=2
 X number of nodes with 2 processors per node
 For Cauldron

#PBS -lncpus=X
 X number of cores
Script Definitions
 Which queue you want to use
 #PBS -q <queue name>
 queues available now






System X OS X partition: production_q
System X Linux partition: linux_q
Cauldron: cauldron_q
Inferno2: inferno2_q
Ithaca: ithaca_q
Ithaca parallel matlab: pmatlab_q
Script Definitions
 Some information about who you are
 Your submission group
 #PBS -W group_list=<group>



For System X it is tcf_user
For Cauldron it is sgiusers
Type `groups` when logged into a head node to check that you belong
to group of the machine you wish to submit to
 Your cpu-hour hat
 #PBS -A <hat>


On Cauldron it is sgim0000
System X users were told their hat in their welcome letters.
Job Script Template
#!/bin/bash
#PBS -lwalltime=01:00:00
#PBS -lncpus=8
#PBS -q cauldron_q
#PBS -W group_list=sgiusers
#PBS -A sgim0000
Job Script
 After the PBS definitions, put in the commands to start
your job
 There are example job scripts found in
/apps/doc(s)
Running Your Job
 Use qsub to submit your job to the queue
 qsub ./jobscript
 To check on your job’s status
 qstat -a <queue name>
 showq -p <partition name>
 OSX, LINUX, or CAULDRON
 checkjob <job id number>
 cstat (on Cauldron)
 To delete a job, use qdel
 qdel <job id number>
Check Status
To display jobs currently in the queue:
-bash-3.1$ showq -p LINUX
active jobs-----------------------JOBID
USERNAME
STATE PROCS
176882
176885
176889
176918
176897
5 active jobs
REMAINING
STARTTIME
jalemkul
Running
24
23:46:56 Mon
jalemkul
Running
24 1:01:37:59 Mon
jalemkul
Running
24 1:02:21:27 Mon
kmsong
Running
44 6:14:25:16 Mon
kmsong
Running
88 15:17:01:30 Tue
118 of 118 processors in use by local
50 of 59 nodes active
(84.75%)
eligible jobs---------------------JOBID
USERNAME
STATE PROCS
0 eligible jobs
Aug
Aug
Aug
Aug
Aug
jobs
2 07:11:24
2 09:02:27
2 09:45:55
2 16:49:44
3 11:25:58
(100.00%)
WCLIMIT
QUEUETIME
blocked jobs----------------------JOBID
USERNAME
STATE PROCS
WCLIMIT
176956
kmsong
Idle
112 33:08:00:00
1 blocked job
QUEUETIME
3 15:15:13
Total jobs:
6
Tue Aug
Check Status
With qstat:
-bash-3.1$ qstat linux_q
Job id
Name
------------------- ---------------176882.queue
yt42_md1
176885.queue
yt42_md3
176889.queue
yt42_md2
176897.queue
DNS
176918.queue
Re1200_2sec
176956.queue
LDNS
User
--------------jalemkul
jalemkul
jalemkul
kmsong
kmsong
kmsong
Note: status give by R – running and Q – queued
Time Use
-------1229:52:
1185:20:
1168:07:
00:00:00
1828:24:
0
S
R
R
R
R
R
Q
Queue
----linux_q
linux_q
linux_q
linux_q
linux_q
linux_q
Qstat -f
-bash-3.1$ checkjob -v 176956
job 176956 (RM job '176956.queue.arc-int.vt.edu’)
AName: LDNS
State: Idle
Creds: user:kmsong group:tcf_user account:engr1003 class:linux_q
WallTime:
00:00:00 of 33:08:00:00
SubmitTime: Tue Aug 3 15:15:13
(Time Queued Total: 1:23:40:11 Eligible: 00:00:19)
NodeMatchPolicy: EXACTNODE
Total Requested Tasks: 112
Total Requested Nodes: 56
Req[0] TaskCount: 112
NodeAccess: SINGLEJOB
TasksPerNode: 2
qos:sysx_qos
Partition: ALL
UMask:
0000
OutputFile:
sysx2.arc-int.vt.edu:/home/kmsong/Turb_channel/Simulation/Re600/176103.queue.arcint.vt.edu/LDNS.o176956
ErrorFile:
sysx2.arc-int.vt.edu:/home/kmsong/Turb_channel/Simulation/Re600/176103.queue.arcint.vt.edu/LDNS.e176956
BypassCount:
305
Partition List: LINUX,SHARED
SrcRM:
SystemX DstRM: SystemX DstRMJID: 176956.queue.arc-int.vt.edu
Submit Args:
-l walltime=800:00:00 -l nodes=56:ppn=2 -Wgroup_list -Aengr1003 -NLDNS -q linux_q -I
Flags:
INTERACTIVE
Attr:
INTERACTIVE,checkpoint
StartPriority: 200
PE:
112.00
NOTE:
job violates constraints for partition OSX (partition OSX not in job partition mask)
Node Availability for Partition LINUX -------available for 2 tasks
- n[925,951-958]
rejected for State
- n[833-1024]
NOTE: job req cannot run in dynamic partition LINUX now (insufficient procs available: 18 <
112)
NOTE: job violates constraints for partition CAULDRON (partition CAULDRON not in job partition
mask)
NOTE: job violates constraints for partition INFERNO2 (partition INFERNO2 not in job partition
mask)
NOTE:
job violates constraints for partition TT (partition TT not in job partition mask)
NOTE:
job violates constraints for partition PECOS (partition PECOS not in job partition mask)
NOTE: job violates constraints for partition ITHACA (partition ITHACA not in job partition
mask)
BLOCK MSG: job 176956 violates active SOFT MAXJOB limit of 2 for class linux_q user (Req: 1
InUse: 2) (recorded at last scheduling iteration)
Queuing Strategies
 Queue early, queue often
 Queue your jobs up!
 Can’t run jobs if they aren’t in the queue
 Don’t wait for the queue to get smaller because the job will
wait, its waiting anyways!
 Possibility for backfill for smaller jobs
 Have an accurate walltime
 Accurate walltimes will help the queue try to backfill in
smaller jobs in between runs of larger jobs, but only if it
won’t effect the start time of the next job
 Try to queue large jobs before downtimes
 If you have a large job that can never seem to have enough
cpus available, queue it up before a downtime.
Queue Strategies
 The command `showbf`
 That command shows cpus available right now, and
for how long
 Showstart
 estimated start time of a job
 checkjob -v
 Checkpointing
 If your code does checkpointing you can exploit
backfill, by queuing jobs to fill the small places but
maybe not running to completion
 Good idea in general, in case of hardware failure
showbf
-bash-3.1$ showbf
Partition
Tasks
------------ALL
146
OSX
4
LINUX
62
PECOS
8
ITHACA
72
Nodes
----43
2
31
1
9
StartOffset
-----------00:00:00
00:00:00
00:00:00
00:00:00
00:00:00
Duration
-----------INFINITY
INFINITY
INFINITY
INFINITY
INFINITY
StartDate
-------------09:29:01_08/10
09:29:01_08/10
09:29:01_08/10
09:29:01_08/10
09:29:01_08/10
showstart
-bash-3.1$ showstart 177165
job 177165 requires 64 procs for 12:00:00
Estimated Rsv based start in
on Tue Aug 10 18:11:40
Estimated Rsv based completion in
on Wed Aug 11 06:11:40
Best Partition: OSX
8:47:53
20:47:53
showstart
-bash-3.1$ showstart 64@12:00:00
job 64@12:00:00 requires 64 procs for 12:00:00
Estimated Rsv based start in
on Tue Aug 10 18:11:40
Estimated Rsv based completion in
on Wed Aug 11 06:11:40
Best Partition: OSX
8:44:19
20:44:19
Documentation
Torque/PBS and Moab scheduler and job submission
documentation:
http://www.clusterresources.com/pages/resources/docu
mentation.php