The backend job scheduler is Gridengine, which functions similarly to the PBS Pro/OpenPBS
Resource Requests
The table below summarises the major resources attributes which are commonly used in most of the jobs. There are also other attributes which are very helping for fine tuning how a job should be scheduled, whose details are provided in the sub sections.
Resources
Attribute
Description
Default Value
Parallel Environments (-pe)
smp
Allocate X number of CPUs on the SAME compute node
Optional, if not specified, a job defaults to use 1 CPU
mpi
Allocate X number of CPUs from multiple compute node, this is mainly used by a job that implements under Open MPI framework.
Resource request list (-l)
mem
The amount of memory limit can be used by a job
Default 1024M
jobfs
The amount of disk space limit can be used by a job
Default 1G
walltime
The run time limit (elapsed time) before a job gets killed by the job scheduler
Default 0:30:0
ngpus
The number of GPGPUs
N/A
project_name (-P)
project_name
Request a job to consume resource quota defined via Project. Check Project sub section for details.
Optional, if not specified, default per user quota is consumed instead.
GPGPU
Two attributes gpu_model and gpu_code allow jobs to request to run on a particular GPGPU card model or chip. To find out what are available:
A batch job can be submitted by using command qsub, in the following pattern:
# submit a job which calls a script (bash, shell, python scripts etc)
qsub -N JOB_NAME -pe smp NUMBER_OF_CPU -l ATTR1=VAL1,ATTR2=VAL2 SCRIPT
# submit a job which calls a BINARY (anything which are not script, such as sleep, dd etc)
qsub -N JOB_NAME -pe smp NUMBER_OF_CPU -l ATTR1=VAL1,ATTR2=VAL2 -b y BINARY
Examples
# a very big sleep job that needs 16 x CPUs, 2 x GPGPUs, 64GB memory, 10G disk space
qsub -b y -N generic_gpgpu -pe smp 16 -l ngpus=2,mem=65G,jobfs=10G sleep 1m
# a smaller sleep job that requires the specific A2 GPGPU...
qsub -b y -N t1000_gpgpu -pe smp 8 -l ngpus=2,gpgpu_model=A2,mem=16G,jobfs=10G sleep 1m
# a big job runs on multiple H100 nodes inside the same physical rack/cabinet F (rack awareness)
qsub -b y -N h100_gpgpu -pe mpi 256 -l ngpus=2,gpgpu_model=H100,rack=f,mem=128G,jobfs=100G sleep 1m
Submission Script
For larger and more complex analyses, the qsub submission script can be very useful. A submission script contains pre-populated qsub parameters, can be reused, distributed and version controlled easily. It looks like:
#!/bin/bash
#
# It prints the actual path of the job scratch directory.
#$ -pe smp 8
#$ -j y
#$ -e logs/$JOB_ID_$JOB_NAME.out
#$ -o logs/$JOB_ID_$JOB_NAME.out
#$ -cwd
#$ -N dd_smp
#$ -l mem=1G,jobfs=110G,tmpfree=150G,walltime=00:30:00
#
echo "$HOST $tmp_requested $TMPDIR"
# about 107GB
dd if=/dev/zero of=$TMPDIR/dd.test bs=512M count=200
To submit
z1234567@login01:~$ qsub sge_dd_smp.sh
Your job 31 ("dd_smp") has been submitted