OpenPBS software optimizes job scheduling and workload management in high-performance computing (HPC) environments – clusters, clouds, and supercomputers – improving system efficiency and people’s productivity.

PBS Cheatsheet –

pbsnodes -a	Shows you the list of nodes with all details
pbsnodes -aSj	Shows you detail overview of resources
qsub	use to submit a job
qdel jobid	delete a submitted job
qstat -a	shows you the status of all jobs
tracejob jobid	shows you the details status of a job

Sample OpenPBS script –

#!/bin/bash
#PBS -N Jobname
#PBS -q Queue_name
#PBS -l select=16:ncpus=1:mpiprocs=1
#PBS -l walltime=hh:mm:ss
#PBS -j oe
#PBS -V
#PBS -o log.out

cd $PBS_O_WORKDIR
cat $PBS_NODEFILE > ./pbsnodes
PROCS1=$(cat $PBS_NODEFILE | wc -l)

mpirun -machinefile $PBS_NODEFILE -np 16  ./filename
/bin/hostname

Where the line “-l select=2:ncpus=16 ” is the number of processors required for the job. ‘SELECT’ specifies the number of nodes (or chunks of resource) required.

ncpus indicates the number of CPUs per chunk required. Per CPU core is also considered as a resource. So you always keep the value of ‘SELECT’ as the number of cores you want if you are not bothered of splitting your codes and want the PBS to decide and schedule it. So for a 64 core job, the value will be

#PBS -l select=64:ncpus=1 

(In this case, PBS will start to fill on all the cores available from node 1 and continue till it satisfies 64 core request)

If we want the 64 core job to run on 4 nodes with 16 core on each node (i.e. 16 core x 4 nodes = 64 cores)

Your PBS value should be –

#PBS -l select=4:ncpus=16 
#PBS -l place=scatter

(In this case, PBS will look for all 4 nodes available with free 16 cores and start to fill it to satisfy 64 core request. Place value is important here, by default the place value is free)

In case you want your job to run on single node and does not want PBS to split it as per the availability of the cores from node1. (i.e if node1 has 6 core free and node2 has 32 core free, your 32 core job will be distributed among node1 and node2 , PBS will first fill the 6 core available on node1 and then rest 26 core no node2)

To avoid this you can use place=pack, So your request should look like –

#PBS -l select=32:ncpus=1 
#PBS -l place=pack

Our cluster has 32 core on each node via hyper-threading, so you cannot have a pack request for a job of 64 core as PBS will not be able to satisfy your request.

‘mpiprocs’ should only be given if your code is running on mpi and the value of mpiprocs will always be equal to ncpus.

You use the place statement to specify how the job’s chunks are placed.
The place statement can contain the following elements in any order: free(default) / pack / scatter .

The above example will submit 16 core jobs on the cluster and place your job as per the availability of resources. Suppose you want your job to run on a single node and do not want it to split. You can use the below parameter

#PBS -l select=16:ncpus=1:mpiprocs=1
#PBS -l place=pack

Suppose you want your code split into two nodes equally, you can use scatter.

#PBS -l select=2:ncpus=8:mpiprocs=8
#PBS -l place=scatter
.
.
.
mpirun -machinefile $PBS_NODEFILE -np 16  ./filename

**In case you want to Submit GPU code, you can do as below –

#PBS -l select=1:ncpus=1:ngpus=1

Above, will submit your CUDA code to one of the GPU nodes available in the cluster.

**Sample OpenPBS Scripts which you can download and modify as per your requirements

• click to see the code – OpenFoam

• click to see the code – Matlab – Use the sample Matlab Code – TestCode.m :-

% TEST CODE
a = 2;
b = 3;
c = a+b
display('Simulation completed successfully!')

OpenPBS job submission –