Multiprocessing MPI Batch Jobs

MPI jobs are typically processed through a predefined mpirun run script. Note that there are several compilers available to choose from. Once a compiler is chosen, a comparable selection should also be made on the mpirun script. For example, if you compiled your source code with an Intel compiler (be it C or Fortran), then you should also use the Intel mpirun script to run that executable. The shell definition ROOT in the following batch script provides the path to homes of the various mpirun scripts for different compilers.

#!/bin/bash
#
# All lines starting with "#PBS" are PBS commands
#
# Request 4 nodes with 1 processor per node (equals 4 processors)
# ppn can either be 1 or 2
#
#PBS -l nodes=4:ppn=1
#
# Set wall clock time to 0 hours, 15 minutes and 0 seconds
#PBS -l walltime=00:15:00

GMCONF=`/usr/local/pbs/util/mknodes`
# cd to working directory
cd $PBS_O_WORKDIR
# name of executable
MYPROG="psor"
# Extract number of processors from GMCONF above
NP=`wc -l $GMCONF |awk '{print $1}'`

echo Number of processors is $NP

# Uncomment the appropriate ROOT definition below
# to specify an alternate version of mpich
# Leaving the variable blank will allow you to use the default
# version selected by your environment variables
ROOT=''
#ROOT='/usr/local/IT/mpichgm-1.2.6..14b/intel/bin'
#ROOT='/usr/local/IT/mpichgm-1.2.6..14b/gcc/bin'

# Run MYPROG with appropriate mpirun script
$ROOT/mpirun -machinefile $GMCONF -np $NP $MYPROG

# make sure to exit the script, else job won't finish properly
exit 0

Reminder: Give the script the necessary execute-attribute with

cootie% chmod +x script_name

Submitting MPI jobs without a script

As alluded to above, PBS's qsub expects a script name and, optionally, PBS runtime parameters, as command line input. It does not, however, accept an executable name and number of processors as command line input and hence most likely you will have to modify the script frequently. In view of this, two scripts, psubi and psub, have been created by SCV as generic batch scripts for Intel and gcc compiler-generated executables for MPI multiprocessing jobs, respectively. These scripts accept executable names, number of processors, and optionally, PBS runtime parameters (such as walltime) as input. These inputs are then used to generate a runscript for the specific executable. A PBS batch job is then submitted for you using this script via qsub. For example, the following submits a job to run the executable a.out on 4 processors (default to 1 processor per node) and a default walltime limit of 30 minutes.

cootie% psubi a.out 4 

The following submits a job to run a.out residing in a parent directory and requires 8 processors (4 nodes with 2 processors per node) and a wall time of 45 minutes (the script has a 30-minutes preset. This example overrides that with "-l walltime=00:45:00".)

cootie% psubi ../../a.out 8 '-l walltime=00:45:00 -l nodes=4:ppn=2'