Programming for the Katana Cluster

On the Katana Cluster, the compilers (for C and FORTRAN) available are from GNU and PGI (Portland Group Inc.). All GNU compilers start with the letter g (e.g., g77 for GNU FORTRAN 77) while PGI compilers begin with "pg" (e.g., pgcc). For MPI programs, special scripts, such as mpiCC for MPI-enabled C++ codes, are available for the compilation of programs in these languages.

Running short jobs interactively on the login node Katana are permitted to facilitate and expedite users' program development and debugging. The interactive run procedures for various serial and parallel processing jobs are demonstrated.

Compilers 

(Sixty-four-bit compilation is the system default. 32-bit compilation is optional.)

Compile a program parallelized with OpenMP directives

• Examples for Portland Group Compilers

  katana:~ % pgf77 -mp -o myexec myprogram.f
  
  katana:~ % pgf95 -mp -o myexec myprogram.f90
  
  katana:~ % pgcc -mp -o myexec myprogram.c

• Examples for GNU Compilers

  katana:~ % gfortran -fopenmp -o myexec myprogram.f
  
  katana:~ % gfortran -fopenmp -o myexec myprogram.f90
  
  katana:~ % gcc -fopenmp -o myexec myprogram.c

• Additional OpenMP information ...
  • If you use gfortran for OpenMP applications, care must be exercised with OpenMP directives' placement. (more details ...)
  • OpenMP Timing comparisons of the IBM pSeries and Katana.

Running interactive OpenMP jobs

For program development and debugging purposes, short OpenMP jobs may run on Katana, the login node. These jobs are limited to 4 processors and 10 minutes of CPU time per processor.

1. Set processor count via the environment variable OMP_NUM_THREADS

   katana:~ % setenv OMP_NUM_THREADS 2

2. Run job on katana

   katana:~ % a.out

Compile a program parallelized with MPI 

Compiling an MPI program is made easy with the use of the scripts mpif77, mpif90, mpicc and mpiCC (see table above). Demonstrated below are examples of using these compiler wrapper scripts to compile FORTRAN 77/90/95, C, and C++ programs.

katana:~ % mpif77 -o myexec myprogram.f

katana:~ % mpif90 -o myexec myprogram.f90

katana:~ % mpicc -o myexec myprogram.c

katana:~ % mpiCC -o myexec myprogram.C

For MPI options, please consult, for example, the mpicc manpage. For code optimizations and other language-specific options, please refer to the PGI compilers' manpages, such as pgf95 (see table above).

Additional MPI information ...

• MPI Timings for the Katana Cluster and other SCV computer systems.

MPI_IMPLEMENTATION and MPI_COMPILER environment variables  

As an alternative to the system default, you may choose the mpich MPI implementation or select the gnu group of compilers for MPI code compilations. If you are interested in using these non-default options, please follow these instructions.

Running interactive MPI jobs

For program development and debugging purposes, short MPI jobs may run on Katana, the login node. These jobs are limited to 4 processors and 10 minutes of CPU time per processor.

katana:~ % mpirun -np 2 a.out

Compiling 32-bit executables  

The mpi** wrapper scripts build 64-bit MPI programs by default. To build 32-bit MPI programs, use the flags appropriate to the underlying compiler, as specified by the MPI_COMPILER environment variable, with the wrapper scripts:

• for the PGI compilers

  katana:~ % katana:~ % setenv MPI_COMPILER pgi
  katana:~ % mpicc -tp p7 myexample.c

• for the GNU compilers

  katana:~ % setenv MPI_COMPILER gnu
  katana:~ % mpicc -m32 myexample.c

Scratch space usage  

The scratch disks are available as temporary storage space. They are open to all SCF users and there is no preset quota for use. Files stored on any scratch disk are NOT BACKED UP and these files can only be kept in scratch for up to 10 days. Files not removed from scratch by the owner will be deleted by the system after 10 days. You can access a specific scratch disk via the pathname "/net/katana-xNN/scratch", where x is either "a", "b", or "c" and NN is a node number from 01-14.

It is important that you type the full path and not rely on the system to complete the word with tabbing.

katana:~ % cd /net/katana-a05/scratch      this works !

katana:~ % cd /net/katana-a05/scr<tab>     this fails !

• The Katana login and compute nodes each have 50 GB of scratch disk space.
• On a compute node, a reference to /scratch points to the local node's /scratch.
• Similarly, if you are on the login node, type "cd /scratch" to access its own scratch.
• If you'd like to use scratch space in a batch job, please use the scratch space of the compute nodes assigned to the job.
• If you'd like to perform I/O on scratch while running a job on the login node, the katana login node's scratch may be used.

Programming Tools  

• Debuggers
• GNU (serial) debugger (gdb)
• PGI debugger (pgdbg, how-to)
• TotalView debugger (TotalView)
• Profilers
• GNU profiler (gprof)
• PGI profiler (pgprof, how-to)