Vendor:  TeamHPC #79290
UD Property Tag:  143988

The five power distribution units (PDUs) can be controlled over the cluster’s interconnect network. The following schematic shows what is attached to each outlet of each unit:

Access to a particular PDU is via telnet; the names at the left of each block in the schematic map to the appropriate IP address from the head node. The administrative username is root with the same password as on the head node itself.

There are two variants each of MPICH and MPICH2 installed on the cluster, compiled with the standard GNU tools (gcc, g++, and g77) and the Portland Suite (pgcc, pgCC, pgf77, and pgf90). The version you use depends on whether you want to compile your code with the commercial package (Portland Suite) or the standard set. Note that the Intel development tools are also installed on this cluster but for the moment I’ve not built an Intel-based MPICH/MPICH2 package on the cluster. If anyone needs one, let me know.

By default, your environment will be setup to use the GNU-based MPICH variant. This means that commands like mpicc and mpif77 can be used on the command line and by default map to /opt/mpich.gcc/bin/mpicc and /opt/mpich.gcc/bin/mpif77, respectively. If you want to use the MPICH variant for the Portland Suite, issue the following command:

% setup-mpich pgi
You can switch to the GNU MPICH2 variant using:
% setup-mpich 2-gcc

or see all the possible variants with:

% setup-mpich --list
2-gcc
2-pgi
default
gcc
pgi

Always remember which development tool set you used to compile your MPI program, since you’ll need to use the appropriate queue submission script when you run it! One easy way to accomplish this is to use a Makefile to build your program, adding the following variable definitions at the beginning of the file:

#
# Setup the MPICH2 variant we want:
#
MPIBASE       = /opt/mpich2.gcc

# Compilers and flags:
CC            = $(MPIBASE)/bin/mpicc
CFLAGS        = -O
FC            = $(MPIBASE)/bin/mpif77
F77           = $(MPIBASE)/bin/mpif77
F90           = $(MPIBASE)/bin/mpif90
FFLAGS        = -O

# Other utils:
AR            = ar -rsv
And then use rules that look like:
derivs.o: derivs.c
        $(CC) -c $(CFLAGS) derivs.c

walk.o: walk.f
        $(FC) -c $(FFLAGS) walk.f

Obviously, if you want to switch the development tool set you can then accomplish it very easily by modifying the MPIBASE and adding/removing elements to/from the CFLAGS and FFLAGS variables. Always recompile all of the code in a project after you switch the development tool set!

To run your MPI code, you’ll need to write a queue script to submit to the GridEngine. Not to worry: I’ve put template queue scripts in /opt/Templates. You need only copy the appropriate script to one of your own directories and change the number of processors you want to request and provide your own command line:

[frey@borelli ~]% cp /opt/Templates/mpich2-gcc-template.sh mpitest-run.sh
[frey@borelli ~]% more mpitest-run.sh
#!/bin/bash
#$ -cwd
#$ -pe mpich2 <nproc>
#$ -j y
#$ -S /bin/bash
#
# Use this qsub script to run programs compiled with the
# gcc variant of mpich2 -- /opt/mpich2.gcc
#
# The number of processors you desire is indicated by replacing
# <nproc> above.
#
# Change the value assigned to USER_EXE to reflect the MPI
# program's command line invocation, e.g.
#
#  USER_EXE="/home/frey/calcpi --showhosts --digits=10"
#

USER_EXE="/home/frey/Developer/mpitest/mpitest"

# Don't change these values!!
MPIEXEC=/opt/mpich2.gcc/bin/mpiexec
export MPIEXEC_RSH=rsh
HOST=`hostname`

# Begin the run by printing some basic info and then
# invoke mpiexec:
echo "GridEngine parameters:"
echo "  nproc       = $NSLOTS"
echo "  machines    = ${HOST}:${TMPDIR}/machines"
echo "  executable  = $USER_EXE"
echo "-- begin MPICH2 run --"
$MPIEXEC -rsh -nopm -n $NSLOTS -machinefile $TMPDIR/machines $USER_EXE
echo "-- end MPICH2 run --"

Be sure to use the canonical path to your MPI program (a’la the USER_EXE in the template). Assuming I edited this file to request 20 processors, the initial few lines would then look like:

#!/bin/bash
#$ -cwd
#$ -pe mpich2 20
#$ -j y
#$ -S /bin/bash

You can also request a range of processors, and as long as the cluster has any number of free processors in that range the job can run:

#!/bin/bash
#$ -cwd
#$ -pe mpich2 10-20
#$ -j y
#$ -S /bin/bash

Hint: Leave out the second number – a’la just “10-” – and the total number of processors in the cluster is the upper bound to the range.

Once you’ve created a queue script you submit it to GridEngine. The queue system then determines what nodes in the cluster are available for your job; if not enough are free, then your job will wait in the queue until that condition changes:

[frey@borelli ~]% qsub mpitest-run.sh
Your job 39 ("mpitest-run.sh") has been submitted

You can monitor the status of all the jobs both waiting in the queue and those that have migrated onto the cluster and are running using the qjobs command:

[frey@borelli ~]% qjobs
===============================================================================
 JID   Owner      State   Submitted as
===============================================================================
39    frey       running mpitest-run.sh
40    frey       running walk2d-01-test.sh
       :
43    frey       queued  walk2d-02-test.sh
===============================================================================
5 jobs total.

There is also a web page that displays GridEngine jobs that are running on the cluster.

As your job runs, anything it writes to the standard output streams (stdin and stderr) will be collected in a file in the same directory as your queue script; this file will have the same name as the queue script but with a “.o<JID>” extension. The mpitest-run.sh example above received a JID of 39, so its output would be found in mpitest-run.sh.o39:

GridEngine parameters:
  nproc       = 80
  machines    = compute15:/scratch/41.1.all.q/machines
  executable  = /home/frey/Developer/mpitest/mpitest
-- begin MPICH2 run --
B[80] = { 0 , 1 , 4 , 9 , 16 , 25 , 36 , 49 , 64 , 81 , 100 , 121 , 144 , 169 , 196 , 225 , 256 , 28
9 , 324 , 361 , 400 , 441 , 484 , 529 , 576 , 625 , 676 , 729 , 784 , 841 , 900 , 961 , 1024 , 1089 
, 1156 , 1225 , 1296 , 1369 , 1444 , 1521 , 1600 , 1681 , 1764 , 1849 , 1936 , 2025 , 2116 , 2209 , 
2304 , 2401 , 2500 , 2601 , 2704 , 2809 , 2916 , 3025 , 3136 , 3249 , 3364 , 3481 , 3600 , 3721 , 38
44 , 3969 , 4096 , 4225 , 4356 , 4489 , 4624 , 4761 , 4900 , 5041 , 5184 , 5329 , 5476 , 5625 , 5776
 , 5929 , 6084 , 6241}
-- end MPICH2 run --