Revision 1 as of 2012-10-17 04:00:16

Clear message
Locked History Actions

InfolabClusterComputeHowtoJobArray

Here is a hypothetical... you have one program that is not multi-threaded nor aware of multiple cores. You have to run that program about a thousand times with different input parameters and different input data. And luckily... the results of a single run are independent of all the other results. This HOWTO describes how one might run such a scenario on the Infolab Compute Cluster.

We presume that you know your qsub basics. If that is not the case, please see InfolabClusterComputeHowtoSingle and InfolabClusterComputeHowtoVariables first.

The submission script

We'll tackle this one the other way around. So let's create our submission script first.

Now that we got the program up and running let's log into the submission node ilhead1 and prepare a submission script. You can download the script here: SingleCore.qsub.sh

   1 #!/bin/bash
   2 #PBS -N JobArray
   3 #PBS -l nodes=1:ppn=1
   4 #PBS -l walltime=00:01:00
   5 
   6 /usr/bin/python2.7 $HOME/tutorial/JobArray/JobArray.py $PBS_ARRAYID 

The only special thing here is that we'll be passing the array id (so the number of the job in the array) to our Python script.

The program

We are going to use just a simple Python script as our main program for this HOWTO. You can download the script here SingleCore.py.

   1 #!/usr/bin/python2.7
   2 
   3 import socket, datetime, time, getpass
   4 
   5 start = datetime.datetime.now()
   6 hostname = socket.gethostname().split('.')[0]
   7 username = getpass.getuser()
   8 time.sleep(10)
   9 end = datetime.datetime.now()
  10 
  11 dfmt = "%Y-%m-%d %H:%M:%S"
  12 print "Started: %s Finished: %s Host: %s User: %s" % (start.strftime(dfmt), end.strftime(dfmt), hostname, username)

The script starts, records the current time, figures out the hostname it is running on and the username it is running as. Then it sleeps for 10 seconds (so we at least have some impact on the cluster), records the time again and prints out a string that may look a little something like this:

Started: 2012-10-16 15:56:55 Finished: 2012-10-16 15:57:05 Host: ilhead1 User: akrevl

It's a good idea to check if the program will run on the target platform. It doesn't make much difference for a Python script, but if you were running a C binary it's worth checking if it runs on the AMD platform. This is where ild1 comes in. The development node ild1 is set up in the same way as the cluster nodes are. So let's test the script on ild1:

/usr/bin/python2.7 /afs/cs.stanford.edu/u/akrevl/tutorial/SingleCore/SingleCore.py

Note that we are using a full path both to the python executable and to the Python script. The result is as expected:

Started: 2012-10-16 17:04:44 Finished: 2012-10-16 17:04:54 Host: ild1 User: akrevl

Submit the job

Nothing left to do but submit the job to the cluster with qsub:

qsub -V /afs/cs.stanford.edu/u/akrevl/tutorial/SingleCore/SingleCore.qsub.sh

If we submitted the job successfully, the resource manager should reply with with the ID of the job and the name of the headnode:

4651.ilhead1.stanford.edu

Check on the job

While the job is running, you can check on it with qstat and showq commands. Please be patient with the showq command as it tends to return timeouts when a lot of jobs are in the queue.

~/ $ qstat
Job id                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
4651.ilhead1               SingleCoreJob    akrevl                 0 R test

~/ $ showq
ACTIVE JOBS--------------------
JOBNAME            USERNAME      STATE  PROC   REMAINING            STARTTIME

4651                 akrevl    Running     1    00:01:00  Tue Oct 16 17:19:29

     1 Active Job        1 of  896 Processors Active (0.11%)
                         1 of   28 Nodes Active      (3.57%)

The results

Once the job is finished it should deposit two files into the directory we ran qsub from:

  • SingleCoreJob.e4651: copy of the standard error stream

  • SingleCoreJob.o4651: copy of the standard output stream

Let's see what does our directory contain:

~/ $ ls /afs/cs.stanford.edu/u/akrevl/tutorial/SingleCore
SingleCoreJob.e4651 
SingleCoreJob.o4651
SingleCore.py
SingleCore.qsub.sh

Now let's see the content of those files:

~/ $ cat SingleCoreJob.e4651
~/ $ cat SingleCoreJob.o4651
Started: 2012-10-16 17:19:29 Finished: 2012-10-16 17:19:39 Host: iln28 User: akrevl

Excellent, the standard error file is empty and the standard output tells us that our job ran on node iln28 and it finished (as expeted) in 10 seconds.