This HOWTO describes how to pass arguments to your program running on the cluster Infolab Compute Cluster. We assume that you are already familiar with the basic principles of submitting jobs to the cluster. If that is not the case please see this tutorial first: InfolabClusterComputeHowtoSingle.

The program

We are going to use the script from InfolabClusterComputeHowtoSingle that we modified just so it prints out the arguments passed to it. You can download the script here SingleCoreVariables.py.

   1 #!/usr/bin/python2.7
   2 
   3 import socket, datetime, time, getpass, sys
   4 
   5 start = datetime.datetime.now()
   6 hostname = socket.gethostname().split('.')[0]
   7 username = getpass.getuser()
   8 time.sleep(10)
   9 end = datetime.datetime.now()
  10 
  11 dfmt = "%Y-%m-%d %H:%M:%S"
  12 print "Started: %s Finished: %s Host: %s User: %s" % (start.strftime(dfmt), end.strftime(dfmt), hostname, username)
  13 print "My arguments:"
  14 print sys.argv

The script starts, records the current time, figures out the hostname it is running on and the username it is running as. Then it sleeps for 10 seconds (so we at least have some impact on the cluster), records the time again and prints out a string that may look a little something like this (if we called it with myarg1, myarg2 and myarg3, of course):

Started: 2012-10-16 18:19:47 Finished: 2012-10-16 18:19:57 Host: ilhead1 User: akrevl
My arguments:
['./SingleCoreVariables.py', 'myarg1', 'myarg2', 'myarg3']

The submission script

Static parameters

In this example we show how to run our program with parameters if these parameters are static and can be coded into the submission script. Perhaps not very useful, but here it goes anyway...

   1 #!/bin/bash
   2 #PBS -N SingleCoreVariablesJob
   3 #PBS -l nodes=1:ppn=1
   4 #PBS -l walltime=00:01:00
   5 
   6 /usr/bin/python2.7 /afs/cs.stanford.edu/u/akrevl/tutorial/SingleCore/SingleCoreVariables.py myarg1 myarg2 myarg3

Once our job completes, the output is:

~/ $ cat SingleCoreVariablesJob.o4656
Started: 2012-10-16 18:51:31 Finished: 2012-10-16 18:51:41 Host: iln28 User: akrevl
My arguments:
['/afs/cs.stanford.edu/u/akrevl/tutorial/SingleCoreVariables/SingleCoreVariables.py', 'myarg1', 'myarg2', 'myarg3']

-V: using environment variables

What if we would like to pass arguments along with the qsub command? We can try a script like this:

   1 #!/bin/bash
   2 #PBS -N SingleCoreVariablesJob
   3 #PBS -l nodes=1:ppn=1
   4 #PBS -l walltime=00:01:00
   5 
   6 /usr/bin/python2.7 /afs/cs.stanford.edu/u/akrevl/tutorial/SingleCore/SingleCoreVariables.py $1 $2 $3

However if we try to run the qsub with our three arguments it will complain that we have supplied wrong arguments (actually it just prints out the usage information, but we should take the hint).

~/ $ qsub -V $HOME/tutorial/SingleCoreVariables/SingleCoreVariables.qsub.sh myarg1 myarg2 myarg3
usage: qsub [-a date_time] [-A account_string] [-b secs]
      [-c [ none | { enabled | periodic | shutdown |
      depth=<int> | dir=<path> | interval=<minutes>}... ]
...

Instead of passing arguments by the command line, we can pass them as environment variables. Let's export our arguments first:

   1 ~/ $ export qsubarg1="myarg1"
   2 ~/ $ export qsubarg2="myarg2"
   3 ~/ $ export qsubarg3="myarg3"

Now let's make the adjustments to the submission script. We need to use qsubargX ((you can download the script here SingleCoreVariables.qsub.sh)):

   1 #!/bin/bash
   2 #PBS -N SingleCoreVariablesJob
   3 #PBS -l nodes=1:ppn=1
   4 #PBS -l walltime=00:01:00
   5 
   6 /usr/bin/python2.7 /afs/cs.stanford.edu/u/akrevl/tutorial/SingleCoreVariables/SingleCoreVariables.py $qsubarg1 $qsubarg2 $qsubarg3

Let's try and submit this to the cluster (do not forget the -V switch):

qsub -V /afs/cs.stanford.edu/u/akrevl/tutorial/SingleCoreVariables/SingleCoreVariables.qsub.sh

qsub made sure that all the environment variables were passed to the execution node and our program ran with the provided arguments.

~/ $ cat SingleCoreVariablesJob.o4657
Started: 2012-10-16 19:21:13 Finished: 2012-10-16 19:21:23 Host: iln28 User: akrevl
My arguments:
['/afs/cs.stanford.edu/u/akrevl/tutorial/SingleCoreVariables/SingleCoreVariables.py', 'myarg1', 'myarg2', 'myarg3']

-v: listing the variables

Another way of passing the arguments to qsub is to just list them as key=value pairs. Let's keep the submission script we used in the previous example (you can download the script here SingleCoreVariables.qsub.sh):

   1 #!/bin/bash
   2 #PBS -N SingleCoreVariablesJob
   3 #PBS -l nodes=1:ppn=1
   4 #PBS -l walltime=00:01:00
   5 
   6 /usr/bin/python2.7 /afs/cs.stanford.edu/u/akrevl/tutorial/SingleCoreVariables/SingleCoreVariables.py $qsubarg1 $qsubarg2 $qsubarg3

We can list the argument values as part of the qsub command using the -v switch (we could actually omit the -V switch, but we just got used to it, so why bother):

qsub -V -v qsubarg1="myarg1",qsubarg2="myarg2",qsubarg3="myarg3" /afs/cs.stanford.edu/u/akrevl/tutorial/SingleCoreVariables/SingleCoreVariables.qsub.sh

qsub made sure that the specified arguments were available on the execution node and our program used them as you can see in the following listing:

~/ $ cat SingleCoreVariablesJob.o4658
Started: 2012-10-16 19:28:55 Finished: 2012-10-16 19:29:05 Host: iln28 User: akrevl
My arguments:
['/afs/cs.stanford.edu/u/akrevl/tutorial/SingleCoreVariables/SingleCoreVariables.py', 'myarg1', 'myarg2', 'myarg3']