Intro

A simple example of how to get Condor to run Athena jobs.

Submitting a Simple Fastsim Job Transform

Make a file called test.cmd. This sets up the basic condor configuration that you're already familiar with. Be sure to change the executable line to the location of the executable you'll be running.

universe=vanilla
executable = /home/muell149/testarea/15.5.1/PhysicsAnalysis/AnalysisCommon/UserAnalysis/run/test.sh

arguments = /home/muell149/testarea/15.5.1/PhysicsAnalysis/AnalysisCommon/UserAnalysis/run/user08.GiaKhoriauli.mcatnlo331.005200.ttbar.EVNT.v14022001.v2.EXT0._02485.pool.root

output=/home/muell149/log.stdout
error=/home/muell149/log.stderr
log=/home/muell149/log
queue 1

Now, make a file called test.sh. In here I set the home directory environmental variable. Then I source my athena setup script and cd into the scratch directory.

#!/usr/bin/env bash
export HOME="/home/muell149/"
source /home/muell149/testarea/15.5.1/athena_setup.sh
cd $_CONDOR_SCRATCH_DIR
`which athena` /home/muell149/testarea/15.5.1/PhysicsAnalysis/AnalysisCommon/UserAnalysis/run/csc_simul_reco_trf.py /home/muell149/testarea/15.5.1/PhysicsAnalysis/AnalysisCommon/UserAnalysis/run/user08.GiaKhoriauli.mcatnlo331.005200.ttbar.EVNT.v14022001.v2.EXT0._02485.pool.root /home/muell149/testarea/15.5.1/PhysicsAnalysis/AnalysisCommon/UserAnalysis/run/AOD.pool.root 1 0 70278 ATLAS-GEO-02-01-00 1 2 QGSP_BERT jobConfig.VertexPosFastIDKiller.py NoTrackSlimming.py,FastSimulationJobTransforms/FastCaloSimAddCellsRecConfig.py NONE NONE

Notice how the arguments follow the transform script just as if executed on the command line. You must be sure the paths to any datasets are complete paths as these instructions are being interpreted by condor.

Now change the permissions on these two files as described below and submit them as shown below.

A simple job

Make a file called test.cmd. This sets up the basic condor configuration that you're already familiar with. Be sure to change the executable line to the location of the executable you'll be running.

universe=vanilla
executable = /home/koll/Athena/testarea/rel_2/PhysicsAnalysis/AnalysisCommon/UserAnalysis/run/test.sh
output=/home/koll/log.stdout
error=/home/koll/log.sterr
log=/home/koll/log
queue 1

Make a file called test.sh. In here I set the home directory environmental variable. Then I source my athena setup script and cd into the scratch directory. Notice that instead of athena or athena.py, I use `which athena.py`.

#!/usr/bin/env bash
export HOME="/home/koll/"
source /home/koll/Athena/testarea/rel_2/athena_setup.sh
cd $_CONDOR_SCRATCH_DIR
`which athena.py`  /path/to/HelloWorldOptions.py

Make sure that all paths referred to in your HelloWorldOptions.py file use absolute path names and do not assume your directory locations. For example, if you normally include a file that is in the directory that you run in, then you must adjust it to work. For example,

include("supplementalFile.py") 

should be replaced with

include("/absolute/path/to/supplementalFile.py") 

Before submitting, be sure to change the permissions on test.cmd and test.sh by doing:

chmod ++x test.sh 

and

chmod ++x test.cmd 

Now you should be able to submit this with condor_submit test.cmd and have it run successfully.

Running many jobs

Of course, making the .sh and .cmd files by hand can be a chore when you have many jobs you want to run. I made the below script in order to help automate this. The first argument it takes is a plain text file in which each line is an athena command. For example, it might look like this:

`which athena.py` /path1/to/HelloWorld1.py
`which athena.py` /path2/to/HelloWorld2.py
`which athena.py` /path3/to/MyCustomOptions.py

Note again that I use `which athena.py` instead of just athena.py. The second argument is a name that you want to give to the run. This name will be attached to the front of the names of all log files that are generated.

#---------------------------------------------------------------------
#source athena_sub.sh <list of athena commands> <name of run>

n=1
while read line
do

# Make .sh file for each command
cat > $2${n}.sh <<EOF
#!/usr/bin/env bash
export HOME="/home/koll/"
source /home/koll/Athena/testarea/rel_2/athena_setup.sh
cd \$_CONDOR_SCRATCH_DIR
${line}
EOF

# Make executable
chmod +x $2${n}.sh

# Make .cmd file for each command
cat > $2${n}.cmd << EOF
universe=vanilla
executable = `pwd`/$2${n}.sh
output=/home/koll/Athena/logs/$2${n}.stdout
error=/home/koll/Athena/logs/$2${n}.sterr
log=/home/koll/Athena/logs/$2${n}.log
queue 1
EOF

#Submit to condor
condor_submit $2cmd${n}.cmd

  ((n=n+1))
done<$1

#------------------------------------------------------------------

So, say I name the above file athena_sub.sh and am going to use it to run athena commands listed in commandList.txt that do some analysis on ttbar events. I would simply run the following in a terminal:

source athena_sub.sh commandList.txt myttbarevents

The jobs would be submitted to condor, and after they finished running I would have log files in /home/koll/Athena/logs/ such as myttbarevents1.stdout, myttbarevents2.stdout, etc.

-- JamesKoll - 22 Oct 2009