The desire was to set up a local installation of the dq2 tools, eg, dq2_get and dq2_ls. Previous setups used by the UM group did not work for a variety of reasons.

Using the local dq2* commands

Below are the complete details of the installation process. In the end, the user must do at most 3 things.

These directions updated on February 11, 2013.

There are two ways to get to the dq2 tools, using either the CVMFS installation, or our own, local installation.

  • CVMFS access
    • source /usr/local/bin/setup/
    • localSetupDQ2Client
  • Local installation access
    • source $DQ2_ENV/

It is now possible to do voms-proxy-init -voms atlas (you must have a grid proxy already set up to do this) and proceed to using the dq2 tools.

Note, for some of these tools, it may be necessary to do the following:


By default, the LOCAL dq2- tools use our own AGLT2 site as their file source. This can be over-ridden using the argument "-L some_other_site" on the tool, eg, "dq2-ls -L BNLDISK etc". Possible values for the -L argument can be obtained using the "dq2-sources" command. This is not recommended though as the copy command selected depends on both the source and destination sites, and use of "-L" implies the specified locale is also your local locale.

The environment variable above is set up only on our interactive machines (, and on personal workstations. The commands are interactive in nature, so no other machine has been enabled in this way.

Examples using the local dq2* commands

  • dq2-ls M5.0029578.Default.L1TT-b00001000.ESD.v13003013
    List all files at AGLT2 in this dataset
  • dq2-ls -L BNL M5.0029578.Default.L1TT-b00001000.ESD.v13003013
    List all files in this dataset assuming you are at BNL
  • dq2-ls -r trig1_misal1_mc12.006535.McAtNloHww200.recon.AOD.v12000605_tid018601
    List sites that have this dataset, and indicate if the dataset is complete or incomplete
  • dq2-get trig1_misal1_mc12.006535.McAtNloHww200.recon.AOD.v12000605_tid018601
    Copy all files in the dataset to the current local directory tree, without specifying the source for the files
  • dq2-get -s AGLT2_SRM trig1_misal1_mc12.006535.McAtNloHww200.recon.AOD.v12000605_tid018601
    Copy all files in the dataset to the current local directory tree, specifying the source as our own dCache

Special case, BNL error

An error in copying files from BNL has been found. The symptom is that copies from BNL to AGLT2 can fail, with the following error recorded, even though dq2-ls has verified the file actually exists at BNL:
Starting transfer: lcg-cp -v --vo atlas -b -D srmv2
[Errno 2] No such file or directory:

Note the mixture of "srmv2" with "managerv1?SFN" in the erring copy command. The solution (hot fix) is to use the following copy syntax, specifying the -p switch:
  • dq2-get -p lcg1 datasetname

NOTICE: Details of the setup

Unless you are really interested in all the details, there is no need to read further on this page

The Installation Procedure

Update history

WLCG-client updated October 19, 2009, with following inclusions:
[ball@umt3int01:~]$ lcg-cp --version

WLCG-client updated January 11, 2010. Inclusions unchanged.
WLCG-client updated May 25, 2010. Inclusions unchanged.
  • On 8/10 it is found that the adler32 command of this install is broken.
    • eg, adler32 /tmp/bob.dat
    • Revert to 1/11/2010 version.
  • On 8/23, investigation finds this ticket that precisely describes the issue, with the solution:
    • cd install_location/srm-client-fermi/lib/globus
    • ln -s cog-jglobus-1.7.0.jar cog-jglobus.jar
  • Fixed link in installed wlcg-client, but not yet switched back to using it.
WLCG-client updated Aug 27, 2010.

WLCG-client updated October 20, 2011. Inclusions remain unchanged, but WLCG version is now 1.8

dq2 tools updated as follows:
Updated to 0.1.28 version on June 2, 2009
Updated to 0.1.29 version on September 22, 2009
Updated to 0.1.31 version on October 19, 2009
Updated to 0.1.33_rc2 version on January 11, 2010
Updated to 0.1.33 version on March 1, 2010
Updated to 0.1.35_rc2 version on June 1, 2010
Updated to 0.1.36_rc4 version on Jan 17, 2011
Updated to 0.1.37_rc1 version on April 7, 2011
Version 0.1.37_rc6 downloaded October 20, 2011, but never set as default
Updated to 2.3.0 version on June 5, 2012

Current procedure -- December 12, 2008

The underlying software is no longer the LCG/UiTarInstall, but is the WLCG-client package installed via pacman. All software is installed on replicated afs volumes, typically mounted read-only. Installation is performed from, sourcing the pacman setup:
source /afs/

Make sure that "opt" afs volume is mounted read/write (below).

# Set up the wlcg-client package.  
# Does not need update with each new DQ2Client.
cd /afs/
mkdir WLCG-client
cd WLCG-client
## Changes with Condor 7.6 to CONDOR locations
## export VDTSETUP_CONDOR_LOCATION=/opt/condor
## export VDTSETUP_CONDOR_CONFIG=/opt/condor/etc/condor_config
export VDTSETUP_CONDOR_CONFIG=/etc/condor/condor_config
pacman -get
cd globus
ln -s /afs/ TRUSTED_CA
cd share
ln -s /afs/ certificates

# New with WLCG version 1.8, October, 2011
# INFO: To enable the pythonloader to search for a python satisfying DQ2 requirements 
# add this symbolic link:
cd atlasosgcompat/bin/
ln -s pythonloader python

# Furthermore, the dq2 tools require a 32-bit python image.  
#   NOTE: This is no longer true as of 0.1.28 version with its new WLCG-client
# Does not need update with each new DQ2Client.
cd /afs/
pacman -get

# Directions from Mario Lassnig for dq2-*, with our local installation modifications:
cd /afs/
mkdir #new version directory#, for example, mkdir 0.1.25
cd #new version directory#
pacman -trust-all-caches -allow tar-overwrite -get

(NOTE: AGLT2_UM deprecated, 10/2010, change to AGLT2_DATADISK)
edit and add this line at the start of the pre-setup section:
. /afs/
Add these 3 lines prior to the post-setup:

edit setup.csh and add this line at the start of the pre-setup section:
source /afs/
Add these 2 lines prior to the post-setup:

No other setup modification is necessary, except to define the default version.
Use the installed directory version in these commands.

cd ..
rm; ln -s 0.1.25/
rm setup.csh; ln -s 0.1.25/setup.csh setup.csh

Current procedure minus 1 -- September 1, 2008

Installation is in the afs directory /afs/

Directions here, picked up 2 files from here, namely glite-UI-3.1.18-0.tar.gz and glite-UI-3.1.18-0-external.tar.gz (September 2008 issue). Then follow the tarball installation directions, with the following modifications.

The following files must be created:
  • touch glite/yaim/etc/groups.conf
  • touch glite/yaim/etc/users.conf
  • touch glite/yaim/etc/wn-list.conf
The site-info.def must be modified to fit our situation. Following the unpacking of the 2 tarballs, an example can be found in .../glite/yaim/examples/siteinfo/site-info.def . Copy this. The following diff output shows the changes made at AGLT2.

[umopt1:gLite]# pwd
[umopt1:gLite]# diff glite/yaim/examples/siteinfo/site-info.def site-info.def
> ## INSTALL_ROOT=/opt
> FUNCTIONS_DIR=${INSTALL_ROOT}/glite/yaim/functions
< WN_LIST=/opt/glite/yaim/etc/wn-list.conf
< USERS_CONF=/opt/glite/yaim/etc/users.conf
< GROUPS_CONF=/opt/glite/yaim/etc/groups.conf
< FUNCTIONS_DIR=/opt/glite/yaim/functions
> ## Modified from /opt/glite/yaim to /afs/
> WN_LIST=${INSTALL_ROOT}/glite/yaim/etc/wn-list.conf
> USERS_CONF=${INSTALL_ROOT}/glite/yaim/etc/users.conf
> GROUPS_CONF=${INSTALL_ROOT}/glite/yaim/etc/groups.conf
> FUNCTIONS_DIR=${INSTALL_ROOT}/glite/yaim/functions
< JAVA_LOCATION="/usr/java/j2sdk1.4.2_12"
> ## JAVA_LOCATION="/usr/java/j2sdk1.4.2_12"
> JAVA_LOCATION="/usr/java/jdk1.5.0_10"
< CRON_DIR=/etc/cron.d
> # CRON_DIR=/etc/cron.d
> CRON_DIR=/dev/null
> ## GLOBUS_TCP_PORT_RANGE="10000,60000"
< SITE_NAME=my-site-name
< SITE_LOC="City, Country"
< SITE_LAT=0.0 # -90 to 90 degrees
< SITE_LONG=0.0 # -180 to 180 degrees
> SITE_LOC="Ann Arbor, Michigan, USA"
> SITE_LAT=45.0 # -90 to 90 degrees
> SITE_LONG=75.0 # -180 to 180 degrees
< SITE_SUPPORT_SITE="my-bigger-site.their_domain"
> ## CE_HOST=my-ce.$MY_DOMAIN
> ## RB_HOST=my-rb.$MY_DOMAIN
> ## PX_HOST=my-px.$MY_DOMAIN

Next, install the dq2 enduser tools.

# source /opt/pacman-3.24/

# pacman -trust-all-caches -allow tar-overwrite -get and setup.csh that are created here must be modified in two ways:
  • Add the initial value for the data source, ie, export DQ2_LOCAL_SITE_ID=AGLT2_UM
    • Values for this can be obtained by running the command "dq2-sources"
  • Add the 32-bit python image directory to the PATH, ie, /afs/

Early version installs -- retained only for historical reasons

The dq2 tools required a local installation of the LCG tools. The also require a java sdk installation. Both are installed in afs, in the newly created volumes
  • /afs/
  • /afs/
So, first things first, install the java sdk in afs. Following java installation directions we pick up the file "j2sdk-1_4_2_13-linux-i586.bin"
  • chmod +x j2sdk-1_4_2_13-linux-i586.bin
  • cd /afs/
  • execute the binary
This needs ~165MB of disk

If /usr/lib/ does not exist, then
  • yum install compat-openldap.[i386|x86_64]

Now, following the directions here, the most recent versions of the two local tar files were copied to UMich from:

These were then unpacked in the afs directory created for this purpose.

  • cd /afs/
  • mkdir LCG
  • cd LCG
  • tar -zxf [location]/gliteUI_WN-2006-10-17.tar.gz
  • tar -xzf [location]/gliteUI_WN-userdeps-2006-10-17.tar.gz

Now, some setup. The following files must be created:
  • mkdir glite/yaim/etc
  • touch glite/yaim/etc/groups.conf
  • touch glite/yaim/etc/users.conf
  • touch glite/yaim/etc/wn-list.conf
Create the file site-info.def . As we have configured it, only the beginning of this file needs modification from the template version found at ./glite/yaim/examples/site-info.def

# YAIM example site configuration file - adapt it to your site!

# Node names
# Note: - SE_HOST -->  Removed, see CLASSIC_HOST, DCACHE_ADMIN, DPM_HOST below
#       - REG_HOST --> There is only 1 central registry for the time being.

# VO-BOX - Set this if you are building a VO-BOX 

# Set this to "yes" your site provides an X509toKERBEROS Authentication Server 
# Only for sites with Experiment Software Area under AFS 

# LFC - Set these if you are installing an LFC

# These are set to default to using the standard database on the same hosts
# as the LFC daemon is on

# If you use a DNS alias in front of your LFC, specify it here

# All catalogues are local unless you add a VO to 
# LFC_CENTRAL, in which case that will be central

# If you want to limit the VOs your LFC serves, add the locals here

# TORQUE - Change this if your torque server is not on the CE
# it's ingored for other batch systems

# These variables tell YAIM where to find additional configuration files.

# Repository settings 
LCG_REPOSITORY="'rpm rhel30 externals Release3.0 updates'"
CA_REPOSITORY="rpm LCG-CAs/current production"
REPOSITORY_TYPE="yum"  #"apt" # or "yum"

# For the relocatable (tarball) distribution, ensure
# that INSTALL_ROOT is set correctly

# You will probably want to change these too for the relocatable dist
#  "/usr/java/j2re-1.4.2_06"

# Set this to '/dev/null' or some other dir if you want
# to turn off yaim installation of cron jobs

# Set this to your prefered and firewall allowed port range

# Choose a good password ! And be sure that this file cannot be read by
# any grid job !

# GRID_TRUSTED_BROKERS: DNs of services (RBs) allowed to renew/retrives 
# credentials from/at the myproxy server. Put single quotes around each trusted DN !!! 

#'broker one'
#'broker two'

# The RB now uses the DLI by default; set VOs here which should use RLS
RB_RLS="" # "atlas cms"

# Space separated list of ldap servers in edg-mkgridmap.conf which authenticate users.
# Ex.: GRIDMAP_AUTH="ldap://,o=registrar,dc=lcg,dc=org ldap://xyz"

# GridIce server host name (usually run on the MON node).

# Site-wide settings
SITE_LAT=45.0 # -90 to 90 degrees
SITE_LONG=75.0 # -180 to 180 degrees

##### The balance of the template file beyond this point remains un-modified #####

Now, install LCG:
  • ./glite/yaim/scripts/configure_node ./site-info.def TAR_UI
A bug exists in srmcp. This should be fixed now, as indicated by the following diff output.
     edits to /public/LCG/d-cache/srm/sbin/srm
         <     SRM_CONFIG_PARENT=dirname $SRM_CONFIG
         >     SRM_CONFIG_PARENT=`dirname $SRM_CONFIG`

Now, nearly finished, install the dq2 commands by following the directions at These are installed in a directory parallel to the LCG directory created above
  • cd /afs/
  • mkdir dq2_enduser
  • cd dq2_enduser
A localized setup script must be created to go along with this, based on the .CERN and .BNL examples copied in here. This script will reference the java installation above, and also set the actual copy command to be used. I have not been able to get srmcp to work, so instead we set an alternate command given by
  • export DQ2_COPY_COMMAND='lcg-cp -v --vo atlas'
The complete and setup.csh.UM are shown here.
----------------------------- --------------------------------
# DQ2 server 
export DQ2_URL_SERVER=''

# local site ID
# e.g., for CERN
# export DQ2_LOCAL_ID=CERN
# if your site doesn't deploy a DQ2 site service
# export DQ2_LOCAL_ID=

export DQ2_LOCAL_ID=

# access protocol to local storage
# for CASTOR
# export DQ2_LOCAL_PROTOCOL=rfio
# for dCache
# export DQ2_LOCAL_PROTOCOL=dcap
# for normal disk storage
# export DQ2_LOCAL_PROTOCOL=unix

export DQ2_LOCAL_PROTOCOL=unix

# root directory of local storage.
# e.g.,
# export DQ2_STORAGE_ROOT=/castor
# export DQ2_STORAGE_ROOT=/pnfs
# if you don't have special mount point for storage
# export DQ2_STORAGE_ROOT=''

export DQ2_STORAGE_ROOT=''

# local SRM host
# e.g.
# export DQ2_SRM_HOST=srm://
# if your site doesn't deploy an SRM server
# export DQ2_SRM_HOST=''

export DQ2_SRM_HOST=''

# local GSIFTP host
# e.g.
# export DQ2_GSIFTP_HOST=gsi
# if your site doesn't deploy a GSIFTP server
# export DQ2_GSIFTP_HOST=''

export DQ2_GSIFTP_HOST=''

export DQ2_COPY_COMMAND='lcg-cp -v --vo atlas'

export DQ2_LFC_HOME=/grid/atlas

# java
export JAVA_HOME=/afs/

if [ $?JAVA_HOME ] ; then
        export PATH=$JAVA_HOME/bin:$PATH

export PATH=/afs/$PATH
#export PATH=/afs/$PATH

# prevent the wild-card expansion
_dq2_ls_helper () {
  unalias dq2_ls 
  dq2_ls $*
  set +f
  alias dq2_ls='set -f; _dq2_ls_helper'
alias dq2_ls='set -f; _dq2_ls_helper'

_dq2_cr_helper () {
  unalias dq2_cr 
  dq2_cr $*
  set +f
  alias dq2_cr='set -f; _dq2_cr_helper'
alias dq2_cr='set -f; _dq2_cr_helper'

_dq2_get_helper () {
  unalias dq2_get 
  dq2_get $*
  set +f
  alias dq2_get='set -f; _dq2_get_helper'
alias dq2_get='set -f; _dq2_get_helper'

----------------------------- setup.csh.UM -------------------------------
if (! $?LCG_LOCATION ) then
        echo 'ERROR : setup Grid first!'

# DQ2 server 
setenv DQ2_URL_SERVER ''

# local site ID
setenv DQ2_LOCAL_ID ""

# access protocol to local storage
setenv DQ2_LOCAL_PROTOCOL unix

# root directory of local storage
setenv DQ2_STORAGE_ROOT ""

# local SRM host
setenv DQ2_SRM_HOST ""

# local GSIFTP host
setenv DQ2_GSIFTP_HOST ""

# use SRM for all data transfer (True or False)
setenv DQ2_USE_SRM True

setenv DQ2_LFC_HOME /grid/atlas

setenv DQ2_COPY_COMMAND 'lcg-cp -v --vo atlas'

# setup dq2_*
setenv PATH /afs/$PATH

# setup python 2.4
#setenv PATH /afs/$PATH

setenv JAVA_HOME /afs/

if ( $?JAVA_HOME ) then
        setenv PATH $JAVA_HOME/bin:$PATH

# prevent the wild-card expansion
alias dq2_ls 'set noglob; \dq2_ls \!*; unset noglob'
alias dq2_cr 'set noglob; \dq2_cr \!*; unset noglob'
alias dq2_get 'set noglob; \dq2_get \!*; unset noglob'

Mounting afs volumes as read-write

cd /afs/
fs lsmount opt
# If shows as #opt, then it is ro, otherwise skip these steps
fs rmmount /afs/
fs mkmount /afs/ opt -rw
vos release root.cell
fs checkvolumes

# Moving on, opt.localDQ2LCG is mounted at localDQ2LCG.  Make it rw as well
cd opt
fs lsmount localDQ2LCG
# If shows as #opt.localDQ2LCG, then it is ro, otherwise skip these steps
fs rmmount /afs/
fs mkmount /afs/ opt.localDQ2LCG -rw
vos release opt.localDQ2LCG
fs checkvolumes

Mounting afs volumes as read-only

# Reverse order, opt.localDQ2LCG is mounted at localDQ2LCG.  Make it ro first
cd opt
fs lsmount localDQ2LCG
# If shows as %opt.localDQ2LCG, then it is rw, otherwise skip these steps
fs rmmount /afs/
fs mkmount /afs/ opt.localDQ2LCG
vos release opt.localDQ2LCG
fs checkvolumes

# Now do the parent opt directory
cd /afs/
fs lsmount opt
# If shows as %opt, then it is rw, otherwise skip these steps
vos release opt
fs rmmount /afs/
fs mkmount /afs/ opt
vos release root.cell
fs checkvolumes

-- BobBall - 09 Jan 2007
Topic revision: r29 - 21 Feb 2014 - 16:35:08 - BobBall

This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback