How to Copy data to dcache

we can use dccp , srmcp or globus-url-copy utilities to copy data to dcache, but dccp dosnt support recursive, srmcp will finally call globus-url-copy as the transfer tool.. so the best tool for this is globus-url-copy.

For example.. we need to copy data from /afs/atlas.umich.edu/umrocks01 to dcache whose destination directory is /pnfs/aglt2.org/data/umrocks01. we could follow the following steps:

1. create the destination directory as root on head02 which runs pnfs server..

head02$mkdir /pnfs/aglt2.org/data/umrocks01

2. create the subdirectories of the destination directory (umrocks01)

I use a scripts to walk through the source directory /afs/atlas.umich.edu/umrocks01 , and create the subdirectories in the destination directory /pnfs/aglt2.org/data/umrocks01. note: though globus-url-copy supports -cd which means it could create the destination directories as the recursion , but it often get confused and cause errors.. better to create the destination directories seperately.

the scripts :

#!/usr/bin/perl
my $destdir="/pnfs/aglt2.org/data/umrocks04-1";
my $dir="/afs/atlas.umich.edu/umrocks04";
searchdir($dir);
sub searchdir
{
        local $path=$_[0];
        #print "searching dir ".$path."....\n";
        opendir(DIR,$path)||die "can not open dir ".$path." \n";
        local @entry=readdir(DIR);
        local $tmp_destdir;
        close DIR;
        foreach(@entry)
        {
                #print "\$_=".$_."\n";
              
              
                local $file=$path."/".$_;
                if((-d $file)and!($_ eq '.')and !($_ eq '..'))
                {
                        if($file=~/$dir(\S*)/)
                        {
                                $tmp_destdir=$destdir.$1;
                        }
                        system("mkdir -p $tmp_destdir");
                        system("chmod 777 $tmp_destdir");
                        searchdir($file);
                }

        }

}

3. run globus-url-copy

nohup globus-url-copy   -r -c  file:////afs/atlas.umich.edu/umrocks04/ gsiftp://umfs08.aglt2.org:2811/pnfs/aglt2.org/data/umrocks04-1/&

globus-url-copy help infomation:
OPTIONS
  -help | -usage
       Print help
  -version
       Print the version of this program
  -versions
       Print the versions of all modules that this program uses
  -c | -continue-on-error
       Do not die after any errors.  By default, program will exit after
       most errors.
  -a | -ascii
       Convert the file to/from ASCII format to/from local file format
  -b | -binary
       Do not apply any conversion to the files. *default*
  -f <filename>
       Read a list of url pairs from filename.  Each line should contain
       <sourceURL> <destURL>
       Enclose URLs with spaces in double qoutes (").
       Blank lines and lines beginning with # will be ignored.
  -cd | -create-dest
       Create destination directory if needed
  -r | -recurse
       Copy files in subdirectories
  -fast
       Recommended when using GridFTP servers. Use MODE E for all data
       transfers, including reusing data channels between list and transfer
       operations.
  -q | -quiet 
       Suppress all output for successful operation
  -v | -verbose 
       Display urls being transferred
  -vb | -verbose-perf 
       During the transfer, display the number of bytes transferred
       and the transfer rate per second.  Show urls being transferred
  -dbg | -debugftp 
       Debug ftp connections.  Prints control channel communication
       to stderr
  -rst | -restart 
       Restart failed ftp operations.
  -rst-retries <retries>
       The maximum number of times to retry the operation before giving
       up on the transfer.  Use 0 for infinite.  Default is 5.
  -rst-interval <seconds>
       The interval in seconds to wait after a failure before retrying
       the transfer.  Use 0 for an exponential backoff.  Default is 0.
  -rst-timeout <seconds>
       Maximum time after a failure to keep retrying.  Use 0 for no
       timeout.  Default is 0.
  -rp | -relative-paths
      The path portion of ftp urls will be interpereted as relative to the
      user's starting directory on the server.  By default, all paths are
      root-relative.  When this flag is set, the path portion of the ftp url
      must start with %2F if it designates a root-relative path.
  -s  <subject> | -subject <subject>
       Use this subject to match with both the source and dest servers
  -ss <subject> | -source-subject <subject>
       Use this subject to match with the source server
  -ds <subject> | -dest-subject <subject>
       Use this subject to match with the destionation server
  -tcp-bs <size> | -tcp-buffer-size <size>
       specify the size (in bytes) of the buffer to be used by the
       underlying ftp data channels
  -bs <block size> | -block-size <block size>
       specify the size (in bytes) of the buffer to be used by the
       underlying transfer methods
  -p <parallelism> | -parallel <parallelism>
       specify the number of parallel data connections should be used.
  -notpt | -no-third-party-transfers
       turn third-party transfers off (on by default)
  -nodcau | -no-data-channel-authentication
       turn off data channel authentication for ftp transfers
  -dcsafe | -data-channel-safe
       set data channel protection mode to SAFE
  -dcpriv | -data-channel-private
       set data channel protection mode to PRIVATE
  -off | -partial-offset
       offset for partial ftp file transfers, defaults to 0
  -len | -partial-length
       length for partial ftp file transfers, used only for the source url,
       defaults the full file.
  -stripe
       enable striped transfers on supported servers
  -striped-block-size | -sbs
       set layout mode and blocksize for striped transfers
       If not set, server defaults will be used.
       If set to 0, Partitioned mode will be used.
       If set to >0, Blocked mode will be used, with this as the blocksize.
  -ipv6
       use ipv6 when available (EXPERIMENTAL)

-- WenjingWu - 14 Nov 2007