How to add a new pool node to dCache/Chimera

  • Provision node via ROCKS (or other method)
  • Install correct kernel (if not done above): currently 2.6.38-UL5/SL5 or 3.2.13-UL1/SL6
  • Configure network 'bonding' if not done above and add bonding monitoring
    • NOTE: On SL 6.x you should create /etc/modprobe.d/bonding.conf containing
      alias bond0 bonding
      . Put the options in the ifcfg-bond0 file like
      BONDING_OPTS="mode=1 arp_ip_target=192.41.230.1,10.10.1.2 arp_interval=250 primary=p1p1"
    • NOTE2: The NM_CONTROLLED option in all the ifcfg-x scripts should be set to
      no
      .
  • Update BIOS/FIRMWARE/DRIVERS if needed (use script /afs/atlas.umich.edu/hardware/Dell_BIOS_FW/update_bios_fw.sh)
  • Make sure lldpd/SL5 or lldpad/SL6 is installed
  • Verify that iptables is configured correctly to allow the various dCache services to operate
  • Verify the the /etc/grid-security directory is setup properly and includes a 'certificates' directory properly updated by a cron-task
  • Install OMSA current version and configure notification/alerting (use script /afs/atlas.umich.edu/hardware/Dell_OMSA_71/update_OMSA.sh)
  • Format pool areas as XFS and setup mounting via labels (use script /afs/atlas.umich.edu/hardware/dCache/fss_setup_filesystems*.sh)
  • Install dCache via scripts in /root/tools/dCache (See also HowToAddNewPoolsToChimera )
    • Create pool areas (currently setup RAID-6, 1 per MD1200/MD1000 shelf)
    • After pools are mounted run the script /afs/atlas.umich.edu/dcache/install/install_dcache.sh)
  • Verify the meta directories are relocated off the pools. If not run the /afs/atlas.umich.edu/dcache/install/relocate_meta.sh script which will check/move all meta directories:
    relocate_meta.sh 
    . Note this step is run as part of install_dcache.sh.
  • Install needed ssh-keys in /root/.ssh/authorized_keys (to allow access from t-head01, head01 and head02)
  • Install 'monit' and configure for dCache (use script /afs/atlas.umich.edu/tools/config/monit/install_monit_dcache.sh)
  • Update head01 configuration to include this new node and pools
  • Update ccc_config.py to include the pools from this node (on t-head01 in /root/ccc)
  • Update head02 list in /pnfs namespace inside 'dcache.conf' to include dcap doors on new node
  • Update DNS round-robin 'dcap.aglt2.org' to include this new node
  • Create new AFS volume 'dcache_hostname' and mount it in /afs/atlas.umich.edu/dcache/hostname.
  • Add ACL to new directory:
    fs setacl /afs/atlas.umich.edu/dcache/hostname aglt2fs rlidwka
  • Add node public and private IPs to AFS pts group 'aglt2fs'
    • Create entry for IP:
      pts createuser IP
    • Add new entry to aglt2fs group:
      pts adduser IP aglt2fs
    • Verify entry:
      pts membership aglt2fs
  • Setup dCache metadata 'rsync' to AFS via crontab entry (Now done as part of install_dcache.sh)
    15 * * * * rsync -rlptDzq /var/dcache/meta /afs/atlas.umich.edu/dcache/`hostname -s`/
  • Install 'iostat-cacti' setup (/afs/atlas.umich.edu/dcache/install/install_iostat_cacti.sh ). You may need to do 'service crond reload' after.
  • Update (or add to Cacti) plots for this new node
    • Add the 10GE NIC histogram to the appropriate in-out Network summary graphs for dCache
    • Make sure the iostat graphs are working (after installing the iostat-cacti above)
  • Add this node to the 'monit' monitoring on head01.aglt2.org
-- ShawnMcKee - 02 Dec 2010
Topic revision: r13 - 30 Oct 2012, ShawnMcKee
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback