How to add a new pool node to dCache/Chimera

• Provision node via ROCKS (or other method)
• Install correct kernel (if not done above): currently 2.6.38-UL5/SL5 or 3.2.13-UL1/SL6
• Configure network 'bonding' if not done above and add bonding monitoring
  • NOTE: On SL 6.x you should create /etc/modprobe.d/bonding.conf containing

    alias bond0 bonding

    . Put the options in the ifcfg-bond0 file like

    BONDING_OPTS="mode=1 arp_ip_target=192.41.230.1,10.10.1.2 arp_interval=250 primary=p1p1"

  • NOTE2: The NM_CONTROLLED option in all the ifcfg-x scripts should be set to

    no

    .
• Update BIOS/FIRMWARE/DRIVERS if needed (use script /afs/atlas.umich.edu/hardware/Dell_BIOS_FW/update_bios_fw.sh)
• Make sure lldpd/SL5 or lldpad/SL6 is installed
• Verify that iptables is configured correctly to allow the various dCache services to operate
• Verify the the /etc/grid-security directory is setup properly and includes a 'certificates' directory properly updated by a cron-task
• Install OMSA current version and configure notification/alerting (use script /afs/atlas.umich.edu/hardware/Dell_OMSA_71/update_OMSA.sh)
• Format pool areas as XFS and setup mounting via labels (use script /afs/atlas.umich.edu/hardware/dCache/fss_setup_filesystems*.sh)
• Install dCache via scripts in /root/tools/dCache (See also HowToAddNewPoolsToChimera )
  • Create pool areas (currently setup RAID-6, 1 per MD1200/MD1000 shelf)
  • After pools are mounted run the script /afs/atlas.umich.edu/dcache/install/install_dcache.sh)
• Verify the meta directories are relocated off the pools. If not run the /afs/atlas.umich.edu/dcache/install/relocate_meta.sh script which will check/move all meta directories:

  relocate_meta.sh 

  . Note this step is run as part of install_dcache.sh.
• Install needed ssh-keys in /root/.ssh/authorized_keys (to allow access from t-head01, head01 and head02)
• Install 'monit' and configure for dCache (use script /afs/atlas.umich.edu/tools/config/monit/install_monit_dcache.sh)
• Update head01 configuration to include this new node and pools
• Update ccc_config.py to include the pools from this node (on t-head01 in /root/ccc)
• Update head02 list in /pnfs namespace inside 'dcache.conf' to include dcap doors on new node
• Update DNS round-robin 'dcap.aglt2.org' to include this new node
• Create new AFS volume 'dcache_hostname' and mount it in /afs/atlas.umich.edu/dcache/hostname.
• Add ACL to new directory:

  fs setacl /afs/atlas.umich.edu/dcache/hostname aglt2fs rlidwka

• Add node public and private IPs to AFS pts group 'aglt2fs'
  • Create entry for IP:

    pts createuser IP

  • Add new entry to aglt2fs group:

    pts adduser IP aglt2fs

  • Verify entry:

    pts membership aglt2fs

• Setup dCache metadata 'rsync' to AFS via crontab entry (Now done as part of install_dcache.sh)

  15 * * * * rsync -rlptDzq /var/dcache/meta /afs/atlas.umich.edu/dcache/`hostname -s`/

• Install 'iostat-cacti' setup (/afs/atlas.umich.edu/dcache/install/install_iostat_cacti.sh ). You may need to do 'service crond reload' after.
• Update (or add to Cacti) plots for this new node
  • Add the 10GE NIC histogram to the appropriate in-out Network summary graphs for dCache
  • Make sure the iostat graphs are working (after installing the iostat-cacti above)
• Add this node to the 'monit' monitoring on head01.aglt2.org