Proxy server

This system has no access to outside world, and in general systems can't talk to non-edu locations.

To be able to download things, use proxy server. For wget, it is sufficient to:

export http_proxy=cache.local:3128

The "pkg-get" util (/opt/csw/bin) is already setup to use the proxy server (/opt/csw/etc or /usr/local/etc pkg-get.conf, check where this really is). Use this to get precompiled Solaris GNU apps and dependencies easily (gcc, lsof, amanda, libthisorthat).

System Setup

Compiled amanda 3.2.2 from source. See later entry regarding compilation. Installed MysqlZRM server package from http://www.zmanda.com/download-zrm.php See SunX4540ConfigSolaris for notes about other packages.

Root zpool

System is configured at install time with root filesystem on zfs. By default system is setup to enable 2 bootable drives, these are the ZFS root pool. Must use text-based installer. No major knowledge of ZFS required, just have to choose 2 disks to be in the pool (more than 2 not supported). Chose not to make /var separate filesystem but that might be a wise thing to make a separate zfs filesystem and limit quota on it so as not to overfill the root pool someday.

config:

        NAME          STATE     READ WRITE CKSUM
        rpool         ONLINE       0     0     0
          mirror      ONLINE       0     0     0
            c4t0d0s0  ONLINE       0     0     0
            c4t4d0s0  ONLINE       0     0     0

Zpool creation

One giant zpool with raidz2 pool constituents. Should be robust. Performance should be ok (? would like to run benchmark).

zpool create pool1 \
raidz2 c0t0d0 c1t0d0 c3t0d0 c5t0d0 c6t0d0 c0t1d0 c1t1d0 c3t1d0 c5t1d0 \
raidz2 c6t1d0 c4t1d0 c0t2d0 c1t2d0 c3t2d0 c4t2d0 c5t2d0 c6t2d0 c0t3d0 \ 
raidz2 c1t3d0 c3t3d0 c4t3d0 c5t3d0 c6t3d0 c0t4d0 c1t4d0 c3t4d0 c5t4d0 \ 
raidz2 c6t4d0 c0t5d0 c1t5d0 c3t5d0 c4t5d0 c5t5d0 c6t5d0 c0t6d0 c1t6d0 \
raidz2 c3t6d0 c4t6d0 c5t6d0 c6t6d0 c0t7d0 c1t7d0 c3t7d0 c4t7d0 c5t7d0 \

zpool add pool1 spare c6t7d0 

Note that we skipped the root zpool disks.

Alternative: The above configuration gives up 10 disks to parity in 5 raidz2 pools. This may also be an option:

zpool create pool1 \ 
raidz1 c0t0d0 c1t0d0 c3t0d0 c5t0d0 c6t0d0 \
raidz1 c0t1d0 c1t1d0 c3t1d0 c4t1d0 c5t1d0 c6t1d0  \
raidz1 c0t2d0 c1t2d0 c3t2d0 c4t2d0 c5t2d0 c6t2d0 \ 
raidz1 c0t3d0 c1t3d0 c3t3d0 c4t3d0 c5t3d0 c6t3d0  \
raidz1 c0t4d0 c1t4d0 c3t4d0 c5t4d0 c6t4d0  \
raidz1 c0t5d0 c1t5d0 c3t5d0 c4t5d0 c5t5d0 c6t5d0 \
raidz1 c0t6d0 c1t6d0 c3t6d0 c4t6d0 c5t6d0 c6t6d0 \
raidz1 c0t7d0 c1t7d0 c3t7d0 c4t7d0 c5t7d0 

zpool add pool1 spare c6t7d0 

We know this config can sustain writing at 500-600MB/s from many threads, and 900MB/s reading (our tests showed a very flat and consistent rate even increasing reading or writing threads to 12). I am sure the raidz2 version would not do quite as well though it might still be enough. Our tests did not check what happens when doing both at once...however to sustain the LTO-4 drives we only need to be able to read at 160 MB/s. I believe amanda would write to cache and dump to tape at the same time...i think it is reasonable that >10 backup processes can top out the IO write capacity whilst still maintaining sufficient read IO to feed the tape drives without stream interruption. Benchmarks soon...

There is still always the option of creating a raid-0 zpool for Amanda disk cache if the performance in real usage is not sufficient to keep data streaming to the 2 LTO-4 drives. In practice we'll only be using one at once so assuming the need to stream to 2 is a generous estimate. (i hope to fill one 800GB tape per night at most).

ZFS filesystem setup

Make some holding disk space, and disk space to take mysql backups. 5TB should be enough for the Amanda holding disk but we can increase that later. Currently the Mysql-ZRM backups use 37GB of space with a 1 month retention policy. We'll increase the retention and not bother with a quota at this point (since amholding is quota limited from overfilling the pool).

zfs create pool1/amholding
zfs set quota=5T pool1/amholding
zfs set reservation=5T pool1/amholding

zfs create pool1/mysql-backup

Compiling Amanda

Taking AFS backups requires compiling both some afs utils and Amanda to point to a script which wraps GNU tar. Though I don't believe the server needs it for backups, it does need the amanda-afs script for restoring (I think). More details about this are at: AmandaBackup

I compiled the latest release of Amanda with these options, source code in dir as listed below:

/opt/download/src/amanda-3.2.2#  ./configure --with-bsdtcp-security --with-bsdudp-security --with-rsh-security --with-ssh-security \ 
--prefix=/usr --libexecdir=/usr --sysconfdir=/etc --with-group=sys --with-gnutar=/usr/sbin/gtar-wrapper.pl --localstatedir=/var --with-fqdn \ 
--with-config=umatlas --with-smbclient=/opt/csw/bin/smbclient 

Other notes:

• Use "gmake" to run the compilation.
• Solaris /usr/bin/egrep complains about syntax errors during phases of the compilation. Compile still finishes. Recommend linking /usr/bin/egrep to GNU grep (/opt/csw/bin/ggrep as installed on bambi). This eliminates the syntax errors.
• It would be desirable to compile with --with-krb5-security (=/usr possibly required) since Solaris has krb5 libs in /usr/lib. It still doesn't find the libs. I tried commenting out all the configure checks and just leaving the line which adds the krb5 libs to the LIBS variable but that caused other issues later during configure runs. The checks in the configure script want static .a libs and there's comments which indicate the author expects to manually link to static libs. Solaris doesn't have the static libs.

Amanda setup

Amanda config is in /opt/csw/etc/amanda. There are two files:

• amanda.conf.common: Configs common to any backup configuration. Tape type is defined here (see http://wiki.zmanda.com/index.php/Tapetype_definitions#LTO4). Here is where you can tweak global things like how many dump processes to run in parallel, how dumps are prioritized for flush to tape, length of backup cycle, and how many tapes amanda will consume in one run. Also here are defined global types of dumps (program used to backup, compression, priority, use indexing). I won't go into it at length. The hardware configuration is also in this file:

  define changer sl500 {
      tpchanger "chg-robot:/dev/scsi/changer/c7t500104F000B87A69d0"         
      property "tape-device" "0=tape:/dev/rmt/0" "1=tape:/dev/rmt/1"        
      property "eject-before-unload" "yes"                       
      # property "use-slots" "1-5,11-20"
  }
  
  tpchanger "sl500"
  

• umatlas/amanda.conf: Config for one specific backup set. In the umatlas directory you will find the "disklist" file defining what we backup. The "amanda.conf" file defines valid tape labels that the backup set will use, and also defines where to keep indexes, logs, and status information.

Labeling tapes

There is a script to apply an Amanda tape label matching the barcode on the tape. Tapes must be in order starting with slot1 in library with no empty slots (script isn't smart, just counts from 1 to X). To label individual tapes use command as in script.

/opt/csw/etc/amanda/label-changer-barcode:
#!/bin/bash

# this needs to be fixed up to deal with non-sequential slots

echo "Before running this be sure tapes are in sequential order in library from slot 1 to slot X with no gaps"

voltaglist=`/opt/csw/sbin/mtx -f /dev/scsi/changer/c7t500104F000B87A69d0 status | ggrep "Storage Element" | awk '{ print $4 }' | ggrep -o L[0-9][0-9][0-9][0-9][0-9]`
slots=`/opt/csw/sbin/mtx -f /dev/scsi/changer/c7t500104F000B87A69d0 status | ggrep "Storage Element [0-9].*\:Full" | awk '{ print $3 }' | ggrep -o "[0-9][0-9]*"`

slot=1
for voltag in $voltaglist 
do
        echo -e "Labeling tape in slot $slot with $voltag\n"
        su amanda -c "amlabel umatlas $voltag slot $slot" 
        slot=$(($slot+1))
done

-- BenMeekhof - 14 Dec 2009