For testing purposes only we used the Redhat Storage Appliance demo which has gluster tools pre-installed. Docs are here: http://docs.redhat.com/docs/en-US/Red_Hat_Storage_Software_Appliance/3.2/pdf/User_Guide/Red_Hat_Storage_Software_Appliance-3.2-User_Guide-en-US.pdf
The docs are largely a repeat of the documentation available from the gluster.org website. In any real usage scenario I would advise using the freely available gluster packages rather than this appliance. There does not seem to be any significant value in terms of ease of use or setup for purchasing the appliance though it would mean we would have full support for the gluster storage cluster. Cost per node is approximately $4000 and the minimum to start is 4 nodes. There is also a premium version for $6000 per node. This quote is from DLT Solutions who seems to work with the university or one may also want to email lsa.licensing to work with them in obtaining a license. I didn't attach the quote as I am not sure if it is ok to make that public.
Upgrading gluster RPMS
As it turns out the RH appliance ships with version 3.2 of gluster. I wanted to test the most current version so I upgraded the system with v 3.3 RPMS from http://download.gluster.org/pub/gluster/glusterfs/3.3/LATEST/RHEL/
[root@umdist08 ~]# rpm -Uvh glusterfs-3.3.0-1.el6.x86_64.rpm glusterfs-geo-replication-3.3.0-1.el6.x86_64.rpmglusterfs-fuse-3.3.0-1.el6.x86_64.rpm glusterfs-server-3.3.0-1.el6.x86_64.rpm
The glusterfs RPM obsoletes/replaces gluster-core from version 3.2.
It should be noted that the version 3.2 server was not compatible with the version 3.3 client. Noticing that was what led to updating the server. Also note that the update lost my volume but it appeared to simply be a case of some config not being copied from an old config dir to a new one (the -server rpm post-install tried but failed because the destination existed from one of the other RPMS). Things moved from /etc/glusterd to /var/lib/glusterd. The meta-data was still on the brick and I'm pretty sure a real update in production following a more careful procedure would be fairly simple (especially if not a major version upgrade).
3 x md1200 shelves, 3TB disks attached to Perc H800 controller.
Created 3 RAID6 volumes, 12 disks each. Adaptive Read-ahead, Write back, 512KB stripe width.
Created 3 XFS filesystems:
mkfs.xfs -f -L glfs_sdc -d su=512k,sw=10 -l su=256k /dev/sdc
Mounted with inode64:
LABEL=glfs_sdc /glfs1 xfs inode64 1 1
Later we need more volumes so for the replicated tests so 6 RAID5 volumes were set up and filesystems done as such:
mkfs.xfs -f -L glfs_sdd -d su=512k,sw=5 -l su=256k /dev/sdd
Iozone tests done with iozone-3.394-1.el5.rf.x86_64 from DAG repository.
We started with a simple volume using one hardware raid volume. Effectively a distributed volume with only 1 member:
[root@umdist08 RHEL6]# gluster volume create testvolume umdist08:/glfs1
Creation of volume testvolume has been successful. Please start the volume to access data.
[root@umdist08 RHEL6]# gluster volume info
Volume Name: testvolume
Number of Bricks: 1
[root@umdist08 RHEL6]# gluster volume start testvolume
Starting volume testvolume has been successful
Other volumes were created as such:
6 Brick distributed replicated
[root@umdist08 ~]# gluster volume create test replica 2 transport tcp umdist08:/glfs1 umdist08:/glfs3 umdist08:/glfs6 umdist08:/glf
s2 umdist08:/glfs4 umdist08:/glfs5
Multiple bricks of a replicate volume are present on the same server. This setup is not optimal.
Do you still want to continue creating the volume? (y/n) y
Creation of volume test has been successful. Please start the volume to access data.
[root@umdist08 ~]# gluster volume status
Status of volume: test
Gluster process Port Online Pid
Brick umdist08:/glfs1 24009 Y 2149
Brick umdist08:/glfs3 24010 Y 2155
Brick umdist08:/glfs6 24011 Y 2160
Brick umdist08:/glfs2 24012 Y 2167
Brick umdist08:/glfs4 24013 Y 2173
Brick umdist08:/glfs5 24014 Y 2179
NFS Server on localhost 38467 Y 2186
Self-heal Daemon on localhost N/A Y 2192
There are RPMS available for EL6 and others here: http://download.gluster.org/pub/gluster/glusterfs/LATEST/
For our SL5 test system dc2-3-35 I built RPMS from the spec included in the source download after installing some prerequisites:
[root@c-3-35 glusterfs-3.3.0]# yum install flex libtool ncurses-devel readline-devel libibverbs-devel
[root@c-3-35 glusterfs-3.3.0]# cp ../glusterfs-3.3.0.tar.gz /usr/src/redhat/SOURCES/
[root@c-3-35 glusterfs-3.3.0]# rpmbuild -ba glusterfs.spec
Then installed glusterfs-3.3.0-1.x86_64.rpm and glusterfs-fuse-3.3.0-1.x86_64.rpm
Mounted at the command line:
[root@c-3-35 x86_64]# mount -t glusterfs -o log-level=WARNING,log-file=/var/log/gluster.log umdist08:/testvolume /mnt/gfs
glusterfs#umdist08:/testvolume 28T 33M 28T 1% /mnt/gfs
1 brick simple volume
From node dc2-3-35, 1Gb ethernet, Intel(R) Xeon(R) CPU X5355 @ 2.66GHz
Test used 6 simultaneous IO threads. Tests below each specify a different record size for the test and are otherwise identical.
iozone -t6 -Rb gfs_general_4.xls -P1 -r4k -s32G -F /mnt/gfs/io1.test /mnt/gfs/io2.test /mnt/gfs/io3.test \
/mnt/gfs/io4.test /mnt/gfs/io5.test /mnt/gfs/io6.test
iozone -t6 -Rb gfs_general_512.xls -P1 -r512k -s32G -F /mnt/gfs/io1.test /mnt/gfs/io2.test /mnt/gfs/io3.test \
/mnt/gfs/io4.test /mnt/gfs/io5.test /mnt/gfs/io6.test
6 brick distributed replicated
This test was done from a client machine with 10Gb network interface (gluster test server has 10G interface in all tests).
iozone -t6 -Rb gfs_dr_10G_general_512.xls -P1 -r512k -s32G -F /mnt/gfs/io1.test /mnt/gfs/io2.test /mnt/gfs/io3.test /mnt/gfs/io4.test /mnt/gfs/io5.test /mnt/gfs/io6.test