Puppet Infrastructure

Setup from our repo:

To bootstrap foreman and puppetmaster from our svn code:

Initial setup notes

Installed from puppetlabs repo: http://yum.puppetlabs.com/el/6/products/i386/puppetlabs-release-6-6.noarch.rpm

puppet.noarch                                       3.0.1-1.el6                        @puppetlabs-products
puppet-dashboard.noarch                             1.2.12-1.el6                       @puppetlabs-products
puppet-server.noarch                                3.0.1-1.el6                        @puppetlabs-products
puppetdb.noarch                                     1.0.2-1.el6                        @puppetlabs-products
puppetdb-terminus.noarch                            1.0.2-1.el6                        @puppetlabs-products
puppetlabs-release.noarch                           6-6                                installed           

Followed guides here:

http://docs.puppetlabs.com/guides/installation.html

and here to setup Apache webserver for puppet:

http://docs.puppetlabs.com/guides/passenger.html

Created module /etc/puppet/production/modules/rack to do this setup with puppet.

I installed mod_passenger from EPEL using yum:

Installing:
 mod_passenger                           x86_64           3.0.17-2.el6.1              epel           213 k
Installing for dependencies:
 libev                                   x86_64           4.03-3.el6                  epel           113 k
 rubygem-fastthread                      x86_64           1.0.7-2.el6                 epel            59 k
 rubygem-passenger                       x86_64           3.0.17-2.el6.1              epel           146 k
 rubygem-passenger-native                x86_64           3.0.17-2.el6.1              epel           492 k
 rubygem-passenger-native-libs           x86_64           3.0.17-2.el6.1              epel            12 k
 rubygem-rack                            noarch           1:1.1.0-2.el6               epel           446 k

Other puppet setup at this point

Puppet was setup to use environments under /etc/puppet/environments and other basic config setup. Used default puppet ca stored under /etc/puppet/ssl

[main]
    environment = production
    server = puppet.aglt2.org
    logdir = /var/log/puppet
    rundir = /var/run/puppet
    confdir = /etc/puppet
    manifest    = $confdir/environments/$environment/manifests/site.pp
    modulepath  = $confdir/environments/$environment/modules
   

A puppet module was created to distribute hostcerts under /etc/puppet/modules.

Hiera

Setup hiera to reflect puppet environments and setup an initial hierarchy. Under /etc/hiera/production setup some files likes global.

---
:backends:
  - yaml
:hierarchy:
  - global
  - %{osfamily}
  - %{location}
  - %{class}
  - %{fqdn}

:yaml:
# datadir is empty here, so hiera uses its defaults:
# - /var/lib/hiera on *nix
# - %CommonAppData%\PuppetLabs\hiera\var on Windows
# When specifying a datadir, make sure the directory exists.
  :datadir:  '/etc/hiera/%{environment}'

Contents of /etc/hiera/production/global.yaml. When looking up key/value pairs with hiera, puppet will go through hierarchy until it finds all values. The %{class} will be set in %{fqdn}...for example syscfg01.aglt2.org.yaml would define that system as being in syscfg class. (I think maybe it works that way but maybe not and this will have to be re-arranged)

---
syscfg: 
- 'syscfg01.aglt2.org'
- 'syscfg02.aglt2.org'

classes:
- hostcerts

Then additional classes included from syscfg.yaml if %{class} is "syscfg":

From /etc/hiera/production/syscfg.yaml

classes:
- hostcerts::osgtools
- rack

I am not sure these hierarchies are final at all.

Foreman setup

(quick notes)

I began by installing foreman with the foreman installer. I configured the installer to only install foreman and smart proxy, not configure puppet or other components. I modified /etc/httpd/conf.d/foreman.conf to utilize our OSG certs in /etc/grid-security. I removed the default ssl virtual host and made the virtual host in foreman.conf the only ssl virtual host. Via the foreman web interface I created an AGLT2 organization and UM location. Under provisioning menu created a script to to partitioning to the smallest available disk on system that's under 5TB.

Partitioning script:

#Dynamic

sizes=`fdisk -l | grep Disk\ \/dev\/sd | awk '{ print $2 $3 }'`

# if there is not a disk smaller than 5TB then we're not doing anything
smallest=5000
usedisk=""

for size in $sizes; do
        disk=`echo $size | awk -F: '{ print $1 }'`
        size=`echo $size | awk -F: '{ print $2 }' | sed "s/\..*//" `
        if [ "$size" -lt "$smallest" ]; then
                smallest=$size
                disk=${disk#/dev/}
                usedisk=$disk
        fi
done

#echo "Using disk: $usedisk"
#echo "Size is: $smallest"

cat << EOF > /tmp/diskpart.cfg
zerombr yes
bootloader --location=mbr --driveorder=$usedisk --append="rhgb quiet selinux=0"
ignoredisk --only-use=$usedisk
clearpart --all --drives=$usedisk --initlabel msdos
part /boot --fstype ext4 --size=200 --ondisk=$usedisk
part pv.2 --size=1 --grow --ondisk=$usedisk
volgroup vg0 --pesize=4096 pv.2
logvol / --fstype ext4 --name=lv_root --vgname=vg0 --size=1024 --grow
logvol /var --fstype ext4 --name=lv_var --vgname=vg0 --size=30720
logvol /tmp --fstype ext4 --name=lv_tmp --vgname=vg0 --size=15360
logvol /var/cache/openafs --fstype ext4 --name=lv_afscache --vgname=vg0 --size=4096  --fsoptions=noatime,data=writeback,barrier=0,nobh,errors=remount-ro
logvol swap --fstype swap --name=lv_swap --vgname=vg0 --size=1000 --grow --maxsize=4096
EOF

Import puppet classes

Add our smart proxy https://syscfg01.aglt2.org:8443 and under "Configuration" go to "Puppet Classes" and import. Have to make sure the puppet private key is readable by the "puppet" group so smart proxy user can read it also.

Puppet DB

Setup PuppetdB:

http://docs.puppetlabs.com/puppetdb/1.2/install_via_module.html https://groups.google.com/forum/?fromgroups=#!topic/puppet-users/ulP9NiV1iDo

Hiera automatically looks up class paramaters like:

puppetdb::listen_address: 'syscfg01.aglt2.org' http://ttboj.wordpress.com/2013/02/20/automatic-hiera-lookups-in-puppet-3-x/

DO NOT Set foreman as puppet ENC, we are using hiera as classifier and foreman can only confuse issues if it also is ENC. http://projects.theforeman.org/wiki/foreman/External_Nodes

Ran puppet db ssl setup (auto setup didn't?) /usr/sbin/puppetdb-ssl-setup Uses puppet Certs and imports into /etc/puppetdb/ssl/keystore.jks

More notes

GIT workflow: https://puppetlabs.com/blog/git-workflow-and-puppet-environments/

Followed account management example: http://blog.scottlowe.org/2012/11/25/using-puppet-for-account-management/

CLI to add new hosts (for example, adding many new cluster nodes in batch): http://agilecat.web.cern.ch/content/ai-foreman-cli

https://twiki.cern.ch/twiki/bin/view/AgileInfrastructure/HardwareInstallation#Interacting_with_Foreman

-- BenMeekhof - 30 Oct 2012