Difference: CephCluster (1 vs. 11)

Revision 11
02 Oct 2015 - Main.BenMeekhof
Line: 1 to 1
 
META TOPICPARENT name="WebHome"

CEPH cluster setup

Line: 18 to 18
 
  • configures repos (not mirrored)
  • installs packages
  • copies /etc/ceph/ceph.conf from stash/ceph
  • Changed:
    <
    <
  • creates /var/lib/ceph/mon/$(name)-$(mon). $(name) is cluster name 'ceph', $(mon) is slist of monitor instances. Started with just "cephmon01". Directions not entirely clear but I interpret them to say that all monitor instances need directories for themselves and for all other monitor instances. If wrong this should be harmless.
  • >
    >
  • creates /var/lib/ceph/mon/$(name)-$(mon). $(name) is cluster name 'ceph', $(mon) is system hostname. Started with just "cephmon01".
  •  
  • populates /var/lib/ceph/bootstrap-mds and bootstrap-osd on all CEPH class hosts with ceph.keyring. This was obtained from corresponding location on cephmon01 after following monitor bootstrap process from manual install docs and copied into stash/ceph/bootstrap-mds and bootstrap-osd.
  • Installs /root/tools/ceph-disk-prepare on hosts in class CEPH_OSD. This simple script run locally on each OSD runs ceph-disk prepare (and activate) on a range of disks.
  • Line: 95 to 95
     
    ceph osd out <osd-id> 
    Take OSD out of cluster. Perhaps for maintenance, etc. Also use before removing an OSD. Use "ceph osd in " to return to service.
    /root/tools/ceph-osd-rm <osd-id>  
    Permanently remove OSD from crush map, auth map, and cluster (3 commands in script). To put back requires re-init of OSD with ceph-disk prepare. Typically would use if replacing an OSD with some other hardware or needing to re-create OSD from scratch for some reason.
    Added:
    >
    >
    High level diagram of hosts in our cluster (all .aglt2.org):
    CephHighLevelArchAnnotated.png
     

    Sample ceph.conf (3/27/2015)

    
    
    Line: 137 to 141
     

    -- BenMeekhof - 27 Mar 2015
    Added:
    >
    >

    META FILEATTACHMENT attachment="CephHighLevelArchAnnotated.png" attr="" comment="" date="1443811953" name="CephHighLevelArchAnnotated.png" path="CephHighLevelArchAnnotated.png" size="212355" user="BenMeekhof" version="1"
    Revision 10
    27 Jul 2015 - Main.BenMeekhof
    Line: 1 to 1
     
    META TOPICPARENT name="WebHome"

    CEPH cluster setup

    Line: 77 to 77
     
    umtest048 x 750GB XFS on two controllers. 2 x AMD Opteron 285 dual core, 12GB RAM (24 disks in system)
    Changed:
    <
    <
    The ceph docs recommend 1 CPU core per OSD, and ~1GB for 1TB of storage per daemon. When I tried to run the above systems in a cluster using 16 or 24 disks it would work to build the cluster, and work for light operations. As soon as any failure event occurred, such as power interruption, it was impossible to recover the cluster. No matter how I tried to arrange restarting OSD at some point the resource needs for re-establishing a clean state would overwhelm resources available on each host and OSD would begin randomly going down due to timeouts. Then the process would have to start over, etc, etc. So...pay attention to the minimum resources recommended at http://ceph.com/docs/master/start/hardware-recommendations/ I took a chance that for a test situation 2 disks per core would be ok without bringing things to a halt as when I had configured all 16 or 24 disks on each system.
    >
    >
    The ceph docs recommend 1 CPU core per OSD, and ~1GB for 1TB of storage per daemon. When I tried to run the above systems in a cluster using 16 or 24 disks it would work to build the cluster, and work for light operations. As soon as any failure event occurred, such as power interruption, it was impossible to recover the cluster. No matter how I tried to arrange restarting OSD at some point the resource needs for re-establishing a clean state would overwhelm resources available on each host and OSD would begin randomly going down due to timeouts. Then the process would have to start over, etc, etc. So...pay attention to minimum hardware requirements (link below). I took a chance that for a test situation 2 disks per core would be ok without bringing things to a halt as when I had configured all 16 or 24 disks on each system.

    UPDATE: This configuration seems to be ok - we can pull the plug on one system and when it comes up the OSD are placed back into the cluster quickly and without the performance lags and timeouts seen when more OSD are configured per system. This iteration of the cluster also has fewer placement groups. Only 128 per pool are configured. With four Openstack pools created this makes for 512 placement groups on the whole cluster. There is some room yet for more pools in this config. The cluster as configured with 24 OSD should not exceed 2048 PG. Each PG is cpu-intensive so in any configuration we not only have to be careful of OSD per CPU/Memory but also of total PG. The cluster will go to state HEALTH_WARN if there are more than a certain number of PG per OSD (100 I believe).

    Hardware requirements: http://ceph.com/docs/master/start/hardware-recommendations/

    More on PG: http://ceph.com/docs/master/rados/operations/placement-groups/#preselection
     

    Ceph administration

    Changed:
    <
    <
    Login to node cephmon01/02/03 (only cephmon01 exists today). Command 'ceph' is aliased to 'ceph --cluster aglt2'. This is not the place to learn about ceph commands, but one exciting one is "ceph -s" where you can see that the cluster is in state "HEALTH_OK". Another exciting one is "ceph osd tree" to see all hosts and osd. There are other commands too.
    >
    >
    Login to node cephmon01/02/03 (only cephmon01 exists today). This is not the place to learn about ceph commands, but one exciting one is "ceph -s" where you can see that the cluster is in state "HEALTH_OK". Another exciting one is "ceph osd tree" to see all hosts and osd. For other commands related to adding/removing OSDs I'd recommend referencing the docs: http://ceph.com/docs/master/rados/

    There are some useful commands:
     
    Changed:
    <
    <

    Notes from ongoing operation

    >
    >
    /root/tools/ceph-disk-prepare xfs /dev/sd a e ondisk
    Usage further explained above, for provisioning new OSD disks singly or in batch. Run without arguments for help
    ceph osd out <osd-id> 
    Take OSD out of cluster. Perhaps for maintenance, etc. Also use before removing an OSD. Use "ceph osd in " to return to service.
    /root/tools/ceph-osd-rm <osd-id>  
    Permanently remove OSD from crush map, auth map, and cluster (3 commands in script). To put back requires re-init of OSD with ceph-disk prepare. Typically would use if replacing an OSD with some other hardware or needing to re-create OSD from scratch for some reason.
     
    Deleted:
    <
    <
    Our OSD systems are extremely under-spec. The recommendation is 1 CPU core per OSD (disk). We have far exceeded that. As a result, in recovery situations many OSD simply time out and go offline. For example, after an unexpected power outage most of the OSDs may not come up. The solution to this is that you have to start them up individually until they have all been put online: "service ceph start osd.5" for example. If you try to start them all at once it will overwhelm the capability of the systems and you'll never get to an online state.
     

    Sample ceph.conf (3/27/2015)

    
    
    Revision 9
    24 Jul 2015 - Main.BenMeekhof
    Line: 1 to 1
     
    META TOPICPARENT name="WebHome"

    CEPH cluster setup

    Line: 72 to 72
     

    Test System Summary
    Changed:
    <
    <
    umtest014 x 500GB Btrfs 2 x AMD Opteron 285 dual core, 16GB RAM
    umtest034 x 750GB XFS, Journal device at disk 8 256GB OCZ-VERTEX4 SSD (/dev/sdh)2 x AMD Opteron 285 dual core, 8GB RAM
    umtest044 x 750GB XFS on two controllers. 2 x AMD Opteron 285 dual core, 12GB RAM
    >
    >
    umtest018 x 500GB Btrfs 2 x AMD Opteron 285 dual core, 16GB RAM (16 disks in system)
    umtest038 x 750GB XFS, Journal device at disk 8 256GB OCZ-VERTEX4 SSD (/dev/sdh) (23 + 1 ssd disks in system)2 x AMD Opteron 285 dual core, 8GB RAM
    umtest048 x 750GB XFS on two controllers. 2 x AMD Opteron 285 dual core, 12GB RAM (24 disks in system)
     
    Changed:
    <
    <
    The ceph docs recommend 1 CPU core per OSD, and ~1GB for 1TB of storage per daemon. When I tried to run the above systems in a cluster using 16 or 24 disks it would work to build the cluster, and work for light operations. As soon as any failure event occurred, such as power interruption, it was impossible to recover the cluster. No matter how I tried to arrange restarting OSD at some point the resource needs for re-establishing a clean state would overwhelm resources available on each host and OSD would begin randomly going down due to timeouts. Then the process would have to start over, etc, etc. So...pay attention to the minimum resources recommended at http://ceph.com/docs/master/start/hardware-recommendations/ They're not kidding!
    >
    >
    The ceph docs recommend 1 CPU core per OSD, and ~1GB for 1TB of storage per daemon. When I tried to run the above systems in a cluster using 16 or 24 disks it would work to build the cluster, and work for light operations. As soon as any failure event occurred, such as power interruption, it was impossible to recover the cluster. No matter how I tried to arrange restarting OSD at some point the resource needs for re-establishing a clean state would overwhelm resources available on each host and OSD would begin randomly going down due to timeouts. Then the process would have to start over, etc, etc. So...pay attention to the minimum resources recommended at http://ceph.com/docs/master/start/hardware-recommendations/ I took a chance that for a test situation 2 disks per core would be ok without bringing things to a halt as when I had configured all 16 or 24 disks on each system.
     

    Ceph administration

    Revision 8
    23 Jul 2015 - Main.BenMeekhof
    Line: 1 to 1
     
    META TOPICPARENT name="WebHome"

    CEPH cluster setup

    Line: 72 to 72
     

    Test System Summary
    Changed:
    <
    <
    umtest0116 x 500GB Btrfs 2 x AMD Opteron 285 dual core, 16GB RAM
    umtest0323 x 750GB XFS, Journal device at disk 8 256GB OCZ-VERTEX4 SSD (/dev/sdh)2 x AMD Opteron 285 dual core, 8GB RAM
    umtest0424 x 750GB XFS on two controllers. 2 x AMD Opteron 285 dual core, 12GB RAM
    >
    >
    umtest014 x 500GB Btrfs 2 x AMD Opteron 285 dual core, 16GB RAM
    umtest034 x 750GB XFS, Journal device at disk 8 256GB OCZ-VERTEX4 SSD (/dev/sdh)2 x AMD Opteron 285 dual core, 8GB RAM
    umtest044 x 750GB XFS on two controllers. 2 x AMD Opteron 285 dual core, 12GB RAM
     
    Added:
    >
    >
    The ceph docs recommend 1 CPU core per OSD, and ~1GB for 1TB of storage per daemon. When I tried to run the above systems in a cluster using 16 or 24 disks it would work to build the cluster, and work for light operations. As soon as any failure event occurred, such as power interruption, it was impossible to recover the cluster. No matter how I tried to arrange restarting OSD at some point the resource needs for re-establishing a clean state would overwhelm resources available on each host and OSD would begin randomly going down due to timeouts. Then the process would have to start over, etc, etc. So...pay attention to the minimum resources recommended at http://ceph.com/docs/master/start/hardware-recommendations/ They're not kidding!
     

    Ceph administration

    Revision 7
    23 Jul 2015 - Main.BenMeekhof
    Line: 1 to 1
     
    META TOPICPARENT name="WebHome"

    CEPH cluster setup

    Line: 70 to 70
      ... etc through sdp.
    Changed:
    <
    <
    umtest0116 x 500GB btrfs fs
    umtest03
    >
    >
    Test System Summary
    umtest0116 x 500GB Btrfs 2 x AMD Opteron 285 dual core, 16GB RAM
    umtest0323 x 750GB XFS, Journal device at disk 8 256GB OCZ-VERTEX4 SSD (/dev/sdh)2 x AMD Opteron 285 dual core, 8GB RAM
    umtest0424 x 750GB XFS on two controllers. 2 x AMD Opteron 285 dual core, 12GB RAM
     
    Deleted:
    <
    <
    On other hosts (umtest03,04) XFS was used. Umtest03 has a 256GB SSD as disk 8 on the controller which is mapped in the OS as /dev/sdh and was specified as journel dev for the 23 OSD on that host.
     

    Ceph administration

    Login to node cephmon01/02/03 (only cephmon01 exists today). Command 'ceph' is aliased to 'ceph --cluster aglt2'. This is not the place to learn about ceph commands, but one exciting one is "ceph -s" where you can see that the cluster is in state "HEALTH_OK". Another exciting one is "ceph osd tree" to see all hosts and osd. There are other commands too.

    Notes from ongoing operation

    Changed:
    <
    <
    Our OSD systems are somewhat under-spec. Recommendation is 1 cpu core per OSD disk. Sometimes there are timeout issues when many OSD start up at once and need to all contact eachother for verification/recovery (such as after power failure). You may need to run "service start ceph" repeatedly on all the OSD systems until they all really start.
    >
    >
    Our OSD systems are extremely under-spec. The recommendation is 1 CPU core per OSD (disk). We have far exceeded that. As a result, in recovery situations many OSD simply time out and go offline. For example, after an unexpected power outage most of the OSDs may not come up. The solution to this is that you have to start them up individually until they have all been put online: "service ceph start osd.5" for example. If you try to start them all at once it will overwhelm the capability of the systems and you'll never get to an online state.
     

    Sample ceph.conf (3/27/2015)

    
    
    Revision 6
    22 Jul 2015 - Main.BenMeekhof
    Line: 1 to 1
     
    META TOPICPARENT name="WebHome"

    CEPH cluster setup

    Line: 70 to 70
      ... etc through sdp.
    Added:
    >
    >
    umtest0116 x 500GB btrfs fs
    umtest03
      On other hosts (umtest03,04) XFS was used. Umtest03 has a 256GB SSD as disk 8 on the controller which is mapped in the OS as /dev/sdh and was specified as journel dev for the 23 OSD on that host.

    Ceph administration

    Login to node cephmon01/02/03 (only cephmon01 exists today). Command 'ceph' is aliased to 'ceph --cluster aglt2'. This is not the place to learn about ceph commands, but one exciting one is "ceph -s" where you can see that the cluster is in state "HEALTH_OK". Another exciting one is "ceph osd tree" to see all hosts and osd. There are other commands too.
    Added:
    >
    >

    Notes from ongoing operation

    Our OSD systems are somewhat under-spec. Recommendation is 1 cpu core per OSD disk. Sometimes there are timeout issues when many OSD start up at once and need to all contact eachother for verification/recovery (such as after power failure). You may need to run "service start ceph" repeatedly on all the OSD systems until they all really start.
     

    Sample ceph.conf (3/27/2015)

    [global]
    
    
    Revision 5
    01 May 2015 - Main.BenMeekhof
    Line: 1 to 1
     
    META TOPICPARENT name="WebHome"

    CEPH cluster setup

    Line: 13 to 13
      Updated cfengine trunk to accommodate SL7 systems.
    Changed:
    <
    <
    Created cfengine policy in branch "ceph" following basic manual provisioning steps. Effects CEPH_NODE, CEPH_NODE_OSD, CEPH_NODE_MDS, CEPH_NODE_MON, and/or CEPH_NODE_OG as appropriate. These acronyms will make sense to you after you read the ceph docs.
    >
    >
    Created cfengine policy in branch "ceph" following basic manual provisioning steps. Effects CEPH, CEPH_OSD, CEPH_MDS, CEPH_MON, and/or CEPH_OG as appropriate. These acronyms will make sense to you after you read the ceph docs.
     
    • configures repos (not mirrored)
    • installs packages
    • copies /etc/ceph/ceph.conf from stash/ceph
    • creates /var/lib/ceph/mon/$(name)-$(mon). $(name) is cluster name 'ceph', $(mon) is slist of monitor instances. Started with just "cephmon01". Directions not entirely clear but I interpret them to say that all monitor instances need directories for themselves and for all other monitor instances. If wrong this should be harmless.
    Changed:
    <
    <
  • populates /var/lib/ceph/bootstrap-mds and bootstrap-osd on all CEPH_NODE class hosts with ceph.keyring. This was obtained from corresponding location on cephmon01 after following monitor bootstrap process from manual install docs and copied into stash/ceph/bootstrap-mds and bootstrap-osd.
  • Installs /root/tools/ceph-disk-prepare on hosts in class CEPH_NODE_OSD. This simple script run locally on each OSD runs ceph-disk prepare (and activate) on a range of disks.
  • Installs /etc/profile.d/ceph-aglt2.sh,csh. This sets the CEPH_CONF environment variable which doesn't seem to have the documented effect on finding the ceph config with ceph utils or with the init script. It also sets 'ceph' as an alias to 'ceph --cluster aglt2' as a workaround. Obviously if we setup other clusters this won't work or you'll have to use the full path to the ceph util to reference them.
  • >
    >
  • populates /var/lib/ceph/bootstrap-mds and bootstrap-osd on all CEPH class hosts with ceph.keyring. This was obtained from corresponding location on cephmon01 after following monitor bootstrap process from manual install docs and copied into stash/ceph/bootstrap-mds and bootstrap-osd.
  • Installs /root/tools/ceph-disk-prepare on hosts in class CEPH_OSD. This simple script run locally on each OSD runs ceph-disk prepare (and activate) on a range of disks.
  •  

    OS Build

    Line: 64 to 63
      For the OSD systems, they were provisioned and configured with policy from ceph branch. Then used /root/tools/ceph-disk-prepare to create xfs or btrfs filesystems. At the moment, disks on umtest01 are btrfs. The rest are XFS. The --test argument causes the script to print but not do. Replace argument 'ondisk' with a separate journal device if using. The ceph-disk util will automatically create a new 10GB partition for each osd on the journal device.
    [root@umtest01 tools]# ./ceph-disk-prepare btrfs /dev/sd a p ondisk --test 
    
    
    Changed:
    <
    <
    ceph-disk prepare --cluster aglt2 --cluster-uuid db861c04-b855-4f47-8d25-c6e28d24a1e2 --fs-type btrfs /dev/sda
    >
    >
    ceph-disk prepare --cluster ceph --cluster-uuid db861c04-b855-4f47-8d25-c6e28d24a1e2 --fs-type btrfs /dev/sda
      ceph-disk activate /dev/sda
    Changed:
    <
    <
    ceph-disk prepare --cluster aglt2 --cluster-uuid db861c04-b855-4f47-8d25-c6e28d24a1e2 --fs-type btrfs /dev/sdb
    >
    >
    ceph-disk prepare --cluster ceph --cluster-uuid db861c04-b855-4f47-8d25-c6e28d24a1e2 --fs-type btrfs /dev/sdb
      ceph-disk activate /dev/sdb ... etc through sdp.
    Line: 97 to 96
      # replicas allowed for writes in degraded state osd pool default min size = 2
    Changed:
    <
    <
    osd pool default pg num = 2100 osd pool default pgp num = 2100
    >
    >
    osd pool default pg num = 2048 osd pool default pgp num = 2048
      #0 for a 1-node cluster. #1 for a multi node cluster in a single rack #2 for a multi node, multi chassis cluster with multiple hosts in a chassis
    Revision 4
    02 Apr 2015 - Main.BenMeekhof
    Line: 1 to 1
     
    META TOPICPARENT name="WebHome"

    CEPH cluster setup

    Line: 73 to 73
     

    On other hosts (umtest03,04) XFS was used. Umtest03 has a 256GB SSD as disk 8 on the controller which is mapped in the OS as /dev/sdh and was specified as journel dev for the 23 OSD on that host.
    Added:
    >
    >

    Ceph administration

    Login to node cephmon01/02/03 (only cephmon01 exists today). Command 'ceph' is aliased to 'ceph --cluster aglt2'. This is not the place to learn about ceph commands, but one exciting one is "ceph -s" where you can see that the cluster is in state "HEALTH_OK". Another exciting one is "ceph osd tree" to see all hosts and osd. There are other commands too.
     

    Sample ceph.conf (3/27/2015)

    [global]
    
    
    Line: 99 to 103
      #1 for a multi node cluster in a single rack #2 for a multi node, multi chassis cluster with multiple hosts in a chassis #3 for a multi node cluster with hosts across racks, etc.
    Changed:
    <
    <
    osd crush chooseleaf type = 3
    >
    >
    osd crush chooseleaf type = 1

    [osd] journal zero on create = true
     

    [mds.01] host = cephmds01
    Revision 3
    01 Apr 2015 - Main.BenMeekhof
    Line: 1 to 1
     
    META TOPICPARENT name="WebHome"

    CEPH cluster setup

    Line: 21 to 21
     
  • creates /var/lib/ceph/mon/$(name)-$(mon). $(name) is cluster name 'ceph', $(mon) is slist of monitor instances. Started with just "cephmon01". Directions not entirely clear but I interpret them to say that all monitor instances need directories for themselves and for all other monitor instances. If wrong this should be harmless.
  • populates /var/lib/ceph/bootstrap-mds and bootstrap-osd on all CEPH_NODE class hosts with ceph.keyring. This was obtained from corresponding location on cephmon01 after following monitor bootstrap process from manual install docs and copied into stash/ceph/bootstrap-mds and bootstrap-osd.
  • Installs /root/tools/ceph-disk-prepare on hosts in class CEPH_NODE_OSD. This simple script run locally on each OSD runs ceph-disk prepare (and activate) on a range of disks.
  • Added:
    >
    >
  • Installs /etc/profile.d/ceph-aglt2.sh,csh. This sets the CEPH_CONF environment variable which doesn't seem to have the documented effect on finding the ceph config with ceph utils or with the init script. It also sets 'ceph' as an alias to 'ceph --cluster aglt2' as a workaround. Obviously if we setup other clusters this won't work or you'll have to use the full path to the ceph util to reference them.
  •  

    OS Build

    Line: 60 to 61
     

    For the monitor system cephmon01, manual monitor bootstrap instructions were followed after applying the cfengine policy to setup directories and config file. That config file is at the end of this document for reference but the up-to-date version is at https://ndt.aglt2.org/viewvc/cfengine/branches/ceph/masterfiles/stash/ceph/
    Changed:
    <
    <
    For the OSD systems, they were provisioned and configured with policy from ceph branch. Then used /root/tools/ceph-disk-prepare to create xfs or btrfs filesystems. At the moment, disks sda-sdl on umtest04 are btrfs. The rest are XFS. The --test argument causes the script to print but not do. Replace argument 'ondisk' with a separate journal device if using. The ceph-disk util will automatically create a new 10GB partition for each osd on the journal device.
    >
    >
    For the OSD systems, they were provisioned and configured with policy from ceph branch. Then used /root/tools/ceph-disk-prepare to create xfs or btrfs filesystems. At the moment, disks on umtest01 are btrfs. The rest are XFS. The --test argument causes the script to print but not do. Replace argument 'ondisk' with a separate journal device if using. The ceph-disk util will automatically create a new 10GB partition for each osd on the journal device.
     
    
    
    Changed:
    <
    <
    [root@umtest04 tools]# ./ceph-disk-prepare btrfs /dev/sd a l ondisk --test ceph-disk prepare --cluster ceph --cluster-uuid db861c04-b855-4f47-8d25-c6e28d24a1e2 --fs-type btrfs /dev/sda
    >
    >
    [root@umtest01 tools]# ./ceph-disk-prepare btrfs /dev/sd a p ondisk --test ceph-disk prepare --cluster aglt2 --cluster-uuid db861c04-b855-4f47-8d25-c6e28d24a1e2 --fs-type btrfs /dev/sda
      ceph-disk activate /dev/sda
    Changed:
    <
    <
    ceph-disk prepare --cluster ceph --cluster-uuid db861c04-b855-4f47-8d25-c6e28d24a1e2 --fs-type btrfs /dev/sdb
    >
    >
    ceph-disk prepare --cluster aglt2 --cluster-uuid db861c04-b855-4f47-8d25-c6e28d24a1e2 --fs-type btrfs /dev/sdb
      ceph-disk activate /dev/sdb
    Changed:
    <
    <
    ceph-disk prepare --cluster ceph --cluster-uuid db861c04-b855-4f47-8d25-c6e28d24a1e2 --fs-type btrfs /dev/sdc ceph-disk activate /dev/sdc ... etc through sdl. sdm - sdx were provisioned with xfs.
    >
    >
    ... etc through sdp.
     
    Added:
    >
    >
    On other hosts (umtest03,04) XFS was used. Umtest03 has a 256GB SSD as disk 8 on the controller which is mapped in the OS as /dev/sdh and was specified as journel dev for the 23 OSD on that host.
     

    Sample ceph.conf (3/27/2015)

    [global]
    
    
    Line: 86 to 87
      osd journal size = 10280 filestore xattr use omap = true
    Changed:
    <
    <
    [osd] osd pool default size = 2
    >
    >
    # How many times to write an object. 3 is the default. osd pool default size = 3

    # replicas allowed for writes in degraded state
      osd pool default min size = 2
    Changed:
    <
    <
    osd pool default pg num = 333 osd pool default pgp num = 333
    >
    >

    osd pool default pg num = 2100 osd pool default pgp num = 2100
      #0 for a 1-node cluster. #1 for a multi node cluster in a single rack #2 for a multi node, multi chassis cluster with multiple hosts in a chassis
    Revision 2
    31 Mar 2015 - Main.BenMeekhof
    Line: 1 to 1
     
    META TOPICPARENT name="WebHome"

    CEPH cluster setup

    Line: 60 to 60
     

    For the monitor system cephmon01, manual monitor bootstrap instructions were followed after applying the cfengine policy to setup directories and config file. That config file is at the end of this document for reference but the up-to-date version is at https://ndt.aglt2.org/viewvc/cfengine/branches/ceph/masterfiles/stash/ceph/
    Changed:
    <
    <
    For the OSD systems, they were provisioned and configured with policy from ceph branch. Then used /root/tools/ceph-disk-prepare to create xfs or btrfs filesystems. At the moment, disks sda-sdl on umtest04 are btrfs. The rest are XFS. The --test argument causes the script to print but not do.
    >
    >
    For the OSD systems, they were provisioned and configured with policy from ceph branch. Then used /root/tools/ceph-disk-prepare to create xfs or btrfs filesystems. At the moment, disks sda-sdl on umtest04 are btrfs. The rest are XFS. The --test argument causes the script to print but not do. Replace argument 'ondisk' with a separate journal device if using. The ceph-disk util will automatically create a new 10GB partition for each osd on the journal device.
     
    
    
    Changed:
    <
    <
    [root@umtest04 tools]# ./ceph-disk-prepare btrfs /dev/sd a l --test
    >
    >
    [root@umtest04 tools]# ./ceph-disk-prepare btrfs /dev/sd a l ondisk --test
      ceph-disk prepare --cluster ceph --cluster-uuid db861c04-b855-4f47-8d25-c6e28d24a1e2 --fs-type btrfs /dev/sda ceph-disk activate /dev/sda ceph-disk prepare --cluster ceph --cluster-uuid db861c04-b855-4f47-8d25-c6e28d24a1e2 --fs-type btrfs /dev/sdb
     
    This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
    Ideas, requests, problems regarding Foswiki? Send feedback