I have a couple of the new R740 reference spec machines with 1.6TB NMVe to add to the cluster. 

Time to adjust my disks:   Old Machines: R740,  2X10 core Xeon, 384GB RAM, 4X240GB SSD, 6X4TB SATA, 2X10Gbps Copper bonded   New Machines: R740,  2X12 core 6130 Xeon Gold, 512GB RAM, 2X240GB SSD, 6X4TB SATA, 4X10Gbps Copper bonded, 1.6TB NVMe card   Proposed Disk Architecture:

1. OS - 240GB Usable, 2XSSD RAID1 Mirror (RAID Controller)
2. Hypervisors: 
        1. LXD - 8TB Usable (ZFS) RAIDz1 Dedupe (zlib2) & Compression, 3XSATA RAW + 1XSSD/240GBXNMVe ARC2
        2. KVM, 3XSATA LVM Stripe+Parity
3. Ceph Storage - 72TB, 36TB Usable (2 replicas), 3xSATA JBOD + 1XSSD OSD/240GB NVMe
4. 1.1TB NVMe remaining space (Docker host shared drives, SSD Cinder Volumes, Special Use Cases)

My previous R740 machines used the nasty little BOSS SATA3 SSD cards with 2X240GB Intel (slow but lots of spare space for wear and tear)   Whenever we go to the data center, I am looking at R740XD, which is not Canonical reference spec.  24X1TB SSD all flash storage would be easier to allocate though. 

I am doing it this way because I have not found any data on how to just convert the entire machine’s storage to tiered (SSD caches + SATA) ZFS with Deduplication (zlib2) + Compression) which would essentially make each machine a Netapp and with dedupe running I would probably never run out of space, ever. 

Here is why:  

Images = VM duplicates = Snapshot duplicates

  With Cinder, Ceph, LXD/KVM, and Glance running on every machine, everything is a dupe.  I am not doing that b/c each charm says “development only” for setting a place on the local filesystem of a local directory as a volume in the disk settings.    Originally I wanted to just mount a big ZFS array (3X4TBSATA Raidz1 + 3X4TBSATA Raidz1 + NVMe ARC2 SSD Cache) at /var/lib to dedupe everything, with separate OSD (RAID0/NVME partition) and OS (RAID1) SSD arrays on the side. 

I ruined my home lab machine by trying to use LVM2 SSD caching (it breaks startup) and still have not figured out the bcache thing.

Is this right? I still haven't gotten approval to purchase support licenses for my lab so I am still having to "wing it."

  • I am attempting to feed the proposed ZFS settings via User Scripts in MaaS, and will then need to attach the cache drives to the ZFS array by a manual cloud-nit step I guess. The issue is with using ZFS to replace the /var filesystem, since that is where the money gets made. WIll update if I ever get past "Commissioning." – spyderdyne Apr 3 at 20:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.