This is an ongoing post about rebuilding my lab infrastructure, see the initial post here.
Today let’s look at VMs and Baremetals and Operating Systems.
Virtual Machines
As I mentioned I’ve been a Linode customer since essentially their day one and have had many 100s of machines there for personal and client needs.
I soured off them quite significantly after they botched their Kubernetes release by not having Europe side support past basic triage initially that led to multiple multi hour outages and so I have been making some changes.
My most recent incarnation had about 5 or 6 Linode machines - Puppet Server, Choria Repos * 3, 2 x DNS, General Services machine.
Today I have 2 Linode machines left. I was reluctant to move them as they were DNS servers for many domains but I changed my way of hosting domains also during this so less of a concern now.
They are now just RPM/Deb repos for Choria and I will move those elsewhere soon also. That’ll be the first time I am without Linode machines since basically 2003, such a shame. One is in the US to provide a nearby mirror there, I might keep it and just scale it down to lower spec. But with the recent changes at Linode it feels a bit like it’s time to consider alternatives.
Previously I had Digital Ocean droplets x 3 for my Kubernetes cluster, as discussed that’s all gone now too.
I used to have quite a selection of Vultr machines, I don’t recall why I left them really I think I felt Linode was just the rolls royce of this kind of Cloud provider and so consolidated to simplify my business accounting etc
Baremetal
In the previous iteration I had only 1 hosted physical machine and that was my Backups machine running Bacula on a Hetzner SX64 (64GB RAM Ryzen 5 3600) with 4 x 16 TB SATA Enterprise HDD 7200rpm. I do not need much from this machine, it wakes up, do backups then sleep again till tomorrow. So the spinning rust is fine for that, I just need lots of it. I rebuilt on a new one just to get a hardware and OS refresh.
Of course I do still use Virtual Machines just managed by Cockpit as per the 2nd part in this series.
I got a pair of Hetzner AX41-NVMe (64GB RAN Ryzen 5 3600) with 2 x 512 NVMe SSDs each. I was expecting to add a 3rd but really these 2 plus the Ryzen at my office turns out to be plenty for my needs. They have some upgrades available - more RAM, Extra Disks, SATA can be added etc. I don’t know if Hetzner supports upgrading running machines but this is nice little platform. At EUR37 per machine that puts them between a 4GB and 8GB shared CPU Linode. You really can’t complain. I might get a 3rd one just for the sake of it and spread my development machines out more. Something for after the summer.
Performance wise moving from my Droplets and other VMs to these machines have been amazing, for an investment equalling 1 Linode I can run several VMs with no additional cost to expand to another VM or to shuffle memory allocations - or just to allocate more since 64GB is way more than I need. This really is a no brainer for my needs.
I’ve been a Hetzner customer also since a very long time, it’s not clear how long but it feels like maybe 2008 or so. They’ve had their ups and downs and dodgy datacenters, dodgy connectivity and dodgy hardware, bad english support, but in the past few years I think they’re really firmed up quite nicely and my machines there have not given me trouble so I felt it’s safe to lean on them a bit more. A few months in now and I’ve not had one minute of problems.
Read on about Operating Systems and more.
Operating System
A bit of history is needed I guess. I started with Softlanding Linux System around 1993, after installing this on a 20MB HDD I removed from a Novell machine the disk promptly died. A bit of hits and misses later I eventually got it quite happy.
Then I moved to Slackware which was the successor to SLS of course. I tried a few things but once RedHat released their first preview in October 1994 on Halloween it was pretty much just RedHat all the way from there. I’ve used all the versions, even the 5.x series that was a total disaster after they moved to ELF but kept at it.
When RedHat removed their free editions I dabbled with Debian but I don’t like the inconsistencies, community, approach or really anything at all about Debian.
Luckily of course CentOS came to the rescue, I even donated hardware to them - a IBM Bladecenter full of Blades. Of course then for some reason they decided to join RedHat, I never did understand the thinking here. Needless to say that did go about as well as it was obvious to anyone it would.
This left a few choices, mainly Rocky Linux and AlmaLinux. I initially went with Rocky Linux it seemed to be on a good track and had the CentOS founder on board, they did seem to loose steam a bit and had a few fits and starts. Now seems fairly solid so I wouldn’t have qualms using them but I settled on AlmaLinux. AlmaLinux seemed at the time to have a bit more backers, a bit more polish and just worked better for my needs.
Then of course RedHat removed the source repos from CentOS Stream this left things quite difficult. Rocky Linux, then, stated they are going to get their SRPMs from a bunch of places to keep the bug-for-bug EL rebuild thing going by means which I really do not think is sustainable. From using paid-for cloud instances and then getting SRPMs using those or using the UBI container images etc, basically they go spelunking all over the place to find source RPMs and then somehow promise to be a complete like-for-like distribution of RHEL. I am not convinced.
Almalinux went another way, they decided to drop the 1:1 RHEL promise and instead moving to promising ABI/Binary compatible with RHEL. They probably also get their SRPMs from some interesting places but they are forging a path of innovation which already resulted in them returning some old hardware support for example. I am quite happy with my choice.
To complicate matters a bit more - or simplify them I guess - the Open Enterprise Linux Association was created to be a
source of SRPMs for EL rebuilders like Alma and Rocky. Their mission is to deliver All sources necessary to achieve a 1:1 / bug-for-bug compatible version of EL which will be distributed via Git, encouraging community collaboration. Hopefully this will help matters along. This is supported
by Oracle and SUSE. Oracle of course have their own Oracle Enterprise Linux that was dunked into the same problem - and no doubt the source of
all the headaches for RedHat.
Anyway, so a lot of history, I like EL. I will keep using EL even if its a slightly different EL. I don’t think its inherently better or worse than other alternatives - and I do not care. The best tool for the job is often the one you know best. I have 30 years of EL experience and that works for me. It’s all just SystemdOS now anyway.
Conclusion
That’s about it really for hardware and OS. The consolidation, moving to Baremetals etc have ended up resulting in quite a bit of money saved. Even if I got a 3rd VM host at Hetzner I’d still be more than a $100 down on monthly expenses but got a lot more.
I have some travel coming up in the next few weeks so the next installment might be a while.