This is the final in a series of posts about rebuilding my lab infrastructure, see the initial post here.
Today we’ll wrap things up with a look at some SaaS tools I use and a general look through some small utilities and things I use to bring it all together.
I’ve been enjoying my summer for the last 3 months hence the hiatus of posts.
Long ago I used to run my own Zimbra but I gave up on that a few years ago, been with Fastmail ever since and really happy with their service.
Delivering email from my 20+ VMs all over the world though is tricky, getting all the various DKIM and other settings right for a big set of IP addresses and ranges quickly becomes a maintenance nightmare. But ther’s a constant trickle of stuff from them - cron job, monitoring, backup statusses and more.
After some looking around at options I found SMTP2GO who have a very generous 1000 / month free tier. This is usually fine for me but I ended up paying them anyway for a annual account. This way I have just one egress point to consider in my various email policy setups and thus far, for delivering system emails, this has been a great time saver.
Read on about DNS, Git, SSO and more.
DNS
Till this rebuild I hosted my own DNS, I’ve had a set of quite stable DNS servers on the same IP addresses for over a decade so it’s worked well.
I’ve always wanted to be rid of hosting this myself but most services are request based billing and I always felt this would not go well since you have no way of controlling how many DNS requests you get.
After some research I found ClouDNS who have a 50 zone / 2000 record plan that allows 200 million queries a month for $5/month. This is plenty and incredibly cheap. They have Geo DNS servers, support is responsive and knowledgeable and have a decent API.
For comparison DNSimple is $30/month and charges per zone and 10c per million queries a month. That’s crazy.
My usage is around 20 million queries a month so I am very comfortable in this level of service and for $5/month there is simply no way to compete with this on any kind of self hosting setup.
They have a bind zone file import feature and I use their API to do daily backups of all the zones into my local Git server using a little tool I wrote call CloudDNS Backup
Git
Speaking of Git hosting, I quite like Gitea though should probably move to Foregejo. The split from Gitea happened just as I was building things up so I am still on that.
Gitea is pretty great, it does a reasonable job of being a GitHub facsimile and uses act (also great for local action testing) to provide reasonably compatible self hosted GitHub Actions.
Gitea and Foregejo are both single binary/single process tools so really easy to get going, for my needs even their SQLite support is ideal.
SSO
Getting all things authenticated and managing users is a nightmare, I use Okta free tier to front almost all my HTTP
stuff and it’s been great. With Apache mod_auth_openidc it’s easy to stick that in-front of most things even
static sites.
Object Storage
I used to use Digitalocean S3 compatible storage (still do a bit tbh), but I am slowly moving all my use of that over to a private Minio instance. I do not need it to be super high available but do need the data on it secure so this runs on my Hetzner Backup server with its 6 disk redundant storage setup.
To be honest I think Minio is just unusable. It’s not that the tool is bad, it’s really great, I really want to love
it, it’s that the project is just crazy with releases. It’s in everyones interest to upgrade and run the latest software
but with 50 releases THIS YEAR ALONE (IT IS AUGUST!), I do not know how anyone use this in any seriousness. This is not
software made to be used by real teams in the real world with real pressures on their time. Further their Release
Notes are of the git log --oneline variety which does not help, commit logs are developer UX not end user UX.
So I am probably looking at an alternative soon. This is kind of why my migration to it have stalled until I can figure out what to do with this.
Security
We all have to stay on top of security alerts, in the good old days you’d just read Bugtraq and know all there is to know. These days though things are much more complex with the amount of things we run and how much goes on the internet.
I use OpenCVE to track and alert me of any CVEs on the main tools I care about.
I’ve also for the first time set my Enterprise Linux machines to auto update themselves, it’s a bit scary to be honest and I exclude kernel updates, but so far it’s been ok. Once a httpd update messed me around a bit but Puppet soon fixed that on next run so was a small inconvenience. On the whole I’d strongly recommend using this.
Monitoring
Apart from the obvious Grafana/Prometheus pair that I do self-host I use a few other things. Graphite to store my IoT data in, it just seems a bit more suitable though alas it’s in a sad state and mostly dying. I might need to revisit what I do there in time.
I have deployed Apprise everywhere and it’s really great it can notify a huge list of services and having it on every machine ready to use is great. I have it integrated in a reusable Gitea action so any failing builds result in alerts.
Apprise sends some alerts and statuses to a dedicated private Mastodon account others to Slack and others to Victorops and to Pushover. This is well worth looking into. Geting Slack/Mastadon reach outs from cron or other tools is really good.
Victorops I use for my Prometheus alerts and some other things, I like its ability to silence and to give me a clear view of the current state of things when I am not around computers. Probably better options now but it’s cheap and just works.
I use Pulsetic to do external checks on my websites, they have a very generous free tier though I did recently upgrade to a paid plan. They allow you to make great external visible or private dashboard like this one for my Choria Project Infrastructure
Letsencrypt
I use LE for TLS like more or less everyone else. ClouDNS is supported for a DNS authenticator in acme.sh so that’s a good fit.
I have a difficult problem in that I want to do a globally redundant hosting of some static files on the same name but I do not want to pay for a GSLB. So I ended up making an Action that will manage those certificates daily, renew them and commit them to my Puppet repository from where they get rolled out to the webservers. This works great.
Awesome Lists
I’ll take liberty and do a plug for my Free-for-dev list that currently has 1500+ services listed. It focus on services that provide generous free tiers generally in the devopsey world many of them suited for home labs. Most of the ones above came from this list, this is literally why I maintain this list. Currently this is the 4th most popular Awesome List with nearly 90 000 stars. Built by 1600+ people collaboratively on GitHub
If you’re not aware I also strongly recommend subscribing to This Week in Self-Hosted if home labs is your thing this will be invaluable.
In general if you are not yet on board with the whole Awesome List movement you really should, check out a giant list in sindresorhus/awesome and you’ll find many tools to track, discover, search and more.
These are the new Yahoo at grass roots level, it’s amazing and one of the most real and relevant resources out there today.
Conclusion
Well that about sums it up.
After this we’ll get back into some general blogging, there’s a fair bit I did not get into like my recent intro into 3D printing and more but this series related to my Home Lab build has more or less run its course.