Lab Infra Rebuild Part 4


This is an ongoing post about rebuilding my lab infrastructure, see the initial post here.

Today I’ll talk about my physical office and office hardware.

Office Space

When my son started going to school I did not look forward to all the driving so figured a office near his school would be good, I’d spend the days there and come home after pick up. I rented a nice place in a town called Mosta, it was nice and had ample storage and would have made a really great maker space as it had about 4 car garages worth of underground storage that was well lit and ventilated.

Unfortunately this place was opposite a school and parking was absolute hell. I ended up just not using it for months at a time since I could drive home in 12 minutes or spend 45 minutes finding parking, no thanks. I gave up trying to find a garage to rent around there, it’s just crazy.

When I started looking the 2nd place in Żebbuġ I saw seemed perfect, a fantastic bright 7m x 7m office with a underground 1.5 car garage at a reasonable price. I took that and just recently extended my lease to 5 years. My sister-in-law is also moving her business to the same street and there’s a 3D printer shop around the corner, bonus. Parking is always easy even on the street and it’s like 5 minutes from my house.

Here I am able to set up 3 workstations, my main desk, one for guests with a little Desktop for my son and big work station for soldering, assembling IoT projects and such. I also have a nice 2 seater sofa with a coffee table. With the garage I have space to put things like a lazer cutter and more in future - though I am eyeing a small space across the road as a workshop space also for that kind of thing.

Location wise I could not want more, it’s a easy walk or cycle from my house through a beautiful car-free valley and the town is pleasant enough with food options and a small corner shop nearby.

Read on for more about the hardware.

Desktop / Laptop

I’ve been using using Apple desktops and laptops almost exclusively since about 2007. I do like the graphical UI but loathe the BSD based shell, so generally my mantra is: MacOS Shells are for SSH to Linux machines.

I’ve had every iMac since the very first plastic white ones, I really liked that form factor, so I just kept buying each model that came out. I especially liked the time when Apple sold visually complimentary displays for these machines and you could have a quite pleasing dual screen setup.

Alas those days are gone, now to be honest, every iMac dual screen setup just looks like rubbish, so I just can’t with that anymore and so it was time to change. I now have 2 useless Display port old Cinema displays, guess they go to the trash.

I have a MacBook 16" M2 PRO 32GB/1TB SSD, which I think is pretty much perfect - despite coffee damage knocking out all the ports on the right side, it’s still awesome.

I’ve always had a disconnect between Laptop and Desktop speeds and this time round the stars aligned reasonably well that one can almost have parity between the 2, so after some looking around I figured a Mac Mini will give me near enough performance and good enough options for displays.

To be honest I was not too convinced about moving to a Mini - but the iMacs were never really speed daemons so maybe it would be fine? So I picked up a Mac Mini M2 PRO 32GB/1TB SSD and really moving between the 2 just feels exactly the same performance wise. This has been a bigger deal than I anticipated for my general happiness when switching setup - something I do all the time.

The mini sitting on my desk is a bit annoying, I’m considering options to mount it on the wall behind the display.

For my display I went with a LG 40" Curved UltraWide 5K2K Nano IPS Monitor. I’ve used a 32 inch ultra wide for a few years, this has been a great upgrade. It’s quite pricey around EUR1200, but it’s a really nice display and as it’s a Thunderbolt monitor it just works flawlessly with the Macs. I did buy BetterDisplay to get some more control, well worth it - I’d say essential if you have a monitor like this. I do wish it was brighter though.

I used to have a Moode Audio system with some high end DACs and Amp paired with JBL monitors for audio, I got those before HomePods existed. Now tossed all that out for just 2 HomePod Minis on my desk, they’re fantastic.

I use a Logitech Brio 4K Ultra HD Webcam - works well on MacOS including it’s management software. My previous camera did not handle the sync frequency of my new office lights well and so was instantly useless.

Today I am replacing my Razer Abyssus (2014) with a new Abyssus that doesn’t look like it’s been to war.

Keyboard wise I use the MS Natural 4000 and have 4 more in boxes so I don’t need to think about keyboard for years, they last about 3 to 4 years each.

Storage

I’ve been a Qnap user since forever, like maybe 2005 or so. The first Qnap I bought was some weird ARM CPU and they ran a patched Linux Kernel to get big partition support. When the machine died I had quite a bit of trouble to access my files but did get there eventually - but did make me reconsider my backup strategy.

Their next gen kit was Intel based and ran stock kernels so I gave them another go. What I got then was a 4 bay TS-439. This machine moved with me from London to Malta and now 14 years later it still got a security update just before I retired it, unbelievable.

Later I got a TS-451+ as the old ones CPU was a bit slow, the old one moved to my office and I synced to it daily as a backup. But as the old one is now 14 years old it seems I am just pushing my luck with this box, so I retired it. Moved the TS-451+ to the position of backup machine and got a new TS-453E.

The TS-453E is interesting because it’s a kind of LTS hardware that will have full hardware support till 2029 and probably software far past that. I got it with an extended warranty also so this should set me up for 5 years at least on this platform.

I set the primary NAS up in 2 x RAID-1 configuration and basically perform backups like this:

  • Real time RAID-1 mirror between 2 disk for the primary volume
  • Daily sync my Dropbox onto the primary NAS
  • Daily sync primary to the backup NAS that is running a RAID-5 setup in another location in Malta
  • Daily sync the primary to a 5TB external USB drive that I can take to a 3rd offsite when I leave the office for months over summer
  • Monthly I do a full bitrot check across all the files on the primary
  • Monthly I do a manual sync between the RAID-1 volumes of the primary NAS
  • Monthly I sync the full primary NAS to Finland on a machine with a RAID-6
  • My primary desktop does TimeMachine backups to the primary regularly

This achieves the following redundancy:

  • +1 disk redundancy for any change I made now thanks to the first RAID-1
  • daily same-country offsite to a RAID-5 storage
  • monthly OS-level integrity checks of all files and the chance to restore any to the primary
  • monthly on-site backup for long term recovery should I notice a accidental file deletion or corruption
  • monthly backup to another country.

In the end every file lands on about 11 disks in duplicate, 3-4 locations, 2 countries, different raid levels and with bitrot detection at multiple RAID levels both standard Linux kernel and NAS kernels and at the OS level. I could make the final Finland archive a 3 month rolling archive since I have enough space there for it.

The machine I mention in Finland is a Hetzner SX64 (64GB RAM Ryzen 5 3600) with 4 x 16 TB SATA Enterprise HDD 7200rpm drives it runs Bacula for my server backups, but for above I just rsync to it. It’s a slow crappy machine tbh but for the purpose of mostly just keepting files on disks mostly idle, it’s perfectly fine and the price is great. I’ve had similar machines for a decade or more, I cycle them every 3 years to new hardware.

Linux Shell

I mentioned earlier that I live by: MacOS Shells are for SSH to Linux machines, so I need a Linux machine.

I used to use the NUC machines but as they were ancient and dying I needed something else. Slack suggested I look at the Asus Mini PC PN51 and what a great suggestion that was.

I have the Ryzen 7 5700U based machine with 64GB RAM and 1TB M.2 SSD in it, this machine runs CentOS and later Almalinux flawlessly. I used to use it bare but that was a bit of a waste now it runs Cockpit as per my previous post and I have a 16GB VM there dedicated as my shell box.

Along with that I have 6-10 VMs elsewhere making up a dev environment.

Miscellaneous Hardware

At my office I have a few other things:

  • One of my old NUC machines is a Linux Desktop for my son, mainly for minecraft/youtube when he visits
  • A Krups Nescafe Dolce Gusto Infinissima Touch Black Automatic Coffee Machine - it’s rubbish, don’t buy these coffee machines
  • A bar fridge
  • Xiaomi air purifier
  • Xiaomi robot vacuum - the nice one with auto bag empty feature so the place keeps clean when I am away for months at a time over summer
  • A Prusa MK4 and it’s many accessories (this is a whole blog post to be made)
  • A 15 year old Brother printer that is enormous and has finally now probably met it’s end, probably fixable, but I want an excuse to buy a smaller printer
  • Ubiquity Dream Machine and a 8 port switch

Everything is new except the Brother printer, but we’ll fix that soon.

I try to not clutter the place up so I keep things to a minimum here and as mentioned I might move my workshop across the road to another unit and get some more 3D printers, Lazer cutters and so.

Conclusion

That’s about it for the physical location and physical hardware. I’ll need to redo my home office as I gutted it and brought most things here but that’s for later, it’s mainly used now for its real purpose of being a media room with a 100 inch 4K project Dolby Atmos audio setup.