Blog

So.  After the last post, I went “first principle” on things.  Now, I have moved to UnRaid and was finding a lot better stability, user experience, and overall performance.  

Except, still getting some reboots and corruptions.  First principles time.  As a multi-week side quest, I pulled it down, opened it up, and started running full diagnostics and setups and resetting everything.  BIOS was definitely outdated.  I had done memory tests and and seen multiple failures, and then remembered I hadn’t updated the BIOS since building the system.  Yes, I know, there are a lot of rookie system builder mistakes on this path, as it had been a very long time since I assembled any PC, and when I last did, the intricacy of DDR5 was most definitely not a thing.  

So.  Back to the BIOS update.  Then memtests again.  All green…. except the fade test.  Weird.  The server was running pretty solidly when I did boot it up at that point, and a fade test is a pretty exhaustive test of the RAM.  And I did finally get the odd crash with ZFS finding inconsistencies in the memory with the very large disk cache.  

Two years in, I decided to see if Corsair warranties were a) still in force and b) covered the fade tests.   Answer:  Yes to both!  As I had gone through everything recommended and that crucial asked ahead of time and done it actually multiple times while trying to troubleshoot, they were very quick to open up the RMA.  I sent the RAM in, they sent me a replacement kit (64 GB of RAM, and for interest, I had bought it for $254 two years ago, and it is now over $1200 (CAD) for the same kit.  Needless to say, this was an insanely valuable warranty and replacement, and props to Corsair for amazing customer service throughout. 

When the new kit showed up, I again dutifully took the time to test each stick multiple times, ALL tests green this time including the fade, and then for good measure tested them as a set multiple times, also all green.  And the server booted up cleanly and has been running brilliantly since.  Time Machine has been ALMOST flawless. There are known issues where interrupted backups due to sleep or network outages can screw up the sparse bundles.  A verify was able to repair it once, but I also take periodic ZFS snapshots on UnRaid, and that was able to keep most of the longer term backups while recovering from a corrupted sparse bundle.  I need to automate the ZFS snapshots, but that’s not part of the ZFS admin plugin, so that requires a bit of sysadmin work I haven’t gotten to.  

Other bonuses?  MUCH better community support and lower technical/time barriers to doing things with the server.  I’ve since added docker containers for ForgeJo, Postgres, and a Jellyfin server and put a number of my DVDs onto the network.  SO much better. 

So overall, TrueNAS probably had a lot of the issues due to my hardware problems, but when it comes to adding docker apps, UnRaid is SO much more user-friendly, and has a much more vibrant user contribution community from what I have seen for things.  I might be wrong as I didn’t dig in for hours.  The results were just smoother and quicker and … well…. friendlier.  And I’m decently technical on server administration.  

So if I was to recommend a setup for someone, I’ve still got a fairly capable server two years in.  I’d probably go with less RAM and possibly even just DDR4 RAM depending on budget, and maybe less disk, but the overall setup, with UnRAID using zfs, is a much more capable and solid upgrade from the Time Machine server I had previously. 

Well it’s been a VERY long time.  So much has changed.  But enough about me and my gap in posting.

Given the discontinuation of Apple’s Time Capsules a while back, I knew that ultimately I would have to put some other sort of NAS storage system on the network to continue to enjoy the seamless backups.  That time came this summer when the Time Capsule was starting to make some unpleasant sounds which often precede a disk failure. It was time. 

Cue a fair bit of research.  Looking at Synology, Western Digital, Buffalo, Asustor, Qnap, Terramaster.  Many great products. All commercial, simple, decently supported.  But pricey for what you get.  I had just been burned by a bad Kickstarter that was promising a good home NAS and server system, and I wanted all that storage and flexibility, and that price.  So I looked at what the state of the DIY sphere was. 

Ultimately it came down to TrueNAS and Unraid for being able to have solid disk arrays, flexibility and resilience.  TrueNAS is more of an enterprise offering and also had what seemed to be an amazing ecosystem with TrueCharts.  Unraid was also a great alternative, simpler, perhaps a bit less powerful, but also booted off of a flash drive, which just didn’t quite sit right with me.  So ultimately, I went TrueNAS. 

The config:  AMD Ryzen 7 8700g with 64 GB of DDR 5 RAM (not ECC, if you’re into the NAS TrueNAS scene, it’s a debate), 1 TB NVME boot drive, Asus Prime B650M-A AX motherboard, 750w Prime Gold 80+ seasonic power supply, Fractal Design Node 804 case (GREAT case for a server with space for 8 3.5” drives, and really easy access), 5 8 TB Iron World NAS drives and 3 4 TB Iron Wolf NAS drives (2 pools and a somewhat constrained budget).  All that was going to get me a lot more storage than the commercial offerings, a lot of performance for running containers/servers at home, transcoding capability if I did put a Plex server together, and 2.5 Gbit ethernet wired connectivity.  Start the adventure. 

I put it all together, installed TrueNAS and got things up and running.  Just started with a pure Time Machine backup on an SMB share configured for multi user Time Machine.  MUCH faster, and worked great.  I made some mistakes though, and the array started getting fatally corrupted. Cue a fair bit of trouble shooting looking for drive failures, long drive testing, etc.  It seems the motherboard SATA controllers might be getting overloaded, and ZFS is a bit temperamental about the hardware (again, the ECC debate comes up, but that wasn’t where it was happening).  So add in a PCI card with 8 SATA connectors on it per TrueNAS forum recommendations with the 9211 JBOD-configured PCI card.  Rebuild the pools.  All is well.  No more disk corruption.  So.  Pretty sure that problem was diagnosed and solved correctly.  

After more successful backups and stability, it’s time to add in a few servers.  Simple.  Gitea (a source code control server based on Git) and the Postgres database server to support it.  Easy install, easy startup, all good.  TrueCharts was as promised.  

Then I started to get random reboots.  Zero logs, just the computer would randomly reboot itself.  No warning.  Cue another investigation.  It seemed to correlate with the start of Time Machine backups and disk activity spikes.  Hmmmm.  Again testing disks, nothing.  Power?  750W was WELL in excess of the disks all startup up at once.  Not sure.  One of my sons has use for a power supply on a gaming build, so I grabbed a 1000W gold 80+ seasonic.  But before that I grabbed a live Ubuntu image on USB, and booted the system clean on that.  Mounted all the drives and did stress tests on the CPU, RAM, and ever disk including the NVME.  For a week.  Perfectly stable.  Tuned the stress tests to maximize the load and minimize wait times on the arrays, and also to burst it all.  Rock solid.  The hardware recommendations from the TrueNAS forums were as promised.  

So.  Full clean install then.  Wiped everything.  EVERYTHING.  Latest 24.04.2.3 version of TrueNAS, clean install, new arrays, and off we go again.  No load.  How long will it run by itself just doing snapshots (DO THESE WITH ZFS.  It’s like Time Machine for your arrays without piles of backup storage, but you DO still need full backups not in the system, that’s a different post). Flawless.  No issues, smooth as silk for a week again.  Start the Time Machine backups.  Also all good.  Ran for over a week.  Zero issues.  Then random reboots started up again.  The reboots stopped when I reinstalled the entire system, and now are starting again?  This smells of a bug now.  Zero errors of any kind reported by ZFS or S.M.A.R.T. on anything in the disks, NVME, or HDD.  But no disk corruption.  Before I had to revert to a snapshot to have Time Machine work again.  Not this time.  Curiouser and Curiouser.  

Seemed semi-stable now though.  I started the servers again.  Same behaviour for another week.  Still generally stable but random reboots.  Keep looking, keep checking firmware and any other clue or possibility I could find in the forums.  Everything SHOULD be solid. 

And now, corruption of the Time Machine backup again.  The array is again fine, as it has been ever since I put the PCI SATA card in the system.  There’s a bug somewhere.  And all through this the TrueNAS system will randomly reprint IP address for the interfaces despite it being a static address (also done with a permanently assigned DHCP address, same behaviour).  

Well, I KNOW TrueNAS is amazing and solid and brilliant for literally thousands of people and that iXSystems has a great offering for thousands of companies.  But I think I’m throwing the towel in on this.  TrueCharts had a major blow-up with the community and abruptly just UNPUBLISHED all of the repo.  So now there’s an open source war on the ecosystem that was also valuable.  It was all built on Helm Charts on k3s (mini Kubernetes, or k8s if you’re into that) and it was another lfit to just take a simple docker container and get that going, but I was less worried about that.  Now it was a pain. 

Add it all up and then Unraid is now supporting ZFS in their beta stream of 7.0.0-beta-x.  Well, then I looked into backup options and experience on that flash boot drive.  Nothing alarming, and generally really great reviews of response and support.  It is a paid license.  But a very reasonable amount and it is a lifetime license.  No call home things of some of the mainstream commercial stuff, all very open, open standards, and generally a much more “grass roots” company, but very successful.  Standard docker support, and again a great and flourishing ecosystem.  Noted as much more user friendly.  

To be clear, I’ve done a fair bit of sysadmin work over my career, and nothing TrueNAS is doing is leaving me in the dust, but at some point, I want the systems to work.  I’m not hacking on these things, they are supporting my hacking efforts.  I’m more software than hardware.  

So I’m going to look into pulling it all down and trying out Unraid now.  I still back up everything to detached SSD drives and the Git and DB stuff is all also locally mirrored so nothing is at risk.  And I have offsite backups on separate HDDs.  This is an experiment to do more and enable more and also move past the Time Capsules without having drives dangling everywhere.  

So I still have a pile of respect for the TrueNAS system and community, and they are moving with the upcoming 24.10 electric eel releases (in release candidate status as of this posting) to support pure docker and put the TrueCharts fiasco behind them.  All great things.  It’s the fact the foundational reason for this all, the backups, is not reliable FOR ME.  It’s obviously reliable for other people, but I’ve invested a lot of time to make it work, and just haven’t had success, and I can’t swap out every piece of hardware just to try to find some edge case issue in the software after all the mainstream recommended stress tests showed all the hardware to be rock solid.  

So hopefully I will actually do some follow up posts for anyone reading this with what I discover.  🙂