On self-hosting

2026-08-04

At one point in time my self-hosting setup was unavailable. The infrastructure is in another country, not where I'm currently based. Someone who took the role of remote hands was out of town, and kept delaying their return. I asked someone else to go and figure out what's going on. The moment they entered the room they told me they didn't see the server. My first thought was: "oh, did someone steal the gear?". I immediately thought of how terrible the data loss would be in this case - I have all my files there. A few seconds later they told me they saw the server. What a relief.

The setup

I spent a good month building my own little server. I used consumer parts. It has pretty decent specs - 24 vCPU, 64 GiB of RAM. The storage is split into multiple layers:

Dedicated OS NVMe drive, expensive one that has high endurance specs
6-disk ZFS raidz2, plus one spare, 3 disks of one vendor, 4 of another to account for manufacturing errors and therefore reduce the likelihood of all of them failing at the same time
2 NVMe drives for ZFS cache and log - mirrored log, striped cache

All of this runs NixOS. I chose NixOS to have a reliable rollback mechanism in place should the update go wrong. All disks are encrypted, automatic OS drive unlock is bound to TPM, or manual entry of passphrase while the boot is at the initrd stage - I can SSH in and unlock the drive.

The network is fancy too, I have a MikroTik router. Three subnets: one for trusted devices, one for untrusted, and a public subnet for Wi-Fi. All VLANs have access to the Internet, yet can't see each other. Isolation is done on the VRF side too. To connect to the network from the outside, I have a choice of either WireGuard or direct SSH connection. The inbound access is protected by port knocking. It has bandwidth rate-limiting configured, so that the server is prioritized, even with high channel utilization through Wi-Fi.

My backup story is simple - everything is backed up to a ZFS-backed storage, and ZFS itself is snapshotted regularly. I pull snapshots to my external HDD, which I carry with me.

What could go wrong?

Running your own hardware incurs maintenance costs. I knew that when I was building my self-hosting machine. What I did not plan for in detail, is the precise failure modes, each taking some troubleshooting time, or simply waiting until someone could resolve it on-site.

Let me run you through the issues that happened over 3 years:

I caught a CPU freeze mid-work. The SSH was unavailable, yet the network works. The fix was simple - wait until someone comes and reboots the machine. Resulted in ~30 minutes of unavailability and a few minutes of checking the logs and understanding the problem.
Disk failure. The solution is simple, just replace the disk. I had to go and buy one first - 30 minutes. Then replace it, another 10 minutes of work.
3 disks started to intermittently fail operations. Very suspicious. These were disks of one vendor, so at first I thought they share some problem, although SMART showed no issues. I've tried to swap the SATA cables. Did not help. What else could that be? I connected one of the disks to another machine and it worked fine. So, it could be that the controller on the motherboard started to malfunction. I purchased a PCI-e extension card, connected disks to that. Resolved. Monetary costs plus around 3 hours of total time troubleshooting and fixing.
My router died. I'm in another country. I asked a friend to purchase a new one, they connected it to the Internet, so I can remotely configure it before the swap. Gladly I had a checklist for setting up a router. Backup is not available, as the server is unreachable. After a few hours of don't-get-locked-out dance, I set the router up and asked them to go and do the swap. It worked. I spent half a day of work in total. Downtime was roughly 3 days - it took a while to get the replacement.
The story I started the post with. Turns out the UPS died. The device whose entire purpose was to protect against power loss had become the cause of it. The fix? Connect the infrastructure directly to the power outlet. Downtime of 6 days, spent ~30 minutes in total.
Unreliable power causing random reboots. It happens once a month on average. Each reboot takes a couple of minutes to restore the sessions (I also use it as a remote workstation). 2 minutes * 36 reboots = 72 minutes of time wasted.

In addition to that, I have these background worries. First, someone could literally steal the hardware. I worked in small town ISP once and we had cases when network devices went unreachable. You go to the location and there is no switch or router. Second, I'm in another country. If the OS SSD dies, I will have a hard time restoring it remotely.

Time and money

It's not just the incidents. It's the routine 30-minute monthly to upgrade the system and software running on it. I worked as an SRE professionally, I know the importance of updates. Yet, due to lack of automation, and due to my laziness, I haven't updated my system for almost a year now, because life gets in the way. Laziness without automation is just neglect.

You may think that my production has these problems. Not really. I run Flatcar in production with auto-updates on long-running instances and constant instance replacements for ephemeral VMs. Cloud solves the problems for you.

So, let's do some arithmetic. Incidents took: 5 + 40 + 180 + 240 + 30 + 72 minutes of work time, that's 9.45 hours, let's round that to 10. Then, 30 minutes monthly for updates is another 6 hours per year, 18 hours over 3 years. 10 + 18 hours in total. If we take the hourly rate of 50 USD, that's 1400 USD over 3 years, ~467 USD every year, ~39 USD monthly. That's not counting the precious time of someone helping me on-site.

The cost of the hardware is 3k USD approximately - ~83 USD monthly across 3 years. I understand that this one should be reduced as the time goes by. Let's say the hardware will live for 7 more years with nothing requiring replacements. That's 25 USD monthly. Plus the initial setup costs, roughly 60 hours of 2 hours per day for 30 days - divided by 10 years that's 300 USD per year or 25 USD monthly. So, the bottom line is 39 + 25 + 25 = ~89 USD monthly.

I believe another source of cost is background worry of your setup working correctly. That occupies mental bandwidth you could otherwise spend on more productive work or simply enjoying life.

An alternative?

Self-hosting solves the problem of privacy. There is, however, a distinction between the data ownership vs. infrastructure ownership. Most want the former, not the latter. So, what could work, is having an encrypted service alternative for certain data.

Self-hosted is individual by definition, if you start operating the service for someone else, you end up with a privacy problem. Spending 5 hours annually to keep the setup running, multiplied across a million of privacy-conscious users is 5 million of collective meta-work producing no direct value. A handful of encrypted services operated by an engineering team that handles infrastructure professionally elimniates most of that.

While it is easier to manage keys than to manage data, the current story of encrypted services where losing your credentials equals losing your data is far from perfect and has to be addressed. There are solutions to that in applied cryptography space, these are just not applied widely enough.

Don't get me wrong, I'm not abandoning my server - I'm rather attached to it. It has its uses. The maintenance burden can be solved by an encrypted cloud-based service - the problems vanish if you hold the keys to the data. The internet is just not at this stage yet. More on that later.