On self-hosting

At one point in time my self-hosting setup was unavailable. The infrastructure is in another country, not where I'm currently based. Someone who took the role of remote hands was out of town, and kept delaying their return. I asked someone else to go and figure out what's going on. The moment they entered the room they told me they didn't see the server. My first thought was: "oh, did someone steal the gear?". I immediately thought of how terrible the data loss would be in this case. A few seconds later they told me they saw the server. What a relief.

The setup

I spent a good month building my own little server from consumer parts. It has pretty decent specs - 24 vCPU, 64 GiB of RAM. The storage is split into multiple layers:

• Dedicated OS NVMe drive, expensive one that has high endurance specs
• 6-disk ZFS raidz2, plus one spare, 3 disks of one vendor, 4 of another to avoid correlated failures
• 2 NVMe drives for ZFS cache and log - mirrored log, striped cache

All of this runs NixOS to have a reliable rollback mechanism in place should the update go wrong. The OS runs on tmpfs with the Nix store and journald bind-mounted to the OS drive, which is why it still warrants the high-endurance spec. All disks are encrypted. The OS drive unlocks automatically via TPM, or manually via passphrase at the initrd stage - including over SSH.

The network is fancy too, handled by a MikroTik router. Three subnets: one for trusted devices, one for untrusted, and a public subnet for Wi-Fi. All VLANs have access to the Internet, yet can't see each other. Isolation is done on the VRF side too. To connect to the network from the outside, there is a choice of either WireGuard or direct SSH connection. The inbound access is protected by port knocking.

My backup story is simple - everything is backed up to a ZFS-backed storage, and ZFS itself is snapshotted regularly. I pull snapshots to my external HDD weekly.

What could go wrong?

Running your own hardware incurs maintenance costs. I knew that when I was building my self-hosting machine. What I did not plan for in detail, is the precise failure modes, each taking some troubleshooting time, or simply waiting until someone could resolve it on-site.

Let me run you through the issues that happened over 3 years:

• I caught a CPU freeze mid-work. The SSH was unavailable, yet the network works. The fix was simple - wait until someone comes and reboots the machine. Resulted in ~30 minutes of unavailability and a few minutes of checking the logs and understanding the problem.
• Disk failure. The solution is simple, just replace the disk. I had to go and buy one first - 30 minutes. Then replace it, another 10 minutes of work.
• 3 disks started to intermittently fail operations. Very suspicious. These were disks of one vendor, so at first I thought they share some problem, although SMART showed no issues. I've tried to swap the SATA cables. Did not help. What else could that be? I connected one of the disks to another machine and it worked fine. So, it could be that the controller on the motherboard started to malfunction. I purchased a PCI-e extension card, connected disks to that. Resolved. Monetary costs plus around 3 hours of total time troubleshooting and fixing.
• My router died. I'm in another country. I asked a friend to purchase a new one, they connected it to the Internet, so I can remotely configure it before the swap. Gladly I had a checklist for setting up a router. Backup is not available, as the server is unreachable. After a few hours of don't-get-locked-out dance, I set the router up and asked them to go and do the swap. It worked. I spent half a day of work in total. Downtime was roughly 3 days - it took a while to get the replacement.
• The story I started the post with. Turns out the UPS died. The device whose entire purpose was to protect against power loss had become the cause of it. The fix? Connect the infrastructure directly to the power outlet. Downtime of 6 days, spent ~30 minutes in total.
• Unreliable power causing random reboots. It happens once a month on average. Each reboot takes a couple of minutes to restore the sessions (it is also used as a remote workstation). 2 minutes * 36 reboots = 72 minutes of time wasted.

In addition to that, I have these background worries. First, someone could literally steal the hardware. I used to work in small ISP once and we had cases when network devices went unreachable. You go to the location and there is no switch or router. Second, I'm in another country. If the OS SSD dies, I will have a hard time restoring it remotely.

Time and money

It's not just the incidents. It's the routine 30-minute monthly to upgrade the system and software running on it. I worked as an SRE professionally. I know the importance of updates. Yet, due to lack of automation, and due to my laziness, I haven't updated my system for a few months now, because life gets in the way. Laziness without automation is just neglect.

You may think that my production environment has these problems. Not really. Production runs Flatcar with auto-updates on long-running instances and constant instance replacements for ephemeral VMs. Cloud solves the problems for you.

So, let's do some arithmetic. Incidents took: 5 + 40 + 180 + 240 + 30 + 72 minutes of work time, that's 9.45 hours, let's round that to 10. Then, 30 minutes monthly for updates is another 6 hours per year, 18 hours over 3 years. 10 + 18 hours in total. If we take the hourly rate of 50 USD, that's 1400 USD over 3 years, ~467 USD every year, ~39 USD monthly. That's not counting the precious time of someone helping me on-site.

The cost of the hardware is 3k USD approximately - ~83 USD monthly across 3 years. I understand that this one should be reduced as the time goes by. Let's say the hardware will live for 7 more years with nothing requiring replacements. That's 25 USD monthly. Plus the initial setup costs, roughly 60 hours of 2 hours per day for 30 days - divided by 10 years that's 300 USD per year or 25 USD monthly. So, the bottom line is 39 + 25 + 25 = ~89 USD monthly.

I believe another source of cost is background worry of your setup working correctly. That occupies mental bandwidth you could otherwise spend on more productive work or simply enjoying life.

I understand I have a remote setup, which adds extra complexity and increases the maintenance overhead, but that's how it ended up for me. If running your own infrastructure is a hobby for you, subtract accordingly.

An alternative?

Self-hosting solves the problem of privacy. There is, however, a distinction between data ownership and infrastructure ownership. I believe most want the former, not the latter. So, if you are after data ownership, what could work is having an encrypted service alternative for certain data. Owning infrastructure on the other hand gives you general-purpose computing, which is useful on its own.

Self-hosted is individual by definition, if you start operating the service for someone else, you end up with a privacy problem. Spending 6 hours annually to keep the setup running, excluding incidents, multiplied across a million of privacy-conscious users is 6 million hours of collective meta-work producing no direct value. A handful of encrypted services operated by an engineering team that handles infrastructure professionally eliminate most of that.

While it is easier to manage keys than to manage data, the current story of encrypted services where losing your credentials equals losing your data is far from perfect - it's solvable, I've built around it for exactly this reason. There are solutions to that in applied cryptography space. These are just not applied widely enough.

To be fair, trust in third-party services is also important. Evaluating cryptographic protocols and key management is a pain, while with self-hosted you get full control over the layers, and all the overhead that entails.

Don't get me wrong, I'm not abandoning my server - I'm rather attached to it. It has its uses. The maintenance burden for some data can be solved by an encrypted cloud-based service - the problems vanish if you hold the keys to the data. The internet is just not at this stage yet. More on that later.

Full disclosure, I've since built one of encypted services myself, which is its own story.