A basic approach for managing TLS certificates

So I have this setup. The application server sits behind a network load balancer and terminates TLS. It's a simple web API thingy. The question is: how do you store and distribute certificates?

The solution I came up with is a slightly weird: just store the certs and keys in version control system and compile them into the binary!

Keeping it secure

Obviously storing plain-text keys is no good. We need a solution to that, and luckily enough there is symmetric encryption. Sure, can make use of that. How do you get the encryption key? It has to be deterministic.

I actually already have a "main" key which is the only secret that is transmitted securely to the server, the key is used indirectly for JWT signatures. We can just run a KDF on that key material, for instance using blake3 with a unique context for the encryption key. So the problem of the symmetric encryption key is somewhat solved.

Then what? I think nothing else really, just use that certs encryption key to encrypt the cert keys stored in VCS and then decrypt them on the server side at startup.

Filesystem layout

In VCS, we can build a simple "database" of certificates. The layout can look like this: ./certs/<identity>/<name>/<version>.(key|crt), where identity is the hex-encoded fingerprint of the encryption key, name is the certificate name, and version is a... version number. For example, ./certs/1e2df112203bc77f/star.example.com+ecdsa384/1.key.

Does not seem to be too hard either. Simply have a little CLI tool to access the certs database, and allow for assigning the keys to new identities by re-encrypting the keys. Let it manage the versions too, add existing certs to the database, generate new certs, inspect the stored certs, and keep the database in the correct format whatever that may be. I use PEM-encoded cert chains and encrypted DER keys.

The distribution problem

Just compile the certs. My application server is in Rust, so it drills down to writing a build.rs script that walks through the filesystem and stores all certs and encrypted keys in a hash map. The compile-time hash table is already solved by phf crate. For versions, pick the latest one for starters.

The map is essentially keyed off <identity>:<name>, where certificate name is supplied as a configuration parameter and identity is computed by taking the fingerprint of the certs encryption key. The value is a tuple of (cert_chain, encrypted_key).

What's the catch?

This approach is really good when you have regular rollouts of the "stateless" server. Cert updates could just sneak into that. Plus certs don't change that frequently, and even if they do trigger recompilation of the binary and requires a rollout, building and rolling it out is not that hard and ideally should be an automated process. Moreover, you usually have many days to solve thse problem if the expiration of certs is monitored, which you really should have.

So far I did not find any major downside of that approach, other than little annoyance when rotating the server key material - you have to re-assign the certs. But that does not happen too frequently too, and the simplicity of not having a separate certificate distribution machinery far outweighs that little annoyance.

Of course you could go the filesystem route, but with ephemeral / immutable infrastructure that becomes a pain. The next on the line is a secrets manager service. You may need that for the main key material too, but then why store all the other secrets there, when you can just use symmetric encryption and KDFs? Plus, the compiled certs approach can work in different environments where secrets manager service is not available and you get the powers of VCS for free. Managing many secrets on an external service feels like extra pain too. I tend to try and not hate myself.