Abusing JWTs
JSON Web Tokens, also known as JWTs, are commonly used for authentication and session management. The server builds a JSON, perhaps with a user identifier, encodes it in a certain way and signs it with their keys and sends it to the client. The client includes the JWT in every request, and the server just has to verify the signature and extract the signed data to authenticate the user. Very cool.
But what if we go further than conventional authentication? Since JWTs contain arbitrary JSON, we can make use of them as state carriers for all sorts of things.
The trick is that JWTs are "stateless" in a sense that they do not require server-side stores. I, by being lazy, don't want to run centralized store infrastructure. It requires operations, monitoring, upgrades, and what not. Plus, when you get into the details of server-side distributed systems, you suddenly start to deal with consensus problems, consistency guarantees between stores, an exponentially growing number of failure modes after adding yet another dependency and so on.
Instead, I choose to push the distributed systems problem to the client. Yes, you can't avoid the problem, just relocate it strategically.
Case 1: Session invalidation
Your typical web service keeps stateful sessions in some sort of a store, and invalidation means erasing that session from that store. "Stateless", which still requires state in reality, is a little tricker. You need to rely on eventual consistency of the invalidation process.
Here's how it looks. In my application, I do authentication via digital signatures, the server stores the public keys per user. What you do is attach a monotonically increasing counter to the public keys, call that a version. Then, issue a JWT with that version included in the claims. To invalidate the session, the user changes the credentials, the public keys version is incremented on the server side, and when the request arrives with a JWT containing the old public keys version, it can be considered stale, rejecting the request.
You may say that this still requires a round-trip to the database. Not really. Versioned data can be cached easily. The only problem is that the cache invalidation will take some time, until the JWT with the new public keys version hits all servers. This is not scalable to a large fleet, but on a reasonably small scale works like a charm. Scaling that may require distributed cache invalidation using message brokers or something, which still seems to be simpler than having a zoo of stores for various purposes.
Another potential downside, compared to stateful sessions, is that with a centralized store the feature of invalidating sessions can be given to the user - they click to log out of some other session. I think this is not really a useful one. If someone got access to the account, they can easily invalidate the sessions of the original owner. The more robust way would be to change the credentials, exactly what the version based invalidation does.
Case 2: Feature flags and subscriptions
I went a crazy route here. I grouped features into feature sets, compiled in the binary. Each feature set is an enum, which expands to the exact feature flags for that set. Think of it like a subscription plan. The feature set information is stored per user in the database, along with the expiration of that feature set.
The client-server interaction looks like this: the server issues a JWT with a feature set enum byte and expiration timestamp. When the request hits the server, it can verify the feature set without hitting the database to see if the operation is allowed. The client-side control of features is just an API endpoint that does a lookup based on the feature set inside a JWT via compile-time mapping, which can easily be polled since it's a "free" handler.
While this works quite well, the update to feature sets requires waiting until the next refresh of the token comes through. No good when the user subscribes - they want the upgrade immediately. I approached this problem with little hacks.
First hack, when the checkout session completes and payment has been processed, yet the subscription state has not been reconciled yet, the server issues a JWT with the subscribed feature set for a short period of time for immediate access for that session.
Another hack is when the user navigates to the subscription management window, the request is made to fetch subscription information, and that handler on the server side looks at the JWT and subscription status, and if the JWT is "worse" than the current subscription plan, it sends a hint to the client to refresh the JWT immediately to pick up the new features. Eventual consistency here again, with minimal load on the server side.
It may not be the best UX, but can be communicated. I will summarize the advantages of doing this later.
Case 3: Read-only mode
I decided to implement a read-only mode for the application server, so I can make the servers enter that mode while performing maintenance or database surgeries, or even restoring from a backup. The application still works for browsing the data, yet the data cannot be uploaded. Again, laziness stopped me from running Redis or something, moreover during outages having an external dependency is just asking for trouble, so here's the poor man's solution.
First, make the read-only mode configurable on the server side at startup. Set something like a timestamp until which it is active, and maybe a message for the users. Let the server issue a JWT with the expiration time of read only mode and check for that field on all mutating handlers.
The trick is to handle the mixed state when the config is updated on the server side - some instances may be read-write, some read-only. I decided to solve that with eventual convergence again. Add a static handler for getting the read-only status, which checks the config and the issued JWT and signals the client to refresh their token. When the server has read-only mode, yet JWT is read-write, immediately signal to refresh to make it read-only. When the server is read-write, and JWT is read-only, allow for a cooldown period before signaling to refresh, which may as well be the rollout window of your application server.
The JWT, again, can be simply checked without going to an external system on all mutating operations. Even read-write servers will treat it as read-only access in a mixed state. In other words, the read-only token is "sticky", and asymmetric refresh prioritizes the read-only state, which is far more important when dealing with emergencies. The "sticky" part is also all-or-nothing. If multiple mutating requests have to be made concurrently, the client won't end up with some requests succeeding and some failing.
So what's good about it?
The real benefit is that you avoid server-side distributed systems problems. The maintenance burden is just too high, especially when running solo and having no team who could manage these systems. This advantage is hard to ignore. You compromise on some properties, such as immediate state propagation, but save on overall complexity and have the happy path work most of the time, with little delays occurring from time to time.
You still need to deal with distributed systems problems in one way or another. Be it server-side or client-facing interaction. Thinking about state transitions with JWTs is also quite tricky, and some logic may be brain-twisting, but it's a one-time investment anyway, not an ongoing operational cost for running Consul, etcd, Redis or whatever that may be.