Rolling deployments & maintaining quorum

[a3kov]

With Khepri we have state in the application itself, so for deployments to work we have to ensure the data is moved to the new build every time. That raises many questions such as:

• if our deployment is not cluster-aware, do we need to share files between releases and make sure 2 builds are not running at the same time ?
• in a cluster aware deployment , how do we maintain the quorum if we only have 1 node ? are we forced to have 3 beam instances even if we only have 1 hardware/vm node ?
• when doing rolling deployment, how to monitor that the quorum is maintained between rolling out a new build for every node ?

In general, I think its good to have a discussion where we can share ideas and best practices on this topic.

[michaelklishin]

Can you please elaborate on the "not cluster aware" part? Khepri was built for deployments with 3, 5, 7 nodes, and so on.

Single node deployments of Khepri in production is something I see little value in. There is no such thing as a "cluster-aware one node deployment".

Multi-node deployments should follow a pretty standard rolling upgrade procedure.

[michaelklishin]

@a3kov there isn't really a whole lot of room for debate. This RabbitMQ upgrade guide explains how multi-node Raft-base systems are upgraded. In fact, most distributed systems are upgraded in a similar fashion, where a minority or overwhelming minority (say, 1 or 3 depending on how many nodes you have in total) of nodes are upgraded and restarted at any given moment, then then next batch, then the next.

In RabbitMQ, there are CLI and HTTP API endpoints that allow you to check if the node is quorum-critical, that is, if restarting it will lead some quorum queues or streams without a quorum.
Nothing like that exists of Khepri but you can quite easily steal the idea.

[a3kov]

Sure. By not cluster aware I mean if you don't set up a cluster (during deployment new build doesn't connect to the old one).

Single node deployments of Khepri in production is something I see little value in. There is no such thing as a "cluster-aware one node deployment".

That's a pity. I see it could be useful if one starts with one node later growing to a cluster. The advantage would be not having to rewrite the code. How many nodes are in the cluster should be handled by the DB itself and not be the client's concern, imo

[michaelklishin]

Well Khepri stores its data both on disk and in memory, so if you want to use a single node deployment, you can. You just have to re-attache the volume where the data directory resides between deployments, and perhaps make sure that the directory does not use any deployment-specific variables, such as the hostname.

[dumbbell]

@a3kov: Are you talking about the Khepri library in general, or are you talking about RabbitMQ with Khepri enabled?

[a3kov]

Khepri in general. RabbitMQ case is covered by RMQ docs as was noted above

[a3kov]

I'm interested in using it in Elixir apps. I understand that its beta software atm, but I think the future is bright for this tech especially with the corporate backing in RMQ. I will be following the development

[michaelklishin]

RabbitMQ with Khepri enabled cannot behave meaningfully differently from standalone Khepri. So the approach outlined in RabbitMQ docs should work, or at the very least serve as a good starting point.