We had a data loss incident on January 9th 2024, and this project is an automatic reproduction of what happened.
Broadly, this is what it does:
- A 3 node cluster is brought up using
docker-compose.yaml - 2000 databases are created, each with a single document in.
- One node is killed off and its volume destroyed, then brought back up fresh.
- While this node is doing its initial replication, we create a database on it.
- Then we wait to see if that database replicates to the other nodes with 0 docs in it.
The reproduction doesn't have a 100% success rate, in my own testing I got it to work maybe 20% of the time.
In one terminal:
$ python -m pip install -r requirements.txt
$ docker compose up
In another terminal:
$ python ./main.py
The script will run through all of the steps and tell you at the end if data was lost or not. If it wasn't, you can re-run the script until it happens. It detects if the setup has already been done and skips it.