GenesisL1 blockchain halted at block 6751399 because of AppHash mismatch because of proposal tx, non determinism happened

[alpha-omega-labs]

Hello,
Finally, after almost 2 years of experiments we first time broke GenesisL1, chain is halted on block 6751399.

What happened:

1. ~68% of network updated to v0.3.1 with patched software to reestablish IBC
2. Someone was a bit hurry and submitted
   tx
   https://ping.pub/genesisL1/tx/F5CB518540246758FB7A5B8990029EFCE5E21ABCE96FEFC4B430284DA1A59992
   @ block
   https://ping.pub/genesisL1/block/6751399
3. Chain stopped with ProposalBlock is invalid err="wrong Block.Header.AppHash. Expected 302F5F6E8F765A70715C289176E00604830D9296D501EEF99093B6B06728DA4B, got 71BA161D6919F462141E4C1D6CDB334F2D96637F3D799AF7BB892F0823CE69CA" height=6751400 module=consensus round=39 server=node

Profit!

Because GenesisL1 is running on pretty outdated chain software, unfortunatelly we dont have rollback command like some advanced ethermint forks e.g. cronos.

Now we have few options to recover it:

1. Make snapthot on latest block prior 6751399 block and statesync validators to there.
2. Rollback with some tool like https://github.com/crypto-com/python-iavl
3. Something else

After fixing this issue we need to think about migration to some adavnced, corporate grade chain software, open source, with frequent up-to-date upgrades, like cronos, because its quality suits GenesisL1 needs and goals and fits important info recorded at GenesisL1 blockchain; to make experiments at least another 2 years and make it harder to break it and easier to recover.

Thank you very much to everyone who participate in solving this issue and helping us!

[github-actions]

This issue is stale because it has been open 45 days with no activity. Remove Status: Stale label or comment or this will be closed in 7 days.