EPIC: Storage Optimizations

[yihuang]

The initial target is to reduce disk size requirement of Cronos nodes(#451), as more and more ideas pop up, the target is gradually expanded to be a full-scale storage refactoring, this EPIC issue is created to track the progress.

Phase 1: VersionDB

VersionDB is the earliest plan to reduce disk size requirement of archived full nodes, it's a new storage format designed to store the historical versions of key-value pairs directly instead of go through a merkle tree, it's much more compact than the IAVL merkle tree.

At this stage, versiondb works together with existing IAVL tree, the GRPC query service is served by versiondb while consensus state machine is still served by existing IAVL tree, the archived full nodes can prune the IAVL tree to reclaim disk space, while still support the historical queries.

The main Improvements at this stage:

• Reduce disk size requirement of archive full node by more than 10x.
• Improve performance of grpc query service.
  Notably, the performance of several heavy json-rpc endpoints eth_call/eth_estimateGas/debug_traceTransaction will be greatly improved.

It's already merged into release/v1.0.xbranch, and planed to be released in v1.0.4 soon. released in v1.0.4 already.

VersionDB migration doc.

Related issues:

• Problem: no compact storage for archive node #791
• Problem: windows build in CI fail #853
• Problem: bottommost compression option don't work as expected in restored application.db #862
• Problem: pruning is not supported in current versiondb PR #824
• Problem: changeset verification is not complete #892

Phase 2: IAVL replacement

There's a plan for an alternative IAVL implementation which is expected to perform tremendously better than existing one. We'll try to make it a drop-in replacement of existing IAVL library.

This phase will greatly improve the storage performance of the consensus state machine.

• EPIC: new iavl library is not implemented #870

[yihuang]

close now, only a few minor issues left.