Compressed block headers & merkle proofs over low bandwidth communications

[willcl-ark]

I hope you don't mind me opening this as an issue here, this is a cross-post of: kyuupichan/electrumx#1007

I have written up an Electrum Protocol proposal which seeks to improve both:

1. Reduction of RPC bandwidth requirements (compressed block headers + protobuf/CBOR (de)serialisation)
2. Reduction of Client requirements when making certain RPC calls (removing block_height requirement)
   The proposal can be found here: https://github.com/willcl-ark/electrum-low-bandwidth/blob/master/spec.adoc

As implementers of an Electrum Server yourselves, I would value your feedback and appreciate any comments on the proposal you may have.

Thanks!

[shesek]

Have you considered using bitcoind's merkleblock SPV proof format? If you're expecting Clients to have multiple transactions per block, it will be more compact because is can encode them together in one go without repeating any leaf information. For a single transaction per block it seems like the Electrum merkle proof format is more compact, though.

[Kixunil]

Why not just sign the replies and verify signatures? Then the whole merkle proof could be dropped. By "working on top of an Electrum Protocol Server." you mean a completely separate proxy?

I like that you actually tried to measure the sizes. Some other ideas to explore:

• Capnproto
• Flat buffers
• Every previous method followed by general-purpose compression algorithm (deflate etc...)
• Measure more than one message to rule-out randomly hitting a special case.

[willcl-ark]

Have you considered using bitcoind's merkleblock SPV proof format? If you're expecting Clients to have multiple transactions per block, it will be more compact because is can encode them together in one go without repeating any leaf information. For a single transaction per block it seems like the Electrum merkle proof format is more compact, though.

Thank you, that is a good suggestion. And yes we have certainly considered the merkle block structure, however I think multiple transactions in one block should be a future optimisation -- I will add it to "future work" section of the proposal spec. For now I would like to focus on improving the existing blockchain.transaction.get_merkle call from the Electrum API for low-bandwidth use.

[willcl-ark]

Why not just sign the replies and verify signatures? Then the whole merkle proof could be dropped.

Sorry, I am not sure I follow what you mean here @Kixunil, we are trying to update this in the most trust-minimised way possible, which I think is achieved via a merkle proof.

By "working on top of an Electrum Protocol Server." you mean a completely separate proxy?

Something like that -- I'm thinking, if for example all the current electrum server implementations don't want to switch to a new RPC wire protocol (e.g. protobuf), then this service could run on top of an Electrum Server, and reflect (deserialise) protobuf requests into Electrum-compatible JSON format, and then serialise the responses. This means we would not break compatibility with all existing wallets and services that are using the JSON format (and might not be as bandwidth-constrained as us).

I like that you actually tried to measure the sizes. Some other ideas to explore:

• Capnproto
• Flat buffers
• Every previous method followed by general-purpose compression algorithm (deflate etc...)
• Measure more than one message to rule-out randomly hitting a special case.

Thanks for the ideas! I have seen the Bitcoin Core devs proposing Cap'n'Proto (for some multiprocess stuff: bitcoin/bitcoin#10102) so will investigate that and flat buffers.

I am hoping to increase the resource and bandwidth tests in the very near future.