ascii85 encoding instead of hex?

[JeremyRand]

The current protocol's usage of hex encoding adds substantial overhead. Especially in the case of Namecoin's usage of blockchain.block.headers without a checkpoint (meaning AuxPoW data is present), things get big enough that I'm getting inquiries from Namecoin users about whether we can improve the encoding efficiency.

Python 3.4 and higher support ascii85 encoding in the standard library (via the base64 package), which only has 25% overhead for random binary data compared to 100% overhead of hex. ascii85 also can further compress data that has lots of zeros, which seems to be the case in Bitcoin headers (since they contain PoW-generated hashes, which have a lot of leading zeros). AFAICT ascii85 seems to be in fairly widespread usage; e.g. the Golang standard library also supports it.

Is there any particular reason why hex encoding is used everywhere? Would a PR be accepted that adds a new protocol version that switches some of the hex-encoded data to ascii85-encoded data?

[kyuupichan]

No good reason other than I'd not researched alternatives. It could be introduced in the next protocol bump; patches are welcome. I was at one stage considering doing a binary protocol implementation, everything necessary is in aiorpcx, but I have other pressing stuff. There is a chance I will do that as an option for a 2.0 protocol, for which I'm currently planning radical changes.

[kyuupichan]

Actually the right place to do this is aiorpcx. It should support a form of BSON, ascii85 and "regular" UTF-8 encoded JSON like now. Clients would specify an ordered list of which they support in the initial version message (most preferred to least) and the server would have a list of suported formats in its features. The server would choose the client's most preferred version in its response that it also supports, and all later messages would use that.

[kyuupichan]

I will get round to this at some stage.

[JeremyRand]

Yes, agreed that a binary protocol is the "right" way to do this, happy to hear that you're open to this possibility.

It should support a form of BSON, ascii85 and "regular" UTF-8 encoded JSON like now.

How about CBOR? CBOR is substantially more compact than BSON, and is more suited to network transmission (BSON seems to be primarily aimed at database storage applications AFAICT). CBOR is also an IETF standard.

[kyuupichan]

Thanks for the pointer. I actually wrote the bulk of an encoder / decoder last night tht is quite similar to CBOR. I'll review where I want to go with this

[kyuupichan]

kyuupichan/aiorpcX#12

[kyuupichan]

Closing this; it's more likely I'll move to a REST API. It's also not an issue for BSV.