Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid/eliminate pickle for storing/loading a code table #17

Open
soxofaan opened this issue Dec 4, 2021 · 1 comment
Open

Avoid/eliminate pickle for storing/loading a code table #17

soxofaan opened this issue Dec 4, 2021 · 1 comment

Comments

@soxofaan
Copy link
Owner

soxofaan commented Dec 4, 2021

PrefixCodec.save() (and .load()) currently use pickle to persist the code table to a file.
While this has the advantage that it handles Python data structures out of the box, it has a couple of disadvantages:

The most obvious alternative is JSON.

@KOLANICH
Copy link

KOLANICH commented Dec 4, 2021

You have options:

  1. Try to utilize some well-known format having a Huffman tree within it. The benefit is that reusing Huffman trees from all files of that format is achieved automatically.
  2. implement an own binary format. To prototype it I recommend Kaitai Struct DSL (it also supports sub-byte stuff, it is not necessary to be limited to whole bytes). It will generate only a parser though. It will still make sense to rewrite the result in vanilla python and mmaps (BTW, feel free to reuse https://github.com/prebuilder/fsutilz.py/blob/master/fsutilz/__init__.py#L8-L47 ).
  3. Use some well-known binary serialization format, as CBOR. It would be less efficient, but more compatible. Also CBOR is not in the stdlib.
  4. Use a text format, like JSON, but it would be inefficient

soxofaan added a commit that referenced this issue Jul 13, 2024
- Refactor out pickle save/load from PrefixCode
- Initial JSON based code table storage
soxofaan added a commit that referenced this issue Jul 13, 2024
- Refactor out pickle save/load from PrefixCode
- Initial JSON based code table storage
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants