v0.1.0

v0.1.0

• package hub: inspect and download files from arbitrary repos. Very functional.
• package tokenizers:
  • Interfaces, types and constants.
  • Gemma tokenizer implementation.
  • Not any other tokenizer implemented yet.
• Examples in README.md.