WordPiece

New Features

• Completely re worked Tokenizer now based on WordPiece
• More pythonic interface for Tokenizer

Important Info

• This build is considered stable enough for a minor release but is the first implimentation of the WordPiece algo. It wont be fast and probably not efficient but it should get the job done with little issue.
• GPU acceleration has been temporarily removed from the Tokenizer in this release due to the ground up rework.
• This build is Windows only and requires Intel python 3.9.15 or newer.