-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Initial Parquet File/Dataset abstraction #501
Conversation
@H-Plus-Time I'm starting to explore an object-store based async JS Parquet reader in this branch. I'm coming to think it's the best solution because I can reuse so much of the same Rust and Python code. I'm thinking of trying to implement a minimal |
I failed at trying to implement a wasm |
Prep work for being able to fetch metadata from a bunch of Parquet files in a folder and load a spatial filter
Change list
ParquetFile
andParquetDataset
Rust structs to read from one or multiple Parquet files. This is generic overAsyncFileReader
, which primarily works with object store.AsyncFileReader
vendored from parquet-wasm. I couldn't get the ObjectStore integration working just yet. Thought object_store_wasm_s3 was updated and that might be an interesting thing to check out.