Skip to content

Commit

Permalink
Initial Parquet File/Dataset abstraction (#501)
Browse files Browse the repository at this point in the history
Prep work for being able to fetch metadata from a bunch of Parquet files
in a folder and load a spatial filter

### Change list

- New `ParquetFile` and `ParquetDataset` Rust structs to read from one
or multiple Parquet files. This is generic over `AsyncFileReader`, which
primarily works with object store.
- Added initial metadata handling, to e.g. read the bounding box of the
file.
- Added Python bindings to each class. This uses ObjectStore.
- Added initial JS bindings to each class. This uses a custom
implementation of `AsyncFileReader` [vendored from
parquet-wasm](https://github.com/kylebarron/parquet-wasm/blob/0ecad6e1d87a377051087122f7ee2db52a2d6533/src/reader_async.rs).
I couldn't get the ObjectStore integration working just yet. Thought
object_store_wasm_s3 was updated and that might be an interesting thing
to check out.
  • Loading branch information
kylebarron authored Mar 20, 2024
1 parent 1f7721c commit 3a2007b
Show file tree
Hide file tree
Showing 26 changed files with 2,632 additions and 285 deletions.
5 changes: 3 additions & 2 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ async-stream = { version = "0.3", optional = true }
async-trait = { version = "0.1", optional = true }
bumpalo = { version = "3", features = ["collections"] }
byteorder = "1"
bytes = { version = "*", optional = true }
bytes = { version = "1.5.0", optional = true }
chrono = "0.4"
# Set default-features = false because async not working in wasm right now
flatgeobuf = { version = "4.1.0", optional = true, default-features = false }
Expand Down Expand Up @@ -79,7 +79,7 @@ rayon = { version = "1.8.0", optional = true }
# Note: geo has a hard dependency on rstar, so there's no point in feature flagging it
rstar = "0.12"
serde = { version = "1", features = ["derive"] }
serde_json = { version = "1" }
serde_json = "1"
sqlx = { version = "0.7", optional = true, default-features = false, features = [
"chrono",
"json",
Expand All @@ -92,6 +92,7 @@ thiserror = "1"

[dev-dependencies]
approx = "0.5.1"
bytes = "1.5.0"
criterion = { version = "0.5", features = ["html_reports"] }
gdal = { version = "0.16", features = ["bindgen"] }
geozero = { version = "0.12", features = ["with-wkb"] }
Expand Down
Loading

0 comments on commit 3a2007b

Please sign in to comment.