-
Notifications
You must be signed in to change notification settings - Fork 61
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[IO] Improved IO with support for reading data from compressed files #308
Conversation
- Add support for opening a gzip, bzip or lzma-compressed file. - Additional tests for io.open_as_file function
This should also work non-local files, right? Like |
01c6c3a
to
c17764b
Compare
c17764b
to
f3052d8
Compare
Can you merge master in please to make sure tests run again |
827ebad
to
a6a288a
Compare
a6a288a
to
0db0ac9
Compare
I couldn't get boto (to mock an S3 bucket) to work on GitHub Actions. In the most recent version, there is this bug and for older versions I can't figure out a set of version constraints between s3fs and boto that works on each Python version. Hence, I propose to disable these tests until the bug is fixed. I recently also found the xopen library for opening compressed files. We could use it as a more efficient and robust replacement of the |
86eb9d8
to
f7b82d6
Compare
f7b82d6
to
7dcc9f6
Compare
7dcc9f6
to
454a245
Compare
Thanks Pieter, great work! |
It is a common practice to store data as compressed files to reduce storage requirements. With this PR it is no longer needed to decompress the file before loading the data with kloppy.
Whether a file is compressed is derived from the file's extension. Currently supports ".gz", ".xz" and ".bz2".