You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We currently allow manifests to contain relative local paths, e.g. test.nc. This is (a) more fragile than an absolute path /test.nc, and (b) not very consistent with cloud bucket urls, which are always absolute.
It would be more robust to ensure that paths in the Manifest are always absolute paths. (If the user wants to move their data around they can always use the .rename_paths method to explicitly adjust the paths in their manifest to point to the files' new locations.)
If all local paths are to be stored as absolute paths then we also might want to use the file URI scheme, so that local paths are stored as file:///test.nc. That way in the Manifest the form of the path is consistent, whether it is local or remote. However as kerchunk does not use this scheme, it will mean extra conversion steps are needed to go between the two formats.
I haven't tried to use fsspec to read data from references that use a file:/// prefix, but the kerchunk readers return dicts containing references that use relative paths. So coercing those relative paths into absolute URIs does break some of VirtualiZarr's kerchunk roundtripping tests.
We currently allow manifests to contain relative local paths, e.g.
test.nc
. This is (a) more fragile than an absolute path/test.nc
, and (b) not very consistent with cloud bucket urls, which are always absolute.It would be more robust to ensure that paths in the
Manifest
are always absolute paths. (If the user wants to move their data around they can always use the.rename_paths
method to explicitly adjust the paths in their manifest to point to the files' new locations.)If all local paths are to be stored as absolute paths then we also might want to use the file URI scheme, so that local paths are stored as
file:///test.nc
. That way in theManifest
the form of the path is consistent, whether it is local or remote. However as kerchunk does not use this scheme, it will mean extra conversion steps are needed to go between the two formats.cc @mpiannucci
The text was updated successfully, but these errors were encountered: