Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Invalidate cache files when they are corrupted #234

Open
malmans2 opened this issue Oct 17, 2023 · 0 comments
Open

Invalidate cache files when they are corrupted #234

malmans2 opened this issue Oct 17, 2023 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@malmans2
Copy link
Contributor

malmans2 commented Oct 17, 2023

Is your feature request related to a problem? Please describe.

Cache files can get corrupted, especially because their paths are listed in public attributes of the source classes. (For EQC, we are running on a shared VM where users have a broad range of expertise).

Similarly to #202, I think cache files should be invalidated when they're modified.

Describe the solution you'd like

In the snippet below, s2 should trigger a new request.

import earthkit.data

args = ("cds", "reanalysis-era5-single-levels")
kwargs = dict(
    variable=["2t", "msl"],
    product_type="reanalysis",
    area=[50, -10, 40, 10],  # N,W,S,E
    grid=[2, 2],
    date="2012-05-10",
    time="12:00",
)

s1 = earthkit.data.from_source(*args, **kwargs)

with open(s1.path, "w") as f:
    pass

s2 = earthkit.data.from_source(*args, **kwargs)
s2.to_xarray()  # NotImplementedError: earthkit.data.readers.text.TextReader.to_xarray()

Describe alternatives you've considered

No response

Additional context

No response

Organisation

B-Open / CADS-EQC

@malmans2 malmans2 added the enhancement New feature or request label Oct 17, 2023
@sandorkertesz sandorkertesz self-assigned this Jan 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

When branches are created from issues, their pull requests are automatically linked.

2 participants