What is the right way to handle JSON schema validation for datasets?

[ianwhale]

Adding a new thread from a short discussion on slack.

JSON schema validation fails for datasets not in the default JSON schema.

The example from the thread was using the ibis.TableDataset.

So the question here is, how should Kedro properly handle thses kedro-catalog-X.json files now that datasets have moved to their new home?

Some options I could imagine:

Custom every time

The suggestion from the Slack thread was:

• Pull kedro-catalog-X.json from github.
• Point your IDE to the local version.
• Add in the dataset you want to use.

This seems a little cumbersome though.

Move kedro-catalog-X.json to kedro-plugins

There's a lot going on in the datasets directory lately!

Perhaps this file should move to a new home and be part of the PR process when a (non-experimental) dataset is added.

[lordsoffallen]

While we discuss this, maybe we can also mention the custom dataset support. I think that would be super nice. I am not sure how to do it yet, i didn't think about it much tho..

[noklam]

Couples of comments:

• We should move the json file to kedro-datasets

Part of the solution is in the vscode-kedro extension, but I also understand not everyone is using VSCode. My vision of this for the extension is this has to be solved at runtime.

My question to both @ianwhale and @lordsoffallen, do you use this json file? We don't have to solve the custom dataset problem immediately with this, moving it to kedro-dataset would be simple. Please open an issue about the outdated json file.

Without any runtime information, it's not possible to figure out:

• if kedro_abc actually exists or not?
• with runtime info, it's easy to support custom dataset, we can probably get the signature of a dataset and do some validation that similar to Pydantic style (validating the argument, or type of argument)

Point your IDE to the local version.

This is something that I wish exist, assume I have move it to kedro-dataset, the IDE can just point to my kedro-datasets installation and figure out where is that file, so it automatically handles different environment (versions) of the dataset. This sounds nice but it still requires some extension so that PyCharm/VSCode know where to pull that file.

[lordsoffallen]

I am using it in pycharm with pointing the schema file in github. I think moving it to kedro-datasets makes sense since this file is a json mapping of class definitions there so it does't really belong to kedro in that regard.

For custom datasets, yes can be thought later as you pointed out.

[ianwhale]

Couples of comments:

• We should move the json file to kedro-datasets

Part of the solution is in the vscode-kedro extension, but I also understand not everyone is using VSCode. My vision of this for the extension is this has to be solved at runtime.

My question to both @ianwhale and @lordsoffallen, do you use this json file? We don't have to solve the custom dataset problem immediately with this, moving it to kedro-dataset would be simple. Please open an issue about the outdated json file.

Without any runtime information, it's not possible to figure out:

• if kedro_abc actually exists or not?
• with runtime info, it's easy to support custom dataset, we can probably get the signature of a dataset and do some validation that similar to Pydantic style (validating the argument, or type of argument)

Point your IDE to the local version.

This is something that I wish exist, assume I have move it to kedro-dataset, the IDE can just point to my kedro-datasets installation and figure out where is that file, so it automatically handles different environment (versions) of the dataset. This sounds nice but it still requires some extension so that PyCharm/VSCode know where to pull that file.

Forgive my slow response!

Yeah I don't use it every time since I usually just work with pretty basic datasets.

On larger projects in the past I would tend to use it though. Specifically on PyCharm.

I'll open the issue though at the very least.

As far as pulling it, just having it locally should be enough I guess? Then a project can just version control its own datasets if new ones are added.