Move NOMAD examples to plugin #448

lukaspie · 2024-10-09T11:09:43Z

build entrypoints
include in MANIFEST.in
include in pyproject.toml
check if all plugins are really up-to-date -> there may be updates in other NOMAD branches that are not included here.
decide whether some of the examples or at least some of the data shall be downloaded instead of stored in Git
can we have a mechanism so that the pynxtools plugins have their own nomad examples and these are brought into the central pynxtools only if the plugin is installed? This will likely be tricky
EDIT: we can just have the data and the nomad plugin entrypoint within the plugins themselves
test for example data and ExampleUploadEntryPoint
test within NOMAD

src/pynxtools/nomad/entrypoints.py

rettigl · 2024-10-09T11:53:50Z

can we have a mechanism so that the pynxtools plugins have their own nomad examples and these are brought into the central pynxtools only if the plugin is installed? This will likely be trickier than

Such a solution would from my point of view be the most logical place to locate these examples, i.e. together with each reader. In any case, we should follow a SSOT approach for these examples.

lukaspie · 2024-10-09T11:56:37Z

can we have a mechanism so that the pynxtools plugins have their own nomad examples and these are brought into the central pynxtools only if the plugin is installed? This will likely be trickier than

Such a solution would from my point of view be the most logical place to locate these examples, i.e. together with each reader. In any case, we should follow a SSOT approach for these examples.

Absolutely, because as of now, we not only need to update them in NORTH and in nomad-FAIR, but also in all distribution branches in nomad-distro. Can we not try to download them from NORTH? We need them there anyway because the CI/CD checks that the examples work with the installed image.

rettigl · 2024-10-09T12:04:57Z

can we have a mechanism so that the pynxtools plugins have their own nomad examples and these are brought into the central pynxtools only if the plugin is installed? This will likely be trickier than

Such a solution would from my point of view be the most logical place to locate these examples, i.e. together with each reader. In any case, we should follow a SSOT approach for these examples.

Absolutely, because as of now, we not only need to update them in NORTH and in nomad-FAIR, but also in all distribution branches in nomad-distro. Can we not try to download them from NORTH? We need them there anyway because the CI/CD checks that the examples work with the installed image.

It feels a bit strange having them in NORTH at all, because there is not a 1:1 mapping of NORTH tools and pynxtools plugins, i.e. there is not a tool for every example/plugin, no? Maybe one could create another example repo with sub-repos for each example, which then gets submoduled where needed? However, then one still needs to update the submodule commits all the time, so also not ideal.

lukaspie · 2024-10-09T13:32:12Z

can we have a mechanism so that the pynxtools plugins have their own nomad examples and these are brought into the central pynxtools only if the plugin is installed? This will likely be trickier than

Such a solution would from my point of view be the most logical place to locate these examples, i.e. together with each reader. In any case, we should follow a SSOT approach for these examples.

Absolutely, because as of now, we not only need to update them in NORTH and in nomad-FAIR, but also in all distribution branches in nomad-distro. Can we not try to download them from NORTH? We need them there anyway because the CI/CD checks that the examples work with the installed image.

It feels a bit strange having them in NORTH at all, because there is not a 1:1 mapping of NORTH tools and pynxtools plugins, i.e. there is not a tool for every example/plugin, no? Maybe one could create another example repo with sub-repos for each example, which then gets submoduled where needed? However, then one still needs to update the submodule commits all the time, so also not ideal.

I think what we could do is have the pynxtools plugins be the SSOT place that have these folders with the example files that shall be used in NOMAD. Then instead of having the files also in NORTH, we can just get them from GitHub in the NORTH CI/CD for testing. And for the NORTH tools that don't have a directly correspondant plugin, we just don't use this.

Then for the NOMAD examples, we use the url option in ExampleUploadEntryPoint (see docs) and point this to the URL of the examples in the repo.

lukaspie · 2024-10-09T13:35:19Z

can we have a mechanism so that the pynxtools plugins have their own nomad examples and these are brought into the central pynxtools only if the plugin is installed? This will likely be trickier than

Such a solution would from my point of view be the most logical place to locate these examples, i.e. together with each reader. In any case, we should follow a SSOT approach for these examples.

Absolutely, because as of now, we not only need to update them in NORTH and in nomad-FAIR, but also in all distribution branches in nomad-distro. Can we not try to download them from NORTH? We need them there anyway because the CI/CD checks that the examples work with the installed image.

It feels a bit strange having them in NORTH at all, because there is not a 1:1 mapping of NORTH tools and pynxtools plugins, i.e. there is not a tool for every example/plugin, no? Maybe one could create another example repo with sub-repos for each example, which then gets submoduled where needed? However, then one still needs to update the submodule commits all the time, so also not ideal.

I think what we could do is have the pynxtools plugins be the SSOT place that have these folders with the example files that shall be used in NOMAD. Then instead of having the files also in NORTH, we can just get them from GitHub in the NORTH CI/CD for testing. And for the NORTH tools that don't have a directly correspondant plugin, we just don't use this.

Then for the NOMAD examples, we use the url option in ExampleUploadEntryPoint (see docs) and point this to the URL of the examples in the repo.

I still think we need to define the ExampleUploadEntryPoint instances directly in pynxtools and cannot delegate that to the plugins because you need to declare the nomad plugin entry-points in the pyproject.toml of the nomad plugin itself, like so:

[project.entry-points.'nomad.plugin']
apm_example = "pynxtools.nomad.entrypoints:apm_example"
ellips_example = "pynxtools.nomad.entrypoints:apm_example"
em_example = "pynxtools.nomad.entrypoints:em_example"

lukaspie · 2024-10-09T14:19:53Z

Then for the NOMAD examples, we use the url option in ExampleUploadEntryPoint (see docs) and point this to the URL of the examples in the repo.

I think we can just use https://download-directory.github.io to directly get the examples from a different repo as a zip file. Or we use git subtree.

rettigl · 2024-10-10T07:22:03Z

This sounds like a reasonable approach. With the requirements for the entry points, I am not sure I understand this well enough

lukaspie · 2024-10-11T12:50:37Z

docs/how-tos/using-pynxtools-test-framework.md

@@ -60,3 +60,5 @@ You can also pass additional parameters to `test.convert_to_nexus`:
 - `caplog_level` (str): Can be either "ERROR" (by default) or "warning". This parameter determines the level at which the caplog is set during testing. If it is "WARNING", the test will also fail if any warnings are reported by the reader.

 - `ignore_undocumented` (boolean): If true, the test skipts the verification of undocumented keys. Otherwise, a warning massages for undocumented keys is raised
+
+## How to write an integration test for a NOMAD example in a reader plugin


This needs to be updated when it's working

lukaspie · 2024-10-11T13:20:58Z

What I hadn't figured out: we can just have the entry points for the examples in the individual plugins, see FAIRmat-NFDI/pynxtools-mpes#32. That makes it so much cleaner because the data can just live in the plugin repo and we don't need to use the url for the entry point.

lukaspie · 2024-10-11T15:02:41Z

.github/workflows/plugin_test.yaml

@@ -22,31 +22,31 @@ jobs:
        include:

          - plugin: pynxtools-ellips
-            branch: main
+            branch: bring-in-examples


reminder to revert

RubelMozumder

This is what I noticed, you may want to confirm that.

src/pynxtools/nomad/entrypoints.py

src/pynxtools/testing/nomad_example.py

.github/workflows/plugin_test.yaml

Co-authored-by: RubelMozumder <[email protected]>

lukaspie marked this pull request as draft October 9, 2024 11:10

lukaspie commented Oct 9, 2024

View reviewed changes

src/pynxtools/nomad/entrypoints.py Outdated Show resolved Hide resolved

lukaspie commented Oct 11, 2024

View reviewed changes

RubelMozumder reviewed Oct 15, 2024

View reviewed changes

src/pynxtools/nomad/entrypoints.py Outdated Show resolved Hide resolved

src/pynxtools/testing/nomad_example.py Outdated Show resolved Hide resolved

src/pynxtools/testing/nomad_example.py Outdated Show resolved Hide resolved

RubelMozumder reviewed Oct 16, 2024

View reviewed changes

.github/workflows/plugin_test.yaml Show resolved Hide resolved

lukaspie force-pushed the move-nomad-examples-to-plugin branch from fffbf41 to 9c6c453 Compare October 27, 2024 11:32

lukaspie added 15 commits November 1, 2024 17:41

build entrypoints for nomad examples

a0b6f0d

remove example files, use url in example entry points

ed8b6c1

add tests for nomad examples hosted in pynxtools

0386795

remove mpes entry point

10e15a3

add test mechanism for example entry points

f1ab36f

change branches for plugin test

f075be0

import from pynxtools testing

6cf4be1

rename test functions

6736069

use entrypoint directly in test

9a54474

update entry point testing

3c284a9

rename testing function

0b3da3b

remove entrypoints for examples that have been moved to plugins

303ed46

activate apm and em plugin tests

915e1a5

remove unused entrypoints

ec7be63

return archive dict in testing function

cfceb90

lukaspie and others added 9 commits November 1, 2024 17:41

use main branch for xrd reader

37a722a

remove stm/sts example entrypoints

6f42762

use relative path in nomad example test, add docstrings

57a4f71

linting

3717263

use pynxtools-stm main branch

e30ba86

alphabetic order of plugin tests

394529b

Update .github/workflows/plugin_test.yaml

4aff4a3

Co-authored-by: RubelMozumder <[email protected]>

fix IV-temp example

f0610a5

Path for pynxtools-stm from pynxtools-stm reader, for test purpose.

c8c6f75

lukaspie force-pushed the move-nomad-examples-to-plugin branch from 42a7bad to c8c6f75 Compare November 1, 2024 16:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move NOMAD examples to plugin #448

Move NOMAD examples to plugin #448

lukaspie commented Oct 9, 2024 •

edited

Loading

rettigl commented Oct 9, 2024

lukaspie commented Oct 9, 2024

rettigl commented Oct 9, 2024

lukaspie commented Oct 9, 2024

lukaspie commented Oct 9, 2024

lukaspie commented Oct 9, 2024

rettigl commented Oct 10, 2024

lukaspie Oct 11, 2024

lukaspie commented Oct 11, 2024

lukaspie Oct 11, 2024

RubelMozumder left a comment

Move NOMAD examples to plugin #448

Are you sure you want to change the base?

Move NOMAD examples to plugin #448

Conversation

lukaspie commented Oct 9, 2024 • edited Loading

rettigl commented Oct 9, 2024

lukaspie commented Oct 9, 2024

rettigl commented Oct 9, 2024

lukaspie commented Oct 9, 2024

lukaspie commented Oct 9, 2024

lukaspie commented Oct 9, 2024

rettigl commented Oct 10, 2024

lukaspie Oct 11, 2024

Choose a reason for hiding this comment

lukaspie commented Oct 11, 2024

lukaspie Oct 11, 2024

Choose a reason for hiding this comment

RubelMozumder left a comment

Choose a reason for hiding this comment

lukaspie commented Oct 9, 2024 •

edited

Loading