Add NX docstring as attribute

[lukaspie]

@rettigl this is a way to add the NX docstrings to the HDF5 files as attributes. The idea is that you can pass the write-docs flag to the dataconverter and then docstrings are added. By default, this is turned off so as to not change any existing workflows. We can discuss if we want this to happen by default.

Here's an example using the xps reader:
output.nxs.zip

The implementation is relatively trivial. There are however two open questions for me

• How to handle inherited docs
  Usually, our appdefs have very sparse documentation since all the docstrings are in the base classes or in appdefs that are further up the chain. My understanding is that the docs are also extended (not overwritten) when you inherit from a class. So technically, one should concatenate the docs of all inherited nodes. However, this gets quite unwieldy and messy as the docs in the files get rather large and sometimes they might even contradict each other. My suggestion (implemented here) was that as we travel up the inherited nodes, as soon as there is a docstring we only use that one (and don't add any docs coming from even earlier nodes). But this is probably not strictly correct. Maybe @sanbrock can comment.

• Docs for NeXus attributes
  NeXus attributes are written as HDF5 attributes already. Since HDF5 attributes cannot have attributes themselves, the question is where to place the docs for these attributes? My solution here: write another attribute <attribute>__docs (e.g. entry/definition/version__docs) to the HDF5 file.

[lukaspie]

reminder to remove before merging

[sherjeelshabih]

• Docs for NeXus attributes
  NeXus attributes are written as HDF5 attributes already. Since HDF5 attributes cannot have attributes themselves, the question is where to place the docs for these attributes? My solution here: write another attribute <attribute>__docs (e.g. entry/definition/version__docs) to the HDF5 file.

Any practical solution works in my opinion. The only issue will be how to report/add this in the NXDL structure. It's a bit meta over the already meta attributes we have in the NXDL. Will something like adding this renameable field, FIELDNAME__docs, in NXobject work out nicely in the NXDL framework?

[lukaspie]

• Docs for NeXus attributes
  NeXus attributes are written as HDF5 attributes already. Since HDF5 attributes cannot have attributes themselves, the question is where to place the docs for these attributes? My solution here: write another attribute <attribute>__docs (e.g. entry/definition/version__docs) to the HDF5 file.

Any practical solution works in my opinion. The only issue will be how to report/add this in the NXDL structure. It's a bit meta over the already meta attributes we have in the NXDL. Will something like adding this renameable field, FIELDNAME__docs, in NXobject work out nicely in the NXDL framework?

I didn't go this far. For groups and fields, I basically added an attribute @docs, whereas for the attributes, I used an additional attribute <attribute>__docs. I guess the problem would be that all of these are undocumented..

[sherjeelshabih]

Ah alright. So it's just for the attributes that you add a suffix.

You're right. They will remain undocumented. Let's say to see how it goes in use for us we can leave it undocumented.

It will make it practically easier to understand Nexus files like this. It makes the Nexus files more self sufficient too. And it seems this is the best we can do without overcomplicating it.

[lukaspie]

Notes from TF meeting:

• Use single underscore for attributes, like for example data and data_units
• Can we render the docs a bit nicer? HTML syntax?
• Don't write the documentation to every single instance of the same concept, use internal / softlink
• External links to documentation? docs and docs_url
• @sanbrock: HDF5 contains data, not information. What happens when we want to reuse the data somewhere else? We could prevent this by writing a parallel structure with the docs and linking to that place.
  • option 1: add a warning if links are used and the docstrings are not actually correct?
  • option 2 (by @rettigl): if we write the docs, we are not using links, but rather we copy the data directly

[sanbrock]

this is called classpath, but this can be ambiguous if two concept of the same type exists next two one another. This is why the NeXus vocabulary follows the path we call html name which is unique for a given concept. E.g. NXexperiment:/ENTRY/INSTRUMENT/mydetector. Note that this does not hold the information of the nexus classes referened along the path, so if it is needed, a classpath needs to be returned, too.