Add HERD gallery to pynwb with streaming

[mavaylon1]

Motivation

What was the reasoning behind this change? Please explain the changes briefly.

Now that HERD is in pynwb, it should have its own gallery. It should also include an example that handles streaming a dandiset in order to add metadata annotations.

• Add small description of the dandiset
• Replace the HerdManager with NWBFile
• Update the examples to use pynwb objects to make it more practical

How to test the behavior?

Show how to reproduce the new behavior (can be a bug fix or a new feature)

Run the doc.

Checklist

• Did you update CHANGELOG.md with your changes?
• Have you checked our Contributing document?
• Have you ensured the PR clearly describes the problem and the solution?
• Is your contribution compliant with our coding style? This can be checked running flake8 from the source directory.
• Have you checked to ensure that there aren't other open Pull Requests for the same change?
• Have you included the relevant issue number using "Fix #XXX" notation where XXX is the issue number? By including "Fix #XXX" you allow GitHub to close issue #XXX when the PR is merged.

[codecov]

Codecov Report

Merging #1781 (52e7a28) into dev (2aceed0) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##              dev    #1781   +/-   ##
=======================================
  Coverage   91.75%   91.75%           
=======================================
  Files          27       27           
  Lines        2619     2619           
  Branches      684      684           
=======================================
  Hits         2403     2403           
  Misses        141      141           
  Partials       75       75           

Flag	Coverage Δ
integration	71.09% <ø> (ø)
unit	83.46% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[oruebel]

Suggested change

	# Add HERD for Experimenter
	entity = herd.get_entity(entity_id='0000-0001-6782-3819')
	entity_uri = None if entity is not None else 'https://orcid.org/0000-0001-6782-3819'
	herd.add_ref(file=read_file,
	container=read_file,
	attribute="experimenter",
	key=read_file.experimenter[0],
	entity_id = '0000-0001-6782-3819',
	entity_uri = entity_uri
	)
	# Add HERD for Experimenter
	entity = herd.get_entity(entity_id='0000-0001-6782-3819')
	entity_uri = None if entity is not None else 'https://orcid.org/0000-0001-6782-3819'
	key = read_file.experimenter[0]
	try:
	# The experimenter name is unique across files in this DANDIset so reuse the key if possible
	key = herd.get_key(key)
	except ValueError:
	pass # If experimenter is not yet in HERD, then add a new key
	herd.add_ref(file=read_file,
	container=read_file,
	attribute="experimenter",
	key=key,
	entity_id = '0000-0001-6782-3819',
	entity_uri = entity_uri
	)

[oruebel]

In this way we can show that reusing keys across mutliple files is allowed

[oruebel]

If we are already fetching the entity, then it would be nice if add_ref accepted the entity as input, rather than having to specify entity_id and entity_uri each time.

[oruebel]

Suggested change

	)
	)
	# Visualize the HERD file
	herd.to_dataframe()
	# Visualize the individual tables. Here we can see that only 2 unique entities were created.
	# We can also see, that for experimenter we only created a single key that is reused across files.
	#
	# **Files**
	herd.files.to_dataframe()
	# **Objects**
	herd.objects.to_dataframe()
	# **Keys**
	herd.keys.to_dataframe()
	# **Object_Keys**
	herd.object_keys.to_dataframe()
	# **Entities**
	herd.entities.to_dataframe()
	# **Entities_Keys**
	display(herd.entity_keys.to_dataframe()

[oruebel]

Suggested change

	# ---------------------------------
	# --------------------------------------

[oruebel]

Suggested change

	er_read = HERD.from_zip(path='./HERD.zip')
	os.remove('./HERD.zip')
	er_read = HERD.from_zip(path='HERD.zip')
	os.remove('HERD.zip')

[oruebel]

This will only show the last table (here entity_keys) in the output. You will either need to call display(er.files.to_dataframe()) for each of the DataFrames or separate them into cell.

[oruebel]

We shouldn't use the example with a species column in the PyNWB tutorial because it will be confusing for NWB users since there is already a Species field in NWB

[oruebel]

I think this is too much low-level detail. This is information that we should point to the tutorial in HDMF for. In the PyNWB tutorial we should focus on NWB-specific examples and walk through the typical use with NWB.

As such, I would suggest a structure that focuses more directly on the different use-cases of NWB Users:

1. Annotate data while creating a new NWBFile:
   • Show and example where Species is wrapped with a TermSetWrapper works
   • Show how wrapping a whole VectorData column with TermSetWrapper works
   • Show how wrapping with TermSetWrapper allows one to: a) validate data and b) automatically create at the end a HERD file
   • Show and example where add_ref is being used to add a custom reference (e.g., for experimenter)
2. Show how annotating an existing NWBFile works (here I think we can use the DANDIset example)
3. Show how reading and querying a HERD file works

[mavaylon1]

Yes it's part of the PR TODO items to do pynwb examples and not hdmf

[rly]

@mavaylon1 reminder to update this when you have a chance