Review of features related to external references

[eslavich]

There are currently four ways to reference an object in an external file:

JSON references
These are the $ref keys that are commonly used in ASDF schemas. They can also be used in ASDF files to pull in objects from external ASDF files.

ndarray URI sources
The ndarray schema permits the source property to contain the URI of another ASDF file. Currently this link is limited to the first block in the external file.

External array reference object
Defined by this schema. This is the description:

  Allow referencing of array-like objects in external files. These files can be
  any type of file and in any absolute or relative location to the asdf file.
  Loading of these files into arrays is not handled by asdf.

FITS references (undocumented)
The ASDF-in-FITS feature of the Python library uses ndarray source values like this to reference data in the surrounding FITS file:

fits:SCI,0

This isn't exactly an external reference, but still seems to have something in common with the others.

I suspect external references is a valuable feature, but it seems to me that we should be able to merge some or all of these individual features into one.

One general concern I have about all of them is this idea of referring to an external file by URI. The way we tend to use these is with a simple filename that is assumed to be in the same path as the referencing file. Unlike with schemas, there's no id field in an ASDF file that we can inspect and see that we're reading the correct file, and none of these methods of referencing objects include checksums. This makes it difficult to ensure data integrity, since you could accidentally overwrite the correct external file with something else and never know.

@perrygreenfield @jdavies-st