Integration tests - how identical?

[blimlim]

#134 added manual integration tests, which compare the results of converting a ppfile with um2nc against data produced by the original um2netcdf4 script. This has been useful to guard against inadvertent changes that might be missed by the unit tests – it would be great to turn these into CI tests that are run during pull requests!

Up until now the output from um2nc has been bitwise identical to that of um2netcdf4.

However, building a new conda environment for um2nc causes the bitwise integration tests to fail. The cause seems to be a newer version of hdf5 producing some (unclear) difference in the resulting files. See #146 (comment) for details. Meanwhile, nccmp suggests that the data and metadata in the netCDF files are identical.

With the release and future updates, how strictly do we want to adhere to bitwise equivalence?

[blimlim]

@atteggiani had the idea here of restricting hdf5 for the initial release, so that the very first version is bitwise identical to the original script, and moving to the newer versions in future releases.

[MartinDix]

h5dump shows different versions
(0): "version=2,netcdf=4.9.2,hdf5=1.14.3"
cf
(0): "version=2,netcdf=4.9.2,hdf5=1.14.4"
which must be embedded in the file somewhere.

I'd be happy using nccmp -Bdegm to compare the data and metadata rather than an exact file comparison.

[atteggiani]

nccmp suggests that the data and metadata in the netCDF files are identical.

If the data and metadata are identical, then I suggest following @MartinDix's advice and use a less-restrictive comparison for the bitwise compatibility.
In that case there is no need to restrict hdf5 in any release.

[blimlim]

Thanks @MartinDix and @atteggiani, using nccmp for the integration tests sounds like a good plan