Automated post-processing

[aidanheerdegen]

The UM outputs are in a proprietary format (PP files) that should be automatically post-processed to netCDF.

This has traditionally been done with python scripts like um2netcdf.py.

This functionality (amongst others) has been subsumed into a new package:

https://github.com/ACCESS-NRI/amami

which will provide a um2nc tool.

This will require having this package available in the payu conda-environment and adding a payu userscript hook to model configurations to call the um2nc function at the appropriate time.

[truth-quark]

Do we have any indication if um2nc will be run from the command line/scripts and/or called as a library from workflow code?

[aidanheerdegen]

Do we have any indication if um2nc will be run from the command line/scripts and/or called as a library from workflow code?

Likely called as a command line script at least in the first instance.

[aidanheerdegen]

What are the requirements for post-processing? Review of command line options. What options can we exclude?

Is everything that is currently done necessary:

• Coordinate rounding
• Removing unreliable cell methods
• Metadata fixes (coordinate naming,
• Masking pressure level field
• Fixing pressure coordinates
• Converting to 32 bit
• Calendar fixes

Any additional metadata required to meet FAIR data standards?

Pinging @MartinDix for feedback.

[atteggiani]

What should be defined here are all the requirements for the ACCESS-ESM1.5 post-processing conversion to netCDF.

Most importantly, I need to understand if there are any requirements that currently cannot be performed in amami um2nc, so I could work on the implementation of them.

Some examples could be:

• Do we expect any processed um fieldsfile to contain time-series fields?

• Do we expect the STASH variables lookup for the naming of the converted netCDF variables to use the same STASHmaster file or we need to allow the choice of a different STASHmaster files to be used for it?

[truth-quark]

It would also be useful to review other details such as any redundant um2nc edge cases. Ideally, anything that should be removed would be done "now", before retrofitting amami unit tests.

[aidanheerdegen]

@MartinDix can you select which of these are required for the ESM1.5 post-processing?

• Coordinate rounding
• Removing unreliable cell methods
• Metadata fixes (coordinate naming)
• Masking pressure level field
• Fixing pressure coordinates
• Converting to 32 bit
• Calendar fixes (proleptic Gregorian conversion)
• Support time-series
• Support choice of STASHmaster file

[truth-quark]

@MartinDix can you select which of these are required for the ESM1.5 post-processing?

• Coordinate rounding
• Removing unreliable cell methods
• Metadata fixes (coordinate naming)
• Masking pressure level field
• Fixing pressure coordinates
• Converting to 32 bit
• Calendar fixes (proleptic Gregorian conversion)
• Support time-series
• Support choice of STASHmaster file

These are raised as issues/requirements here:
https://github.com/ACCESS-NRI/um2nc-standalone/issues

[truth-quark]

Removing unreliable cell methods is handled here: ACCESS-NRI/um2nc-standalone#73