ESMF 8.1.1

Overview

The 8.1.1 is a patch release that fixes a major regression in 8.1.0. The regression is in the area of user level OpenMP thread handling, where we received several reports of reduced performance. Performance in these cases has been restored with ESMF 8.1.1.

Release Notes

• This patch release is backward compatible with ESMF 8.1.0.
• No bit-for-bit changes are expected for this release compared to ESMF 8.1.0.
• No changes were made to the ESMF regrid weight generation methods and applications. The ESMF tables summarizing the ESMF regridding status are unchanged.
• This patch release corrects a regression in the handling of user level OpenMP threads. See the Known Bugs section for details.

Known Bugs

• Same as ESMF_8_1_0 with the following exceptions:
  • The issue with OpenMP thread count being reset to 1 within all ESMF components has been fixed.
  • The PETs of all ESMF components, and any potentially created OpenMP threads under such PETs, are no longer pinned automatically to any specific PEs.
• Platform-specific bugs:
  • The same as ESMF_8_1_0.

ESMF 8.1.0

---

IMPORTANT: This release of ESMF contains a known issue with OpenMP thread counts reset to 1 and incorrect pinning of OpenMP threads, leading to reduced application performance in threaded regions. See Known Bugs below for more details. ESMF release 8.1.1 addresses this issue.

---

Overview

During the 8.1.0 development cycle, a number of exciting new features were added to the ESMF library, ease of use was improved in several areas, and performance was optimized in critical parts of the library. Highlights of the 8.1.0 release are listed below. A detailed list of release notes is also provided further down.

The integration of the MOAB mesh library, developed by the U.S. Department of Energy, into ESMF, has reached a milestone that supports application-level testing. All users of ESMF are encouraged to begin testing their applications with the MOAB option turned ON. Preliminary results indicate that MOAB provides improved performance and scaling, and reduced memory footprint. Applications seeking to go to very high resolution grids should benefit from this new capability. The MOAB mesh feature is still OFF by default in this release, but can easily be turned ON from the application level, any time during the run. The available regridding features supported by the MOAB based implementation are clearly listed in the detailed release notes.

Another area of the library that received significant attention is the key-value storage. The new ESMF_Info class, based on a modern C++ JSON implementation, replaces ESMF_Attribute. Most of the legacy ESMF_Attribute API is preserved for backward compatibility, but users are encouraged to migrate their code to using the ESMF_Info API. As a consequence of the new key-value implementation, the NUOPC initialization time has been reduced. This is most pronounced in applications with large Field and PET counts.

Major ease of use improvements went into the NUOPC Layer. One is the introduction of semantic specialization labels into the NUOPC API. The new approach no longer uses Initialize Phase Definition (IPD) versions or the IPDvXXpY nomenclature when registering methods in the SetServices() method. This leads to clearer, more concise NUOPC “cap” implementations. Another improvement is the seamless integration of the NUOPC profiling options into the ESMF profiling infrastructure. Simply by setting the Profiling attribute on NUOPC Driver, Model, Mediator, or Connector components, it is now possible for the user to generate a detailed NUOPC-level performance profile.

A number of new features were added in the area of resource control, handling of threaded components, and shared memory access. The NUOPC layer now supports resource control and handling of threaded components. This mechanism supports hybrid MPI+OpenMP components with different threading levels, allowing each component to fully utilize HPC resources independently. Coupling between threaded components is supported automatically via the standard NUOPC Connectors. Further, data can now be shared by reference between components that run on the same compute nodes even if running with different threading levels. Both Field-level sharing and Array-level sharing are supported through the ESMF API.

New features were also added to the ESMF regridding implementation. It now supports the “nearest destination”, and “creep nearest destination” extrapolation methods. The “creep fill” extrapolation method, introduced in 8.0.0, is now available through ESMPy.

Finally 8.1.0 includes additions to improve the overall user experience with ESMF. In the area of regridding the user has now the option to specify the ‘checkFlag’ argument to ESMF_FieldRegridStore(). This turns on more comprehensive grid/mesh error checking at the cost of performance. The option should therefore be used during application development and debugging, and not during production runs. Finally, for users faced with debugging failing ESMF applications, a new section was added to the User’s Guide entitled "Debugging of ESMF User Applications" that provides some hints on how to interpret stack traces and errors that appear in the ESMF log files.

Release Notes

• This release is backward compatible with the last major release update, ESMF 8.0.0, for all the interfaces that are marked as backward compatible in the Reference Manual. There were API changes to a few unmarked methods that may require minor modifications to user code that uses these methods. The entire list of API changes is summarized in a table showing interface changes since ESMF_8_0_0, including the rationale and impact for each change.
• Some bit-for-bit changes are expected for this release compared to release ESMF 8.0.0 and patch release ESMF 8.0.1. We observe the following impact with Intel compilers using “-O2 -fp-model precise”:
  • Fixed a problem that could result in erroneously unmapped destinations when going from a very fine source grid to a coarse destination grid (e.g. 1/20 degree to 10x15 degree). Expected bit-for-bit changes:
    • ESMF_REGRIDMETHOD_CONSERVE_2ND: roundoff level changes in weight values because of a change in the order of calculation.
    • All regrid methods: Missing weights being added for very fine source grid to coarse destination grid regridding cases as this fix comes into play.
  • Fixed a problem where using the bilinear method between two identical grids doesn't result in an identity matrix for the regridding weights. It also improves the efficiency of the code when using bilinear or patch between identical grids. Expected bit-for-bit changes:
    • ESMF_REGRIDMETHOD_BILINEAR: small changes in regridding weights when a destination point exactly matches a source point.
    • ESMF_REGRIDMETHOD_PATCH: small changes in regridding weights when a destination point exactly matches a source point.
  • Fixed a problem where a set of points with latitudes set at exactly -90.0 are not all mapped to the same point. Expected bit-for-bit changes:
    • All regrid methods: Small weight changes when a point in the grid lies at exactly -90.0 latitude.
  • Optimized the creep fill so the memory doesn't increase as quickly for large numbers of creep levels. Expected bit-for-bit changes:
    • ESMF_EXTRAPMETHOD_CREEP: Small weight changes for the extrapolated destination locations.
  • Fixed an issue where in some cases creep fill weights can trigger an assertion in the code that redistributes weights to their final decomposition. Expected bit-for-bit changes:
    • ESMF_EXTRAPMETHOD_CREEP: Small weight changes for the extrapolated destination locations.
• Tables summarizing the ESMF regridding status have been updated. These include supported grids and capabilities of the offline and integrated regridding, and numerical results of some specific test cases.
• The ESMF_Info class was introduced as a replacement for ESMF_Attribute. ESMF_Info is based on a modern C++ JSON implementation to provide efficient key-value pair storage. Most of the legacy ESMF_Attribute API is preserved for backward compatibility.
• ESMF is in the process of upgrading the internal mesh representation to use the MOAB mesh library developed by the U.S. Department of Energy. In this release, ESMF capabilities using MOAB have been significantly optimized and expanded, allowing for application-level testing of ESMF with MOAB as the underlying mesh representation. MOAB is built into the ESMF library by default, but its use must be enabled at run-time by calling ESMF_MeshSetMOAB(). When MOAB is activated, the following new capabilities are supported in this release:
  • The Mesh creation, conservative regridding, and bilinear regridding algorithms when MOAB is active have been optimized to reduce their memory use and expand the size of Meshes they can be used on.
  • Grids can now be explicitly converted to a Mesh when MOAB is active, using the [ESMF...

ESMF 8.0.1

Overview

The 8.0.1 release fulfills two purposes: to patch a number of bugs discovered in 8.0.0, and to introduce a small selection of critical new performance features. These features were needed by operational centers in an official release that is fully backward compatible with 8.0.0. No bit-for-bit changes from ESMF regridding functions are expected in 8.0.1 relative to release 8.0.0.

One of the performance improvements introduced by ESMF 8.0.1 is message aggregation on the ESMF_VM level. This mechanism significantly improves the efficiency of inter-component data exchanges, especially in situations where there is an imbalance between the sending and receiving side. The imbalance can either be in the total number of sending vs receiving PETs, or in the timing, where the receiving PETs arrive at the exchange late, and out of sync with the sending PETs. The feature is automatically leveraged by the NUOPC level when executing the NUOPC_Connector between components on disjoint sets of PETs.

The other performance improvement added is on the ESMPy level. ESMPy now supports writing/reading of ESMF RouteHandles to/from file. This allows a user to perform the costly RouteHandle generation once, and re-use it in subsequent runs. This provides a more efficient approach to Regridding in the situation where the number of PETs does not change between runs.

Release Notes

• This release is backward compatible with ESMF 8.0.0. Two new interfaces were added to the Fortran API: ESMF_VMEpochEnter() and ESMF_VMEpochExit().
• No bit-for-bit changes are expected for this release compared to ESMF 8.0.0. This has been verified for a large number of regridding tests with Intel compilers using flags "-O2 -fp-model precise".
• No changes were made to the ESMF regrid weight generation methods and applications. The ESMF tables summarizing the ESMF regridding status are unchanged.
• The ESMF Virtual Machine (ESMF_VM) now supports message aggregation to improve performance for some very common communication patterns. Two new methods, ESMF_VMEpochEnter() and ESMF_VMEpochExit(), allow explicit use of this feature.
• The generic NUOPC_Connector automatically takes advantage of VMEpoch message aggregation when used between components on disjoint petLists.
• The ability to write/read ESMF RouteHandles to/from file was added to the ESMPy layer. This allows a user to perform the costly RouteHandle generation once, and re-use it in subsequent runs. This provides a more efficient approach to Regridding in the situation where the number of PETs does not change between runs.
• The pole_kind parameter was added to allow specification of pole behavior when creating an ESMPy Grid.

Known Bugs

• Same as ESMF_8_0_0 with the following exceptions:
  • The race condition in the ESMF_FileRegrid() method and ESMF_Regrid application has been fixed. Now if the destination grid is a multi-tile grid in GRIDSPEC MOSAIC format and the tile is distributed into multiple PETs, the regridded field is written correctly into the output file.
  • Applying the sparse matrix multiplication to cases where the local data allocation is above the 32-bit limit now works reliably.
  • The ESMF_GridCreate1PeriDim() method can now be used to create a bipole connection on the lower side.
  • The ESMF_ArrayCreate() method now succeeds when called from inside a component, requesting DE sharing.
  • The ESMF_DistGridGet() method now correctly returns "de" and "tile" information for a "localDe".
  • ESMF now correctly links against the Fortran bindings of NetCDF, even when the C and Fortran bindings of NetCDF are provided in different locations.
  • The ESMF install target now reliably works when installing ESMF into a location that is shared with other library installations.
  • The Darwin.intelclang.default build configuration now works correctly.
• Platform-specific bugs:
  • The same as ESMF_8_0_0 with the following exceptions:
    • The GNU and Intel compilers require GCC>=4.8 for C++11 support (Intel uses the GCC headers). By default ESMF now uses the C++11 standard and cannot be downgraded. If you run into build issues due to the C++11 dependency, you must make sure a GCC>=4.8 is loaded.

ESMF 8.0.0

Overview

The ESMF 8.0.0 release concludes another phase of evolving and improving the library. The number of applications in which the library is used continues to grow. Requirements from these applications shaped and guided the developments included in this release.

The four typical ways of using ESMF have not changed: 1) to create high-performance, interoperable component-based modeling systems; 2) as a source of data communication, time management, metadata handling, and other libraries; 3) as a fast, parallel generator of interpolation weights from file for many different grids (see the ESMF_RegridWeightGen application website); and 4) as a Python grid remapping library (see the ESMPy website).

Highlights of 8.0.0 include improved component timing profiles and the introduction of a community based NUOPC Field Dictionary. Further extensions to the NUOPC layer include improved performance, and support for driving a NUOPC system from a higher level such as a data assimilation (DA) system. The ESMPy interface now offers in-memory weight access, and the ESMF regridding implementation was extended to include "creep-fill" extrapolation. Core ESMF data classes were extended: A C interface was added to ESMF_XGrid, the creation and use of ESMF_Mesh has been simplified, ESMF_FieldBundles can be created for packed data allocations, and a shared memory access capability was added to ESMF_Array.

More details of the highlighted items are provided in the following paragraphs for convenience.

The tracing capability that was introduced in the previous release now supports a simple mechanism to generate component timing profiles in text files. A single summary timing file can be generated at the end of a run that provides timing statistics across all the PETs. This provides a simple way to understand the relative cost (in terms of wall clock time) of each component in a coupled application.

Management of the NUOPC Field Dictionary has been made more flexible. The dictionary can now be loaded from file during run-time. A community version of the NUOPC Field Dictionary resides in a dedicated public repository and can evolve independently of the NUOPC Layer.

The NUOPC layer now supports situations where a NUOPC system is driven by a higher level driver (outside of NUOPC). One application of this feature is the ability to integrate NUOPC-based forecast systems (such as UFS) with DA systems that require their own driver layer (such as JEDI). Further the NUOPC run sequence was extended to support switching between different run sequence sections during execution. For example, this capability allows changing which components are active at different stages during a run. The overall performance of the NUOPC layer was improved by eliminating unnecessary synchronization, allowing greater opportunity for component concurrency.

The ESMF regridding system was extended in several areas. The "creep-fill" extrapolation method was added to allow the user to spread data from mapped destination points to neighboring unmapped destination points. Regridding weights can now be returned through the ESMPy interface in-memory to eliminating the need to go through netCDF files when accessing weights from Python. The ESMF_Regrid application has been updated to support GRIDSPEC Mosaic files, and regridding on different stagger locations. The ESMF_RegridCheck external demo was added to test the ESMF_Regrid application and the ESMF regridding system with a collection of test grids and data sets.

The ESMF_XGrid interface was wrapped with ESMC bindings, providing simplified access to the ESMF exchange grid implementation through native interfaces from model code written in the C programming language.

The use of the unstructured mesh class, ESMF_Mesh, has been simplified. ESMF_Mesh objects can now be created directly from a structured ESMF_Grid object, using a simple ESMF_MeshCreate() call. Further, a Mesh object can now be queried for its mask and area information making this information accessible to model code that depends on it.

The ESMF_FieldBundle class was extended to cover the case where multiple Fields are packed into a single data allocation. Fields can be interleaved along any dimension of the packed data allocation. The current implementation of the packed feature is limited to cases where the user provides the data allocation to ESMF. Communication calls are supported going from a packed source FieldBundle to a packed destination FieldBundle, however, both sides must provide the same number of Fields, and the order of Fields must be the same on both sides. Further, the NUOPC layer does not currently support exchanging data via FieldBundles, packed or unpacked. Future ESMF releases are planned to address these limitations.

The ESMF_Array class now allows sharing of data between PETs that execute on the same single system image (SSI). This feature can be used for shared memory data access between components that run on the same set of compute nodes (i.e. same SSIs), but run with different number of PETs on each node. This situation is typically encountered when components use different number of threads under each PET. Exchanging data by shared memory access is usually more efficient than having to transfer the data between PETs.

There are many other features and options added throughout ESMF, detailed in the release notes (see link below). Backward compatibility of the Fortran user interface with the ESMF 5.2.0r release series was preserved for methods that are labeled backward compatible in the Reference Manual; the majority of methods fall into this category.

Release Notes

• This release is backward compatible with the last release, ESMF 7.1.0r, for all the interfaces that are marked as backward compatible in the Reference Manual. There were API changes to a few unmarked methods that may require minor modifications to user code that uses these methods. A number of new interfaces were added. The entire list of API changes is summarized in a table showing interface changes since ESMF_7_1_0r, including the rationale and impact for each change.
• Some bit-for-bit changes are expected for this release compared to the last release, ESMF 7.1.0r. We observe the following impact with Intel compilers using "-O2 -fp-model precise":
  • Roundoff level differences in conservative regridding due to an improvement in an area calculation algorithm
  • Roundoff level differences in regridding when used on a Mesh created from a SCRIP format file that contains longitudes <=0 degrees. This change was due to removing a conversion for non-positive longitudes to improve consistency
  • Minor differences in 2nd order conservative regridding for cells that protrude outside their neighbors (e.g. a peninsula made up of a single cell) due to a bug fix in the weight calculation algorithm for that regridding method
• Tables summarizing the ESMF regridding status have been updated. These include supported grids and capabilities of the offline and integrated regridding, and numerical results of some specific test cases.
• Added an option to output component timing profiles in text format by setting the ESMF_RUNTIME_PROFILE environment variable. ESMF and NUOPC component phases are automatically instrumented and user-defined timed regions are also supported. Timing profiles can be written to the end of the ESMF PET log files, to separate per-PET text files, and/or to a single timing summary file. The summary file provides timing statistics across all the PETs. This provides a simple way to understand the relative cost (in terms of wall clock time) of each component in a coupled application.
• The NUOPC Layer contained in this release has been improved in the following specific technical areas:
  • The NUOPC Field Dictionary can now be ingested from a community-based YAML file which resides in a dedicated public repository and can evolve independently of the NUOPC Layer. YAML Ain't Markup Language (YAML) is a human friendly, Unicode-based data serialization language for all programming languages.
  • A NUOPC_Driver can now be called from a higher level driver outside of NUOPC, going through an ESMF API. This is useful for systems that come with their own driver layer, but need to drive a NUOPC system (e.g. Data Assimilation).
  • The NUOPC run sequences now supports the "*" wildcard character in the "@" timestep syntax. This allows the timestep length to be set in code via driver specialization for run sequences ingested from a text file.
  • NUOPC now allows switching between different run sequence sections during execution. For example, this capability allows changing which components are active at different stages during a run. To enable this, the NUOPC_DriverIngestRunSequence() method now supports specification of a run duration, run sequence concatenation, and component run sequence elements outside loops.
  • ConnectionOptions is now an official NUOPC level Connector attribute. The attribute is read during RunSequence ingestion (for each Connector line), and appended by default to all of the CplList entries of the Connector.
• The NUOPC_DriverAddComp() method now supports adding components with a SetVM() method, allowing the component to configure its own VM, e.g. for PET idling for PE reuse under threaded PETs.
• Fields that are mirrored now arrive on the acceptor side with attributes set to reflect information about the provider side Field (TypeKind, GeomLoc, MinIndex, MaxIndex, ArbDimCount, GridToFieldMap, UngriddedLBound, UngriddedUBound). These attributes can then be used when creating Fields on the acceptor side.
  • The NUOPC Mesh transfer protocol was extended to correctly transfer either the node DistGrid, the element DistGrid...