ufs-weather-model build on MacBook M3 Sonoma 14.5

[barlage]

I'm attempting to build ufs-weather-model on a MacBook M3 Sonoma 14.5 using spack-stack.

I have successfully built spack-stack 1.7.0 using the ufs-weather-model template, though as a side note, the current build of 1.7.0 will not work with current ufs-weather-model develop due to the UFS dependency on the sp library, which will be removed in a current spack-stack build that uses the ip v5 library. I got around this temporarily by pinning the ip library version to 4.3.0 (same IP version as spack-stack 1.6.0). This at least allows the UFS build to start.

Bottom line is I get an error in the final link that looks like this:

Undefined symbols for architecture arm64:
  "___kmpc_for_static_fini", referenced from:
      _.omp_outlined..1 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..3 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..6 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..7 in libfms_r4.a(create_xgrid.c.o)
  "___kmpc_for_static_init_4", referenced from:
      _.omp_outlined..1 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..3 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..6 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..7 in libfms_r4.a(create_xgrid.c.o)
  "___kmpc_fork_call", referenced from:
      _create_xgrid_2dx2d_order1 in libfms_r4.a(create_xgrid.c.o)
      _create_xgrid_2dx2d_order2 in libfms_r4.a(create_xgrid.c.o)
ld: symbol(s) not found for architecture arm64

My module loading and environment setup prior to building:

module load stack-apple-clang/15.0.0 stack-openmpi/5.0.1 ufs-weather-model-env/1.0.0
source modulefiles/ufs_macosx.gnu
export CMAKE_FLAGS="-DAPP=ATM -D32BIT=ON -DCCPP_SUITES=FV3_GFS_v17_p8"
./build.sh

if I build without 32BIT=ON, I get a similar error:

Undefined symbols for architecture arm64:
  "___kmpc_for_static_fini", referenced from:
      _.omp_outlined..1 in libfms_r8.a(create_xgrid.c.o)
      _.omp_outlined..3 in libfms_r8.a(create_xgrid.c.o)
      _.omp_outlined..6 in libfms_r8.a(create_xgrid.c.o)
      _.omp_outlined..7 in libfms_r8.a(create_xgrid.c.o)
  "___kmpc_for_static_init_4", referenced from:
      _.omp_outlined..1 in libfms_r8.a(create_xgrid.c.o)
      _.omp_outlined..3 in libfms_r8.a(create_xgrid.c.o)
      _.omp_outlined..6 in libfms_r8.a(create_xgrid.c.o)
      _.omp_outlined..7 in libfms_r8.a(create_xgrid.c.o)
  "___kmpc_fork_call", referenced from:
      _create_xgrid_2dx2d_order1 in libfms_r8.a(create_xgrid.c.o)
      _create_xgrid_2dx2d_order2 in libfms_r8.a(create_xgrid.c.o)
ld: symbol(s) not found for architecture arm64

Just wondering if anyone out there is using Mac/ARM with spack-stack and running into a similar problem. Or has anyone with Mac M chips been able to build ufs-weather-model?

[barlage]

I now see that OPENMP is forced OFF for Mac builds. However, the spack-stack seems to build with openmp for FMS. Is this the issue?

[barlage]

Some additional tests:

Removing the force of OPENMP OFF for clang builds in configure_macosx.gnu.cmake, same error:

Undefined symbols for architecture arm64:
  "___kmpc_for_static_fini", referenced from:
      _.omp_outlined..1 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..3 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..6 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..7 in libfms_r4.a(create_xgrid.c.o)
  "___kmpc_for_static_init_4", referenced from:
      _.omp_outlined..1 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..3 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..6 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..7 in libfms_r4.a(create_xgrid.c.o)
  "___kmpc_fork_call", referenced from:
      _create_xgrid_2dx2d_order1 in libfms_r4.a(create_xgrid.c.o)
      _create_xgrid_2dx2d_order2 in libfms_r4.a(create_xgrid.c.o)
ld: symbol(s) not found for architecture arm64

Attempt to remove openmp from fms build of spack-stack:

CMake Error at /packages/spack-stack/envs/ufs-plus_noopenmp2/install/apple-clang/15.0.0/crtm-2.4.0.1-jhrjxvw/lib/cmake/crtm/crtm-targets.cmake:61 (set_target_properties):
  The link interface of target "crtm::crtm" contains:

    OpenMP::OpenMP_Fortran

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.

Call Stack (most recent call first):
  /packages/spack-stack/envs/ufs-plus_noopenmp2/install/apple-clang/15.0.0/crtm-2.4.0.1-jhrjxvw/lib/cmake/crtm/crtm-config.cmake:30 (include)
  FV3/upp/CMakeLists.txt:51 (find_package)


-- Generating done (0.1s)
CMake Generate step failed.  Build files cannot be regenerated correctly.

A few additional tests of trying to remove openmp from spack-stack always results in CRTM cmake failure, presumably because there is no way to build CRTM without openmp in spack-stack (?).

Additional test on MacBook Pro intel chip with Ventura 13.6.7:

• successfully built spack-stack 1.7.0 (with IP pinned to 4.3.0 as mentioned above)
• UWM build using the same configuration as my original test results in the same FMS error referenced in the first post above

Undefined symbols for architecture x86_64:
  "___kmpc_for_static_fini", referenced from:
      _.omp_outlined..1 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..3 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..6 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..7 in libfms_r4.a(create_xgrid.c.o)
  "___kmpc_for_static_init_4", referenced from:
      _.omp_outlined..1 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..3 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..6 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..7 in libfms_r4.a(create_xgrid.c.o)
  "___kmpc_fork_call", referenced from:
      _create_xgrid_2dx2d_order1 in libfms_r4.a(create_xgrid.c.o)
      _create_xgrid_2dx2d_order2 in libfms_r4.a(create_xgrid.c.o)
ld: symbol(s) not found for architecture x86_64

[gspetro-NOAA]

Hi Mike!

I forwarded your question yesterday to the team that usually helps troubleshoot spack-stack dependencies. @natalie-perlin mentioned that she is working on building spack-stack (v1.6.0) on Mac but hasn't succeeded yet. Most of our other team members are on Linux systems, I believe. I will see if I can get any more information from them, but several people are either on PTO or about to travel to UIFCW, so there may be a bit of delay. Didn't want you to think we were ignoring the question though!

Best,
Gillian

[climbfuji]

This will be fixed in FMS with an update to [email protected] or newer. Until then, you need to force linking against libomp.dylib on macOS because FMS doesn't declare its FMS C dependency correctly. Use -DCMAKE_SHARED_LINKER_FLAGS="${llvm_openmp_ROOT}/lib/libomp.dylib" and the same for CMAKE_EXE_LINKER_FLAGS. See also https://github.com/JCSDA/ufs-bundle/blob/develop/README.md.

[barlage]

Using the suggestion from @climbfuji I was able to compile with this command:

export CMAKE_FLAGS="-DCMAKE_SHARED_LINKER_FLAGS="${llvm_openmp_ROOT}/lib/libomp.dylib" -DCMAKE_EXE_LINKER_FLAGS="${llvm_openmp_ROOT}/lib/libomp.dylib" -DAPP=ATM -D32BIT=ON -DCCPP_SUITES=FV3_GFS_v17_p8"
./build.sh

[barlage]

After the successful compile, the model still doesn't not run. I have confirmed with another Mac user who gets the same error (I believe using spack-stack 1.6.0 on an M2):

* . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * .
     PROGRAM ufs-weather-model HAS BEGUN. COMPILED       0.00     ORG: np23
     STARTING DATE-TIME  JUL 22,2024  17:19:27.024  204  MON   2460514


MPI Library = Open MPI v5.0.1, package: Open MPI  Distribution, ident: 5.0.1, repo rev: v5.0.1, Dec 20, 2023
MPI Version = 3.1
libc++abi: terminating due to uncaught exception of type nlohmann::json_abi_v3_11_2::detail::out_of_range: [json.exception.out_of_range.403] key 'NUOPC' not found

Anyone seeing or seen something similar or have any suggestions?

I'm attempting to run an approximation of the control_p8 regression test setup.

[gspetro-NOAA]

@climbfuji Do you happen to have any suggestions for Mike?

[climbfuji]

Talk to the ESMF team. This looks very much like the bug I filed 3+ years ago that the UFS crashes immediately in ESMF on macOS with apple-clang ...

[climbfuji]

The good news is that your crash report at least tells us what the problem is (mine never did)

[gspetro-NOAA]

@barlage Were you able to get in touch with the ESMF folks to work on the issue, or do you need more assistance from us?

[uturuncoglu]

@barlage There is no need to be a super ESMF user. Please open a discussion in ESMf side and add all the information that you could get. Definitely, output of the make info would help. If you could also run the unit tests using make all_tests in your installation, that would help to track down the issue. We also discuss it on ESMF Core call yesterday and we need to have more information to see what is going on. @climbfuji Once @barlage open a discussion, please fell free to add any comment that you think it would be helpful. We have couple of people in the team that has ESMF build on Mac without any issue at this point.

[barlage]

@uturuncoglu maybe "power user" is a bit much, but for example, to do what you asked, I believe I have to rebuild spack-stack and prevent cleaning the build (I currently don't know how to do this) because I don't see where I can run make info or make all_tests in the current installation. I can follow the (excellent) instructions for building spack-stack on Mac but it will take me some time if I need to go off script.

[uturuncoglu]

@barlage Yes, you are right since you are using spack-stack, it would be hard to run tests. I think you could still access to make info information without building and running anything. Just find the path that ESMf installed and there is a .spack/ directory in it. I think spack-build-out.txt file has the make info information. spack-build-env.txt will also help. @climbfuji @theurich I am not sure spack allow it but it would be nice to add variant to esmf spack package to run make all_tests command as a part of installation. Of course this could be issue if the installation node does not allow to run MPI jobs in the compute node. So, maybe we could only run part of it.

[climbfuji]

spack build-env esmf ... would be one way to build and run the tests manually. We should look into running the tests as part of the spack installs (there's a ticket open in spack-stack to expand unit testing of packages during the installation).

To be clear, there is no issue building ESMF on the mac, the problem is launching the ufs-weather-model executable. I vaguely remember that the problem only occurs when using apple-clang, not with gcc - but I am not sure. The issue with that is that it is getting increasingly difficult to build the entire software stack on the mac with gcc.

[gspetro-NOAA]

@barlage I think @natalie-perlin has worked on a Mac with the SRW App, but unfortunately, we have so much to do supporting the Tier-1 systems that any work on a Mac has generally been an afterthought/low priority. I will put in a request to management to provide expanded support for UFS on a Mac, since I think it falls well within our "community engagement" goals, but I'm guessing it will receive low priority. I think the fastest path forward would be, as @uturuncoglu said, to file a ticket with the ESMF team. If you can include more detail/explanation as to why it's important for you (and others) to be able to run on a Mac, I can include that with the request to management. I know you're not the only stakeholder who would like to run on a Mac, so the more of these requests we can gather, the more likely it is to happen, even if it takes a while.

[DusanJovic-NOAA]

@barlage If you don't need all the libraries that spack-stack provides and you don't need the entire SRW app, but just need to be able to compile ufs-weather-model, UFS_UTILS or UPP, on macOS you can take a look at simple-ufs (https://github.com/DusanJovic-NOAA/simple-ufs). I just re-enabled macos github workflow and it seems it built the model and ran a very short c96 test configuration successfully. The github workflow runs on 'macos-14-arm64', which I believe is Sonoma on M chip. I don't know which M chip exactly. See: https://github.com/DusanJovic-NOAA/simple-ufs/actions/runs/10386707623/job/28758562692a

[natalie-perlin]

@barlage -
I'm working on this issue at the moment. A develop branch of UFS-WM has been successfully built with the spack-stack-1.6.0 upp-addon-env on MacOS (Intel at the moment). Openmpi/4.1.6 was build as part of the spack-stack. To complete the linkage stage, it did require installing libomp.dylib and setting setenv("CMAKE_EXE_LINKER_FLAGS", "/usr/local/opt/libomp/lib/libomp.dylib") in ufs_macosx.gnu.lua modulefile.
Still need to figure out how to run any test/ RT on a MacOS and configure it to use 8 cpu, and will update asap.

[natalie-perlin]

@barlage @climbfuji -

I'm getting a similar runtime error for UFS-WM on Mac OS with Intel, when running it as a forecast workflow task in the SRW. All other SRW workflow tasks that use the same spack-stack-1.6.0 environment complete successfully (after some additional flags set), with or without MPI.
So it looks that the error is related to the weather model configuration or settings for the Clang/MacOS.

* . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * .
     PROGRAM ufs-weather-model HAS BEGUN. COMPILED       0.00     ORG: np23
     STARTING DATE-TIME  OCT 02,2024  23:48:30.833  276  WEN   2460586


MPI Library = Open MPI v4.1.6, package: Open MPI [email protected] Distribution, ident: 4.1.6, repo rev: v4.1.6, Sep 30, 2023 
MPI Version = 3.1
libc++abi: terminating with uncaught exception of type nlohmann::json_abi_v3_11_2::detail::out_of_range: [json.exception.out_of_range.403] key 'NUOPC' not found

Program received signal SIGABRT: Process abort signal.

[natalie-perlin]

My further digging into the issue shows that this runtime issues appear indeed due to ESMF is built (mainly the way it is linked) for the clang/clang++/gfortran combination.

[DusanJovic-NOAA]

Maybe that's why in my github workflow test here I build everything with the same compiler, GNU in this case.

Why would anyone use C/C++ compiler from one vendor and Fortran compiler from another?

[climbfuji]

(1) Because you can't build anything related to Python with gcc on macOS (Apple makes this impossible), therefore if one wants a consistent software environment using spack-stack, the only option is clang/clang++ for the C/C++ compilers
(2) clang/clang++ are generally considered good compilers, and it's nice to be able to test with those
(3) Because flang isn't anywhere close to being ready, one needs to combine with a different Fortran compiler. gfortran is the obvious choice. My preference would be NAG because it is generally considered one of the best Fortran compilers available, but (a) it is expensive and (b) it is very strict and most of our applications (or even 3rdparty libraries) don't compile with it

I've moved away from macOS due to these problems and I am now happily using different Linux distros on my Windows 11 system with WSL2.

[DusanJovic-NOAA]

I heard about NAG but never used it. Is it available on any of the NOAA machines we have access to.

Regarding the Apple vs. Python with GCC issue, since model uses python only in configuration phase system python should work, which means one can build a subset of spack-stack (without Python modules) with GNU and be able to run the model.

About the macOS, I agree, Linux is a much better OS for model development.

[natalie-perlin]

@climbfuji @DusanJovic-NOAA -
Thank you for your comments!

[mathomp4]

As @climbfuji says, ESMF does really well with ESMF_COMPILER=gfortranclang, so that should work on most systems. That's how I run ESMF on my macs (though full GNU can work too).

[natalie-perlin]

Some updates on the diagnostics: ESMF builds and passes all tests on MacOS with either combination, using clang or gcc:

1. clang/clang++ v14.0.0 , gfortran /gnu.12.3, with openmpi/1.4.6, all exhaustive multi-processor 14K+ unit tests and all system tests pass successfully. ESMF libraries and tests are built after loading other dependent spack-stack-1.6.0 modules (MacOS Monterey x86_64)
2. gcc/g++/gfortran from gnu/13.3.0 , with openmpi/1.4.6, all non-exchaustive multi-processor 9K+ unit tests and all system tests pass successfully. (MacOS Ventura, M1)

These tests also include unit tests in $ESMF_DIR/src/Infrastructure/Base/tests that use nlohmann::json pointers for c++, version 3.11.2; a header json.hpp could be found in $ESMF_DIR/src/include/ and in $ESMF_DIR/src/Infrastructure/Base/include/nlohmann/json/ directories.

A ufs_model binary built with the spack-stack (with all the ESMF tests passing) still produces the same error as @barlage reported, when it is run with no specific conda environment (such as srw_app conda environment):

libc++abi: terminating with uncaught exception of type nlohmann::json_abi_v3_11_2::detail::out_of_range: [json.exception.out_of_range.403] key 'NUOPC' not found

Few things are mentioned in this error may draw the attention: libc++abi, nlohmann::json_abi_v3_11_2, and NUOPC json key. I've explored several possible ways to solve the issue, as listed below, but with no avail so far.
a) installing additional libc++ and libc++abi libraries, and using these libraries linked under /usr/local/opt/llvm/lib/ for ufs_model building and runtime ( -lc++ -lc++abi -lunwind) ; and also rebuilding the esmf when these are used.
b) Including json.hpp (v3.11.2) from the ESMF location, when building and linking the ufs_model;
c) Installing the nlohmann-json library (v3.11.3) via brew ; rebuilding the esmf and ufs_model.

Addressing the issue with NUOPC json missing key is likely something to get help from the ESMF developers, It is post this issue in the ESMF repo as well.

[natalie-perlin]

Thanks to the support at https://github.com/orgs/esmf-org/discussions/311 (comment)
building and running ufs_model is successful when CXX linker is used instead of Fortran linker.
This includes the following change in main CMakeLists.txt for the ufs_model:

set_target_properties(ufs_model PROPERTIES LINKER_LANGUAGE CXX)

(note use of "CXX" instead of the currently used "Fortran" in the ufs wm repo)

So the final linking of the executable is the key issue. I did test options to do the last step of linking the ufs_model by hand, and replacing "mpicxx " in the string that assembles a successful ufs_model binary by "mpifort" with the same list of libraries, which did not help to solve the problem.
It seems to boil down to the way clang++ libraries are accessed.

[climbfuji]

Yay! Great detective work, @natalie-perlin