AOMP Release 19.0-2

These are the release notes for AOMP 19.0-2 AOMP uses AMD developer modifications to the upstream LLVM development trunk. These differences are managed in a branch called the "amd-staging". This branch is found in a mirror of upstream LLVM found at https://github.com/ROCm/llvm-project. The amd-staging branch is constantly changing as it merges the upstream development trunk with its downstream development updates. The AMD modifications are experimental and/or/while contributions under review for the upstream trunk. AOMP uses a snapshot of amd-staging at the commit ids and dates listed below. AOMP also includes builds of related ROCm components. We call AOMP a "standalone" build as it does not use or require ROCm with the exception of the kernel module (dkms) and libdrm which are often part of the Linux distribution. AOMP is isolated from any ROCm installations by installing into /usr/lib/aomp and the use of RPATH for runtime libraries.

For AOMP 19.0-2, the last trunk commit is 6012de2b4ec24826574fe9f2d74c7d2ff2b52f23on June 20, 2024. The last amd-only commit is c3a455408b118b8c22f23c7a65d2b5dbf491ab56 on June 20, 2024. These commits forms a frozen branch now called "aomp-19.0-2". See https://github.com/ROCm/llvm-project/tree/aomp-19.0-2.

The integrated ROCm components for this AOMP release were built with ROCM 6.1.2 sources.
This is the 2nd AOMP release based on LLVM 19 development.

AOMP 19.0-1 was tagged, but will not be released.

Changes since AOMP 19.0-0:

• Significant runtime features to support zero-copy for CPU-GPU unified shared memory. See subsections below.
• Merge of the LLVM upstream relocation of libomptarget into the high level "offload" directory. This establishes the long term objective of the LLVM community to unify offload support for different offloading programming models.
• The integrated ROCm components (non-compiler) were built from ROCM 6.1.2 sources.
• Significant enhancements to the gpurun utility including: support for multiple devices, heterogeneous devices, malloc control inherited from numa-ctl -m -l options, and CPU core binding to same numa node as selected GPU. These changes preserve gpurun's ability to oversubscribe (run multiple processes per GPU) by segmenting a GPUs CUs to different processes. To be fixed in 19.0-3, gpurun fails in TPX mode on MI300X.
• Changes in runtime library locations unique to CPU target triple including fixes for lib64 in Red Hat package.
• Support for fp16 and bfloat16 reductions
• Removed long deprecated utilities mygpu, mymcpu, aompcc, aompExtractRegion, clang-ocl, and cloc.sh.

Errata for AOMP 19.0-2

• gpurun fails in TPX mode for MI300X
• LIBOMPTARGET_SYNC_COPY_BACK default is still true. This is to circumvent a long-standing SDMA problem where signal values appear incorrect to SDMA engines.
• Failure in dynamic_module_load which impacts application termination that uses offloading in shared objects.

Implicit Zero-Copy behavior on MI300A

OpenMP provides a relaxed shared memory model. Map clauses provided in the source code indicate how data is used and copied to and from the GPU device for each target region. On GPUs that provide unified shared memory like the MI300A, these clauses are optional but provide portability to discreet memory GPUs. There is an OpenMP pragma called "requires unified_shared_memory" which tell the compiler and runtime that the code is NOT portable to discreet memory GPUs, and must be compiled and executed on GPUs such as the MI300A. The MI300A is one of several AMD GPUs that has a feature to disable/enable page migration between CPU and GPU called xnack. In this release of the compiler and runtime, we set the runtime behavior depending on the status of xnack and existence of the pragma "requires unified_shared_memory".

MI300A	NO requires unified_shared_memory	requires unified_shared_memory
XNACK enabled	Implicit Zero-Copy	Zero-Copy
XNACK disabled	Copy	Runtime warning*

(*) The runtime warning when running an application using #pragma omp requires unified_shared_memory in XNACK disabled mode can be turned into a runtime error by setting environment variable OMPX_STRICT_SANITY_CHECKS to true (e.g., OMPX_STRICT_SANITY_CHECKS=true ./app_exec).

Implicit Zero-Copy on MI200 and MI300X and any other discrete GPU:

• On discrete memory GPUs, for applications not using #pragma omp requires unified_shared_memory, turn on implicit zero-copy behavior by running applications in XNACK enabled environment and setting to true the environment variable OMPX_APU_MAPS (e.g. HSA_XNACK=1 OMPX_APU_MAPS=1 ./app_exec)
• All other configurations, for applications not using #pragma omp requires unified_shared_memory, will be run in copy behavior.

MI200, MI300X, etc.	not unified_shared_memory	unified_shared_memory
XNACK enabled and OMPX_APU_MAPS=1	Implicit Zero-Copy	Zero-Copy
XNACK enabled	Copy	Zero-Copy
XNACK disabled	Copy	Runtime warning(*)

MI300A host memory pre-faulting in Zero-Copy modes

On MI300A, host memory TLB prefaulting applies when running in in Implicit Zero-Copy and when using #pragma omp requires unified_shared_memory

• By default, for all memory copies with size larger or equal to 1MB, the OpenMP runtime makes the copied host memory visible to the target device agent before calling the copy function
• The environment variable LIBOMPTARGET_APU_PREFAULT_MEMCOPY controls this behavior and it is set to true by default. Setting it to false will disable prefaulting for all memory copy sizes (e.g., disable prefaulting with LIBOMPTARGET_APU_PREFAULT_MEMCOPY=false ./app_exec)
• The environment variable LIBOMPTARGET_APU_PREFAULT_MEMCOPY_SIZE controls the minimum size after which prefaulting is performed. It is currently set to 1MB, meaning that all memory copies that are performed in a synchronous way will have the host memory first prefaulted. Changing the minimum size enables prefaulting at sizes different than larger or equal to 1MB (e.g., to prefault all memory copies larger than 1KB, run with LIBOMPTARGET_APU_PREFAULT_MEMCOPY_SIZE=1024 ./app_exe)