Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update llvmdev to build compiler-rt #979

Closed
wants to merge 15 commits into from

Conversation

oliverhu
Copy link

@oliverhu oliverhu commented Aug 9, 2023

As part of #976 discussion, split the llvmdev build into a separate PR for easier testing.

Validated on Ubuntu, it builds llvm with compiler-rt properly.

❯ tar xvf llvmdev-14.0.6-h84b9e52_3.tar.bz2 
❯ find . -name "libclang*"
./lib/clang/14.0.6/lib/linux/libclang_rt.builtins-x86_64.a

@gmarkall
Copy link
Member

gmarkall commented Aug 9, 2023

I'm experimenting with building the package locally on Linux to begin with - will post any feedback after some experimentation.

@gmarkall
Copy link
Member

Something odd about the build on macOS AArch64 - a snippet of the output:

ld: warning: ignoring file /Users/gmarkall/mambaforge/conda-bld/llvmdev_1691648028380/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/libc++.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
[ 65%] Building ASM object projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins_arm64_ios.dir/outline_atomic_helpers.dir/outline_atomic_ldadd1_3.S.o
[ 65%] Building ASM object projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins_arm64e_osx.dir/outline_atomic_helpers.dir/outline_atomic_ldadd2_2.S.o
cd $SRC_DIR/build/projects/compiler-rt/lib/builtins && $BUILD_PREFIX/bin/arm64-apple-darwin20.0.0-clang -D_DEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -DL_ldadd -DMODEL=3 -DSIZE=1 -I$SRC_DIR/compiler-rt/lib/builtins -I$SRC_DIR/build/projects/compiler-rt/lib/builtins -I$SRC_DIR/compiler-rt/lib/builtins -I$SRC_DIR/build/include -I$SRC_DIR/llvm/include -O3 -DNDEBUG -arch arm64  -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS16.4.sdk -miphoneos-version-min=6.0 -fPIC -O3 -fvisibility=hidden -DVISIBILITY_HIDDEN -Wall -fomit-frame-pointer  -DHAS_ASM_LSE -arch arm64 -o CMakeFiles/clang_rt.builtins_arm64_ios.dir/outline_atomic_helpers.dir/outline_atomic_ldadd1_3.S.o -c $SRC_DIR/build/projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_ldadd1_3.S
cd $SRC_DIR/build/projects/compiler-rt/lib/builtins && $BUILD_PREFIX/bin/arm64-apple-darwin20.0.0-clang -D_DEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -DL_ldadd -DMODEL=2 -DSIZE=2 -I$SRC_DIR/compiler-rt/lib/builtins -I$SRC_DIR/build/projects/compiler-rt/lib/builtins -I$SRC_DIR/compiler-rt/lib/builtins -I$SRC_DIR/build/include -I$SRC_DIR/llvm/include -O3 -DNDEBUG -arch arm64e  -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk -mmacosx-version-min=10.5 -fPIC -O3 -fvisibility=hidden -DVISIBILITY_HIDDEN -Wall -fomit-frame-pointer  -DHAS_ASM_LSE -arch arm64e -o CMakeFiles/clang_rt.builtins_arm64e_osx.dir/outline_atomic_helpers.dir/outline_atomic_ldadd2_2.S.o -c $SRC_DIR/build/projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_ldadd2_2.S
[ 65%] Building ASM object projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.cc_kext_arm64_osx.dir/outline_atomic_helpers.dir/outline_atomic_ldadd2_1.S.o
cd $SRC_DIR/build/projects/compiler-rt/lib/builtins && $BUILD_PREFIX/bin/arm64-apple-darwin20.0.0-clang -DKERNEL_USE -D_DEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -DL_ldadd -DMODEL=1 -DSIZE=2 -I$SRC_DIR/compiler-rt/lib/builtins -I$SRC_DIR/build/projects/compiler-rt/lib/builtins -I$SRC_DIR/compiler-rt/lib/builtins -I$SRC_DIR/build/include -I$SRC_DIR/llvm/include -I$SRC_DIR/compiler-rt/lib/builtins/../../include -O3 -DNDEBUG -arch arm64  -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk -mmacosx-version-min=10.5 -fPIC -O3 -fvisibility=hidden -DVISIBILITY_HIDDEN -Wall -fomit-frame-pointer  -DHAS_ASM_LSE -arch arm64 -mkernel -o CMakeFiles/clang_rt.cc_kext_arm64_osx.dir/outline_atomic_helpers.dir/outline_atomic_ldadd2_1.S.o -c $SRC_DIR/build/projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_ldadd2_1.S
Undefined symbols for architecture x86_64:
  "std::__1::__call_once(unsigned long volatile&, void*, void (*)(void*))", referenced from:
      _wrap__ZNSt3__111__call_onceERVmPvPFvS2_E in tsan_interceptors_mac.cpp.o
      __tsan::substitution__ZNSt3__111__call_onceERVmPvPFvS2_E in tsan_interceptors_mac.cpp.o
     (maybe you meant: _wrap__ZNSt3__111__call_onceERVmPvPFvS2_E)
  "std::__1::__shared_count::__release_shared()", referenced from:
      _wrap__ZNSt3__114__shared_count16__release_sharedEv in tsan_interceptors_mac.cpp.o
      __tsan::substitution__ZNSt3__114__shared_count16__release_sharedEv in tsan_interceptors_mac.cpp.o
     (maybe you meant: _wrap__ZNSt3__114__shared_count16__release_sharedEv)
  "std::__1::__shared_weak_count::__release_shared()", referenced from:
      _wrap__ZNSt3__119__shared_weak_count16__release_sharedEv in tsan_interceptors_mac.cpp.o
      __tsan::substitution__ZNSt3__119__shared_weak_count16__release_sharedEv in tsan_interceptors_mac.cpp.o
     (maybe you meant: _wrap__ZNSt3__119__shared_weak_count16__release_sharedEv)
ld: symbol(s) not found for architecture x86_64
clang-16: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [projects/compiler-rt/lib/tsan/rtl/CMakeFiles/clang_rt.tsan_osx_dynamic.dir/build.make:787: lib/clang/14.0.6/lib/darwin/libclang_rt.tsan_osx_dynamic.dylib] Error 1
make[2]: Leaving directory '$SRC_DIR/build'
make[1]: *** [CMakeFiles/Makefile2:31299: projects/compiler-rt/lib/tsan/rtl/CMakeFiles/clang_rt.tsan_osx_dynamic.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

I'll have to check whether the package builds on main next, or whether this is an issue introduced with this PR.

@gmarkall
Copy link
Member

The package built OK on main on macOS AArch64, so I guess the issue is with the addition of building compiler-rt.

@gmarkall
Copy link
Member

The error seems to be coming from the ThreadSanitizer build, which should be supported even in 14 on macOS AArch64: https://releases.llvm.org/14.0.0/tools/clang/docs/ThreadSanitizer.html

@gmarkall
Copy link
Member

I found a hardcoded x86_64 in the build.sh - will post more info if changing that fixes the issue.

@gmarkall
Copy link
Member

The following (or something a little more elegant) also needs to be applied because it influences the build of compiler-rt:

diff --git a/conda-recipes/llvmdev/build.sh b/conda-recipes/llvmdev/build.sh
index ef21f54..6a5558d 100644
--- a/conda-recipes/llvmdev/build.sh
+++ b/conda-recipes/llvmdev/build.sh
@@ -10,8 +10,10 @@ LLVM_TARGETS_TO_BUILD=${LLVM_TARGETS_TO_BUILD:-"all"}
 # This is the clang compiler prefix
 if [[ $build_platform == osx-arm64 ]]; then
     DARWIN_TARGET=arm64-apple-darwin20.0.0
+    DARWIN_ARCH=arm64
 else
     DARWIN_TARGET=x86_64-apple-darwin13.4.0
+    DARWIN_ARCH=x86_64
 fi
 
 mv llvm-*.src llvm
@@ -74,7 +76,7 @@ if [[ $(uname) == Darwin ]]; then
   # Once we are using our libc++ (not until llvm_build_final), it will be single-arch only and not setting
   # this causes link failures building the santizers since they respect DARWIN_osx_ARCHS. We may as well
   # save some compilation time by setting this for all of our llvm builds.
-  _cmake_config+=(-DDARWIN_osx_ARCHS=x86_64)
+  _cmake_config+=(-DDARWIN_osx_ARCHS=${DARWIN_ARCH})
 elif [[ $(uname) == Linux ]]; then
   _cmake_config+=(-DLLVM_USE_INTEL_JITEVENTS=ON)
 #  _cmake_config+=(-DLLVM_BINUTILS_INCDIR=${PREFIX}/lib/gcc/${cpu_arch}-${vendor}-linux-gnu/${compiler_ver}/plugin/include)

@oliverhu
Copy link
Author

oliverhu commented Aug 11, 2023

Got this error on my macos (M1), looking

Testing compiler for supporting FLAGS-fxray-instrument:
Command: /Users/khu/miniconda3/envs/rt/conda-bld/llvmdev_1691732858678/_build_env/bin/arm64-apple-darwin20.0.0-clang;;;-ftree-vectorize;-fPIC;-fPIE;-fstack-protector-strong;-O2;-pipe;-isystem;/Users/khu/miniconda3/envs/rt/conda-bld/llvmdev_1691732858678/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/include;-fdebug-prefix-map=/Users/khu/miniconda3/envs/rt/conda-bld/llvmdev_1691732858678/work=/usr/local/src/conda/llvmdev-14.0.6;-fdebug-prefix-map=/Users/khu/miniconda3/envs/rt/conda-bld/llvmdev_1691732858678/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh=/usr/local/src/conda-prefix;-fPIC;-Werror=date-time;-Werror=unguarded-availability-new;-Wall;-Wextra;-Wno-unused-parameter;-Wwrite-strings;-Wmissing-field-initializers;-pedantic;-Wno-long-long;-Wc++98-compat-extra-semi;-Wimplicit-fallthrough;-Wcovered-switch-default;-Wstring-conversion;-Wmisleading-indentation;-Wall;-Wno-unused-parameter;-fxray-instrument;-o;/Users/khu/miniconda3/envs/rt/conda-bld/llvmdev_1691732858678/work/build/CMakeFiles/test.o;-c;/Users/khu/miniconda3/envs/rt/conda-bld/llvmdev_1691732858678/work/build/CMakeFiles/src.c

clang-14: error: the clang compiler does not support '-fxray-instrument on arm64-apple-darwin20.0.0'

@oliverhu
Copy link
Author

Skipping those binaries for now:

_cmake_config+=(-DCOMPILER_RT_BUILD_BUILTINS:BOOL=ON)
_cmake_config+=(-DCOMPILER_RT_BUILD_LIBFUZZER:BOOL=OFF)
_cmake_config+=(-DCOMPILER_RT_BUILD_CRT:BOOL=OFF)
_cmake_config+=(-DCOMPILER_RT_BUILD_MEMPROF:BOOL=OFF)
_cmake_config+=(-DCOMPILER_RT_BUILD_PROFILE:BOOL=OFF)
_cmake_config+=(-DCOMPILER_RT_BUILD_SANITIZERS:BOOL=OFF)
_cmake_config+=(-DCOMPILER_RT_BUILD_XRAY:BOOL=OFF)
_cmake_config+=(-DCOMPILER_RT_BUILD_GWP_ASAN:BOOL=OFF)
_cmake_config+=(-DCOMPILER_RT_BUILD_ORC:BOOL=OFF)
_cmake_config+=(-DCOMPILER_RT_INCLUDE_TESTS:BOOL=OFF)

@oliverhu
Copy link
Author

oliverhu commented Aug 11, 2023

Validated in macOS & Linux

❯ nm -a ./lib/clang/14.0.6/lib/darwin/libclang_rt.osx.a | grep f2h
000000000000011c T ___gnu_f2h_ieee
./lib/clang/14.0.6/lib/linux/libclang_rt.builtins-x86_64.a
❯ readelf -s -A ./lib/clang/14.0.6/lib/linux/libclang_rt.builtins-x86_64.a | grep f2h
     7: 0000000000000000     0 SECTION LOCAL  DEFAULT    5 .text.__gnu_f2h_ieee
    12: 0000000000000000     5 FUNC    GLOBAL DEFAULT    5 __gnu_f2h_ieee

@oliverhu
Copy link
Author

Windows:


C:\Users\test\source\repos>dumpbin /symbols C:\Users\khu\llvm\build\llvmdev-14.0.6-4\Library\lib\windows\clang_rt.builtins-x86_64.lib
...
00B 00000000 SECT4  notype ()    External     | __truncsfhf2
00C 00000000 SECT3  notype ()    External     | __gnu_f2h_ieee

@oliverhu
Copy link
Author

Validated llvmdev_manylinux2014 via ./buildscripts/manylinux_2_17/docker_run_x64.sh build_llvmdev.sh, instead of conda-build llvmdev_manylinux2014

+ exit 0

Resource usage statistics from testing llvmdev:
   Process count: 2
   CPU time: Sys=-, User=0:00:00.0
   Memory: 12.0M
   Disk usage: 16B
   Time elapsed: 0:00:02.0


TEST END: /root/miniconda3/envs/buildenv/conda-bld/linux-64/llvmdev-14.0.6-manylinux2014.tar.bz2

@oliverhu
Copy link
Author

oliverhu commented Sep 2, 2023

Added a patch for macOS build in latest commit to expose builtin symbols

Copy link
Member

@gmarkall gmarkall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is looking good and all outstanding issues are resolved.

We should note (in any announcements following the merge) that we're not building all of compiler-rt, but this is just building the compiler-rt builtins.

gmarkall added a commit to gmarkall/llvmlite that referenced this pull request Apr 15, 2024
This enables the build of compiler-rt builtins with the following
changes:

- Add `compiler-rt` to the list of LLVM projects to build
- Disable the build of all `compiler-rt` components except for the
  builtins. Whilst it would be interesting to add these, they seem to
  require clang to be built at the same time.
- Setting the `COMPILER_RT_BUILTINS_HIDE_SYMBOLS` CMake option to `OFF`,
  to ensure that we can link against those symbols in the general case.

Note that these changes are simpler than previous efforts towards
enabling `compiler-rt` builtins (e.g. numba#979) - this is because we now
build as part of the whole `llvm-project` tree, which simplifies things
and removes issues that had to be worked around when building it
separately.
@gmarkall
Copy link
Member

gmarkall commented Jul 1, 2024

The changes in this PR are integrated into the now-merged #1036 - many thanks @oliverhu!

@gmarkall gmarkall closed this Jul 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants