Static stdlib linking is broken on main and 5.9

[FranzBusch]

Description
Trying to compile a simple Swift program on either main or the 5.9 release branch is currently failing to link when using --static-swift-stdlib. It produces the following linker error

#4 6.932 error: link command failed with exit code 1 (use -v to see invocation)
#4 6.932 clang version 13.0.0 (https://github.com/apple/llvm-project.git d41f263a5bcfd9715eaf9a3351f00b4566f13231)
#4 6.932 Target: aarch64-unknown-linux-gnu
#4 6.932 Thread model: posix
#4 6.932 InstalledDir: /usr/bin
#4 6.932 Found candidate GCC installation: /usr/bin/../lib/gcc/aarch64-redhat-linux/11
#4 6.932 Selected GCC installation: /usr/bin/../lib/gcc/aarch64-redhat-linux/11
#4 6.932 Candidate multilib: .;@m64
#4 6.932 Selected multilib: .;@m64
#4 6.932  "/usr/bin/ld.gold" -pie -EL --hash-style=gnu --eh-frame-hdr -m aarch64linux -dynamic-linker /lib/ld-linux-aarch64.so.1 -o /.build/aarch64-unknown-linux-gnu/release/Test /usr/bin/../lib/gcc/aarch64-redhat-linux/11/../../../../lib64/Scrt1.o /usr/bin/../lib/gcc/aarch64-redhat-linux/11/../../../../lib64/crti.o /usr/bin/../lib/gcc/aarch64-redhat-linux/11/crtbeginS.o -L/usr/lib/swift_static/linux -L/.build/aarch64-unknown-linux-gnu/release -L/usr/lib -L/usr/bin/../lib/gcc/aarch64-redhat-linux/11 -L/usr/bin/../lib/gcc/aarch64-redhat-linux/11/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/lib -L/usr/lib /usr/lib/swift_static/linux/aarch64/swiftrt.o --start-group /.build/aarch64-unknown-linux-gnu/release/TestExecutable.build/Program.swift.o /.build/aarch64-unknown-linux-gnu/release/TestLib.build/File.swift.o -lswiftCore -lswift_Concurrency -lswift_StringProcessing -lswift_RegexParser -lswiftGlibc -lBlocksRuntime -ldispatch -lDispatchStubs -lswiftDispatch -lCoreFoundation -lFoundation -luuid -licui18nswift -licuucswift -licudataswift -lm -lpthread -lutil -ldl --end-group -ldl -lpthread -lswiftCore -lstdc++ -lm -export-dynamic --exclude-libs ALL --gc-sections --defsym main=TestExecutable_main "-rpath=\$ORIGIN" -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/bin/../lib/gcc/aarch64-redhat-linux/11/crtendS.o /usr/bin/../lib/gcc/aarch64-redhat-linux/11/../../../../lib64/crtn.o
#4 6.932 /usr/lib/swift_static/linux/libswiftCore.a(Metadata.cpp.o):Metadata.cpp:function swift::blockOnMetadataDependency(swift::MetadataDependency, swift::MetadataDependency): error: undefined reference to 'std::__glibcxx_assert_fail(char const*, int, char const*, char const*)'
#4 6.932 /usr/lib/swift_static/linux/libswiftCore.a(MetadataLookup.cpp.o):MetadataLookup.cpp:function swift::Demangle::__runtime::TypeDecoder<(anonymous namespace)::DecodedMetadataBuilder>::decodeMangledType(swift::Demangle::__runtime::Node*, unsigned int, bool): error: undefined reference to 'std::__glibcxx_assert_fail(char const*, int, char const*, char const*)'
#4 6.932 /usr/lib/swift_static/linux/libswiftCore.a(MetadataLookup.cpp.o):MetadataLookup.cpp:function swift::Demangle::__runtime::TypeDecoder<(anonymous namespace)::DecodedMetadataBuilder>::decodeMangledType(swift::Demangle::__runtime::Node*, unsigned int, bool): error: undefined reference to 'std::__glibcxx_assert_fail(char const*, int, char const*, char const*)'
#4 6.932 /usr/lib/swift_static/linux/libswiftCore.a(MetadataLookup.cpp.o):MetadataLookup.cpp:function __swift::__runtime::llvm::ArrayRef<swift::MetadataOrPack>& std::vector<__swift::__runtime::llvm::ArrayRef<swift::MetadataOrPack>, std::allocator<__swift::__runtime::llvm::ArrayRef<swift::MetadataOrPack> > >::emplace_back<swift::MetadataOrPack*, unsigned int>(swift::MetadataOrPack*&&, unsigned int&&): error: undefined reference to 'std::__glibcxx_assert_fail(char const*, int, char const*, char const*)'
#4 6.932 clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
#4 6.932 error: fatalError

Steps to reproduce

Try to compile a simple swift package that contains an executable target that contains just this

@main
struct Main {
    static func main() async throws {}
}

Environment

• Swift compiler version info: April 11, 2023 main or 5.9 snapshots.

[al45tair]

Could this be related to #64312, I wonder?

[tomerd]

cc @etcwilde @MaxDesiatov

[gwynne]

@al45tair From what I can tell, it's an issue in libstdc++ which was fixed in March of last year¹². I'm guessing (emphasis on "guess") that it's being exposed by #64312 as a downstream result of changes in LLVM³. Someone with more domain experience than I might be better-qualified to figure out the appropriate remedy; I'll try to repro locally so I can juggle the ordering to find a workaround when I get a chance if someone else hasn't solved it already.

Footnotes

1. libstdc++: Ensure __glibcxx_assert_fail has default visibility ↩

2. RedHat Bug 2060755 - Firefox: GCC 12 linking error ↩

3. D142279 - [cmake] Use LLVM_ENABLE_ASSERTIONS to enable assertions in libstdc++ ↩

[gwynne]

(For the record, if it's that same issue and it is being exposed by #64312, it was always a lurking concern; there's never been anything forcing any particular ordering in the autolink-generated part of the link command in the first place - the #64312 fix actually adds an explicit ordering for the first time, thanks to the bug in XCTest linkage.)

[gwynne]

Update: This issue can't have been caused by #64312; it also exists in the Swift 5.7.3 and 5.8 release toolchains!

swift:5.7-jammy:

root@4fcf0d17c25a:/foo# swift build -Xlinker -v -Xswiftc -v -c release --static-swift-stdlib
Building for production...
Swift version 5.7.3 (swift-5.7.3-RELEASE)
Target: aarch64-unknown-linux-gnu
<SNIP>
 "/usr/bin/ld.gold" -pie -EL -z relro --hash-style=gnu --eh-frame-hdr -m aarch64linux -dynamic-linker /lib/ld-linux-aarch64.so.1 -o /foo/.build/aarch64-unknown-linux-gnu/release/foo /lib/aarch64-linux-gnu/Scrt1.o /lib/aarch64-linux-gnu/crti.o /usr/bin/../lib/gcc/aarch64-linux-gnu/9/crtbeginS.o -L/usr/lib/swift_static/linux -L/foo/.build/aarch64-unknown-linux-gnu/release -L/usr/lib -L/usr/bin/../lib/gcc/aarch64-linux-gnu/9 -L/lib/aarch64-linux-gnu -L/usr/lib/aarch64-linux-gnu -L/usr/bin/../lib -L/lib -L/usr/lib /usr/lib/swift_static/linux/aarch64/swiftrt.o --start-group /foo/.build/aarch64-unknown-linux-gnu/release/foo.build/foo.swift.o -lswift_StringProcessing -lswift_Concurrency -lswiftCore --end-group -ldl -lpthread -lswiftCore -lstdc++ -lm -export-dynamic --exclude-libs ALL --gc-sections --defsym main=foo_main "-rpath=\$ORIGIN" -v -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/bin/../lib/gcc/aarch64-linux-gnu/9/crtendS.o /lib/aarch64-linux-gnu/crtn.o
/usr/lib/swift_static/linux/libswift_Concurrency.a(GlobalExecutor.cpp.o):GlobalExecutor.cpp:function swift_task_enqueueGlobalImpl(swift::Job*): error: undefined reference to '_dispatch_queue_attr_concurrent'
/usr/lib/swift_static/linux/libswift_Concurrency.a(GlobalExecutor.cpp.o):GlobalExecutor.cpp:function swift_task_enqueueGlobalImpl(swift::Job*): error: undefined reference to '_dispatch_queue_attr_concurrent'
<SNIP>
/usr/lib/swift_static/linux/libswift_Concurrency.a(Task.cpp.o):Task.cpp:function swift_task_asyncMainDrainQueueImpl(): error: undefined reference to 'dispatch_main'
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)

swift:5.8-jammy:

root@ed014baf2277:/foo# swift build -Xlinker -v -Xswiftc -v -c release --static-swift-stdlib
Building for production...
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
Swift version 5.8 (swift-5.8-RELEASE)
Target: aarch64-unknown-linux-gnu
<SNIP>
 "/usr/bin/ld.gold" -pie -EL -z relro --hash-style=gnu --eh-frame-hdr -m aarch64linux -dynamic-linker /lib/ld-linux-aarch64.so.1 -o /foo/.build/aarch64-unknown-linux-gnu/release/foo /lib/aarch64-linux-gnu/Scrt1.o /lib/aarch64-linux-gnu/crti.o /usr/bin/../lib/gcc/aarch64-linux-gnu/9/crtbeginS.o -L/usr/lib/swift_static/linux -L/foo/.build/aarch64-unknown-linux-gnu/release -L/usr/lib -L/usr/bin/../lib/gcc/aarch64-linux-gnu/9 -L/lib/aarch64-linux-gnu -L/usr/lib/aarch64-linux-gnu -L/lib -L/usr/lib /usr/lib/swift_static/linux/aarch64/swiftrt.o --start-group /foo/.build/aarch64-unknown-linux-gnu/release/foo.build/Foo.swift.o -lswift_RegexParser -lswift_StringProcessing -lswift_Concurrency -lswiftCore --end-group -ldl -lpthread -lswiftCore -lstdc++ -lm -export-dynamic --exclude-libs ALL --gc-sections --defsym main=foo_main "-rpath=\$ORIGIN" -v -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/bin/../lib/gcc/aarch64-linux-gnu/9/crtendS.o /lib/aarch64-linux-gnu/crtn.o
/usr/lib/swift_static/linux/libswift_Concurrency.a(GlobalExecutor.cpp.o):GlobalExecutor.cpp:function swift_task_enqueueGlobalImpl(swift::Job*): error: undefined reference to '_dispatch_queue_attr_concurrent'
/usr/lib/swift_static/linux/libswift_Concurrency.a(GlobalExecutor.cpp.o):GlobalExecutor.cpp:function swift_task_enqueueGlobalImpl(swift::Job*): error: undefined reference to '_dispatch_queue_attr_concurrent'
<SNIP>
/usr/lib/swift_static/linux/libswift_Concurrency.a(Task.cpp.o):Task.cpp:function swift_task_asyncMainDrainQueueImpl(): error: undefined reference to 'dispatch_main'
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)

swiftlang/swift:nightly-5.9-jammy (swift-5.9-DEVELOPMENT-SNAPSHOT-2023-04-12-a):

root@667e5d417d3a:/foo# swift build -Xlinker -v -Xswiftc -v -c release --static-swift-stdlib
Building for production...
remark: Incremental compilation has been disabled: it is not compatible with whole module optimization
Swift version 5.9-dev (LLVM 6676f64aea2989b, Swift 563750fbe935a01)
Target: aarch64-unknown-linux-gnu
<SNIP>
 "/usr/bin/ld.gold" -pie -EL -z relro --hash-style=gnu --eh-frame-hdr -m aarch64linux -dynamic-linker /lib/ld-linux-aarch64.so.1 -o /foo/.build/aarch64-unknown-linux-gnu/release/foo /lib/aarch64-linux-gnu/Scrt1.o /lib/aarch64-linux-gnu/crti.o /usr/bin/../lib/gcc/aarch64-linux-gnu/9/crtbeginS.o -L/usr/lib/swift_static/linux -L/foo/.build/aarch64-unknown-linux-gnu/release -L/usr/lib -L/usr/bin/../lib/gcc/aarch64-linux-gnu/9 -L/lib/aarch64-linux-gnu -L/usr/lib/aarch64-linux-gnu -L/lib -L/usr/lib /usr/lib/swift_static/linux/aarch64/swiftrt.o --start-group /foo/.build/aarch64-unknown-linux-gnu/release/foo.build/Foo.swift.o -lswiftCore -lswift_Concurrency -lswift_StringProcessing -lswift_RegexParser --end-group -ldl -lpthread -lswiftCore -lstdc++ -lm -export-dynamic --exclude-libs ALL --gc-sections --defsym main=foo_main "-rpath=\$ORIGIN" -v -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/bin/../lib/gcc/aarch64-linux-gnu/9/crtendS.o /lib/aarch64-linux-gnu/crtn.o
/usr/lib/swift_static/linux/libswift_Concurrency.a(GlobalExecutor.cpp.o):GlobalExecutor.cpp:function swift_task_enqueueGlobalImpl(swift::Job*): error: undefined reference to '_dispatch_queue_attr_concurrent'
/usr/lib/swift_static/linux/libswift_Concurrency.a(GlobalExecutor.cpp.o):GlobalExecutor.cpp:function swift_task_enqueueGlobalImpl(swift::Job*): error: undefined reference to '_dispatch_queue_attr_concurrent'
<SNIP>
/usr/lib/swift_static/linux/libswift_Concurrency.a(Task.cpp.o):Task.cpp:function swift_task_asyncMainDrainQueueImpl(): error: undefined reference to 'dispatch_main'
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)

Also verified with swiftlang/swift:nightly-5.9-amazonlinux2 (swift-5.9-DEVELOPMENT-SNAPSHOT-2023-04-12-a).

Adding import Dispatch is sufficient to prevent the error in all four versions.

The error is the result of failing to include -lswiftDispatch, -ldispatch, -lBlocksRuntime, and -lDispatchStubs (edit: and -lstdc++) in the link command when using Concurrency (such as an async entry point) while linking statically. They only need to be specified explicitly for static linking; -lswiftConcurrency links them in the dynamic case. This has apparently been a problem for quite some time now.

Edit: -lstdc++ is also conspicuously absent from the failed link commands (it is present when Dispatch is linked; my guess is the linker gave me a different set of missing symbols than Franz got due to some change in the 4/12 snapshot (probably something cxx-interop-related, I'd imagine).

[etcwilde]

You two have different missing symbols. _dispatch_queue_attr_concurrent and the symbols that the dispatch-based concurrency backend uses missing is due to the missing dispatch link. It's unfortunate, but sort of a property of statically linking that requires the developer to know to link that stuff instead of letting the dynamic loader do it. There might be some things we could do from the compiler/runtime side, but it would be brittle (like in cases where someone isn't using the dispatch backend, tools shouldn't be auto-linking the dispatch libraries because they may not even exist, for instance).

The missing __glibcxx_assert_fail symbol from the original linker diagnostic is maybe more interesting because that's supposed to be coming from libstdc++ itself. From a little high-level investigation, it looks like there is a visibility bug on the GCC side of things. https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=4cb935cb69f12088975fa7f6907c6ace0580e2dd
(this caused the same build failures in asserts-builds of Firefox https://bugzilla.mozilla.org/show_bug.cgi?id=1758005). So it looks like we're shipping runtimes with assertions enabled on Linux for some reason. While that's great for debugging, I can't image it's doing good things for our runtime performance, and in this case is blocking our ability to link at all, which is bad. I haven't had a chance to go deeper and do additional verification because I'm typing this while waiting for my waffle cook, and it's also the weekend.

Edit: Just checked the package bot and we're definitely passing --no-swift-stdlib-assertions to the Redhat-UBI9 toolchain build. Will need to keep investigating on Monday.

[gwynne]

@etcwilde You literally dug up the same GCC and Firefox references to that issue that I found (#65097 (comment)) 🙂. But I haven't been able to reproduce the missing C++ symbols that Franz originally reported - nor were those same the missing symbols we were originally investigating when he filed this bug. It's my (revised) opinion that the oddity has/had to do with the additional absence of -lstdc++ for linking when Dispatch isn't explicitly specified; as such I crossed out my original two comments, since I no longer think the GCC issue was involved.

[etcwilde]

Okay, so I think I've fully tracked down the issue. The Clang package on UBI9 installs the gcc-12-aligned devtoolset and libstdc++, which defines the std::__glibcxx_assert_fail symbol. In gcc-11 libstdc++, the function had a different name, and doesn't exist in the libstdc++ that's installed on the base image the function went by a different name and has a different symbol. Clang will use the newest gcc toolchain version, 12 is newer than 11, so it goes with 12, which of course isn't installed on the base system.

Why does the dynamic library work then? Because devtoolset-12 has a libstdc++_noshared.a static archive, which gets embedded into the shared stdlib, because you can do that to dynamic libraries. (Not unlike how our compatibility libraries work in Swift.) Static libraries, you can't really do that in general without getting into some trouble.

I seem to be having success with --gcc-toolchain=/usr while building the stdlib, so that it's built against what is actually on a base UBI9 image instead of the devtoolset-12 toolchain.

As for the question why 5.9/main hits this and 5.8 hits this, 5.9 is built on a newer llvm, which enabled GLIBCXX_ASSERTIONS to get better assertion messages; https://github.com/apple/llvm-project/blob/45eb1af0a22749fde5420562033e0c18e69d09d0/llvm/cmake/modules/HandleLLVMOptions.cmake#L85
(https://reviews.llvm.org/D142279). (Note that folks in the review hit pretty much exactly this too)

So we may need something like

[preset: buildbot_ubi9]
mixin-preset=
  buildbot_linux

extra-cmake-options=
 -DCMAKE_C_FLAGS="--gcc-toolchain=/usr"
 -DCMAKE_CXX_FLAGS="--gcc-toolchain=/usr"
 -DCMAKE_Swift_FLAGS="-Xcc --gcc-toolchain=/usr -Xclang-linker --gcc-toolchain=/usr

to force that through on ubi9. It might not be a terrible idea to force that on all Linux distros to ensure that the runtime will actually work on the distro we're building for, out-of-the-box. I'll try a PR and see if we take down the Linux nightlies off of main.

[etcwilde]

@FranzBusch, is this still reproducing? I just tried building with the 5.9.2 and 5.10.1 docker images and am unable to reproduce this behavior anymore (both with SwiftPM and just a raw swiftc invocation).

[FranzBusch]

I think this has been fixed by some change. Let's close it for now.