feat(default.nix): Init in /compiler+runtime #94

haruki7049 · 2024-10-03T21:03:20Z

Set up automated packaging for various distros #26

Adding Nix-build support for /compiler+runtime directory.

haruki7049 · 2024-10-04T04:30:48Z

Hey @jeaye, In 7585782, An error is occured about LLVM supported version. However github.com/llvm/llvm-project has no version, named as 19.0.x. What LLVM version should I use?

CMake Error at cmake/llvm.cmake:78 (message):
  Found unsupported version: LLVM 19.1.1;

  Please set llvm_dir pointing to the llvm version 19.0 to 19.0.x build or
  installation folder
Call Stack (most recent call first):
  CMakeLists.txt:113 (include)

haruki7049 · 2024-10-04T04:31:56Z

Fixed typo in PR's title...

jeaye · 2024-10-04T05:01:33Z

Hey @jeaye, In 7585782, An error is occured about LLVM supported version. However github.com/llvm/llvm-project has no version, named as 19.0.x. What LLVM version should I use?
CMake Error at cmake/llvm.cmake:78 (message):
  Found unsupported version: LLVM 19.1.1;

  Please set llvm_dir pointing to the llvm version 19.0 to 19.0.x build or
  installation folder
Call Stack (most recent call first):
  CMakeLists.txt:113 (include)

Hey! I'd say, in order to get things compiling, we can just patch that CMake check to allow your 19.1.1 version. Everyone working on jank, right now, compiles Clang/LLVM from source and it's roughly 19.0.0 but it has a particular fix which hasn't made it into the 19.x release yet. I have a ticket for that here: llvm/llvm-project#111068

By using 19.1.1, we may run into some hard crashes when we run jank, but the main task here is getting it compiling with Nix. Using a different Clang version after we get things compiling should be much easier, especially if I can get that commit backported to 19.x.

jeaye · 2024-10-04T05:27:27Z

I went through our vcpkg.json to see what we can do about each of the deps. Looks like only one of them needs compilation. The majority can be global distro packages. A few we'll need to bring in as submodules (like compiler+runtime/third-party/bpptree and others).

bdwgc: distro package (boehmgc or bdwgc, same thing)
boost-algorithm: distro package (boost)
boost-filesystem: distro package (boost)
boost-smart-ptr: distro package (boost)
boost-variant: distro package (boost)
boost-preprocessor: distro package (boost)
fmt: distro package (fmt)
readline: distro package (readline)
cli11: header-only; submodule https://github.com/CLIUtils/CLI11 into compiler+runtime/third-party/cli11
immer: header-only; submodule https://github.com/arximboldi/immer into compiler+runtime/third-party/immer
magic-enum: header-only; submodule https://github.com/Neargye/magic_enum into compiler+runtime/third-party/magic_enum
libassert: skip this, we're not using it right now
libzippp:
- needs compilation
- submodule https://github.com/ctabin/libzippp into compiler+runtime/third-party/libzippp
- uses cmake, so we should be able to build it as an external project

Hopefully this is helpful.

haruki7049 · 2024-10-04T20:19:40Z

Tasks:

Build boost
Build folly (Resolved by using local submodule)

haruki7049 · 2024-10-04T20:22:25Z

Memo: What is the DCHECK?

/build/source/folly/logging/LoggerDB.cpp: In constructor 'folly::LoggerDB::LoggerDB()':
/build/source/folly/logging/LoggerDB.cpp:78:3: error: 'DCHECK' was not declared in this scope
   78 |   DCHECK(ret.second);
      |   ^~~~~~
/build/source/folly/logging/LoggerDB.cpp: In member function 'folly::LogCategory* folly::LoggerDB::createCategoryLocked(LoggerNameMap&, folly::StringPiece, folly::LogCategory*)':
/build/source/folly/logging/LoggerDB.cpp:473:3: error: 'DCHECK' was not declared in this scope
  473 |   DCHECK(ret.second);
      |   ^~~~~~
make[2]: *** [CMakeFiles/folly_base.dir/build.make:3366: CMakeFiles/folly_base.dir/folly/logging/LoggerDB.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:141: CMakeFiles/folly_base.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

jeaye · 2024-10-04T20:24:55Z

Hmm, so jank builds what it needs from folly as part of a custom target: https://github.com/jank-lang/jank/blob/main/compiler%2Bruntime/CMakeLists.txt#L323

Folly itself is a pain in the ass to build, so I've stripped out just the bits that jank needs and patched them to compile more easily. Are you building folly on its own, or using jank's CMake target for it?

haruki7049 · 2024-10-04T20:29:47Z

I was building folly only, thinking that I could just build folly as is. it is quite hard to use git submodule in Nix.

jeaye · 2024-10-04T20:36:31Z

I was building folly only, thinking that I could just build folly as is. it is quite hard to use git submodule in Nix.

I think getting Nix to clone recursively (or just work with submodules in some way) would be beneficial for two reasons:

Lots of people will need to build jank without Nix and submodules will work easily for them
We don't want the Nix build to be building from different sources than what's in the actual repo (i.e. the submodules)

You know Nix better than I do, for sure. What I know, though, is that:

If we're just building from the local filesystem (i.e. submodules have already been initialized), Nix doesn't need to know anything about submodules
If we're building from scratch, fetchGit supports a deepClone = true flag

Are we able to work with that?

haruki7049 · 2024-10-04T20:46:56Z

Now my local repository was not cloned using git clone --recurse-submodules {repo_url}, but simply cloned and used using git clone {repo_url}. Even in this case, to make nix-build available, it was taken from the URL.

Sure, if we have those files locally, we can reference them.

Make git clone --recurse-submodules {repo_url} mandatory?

But I'd like to say no, because I believe that building in Nix should be possible with or without submodules.

jeaye · 2024-10-04T20:58:07Z

Now my local repository was not cloned using git clone --recurse-submodules {repo_url}, but simply cloned and used using git clone {repo_url}. Even in this case, to make nix-build available, it was taken from the URL.

Sure, if we have those files locally, we can reference them.

Make git clone --recurse-submodules {repo_url} mandatory?

But I'd like to say no, because I believe that building in Nix should be possible with or without submodules.

Initializing submodules (either with git clone --recurse-submodules or git submodule update --recursive --init) is currently mandatory, outside of Nix. Since we want Nix to use the same sources, I think Nix should also use the submodules. Otherwise, maintaining the Nix versions of our dependencies will require duplicate effort. Whenever we update a submodule, we'll also need to update the Nix expression.

Submodules are pure and reproducible, like any git commit. A single git commit in jank represents not only a snapshot of jank's source but also all of its submodules.

Hopefully that's a good enough point. Thanks for working with me here. 🙂 すみません

haruki7049 · 2024-10-04T21:02:05Z

That may indeed be true. Ok, I'll take the method of using local submodules on Nix.

And, no need to apologize. It's okay. NO PROBLEM!!
だいじょうぶ!!

haruki7049 · 2024-10-04T21:15:47Z

Wait, The folly submodule should contain files other than those listed below. If you do not write the file exclusion settings in the submodule, you will end up writing the source file exclusion in Nix as well.

set(
  folly_sources
  third-party/folly/folly/ScopeGuard.cpp
  third-party/folly/folly/SharedMutex.cpp
  third-party/folly/folly/concurrency/CacheLocality.cpp
  third-party/folly/folly/synchronization/ParkingLot.cpp
  third-party/folly/folly/synchronization/SanitizeThread.cpp
  third-party/folly/folly/lang/SafeAssert.cpp
  third-party/folly/folly/lang/ToAscii.cpp
  third-party/folly/folly/system/ThreadId.cpp
  third-party/folly/folly/system/AtFork.cpp
  third-party/folly/folly/detail/Futex.cpp
)

jeaye · 2024-10-04T21:18:12Z

Wait, The folly submodule should contain files other than those listed below. If you do not write the file exclusion settings in the submodule, you will end up writing the source file exclusion in Nix as well.

set(
  folly_sources
  third-party/folly/folly/ScopeGuard.cpp
  third-party/folly/folly/SharedMutex.cpp
  third-party/folly/folly/concurrency/CacheLocality.cpp
  third-party/folly/folly/synchronization/ParkingLot.cpp
  third-party/folly/folly/synchronization/SanitizeThread.cpp
  third-party/folly/folly/lang/SafeAssert.cpp
  third-party/folly/folly/lang/ToAscii.cpp
  third-party/folly/folly/system/ThreadId.cpp
  third-party/folly/folly/system/AtFork.cpp
  third-party/folly/folly/detail/Futex.cpp
)

The folly submodule contains more files, yes. The ones listed there are just the files we're compiling, but we still need other files, such as all of the headers that those source files include. I don't think we should be excluding any files using Nix.

haruki7049 · 2024-10-04T21:21:11Z

Then, How should I do for #94 (comment) ?

jeaye · 2024-10-04T21:24:56Z

So, I think you shouldn't need to do anything for folly. We already have a submodule for it and it already builds as part of jank. There shouldn't be anything you need to do there, unless you run into errors compiling the folly which is part of jank. But I believe the error you ran into was when you were compiling a different version of folly. Correct me if I'm wrong.

haruki7049 · 2024-10-04T21:28:38Z

Ah, you are referring to a submodule that is local. I finally understand. Thanks, Jeaye!!

jeaye · 2024-10-14T18:15:50Z

https://cmake.org/cmake/help/latest/command/target_precompile_headers.html#command:target_precompile_headers

@jeaye I wonder if I can use cmake commands like above... I don't want to use shell scripts in cmake...

I looked into it, before making the script. CMake's PCH tooling is very limited. jank is already using it for the AOT PCH. But the JIT PCH requires different flags. Ultimately, a script seemed the only viable approach.

I'd be open to getting that entirely put into CMake, but I didn't succeed when I tried.

Is having it in a script causing trouble with Nix?

haruki7049 · 2024-10-14T18:34:54Z

[87/97] Generating incremental.pch
FAILED: incremental.pch /build/source/build/incremental.pch
cd /build/source/build && /build/source/bin/build-pch /nix/store/6a2f67zl2j0nxi551hp7q4cpid67k3aw-clang-19.1.1 /build/source/include /build/source/build/incremental.pch -std=gnu++20\ -DIMMER_HAS_LIBGC=1\ -DIMMER_TAGGED_NODE=0\ -DHAVE_CXX14=1\ -DFMT_HEADER_ONLY=1\ -DJANK_DEBUG\ -w -isystem\ /nix/store/ai4g3129rzppvqmxfb7fg6xh3ihhxyyz-clang-19.1.1-dev/include\ -isystem\ /nix/store/wiyw2njh783ad5pil81xi1jwdxsqbxxx-llvm-19.1.1-dev/include\ -I/build/source/include/cpp\ -I/build/source/third-party/nanobench/include\ -I/build/source/third-party/folly\ -I/build/source/third-party/bpptree/include\ -isystem\ /nix/store/8xyxg1mp3f0bb8nsll3hbm4lwpisfwz2-boost-1.86.0/include\ -isystem\ /nix/store/wiyw2njh783ad5pil81xi1jwdxsqbxxx-llvm-19.1.1-dev/include\
/build/source/include/cpp/jank/prelude.hpp:3:10: fatal error: 'random' file not found
    3 | #include <random>
      |          ^~~~~~~~
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: /nix/store/6a2f67zl2j0nxi551hp7q4cpid67k3aw-clang-19.1.1/bin/clang++ -std=gnu++20 -DIMMER_HAS_LIBGC=1 -DIMMER_TAGGED_NODE=0 -DHAVE_CXX14=1 -DFMT_HEADER_ONLY=1 -DJANK_DEBUG -w -isystem /nix/store/ai4g3129rzppvqmxfb7fg6xh3ihhxyyz-clang-19.1.1-dev/include -isystem /nix/store/wiyw2njh783ad5pil81xi1jwdxsqbxxx-llvm-19.1.1-dev/include -I/build/source/include/cpp -I/build/source/third-party/nanobench/include -I/build/source/third-party/folly -I/build/source/third-party/bpptree/include -isystem /nix/store/8xyxg1mp3f0bb8nsll3hbm4lwpisfwz2-boost-1.86.0/include -isystem /nix/store/wiyw2njh783ad5pil81xi1jwdxsqbxxx-llvm-19.1.1-dev/include\\ -I /build/source/include -I /build/source/include/cpp -Xclang -fincremental-extensions -Xclang -emit-pch -x c++-header -o /build/source/build/incremental.pch -c /build/source/include/cpp/jank/prelude.hpp
1.      /nix/store/8xyxg1mp3f0bb8nsll3hbm4lwpisfwz2-boost-1.86.0/include/boost/mp11/detail/mp_value.hpp:13:1: current parser token 'namespace'
2.      /nix/store/8xyxg1mp3f0bb8nsll3hbm4lwpisfwz2-boost-1.86.0/include/boost/unordered/detail/type_traits.hpp:36:1: parsing namespace 'boost'
 #0 0x00007fffeb5e38fe llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/nix/store/2f53shlvx25ng9kbmlpar45fg0zrjj35-llvm-19.1.1-lib/lib/libLLVM.so.19.1+0xfe38fe)
 #1 0x00007fffeb5e167c llvm::sys::CleanupOnSignal(unsigned long) (/nix/store/2f53shlvx25ng9kbmlpar45fg0zrjj35-llvm-19.1.1-lib/lib/libLLVM.so.19.1+0xfe167c)
 #2 0x00007fffeb4dac58 CrashRecoverySignalHandler(int) (/nix/store/2f53shlvx25ng9kbmlpar45fg0zrjj35-llvm-19.1.1-lib/lib/libLLVM.so.19.1+0xedac58)
 #3 0x00007fffea0495c0 __restore_rt (/nix/store/3dyw8dzj9ab4m8hv5dpyx7zii8d0w6fi-glibc-2.39-52/lib/libc.so.6+0x405c0)
 #4 0x00007ffff47d3d80 clang::Decl::castFromDeclContext(clang::DeclContext const*) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0xfd3d80)
 #5 0x00007ffff50ee530 clang::Sema::PopDeclContext() (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0x18ee530)
 #6 0x00007ffff51b629d clang::Sema::ActOnFinishNamespaceDef(clang::Decl*, clang::SourceLocation) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0x19b629d)
 #7 0x00007ffff4506535 clang::Parser::ParseNamespace(clang::DeclaratorContext, clang::SourceLocation&, clang::SourceLocation) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0xd06535)
 #8 0x00007ffff44eb2df clang::Parser::ParseDeclaration(clang::DeclaratorContext, clang::SourceLocation&, clang::ParsedAttributes&, clang::ParsedAttributes&, clang::SourceLocation*) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0xceb2df)
 #9 0x00007ffff45cd3ef clang::Parser::ParseExternalDeclaration(clang::ParsedAttributes&, clang::ParsedAttributes&, clang::ParsingDeclSpec*) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0xdcd3ef)
#10 0x00007ffff45cecb5 clang::Parser::ParseTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&, clang::Sema::ModuleImportState&) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0xdcecb5)
#11 0x00007ffff44b5172 clang::ParseAST(clang::Sema&, bool, bool) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0xcb5172)
#12 0x00007ffff68ad499 clang::FrontendAction::Execute() (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0x30ad499)
#13 0x00007ffff68295fb clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0x30295fb)
#14 0x00007ffff6941c5b clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0x3141c5b)
#15 0x0000000000413a51 cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/nix/store/6a2f67zl2j0nxi551hp7q4cpid67k3aw-clang-19.1.1/bin/clang+++0x413a51)
#16 0x000000000040d60a ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&, llvm::ToolContext const&) (/nix/store/6a2f67zl2j0nxi551hp7q4cpid67k3aw-clang-19.1.1/bin/clang+++0x40d60a)
#17 0x00007ffff64485e9 void llvm::function_ref<void ()>::callback_fn<clang::driver::CC1Command::Execute(llvm::ArrayRef<std::optional<llvm::StringRef>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, bool*) const::'lambda'()>(long) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0x2c485e9)
#18 0x00007fffeb4db093 llvm::CrashRecoveryContext::RunSafely(llvm::function_ref<void ()>) (/nix/store/2f53shlvx25ng9kbmlpar45fg0zrjj35-llvm-19.1.1-lib/lib/libLLVM.so.19.1+0xedb093)
#19 0x00007ffff6448cf1 clang::driver::CC1Command::Execute(llvm::ArrayRef<std::optional<llvm::StringRef>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, bool*) const (.part.0) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0x2c48cf1)
#20 0x00007ffff640dc14 clang::driver::Compilation::ExecuteCommand(clang::driver::Command const&, clang::driver::Command const*&, bool) const (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0x2c0dc14)
#21 0x00007ffff640e73d clang::driver::Compilation::ExecuteJobs(clang::driver::JobList const&, llvm::SmallVectorImpl<std::pair<int, clang::driver::Command const*>>&, bool) const (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0x2c0e73d)
#22 0x00007ffff6422da4 clang::driver::Driver::ExecuteCompilation(clang::driver::Compilation&, llvm::SmallVectorImpl<std::pair<int, clang::driver::Command const*>>&) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0x2c22da4)
#23 0x0000000000410e52 clang_main(int, char**, llvm::ToolContext const&) (/nix/store/6a2f67zl2j0nxi551hp7q4cpid67k3aw-clang-19.1.1/bin/clang+++0x410e52)
#24 0x000000000040cff6 main (/nix/store/6a2f67zl2j0nxi551hp7q4cpid67k3aw-clang-19.1.1/bin/clang+++0x40cff6)
#25 0x00007fffea03314e __libc_start_call_main (/nix/store/3dyw8dzj9ab4m8hv5dpyx7zii8d0w6fi-glibc-2.39-52/lib/libc.so.6+0x2a14e)
#26 0x00007fffea033209 __libc_start_main@GLIBC_2.2.5 (/nix/store/3dyw8dzj9ab4m8hv5dpyx7zii8d0w6fi-glibc-2.39-52/lib/libc.so.6+0x2a209)
#27 0x000000000040d055 _start (/nix/store/6a2f67zl2j0nxi551hp7q4cpid67k3aw-clang-19.1.1/bin/clang+++0x40d055)
clang++: error: clang frontend command failed with exit code 139 (use -v to see invocation)
clang version 19.1.1
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /nix/store/6a2f67zl2j0nxi551hp7q4cpid67k3aw-clang-19.1.1/bin
clang++: error: unable to execute command: Segmentation fault (core dumped)
clang++: note: diagnostic msg: Error generating preprocessed source(s).
ninja: build stopped: subcommand failed.

This is the compile log by 2d45d67 . I want to take libstdcxx to the shell script, so I tried to change the stdenv's compiler, such as pkgs.llvmPackages_19.libstdcxxClang, but there is no effects.

I don't know the way to serve libstdcxx to the shell script...

haruki7049 · 2024-10-14T18:47:01Z

stderr.log

{
  pkgs ? import (builtins.fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/75b209227dff3cbfac19f510a62f9446c92beac4.tar.gz";
    sha256 = "166varzdamn2kkw5zslarzl6026q87wjnss1hlhdsq89hwvfgd2d";
  }) { },
}:

let
  lib = pkgs.lib;

  # Deps
  boost = pkgs.callPackage ./nix/boost.nix { };
  boehmgc = pkgs.callPackage ./nix/boehmgc.nix { };
  libzippp = pkgs.callPackage ./nix/libzippp.nix { };
  openssl = pkgs.callPackage ./nix/openssl.nix { };
in

pkgs.llvmPackages_19.stdenv.mkDerivation {
  pname = "jank-lang";
  version = "dev";
  src = lib.cleanSource ./.;

  nativeBuildInputs = with pkgs; [
    cmake
    git
    ninja
  ];

  patchPhase = ''
    patchShebangs --host "${pkgs.bash}" bin/build-pch
  '';

  buildInputs = [
    boost
    boehmgc
    libzippp
    openssl
    pkgs.doctest
    pkgs.double-conversion
    pkgs.readline
    pkgs.libzip
    pkgs.immer
    pkgs.cli11
    pkgs.magic-enum
    pkgs.fmt
    pkgs.llvmPackages_19.clang-unwrapped
    pkgs.llvmPackages_19.llvm
  ];

  cmakeBuildType = "Debug";
  CC = "${pkgs.llvmPackages_19.libstdcxxClang}/bin/clang";
  CXX = "${pkgs.llvmPackages_19.libstdcxxClang}/bin/clang++";

  cmakeFlags = [
    "-Djank_tests=on"
  ];
}

This default.nix can change default compiler, by changing CC and CXX. This can affect jank's compile but not incremental-pch.

jeaye · 2024-10-14T18:57:49Z

Hm, I think we can just try to compile the PCH outside of CMake first, using clang within your nix environment. Once we get that working, then we can hook it into CMake.

Right now, the script is expecting to receive the CLANG_INSTALL_PREFIX, which is the place where Clang lives. Are you setting that to be where Nix has clang?

What happens if you just invoke bin/build-pch from your Nix shell? It needs two args:

The clang prefix
The path to jank's compiler+runtime/include dir

jeaye · 2024-10-29T02:02:56Z

Hi @haruki7049! Just checking in to see the status of this and if you need any help.

haruki7049 · 2024-10-29T02:20:27Z

Hi @jeaye. Thanks for this mention. Perhaps are you in a hurry? If you are in a hurry, I am sorry for that.

I am interested in CI's result on main branch. It is not a good thing that most of the CI results on the main branch have failed. Are you know the reasons?

haruki7049 · 2024-10-29T02:39:55Z

Hey jeaye, can we build jank compiler without incremental-pch? I think that it is easy to build jank compiler without incremental-pch.

jeaye · 2024-10-29T04:27:48Z

Hi @jeaye. Thanks for this mention. Perhaps are you in a hurry? If you are in a hurry, I am sorry for that.

Nope, not in a hurry. No need to be sorry. Just trying to keep things progressing, since you've had really good progress!

I am interested in CI's result on main branch. It is not a good thing that most of the CI results on the main branch have failed. Are you know the reasons?

The reasons boil down to jank previously requiring devs to build clang/llvm from source, to get the latest fixes. This takes 2.5 hours to do in CI and makes debugging any other issues a real pain. The latest llvm 19.1.2 release has everything we need, but it's not available as a binary for most distros yet. Once it is, I can update CI to use that and then resolve anything else.

I agree that having main fail regularly is not ideal. It'll be fixed soon!

Hey jeaye, can we build jank compiler without incremental-pch? I think that it is easy to build jank compiler without incremental-pch.

jank will need the incremental PCH to work. We could build it as a CMake target, though. Perhaps Nix would like that more. We'd just need to use CMAKE_CXX_COMPILER to run the command as a custom CMake target.

We just need to make sure that the changes we make will work outside of Nix, too. Removing vcpkg is a big win for everyone, but we will still need to be able to access the necessary dependencies, either from the OS's package manager or from within the repo as a submodule.

haruki7049 · 2024-10-29T06:39:17Z

The latest llvm 19.1.2 release has everything we need, but it's not available as a binary for most distros yet.

Wait, Is this project using LLVM 19.1.2? Not LLVM 19.1.1?

jeaye · 2024-10-29T06:53:04Z

The latest llvm 19.1.2 release has everything we need, but it's not available as a binary for most distros yet.

Wait, Is this project using LLVM 19.1.2? Not LLVM 19.1.1?

When you started this PR, jank was using an unreleased version of LLVM. That's why we had to compile it from source. I told you not to worry about it, since the issues would show up as hard crashes at runtime but we should focus on getting things compiling first. Since then, 19.1.2 has been released with the necessary fixes, so we can go forward using that.

haruki7049 · 2024-10-29T07:20:10Z

@jeaye Can I close the pull request once I have found it very difficult to write up the default.nix for this project?　Instead, I am going to add some more shell.nix.

jeaye · 2024-10-29T16:28:11Z

How about I remove vcpkg (and add submodules) and the build-pch script? That should make things easier for you.

…

On October 29, 2024 12:20:31 AM PDT, haruki7049 ***@***.***> wrote: @jeaye Can I close the pull request once I have found it very difficult to write up the `default.nix` for this project?　Instead, I am going to add some more `shell.nix`. -- Reply to this email directly or view it on GitHub: #94 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

haruki7049 · 2024-10-29T23:29:05Z

If those could be done, it would be quite easy.... But are they feasible?

jeaye · 2024-10-30T00:06:38Z

Yeah. Give me a few days and I'll report back.

…

On October 29, 2024 4:29:27 PM PDT, haruki7049 ***@***.***> wrote: If those could be done, it would be quite easy.... But are they feasible? -- Reply to this email directly or view it on GitHub: #94 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

jeaye · 2024-10-30T05:16:31Z

Yeah. Give me a few days and I'll report back.
…
On October 29, 2024 4:29:27 PM PDT, haruki7049 @.> wrote: If those could be done, it would be quite easy.... But are they feasible? -- Reply to this email directly or view it on GitHub: #94 (comment) You are receiving this because you were mentioned. Message ID: @.>

I've moved the incremental PCH compilation out of the script and into cmake as a custom target in 319f0b2

Hopefully that helps. I'll work on removing vcpkg next. That'll take longer.

haruki7049 · 2024-10-30T05:57:13Z

Commit hash: 3f60ab5
Command: nix-build
Directory: /compiler+runtime

This error is generated. What is this?
err.log

`nix-shell -p nix-info --run "nix-info -m"`

 - system: `"aarch64-darwin"`
 - host os: `Darwin 24.0.0, macOS 15.0.1`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.18.2`
 - channels(root): `"nixpkgs"`
 - nixpkgs: `/nix/store/63lvc59rjz075kb1f11xww2x2xqpw1pn-source`

jeaye · 2024-10-30T06:15:45Z

Commit hash: 3f60ab5 Command: nix-build Directory: /compiler+runtime

This error is generated. What is this? err.log

nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-darwin"`
 - host os: `Darwin 24.0.0, macOS 15.0.1`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.18.2`
 - channels(root): `"nixpkgs"`
 - nixpkgs: `/nix/store/63lvc59rjz075kb1f11xww2x2xqpw1pn-source`

The error happens when building folly. From what I gather, the issue is because you're using libc++ instead of libstdc++. We can see that in the errors here:

In file included from /nix/store/7x3an9qwvkmird3n67qw09jccmd1dd7g-libcxx-19.1.1-dev/include/c++/v1/functional:552:
In file included from /nix/store/7x3an9qwvkmird3n67qw09jccmd1dd7g-libcxx-19.1.1-dev/include/c++/v1/__functional/boyer_moore_searcher.h:27:
In file included from /nix/store/7x3an9qwvkmird3n67qw09jccmd1dd7g-libcxx-19.1.1-dev/include/c++/v1/vector:325:
In file included from /nix/store/7x3an9qwvkmird3n67qw09jccmd1dd7g-libcxx-19.1.1-dev/include/c++/v1/__format/formatter_bool.h:16:
In file included from /nix/store/7x3an9qwvkmird3n67qw09jccmd1dd7g-libcxx-19.1.1-dev/include/c++/v1/__format/concepts.h:16:
In file included from /nix/store/7x3an9qwvkmird3n67qw09jccmd1dd7g-libcxx-19.1.1-dev/include/c++/v1/__format/format_parse_context.h:16:

jank needs to be built with libstdc++. Generally, clang will use this by default on Linux, but Nix must be doing something weird.

jeaye · 2024-11-02T01:54:25Z

vcpkg is gone: #122

We rely on boost and readline globally, while the rest has been brought in via submodules.

compiler+runtime/CMakeLists.txt

haruki7049 · 2024-11-10T00:11:03Z

@jeaye
I don't know, because I don't know C++, but is the fmt/format really imported?

Commit hash: c221ce9

System:

$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-linux"`
 - host os: `Linux 6.6.31, NixOS, 24.11 (Vicuna), 24.11pre703931.4aa36568d413`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.24.10`
 - channels(root): `"nixos, nixos-hardware"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

err.log

jeaye · 2024-11-10T00:19:45Z

@jeaye I don't know, because I don't know C++, but is the fmt/format really imported?

Commit hash: c221ce9

System:

$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-linux"`
 - host os: `Linux 6.6.31, NixOS, 24.11 (Vicuna), 24.11pre703931.4aa36568d413`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.24.10`
 - channels(root): `"nixos, nixos-hardware"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

err.log

Looks like you may not be on the latest main. Your compilation line there is missing the include path to fmt. It was added here: 8166273#diff-084c38e4be3e607f9cb1264d6a09cf1234173392296c938b42838cfffb28176dR236

haruki7049 · 2024-11-10T00:48:56Z

Hey @jeaye, Are you know this error's meaning?

https://gist.github.com/haruki7049/b04d1772c570f6019bd02808a0fc3647#file-err-log

Commit hash: 3e3a3bd

jeaye · 2024-11-10T01:33:59Z

Hey @jeaye, Are you know this error's meaning?

https://gist.github.com/haruki7049/b04d1772c570f6019bd02808a0fc3647#file-err-log

Commit hash: 3e3a3bd

The error is saying that the incremental PCH we're loading was compiled with different flags than the jank binary. So when the jank binary tries to load the incremental PCH at runtime, clang raises this error.

I saw this before and I think I fixed it by ensuring all the same compilation flags are used for both jank and the incremental.pch. Though, I think I've only seen it when using a prebuilt clang/llvm for building jank. I've never been able to fully get that working, for one reason or another.

jeaye · 2024-11-10T01:45:14Z

If that incremental.pch is the only thing getting in the way, perhaps we can try without it. There would be a severe performance penalty for startup time, though. That's why I'm hopeful that we can get it loaded normally.

haruki7049 added 2 commits October 4, 2024 06:02

feat(default.nix): Init in /comipler+runtime

d4508cd

chore(default.nix): Add pname, version and src

f4472b8

haruki7049 mentioned this pull request Oct 3, 2024

feat(shell.nix): Edit shell.nix to use reproducible development environment #93

Merged

haruki7049 added 3 commits October 4, 2024 07:01

WIP

fe68684

WIP

bc61d1c

chore(default.nix): Edit

7585782

haruki7049 changed the title ~~feat(default.nix): Init in /comipler+runtime~~ feat(default.nix): Init in /compiler+runtime Oct 4, 2024

haruki7049 added 5 commits October 4, 2024 18:37

fix(cmake/llvm.cmake): Edit versions and logics

0a25489

chore(default.nix): Add some buildInputs

c9d00c7

deps(nix-dir): Add some deps written by nix

b7348a1

chore(CmakeLists.txt): Comment out to not copy clang-format

cc7168b

deps(folly): Init, but WIP

261d738

chore(folly): Change src to use local submodule

52a34c0

Merge main branch

3f60ab5

Merge main branch

c221ce9

haruki7049 commented Nov 2, 2024

View reviewed changes

compiler+runtime/CMakeLists.txt Show resolved Hide resolved

haruki7049 force-pushed the feat/setup-nix-build-compiler branch from ed864ba to c221ce9 Compare November 2, 2024 05:32

This comment was marked as outdated.

Sign in to view

Merge main branch

3e3a3bd

chore(nix/readline.nix): Remove because this is unused

cd31331

feat(default.nix): Init in /compiler+runtime #94

Are you sure you want to change the base?

feat(default.nix): Init in /compiler+runtime #94

Conversation

haruki7049 commented Oct 3, 2024

haruki7049 commented Oct 4, 2024

haruki7049 commented Oct 4, 2024

jeaye commented Oct 4, 2024

jeaye commented Oct 4, 2024 • edited Loading

haruki7049 commented Oct 4, 2024 • edited Loading

haruki7049 commented Oct 4, 2024

jeaye commented Oct 4, 2024

haruki7049 commented Oct 4, 2024

jeaye commented Oct 4, 2024

haruki7049 commented Oct 4, 2024

jeaye commented Oct 4, 2024

haruki7049 commented Oct 4, 2024

haruki7049 commented Oct 4, 2024

jeaye commented Oct 4, 2024

haruki7049 commented Oct 4, 2024

jeaye commented Oct 4, 2024

haruki7049 commented Oct 4, 2024

jeaye commented Oct 14, 2024

haruki7049 commented Oct 14, 2024

haruki7049 commented Oct 14, 2024

jeaye commented Oct 14, 2024

jeaye commented Oct 29, 2024

haruki7049 commented Oct 29, 2024

haruki7049 commented Oct 29, 2024

jeaye commented Oct 29, 2024

haruki7049 commented Oct 29, 2024

jeaye commented Oct 29, 2024

haruki7049 commented Oct 29, 2024

jeaye commented Oct 29, 2024 via email

haruki7049 commented Oct 29, 2024

jeaye commented Oct 30, 2024 via email

jeaye commented Oct 30, 2024

haruki7049 commented Oct 30, 2024 • edited Loading

nix-shell -p nix-info --run "nix-info -m"

jeaye commented Oct 30, 2024

nix-shell -p nix-info --run "nix-info -m"

jeaye commented Nov 2, 2024

This comment was marked as outdated.

haruki7049 commented Nov 10, 2024 • edited Loading

jeaye commented Nov 10, 2024

haruki7049 commented Nov 10, 2024

jeaye commented Nov 10, 2024 • edited Loading

jeaye commented Nov 10, 2024

jeaye commented Oct 4, 2024 •

edited

Loading

haruki7049 commented Oct 4, 2024 •

edited

Loading

haruki7049 commented Oct 30, 2024 •

edited

Loading

`nix-shell -p nix-info --run "nix-info -m"`

`nix-shell -p nix-info --run "nix-info -m"`

haruki7049 commented Nov 10, 2024 •

edited

Loading

jeaye commented Nov 10, 2024 •

edited

Loading