Isolate macOS wheel builds from Homebrew #8497

freakboy3742 · 2024-10-25T01:55:57Z

The existing macOS cibuildwheel configuration relies on Homebrew to provide some dependencies. This clearly works in practice, but there are some issues associated with this choice.

Firstly, it is destructive when used on a local build machine. The wheel dependency script invokes brew remove --ignore-dependencies (which can leave the host machine in a broken, and potentially difficult to restore state); and requires the use of sudo to make changes in the /usr/local tree. This means it is difficult to test cibuildwheel changes locally, or to recommend using cibuildwheel as a local build solution.

Secondly, Homebrew builds could be incompatible with Pillow builds. The MACOSX_DEPLOYMENT_TARGET for Homebrew dependencies is fixed by the Homebrew build chain, rather than the Pillow build tools, so a change in Pillow's configuration may not be satisfied by Homebrew's build configuration. This isn't an issue at present, but it could easily become one in future.

Related to this, there is also the potential that a change in a Homebrew recipe could lead to the inadvertent introduction of additional binary libraries into the delocated macOS binaries. This is especially problematic with the fribidi, raqm and freetype dependedencies which are included as "vendor" sources.

However, the real motivation for proposing this change is that it is a precursor for PEP 730-compliant iOS builds. If Homebrew is on the build path when compiling iOS binaries, compiler tooling will often find an ARM64 Homebrew binary and attempt link it into an iOS library - which then (predictably) breaks. So - Homebrew isolation is essential for iOS compilation. Since the build processes for iOS and macOS are very similar, and Homebrew isolation was necessary for iOS, adding Homebrew isolation for macOS will simplify any future iOS patch. It also provides a way to submit iOS support in a series of smaller pieces, rather than a single "monster" patch. Plus, it provides all the side benefits of isolation described above.

An overview of the changes:

Uses the build folder as a location for all dependency builds. This prevents the root checkout from becoming dirtied by dependency artefacts, simplifying cleanup, and avoiding the need for extensive modifications to the .gitignore file.
Sets up an isolated build prefix in the build folder (./build/deps) where dependencies can be installed, and make installs all dependencies into that path.
Forces PATH to be a "clean" environment that only includes bare system tools, plus the Python binary being used for the build, and the isolated build prefix. macOS doesn't include most of it's development libraries in /usr or /usr/local, instead using a path provided by the macOS SDK (which is configured as part of the compiler toolchain).
Adds a build of pkg-config to the dependencies. This is the one build tool that autotools and cmake often require that Xcode doesn't provide. Providing a custom build of pkg-config also ensures that no dependencies other than the ones provided by cibuildwheel are included. The build recipe that is used is derived from the Homebrew recipe.
Adds a build of libXdmcp, to avoid a dependency on the Homebrew-provided version.
Adds builds of fribidi and raqm, rather than using the Homebrew versions as "vendored" versions. The setup.py configuration passed in by cibuildwheel has been modified from the default on other platforms to not use the vendor version.
Updates the pinned multibuild version to the current devel branch hash. This is to incorporate Python3.13 support, plus a number of fixes required to support installs into locations other than /usr/local, and corrections to LIBDIR handling for platforms that use the lib64 suffix.
Adds a dependencies-prefix configuration option to setup.py. This allows the user to provide a specific location to look for dependencies, rather than the series of Fink/MacPorts/Homebrew fallbacks that currently exist.

.github/workflows/wheels-dependencies.sh

.github/workflows/wheels-test.sh

radarhere · 2024-10-25T02:13:13Z

.github/workflows/wheels-dependencies.sh

@@ -23,6 +23,8 @@ OPENJPEG_VERSION=2.5.2
 XZ_VERSION=5.6.3
 TIFF_VERSION=4.6.0
 LCMS2_VERSION=2.16
+RAQM_VERSION=0.7.1


Suggested change

RAQM_VERSION=0.7.1

RAQM_VERSION=0.10.2

It definitely makes sense to use the most recent available version; but pillow-depends-main doesn't include this version. What's the procedure for updating that repo?

(Related: pkg-config, libXdmcp, and an updated webp sources should probably be added to that project)

The theory is that pillow-depends-main is just a more reliable source for the dependencies, and PRs should work without it.

Once this is merged, I will likely add them there.

freakboy3742 · 2024-10-25T02:41:02Z

The manylinux failures look like they'll be fixed by this PR to multi build.

The x86_64 macOS failures are probably a leakage from /usr/local that I haven't accounted for. I'm looking into that one now.

I'm not sure how to interpret the coverage drop, though...

freakboy3742 · 2024-10-25T05:53:53Z

@radarhere I've updated the pin for multibuild to include the (just merged) PR reverting the libjpeg-turbo CMAKE_INSTALL_LIBDIR change; and I've addressed some additional locations where Homebrew can leak in on x86_64.

nulano · 2024-10-25T09:32:17Z

Related to this, there is also the potential that a change in a Homebrew recipe could lead to the inadvertent introduction of additional binary libraries into the delocated macOS binaries.

This has already happened many times in the past, so thanks for working on this.

Adds builds of fribidi and raqm, rather than using the Homebrew versions as "vendored" versions. The setup.py configuration passed in by cibuildwheel has been modified from the default on other platforms to not use the vendor version.

I'm not sure if I understand what you mean here. We need to use a vendored version of raqm in wheels for license reasons. When using raqm=vendor, we build our own version of raqm from https://github.com/python-pillow/Pillow/tree/main/src/thirdparty/raqm. Similarly, using fribidi=vendor does not actually build fribidi, but instead custom code that can load fribidi at runtime, see https://github.com/python-pillow/Pillow/tree/main/src/thirdparty/fribidi-shim. The Homebrew version of fribidi is only used for testing the built wheels.

freakboy3742 · 2024-10-28T01:23:18Z

Adds builds of fribidi and raqm, rather than using the Homebrew versions as "vendored" versions. The setup.py configuration passed in by cibuildwheel has been modified from the default on other platforms to not use the vendor version.

I'm not sure if I understand what you mean here. We need to use a vendored version of raqm in wheels for license reasons. When using raqm=vendor, we build our own version of raqm from https://github.com/python-pillow/Pillow/tree/main/src/thirdparty/raqm. Similarly, using fribidi=vendor does not actually build fribidi, but instead custom code that can load fribidi at runtime, see https://github.com/python-pillow/Pillow/tree/main/src/thirdparty/fribidi-shim. The Homebrew version of fribidi is only used for testing the built wheels.

Thanks - that's some helpful context. I was seeing the missing fribidi detection during the build and interpreting that as something that was part of the move away from homebrew; but I see now that Pillow is loading from Homebrew by design in this instance. That also might explain the coverage drop (as supplying fribidi means any runtime loading code won't be in use). I'll take another swing at that part (and resolve the linux build issues as well).

Out of interest - what's the license issue with raqm? The source code indicates It looks to be MIT licensed... did it used to be GPL (or something else problematic?)

hugovk · 2024-10-28T05:07:36Z

Out of interest - what's the license issue with raqm? The source code indicates It looks to be MIT licensed... did it used to be GPL (or something else problematic?)

From #2753 in 2017:

We can't currently distribute a binary of Pillow with support for raqm due to license incompatibility, because while libraqm is MIT licensed, it links to GPL libraries.

FriBiDi is LGPL.

Since then, Raqm added support for Apache-licensed SheenBiDi in 2021: HOST-Oman/libraqm#138, HOST-Oman/libraqm#139.

Perhaps we could switch.

freakboy3742 · 2024-10-29T21:42:26Z

@hugovk /me casually drops this article into the chat :-)

FWIW - BeeWare ditched Codecov a couple of years ago specifically because of weird inconsistencies like this one; and although the output of a raw coverage report is marginally less pretty (i.e., no annotated code listings), you can get the same report locally that you get in CI, and you get none of the weird outages and reporting issues that seem to plague Coverage. YMMV; but if there's interest, it's a relatively straightforward switch, especially if coverage reporting is already in place.

Co-authored-by: Andrew Murray <[email protected]>

freakboy3742 · 2024-10-29T22:27:52Z

The (seemingly inconsistent) Windows/pypy3.10 test failure also fascinates me... is this the same cache issue mentioned by #8513?

radarhere · 2024-10-30T07:58:09Z

.github/workflows/wheels-test.sh

+    # However, we *do* need Homebrew to provide a copy of fribidi for
+    # testing purposes so that we can verify the fribidi shim works as expected.
+    if [[ "$(uname -m)" == "x86_64" ]]; then
+        HOMEBREW_HOME=/usr/local/homebrew


Suggested change

HOMEBREW_HOME=/usr/local/homebrew

HOMEBREW_HOME=/usr/local

I would have thought just '/usr/local', based on https://docs.brew.sh/Installation

The script installs Homebrew to its default, supported, best prefix (/opt/homebrew for Apple Silicon, /usr/local for macOS Intel

Turns out both work (for brew at least) - Homebrew keeps the "original" versions in /usr/local/homebrew. However, I agree that /usr/local is the generally intended entry point.

I take that back - it does matter (and making this change caused test failures on x86_64).

Firstly, it's possible for homebrew to have packages installed, but not linked. Using the /usr/local/Homebrew location ensures that the binary can be used regardless (note also the capitalisation fix - doesn't strictly matter on HFS+, but for accuracy).

Secondly, it's possible for other libraries to be in /usr/local/lib. Using /usr/local/Homebrew/lib for the DYLD_LIBRARY_PATH avoids this.

hugovk · 2024-10-30T12:54:08Z

The (seemingly inconsistent) Windows/pypy3.10 test failure also fascinates me... is this the same cache issue mentioned by #8513?

Yep, we added a new licence (well, old: MIT CMU) to OSI and Trove classifiers, so that test needs the new trove-classifiers. #7942 has the details.

I've opened #8514 as an alternative to #8513.

Co-authored-by: Andrew Murray <[email protected]>

nulano · 2024-11-04T19:04:23Z

The shim fribidi builds is used in the wheel, with Homebrew providing the fribidi binary on macOS in the test environment

Thanks, that looks better.

There still seems to be something wrong with freetype on arm64, https://github.com/python-pillow/Pillow/actions/runs/11604707482/job/32314276332?pr=8497#step:5:14308

  SKIPPED [2] Tests/test_imagefont.py:1081: FreeType compiled without brotli or WOFF2 support

But it was built with brotli, https://github.com/python-pillow/Pillow/actions/runs/11604707482/job/32314276332?pr=8497#step:5:7768:

    brotli:        yes (pkg-config)

We need to support this for #6554.

We could also consider compiling libwebp before libtiff so that WEBP-compressed tiff files can be read (AFAIK webp only looks for libtiff so that it can compile companion tools, but does not actually use it in the library - as I mentioned in #6562). But we don't need to do it now since the test was already being skipped on macOS and Linux before this PR.
https://github.com/python-pillow/Pillow/actions/runs/11604707482/job/32314276332?pr=8497#step:5:15489:

  SKIPPED [1] Tests/test_file_libtiff.py:905: WEBP compression support is not configured

freakboy3742 · 2024-11-04T22:04:13Z

There still seems to be something wrong with freetype on arm64, https://github.com/python-pillow/Pillow/actions/runs/11604707482/job/32314276332?pr=8497#step:5:14308

Weird. My immediate guess is that it might have something to do with the fact that a vendored version of fribidi is being used, which might not be triggering brotli support (or at least not triggering the test). I'll take a look and see what is going on.

We could also consider compiling libwebp before libtiff so that WEBP-compressed tiff files can be read (AFAIK webp only looks for libtiff so that it can compile companion tools, but does not actually use it in the library - as I mentioned in #6562). But we don't need to do it now since the test was already being skipped on macOS and Linux before this PR. https://github.com/python-pillow/Pillow/actions/runs/11604707482/job/32314276332?pr=8497#step:5:15489:

I saw that, and ultimately gave up trying to fix it - firstly because I was trying to limit the scope of the changes, but also because I'm not sure it can be fixed. libwebp has a dependency on tiff... but then lcms2 needs tiff; and openjpeg needs libpng, tiff, and lcms2. So - I'm not sure this can be fixed (or, at least, it can't be fixed without some sort of 2-pass compile and the use of dynamic linking, which iOS can't use anyway).

nulano · 2024-11-04T22:11:52Z

Weird. My immediate guess is that it might have something to do with the fact that a vendored version of fribidi is being used, which might not be triggering brotli support (or at least not triggering the test). I'll take a look and see what is going on.

The test skip is triggered by a specific error raised by the freetype library which itself does not use fribidi.

libwebp has a dependency on tiff... but then lcms2 needs tiff; and openjpeg needs libpng, tiff, and lcms2

AFAIK all of these dependencies are just for the binary utilities that we don't need so we can disable them with relevant compile flags; the libraries themselves do not have dependencies between them (except for libtiff optionally using libwebp). But I agree that it might be best to leave that for a separate PR.

freakboy3742 · 2024-11-06T06:28:48Z

@nulano I've found the culprit - Homebrew's copy of freetype was leaking into the test environment.

By using DYLD_LIBRARY_PATH pointed at Homebrew's lib foldler, all of Homebrew's libraries become available, and take take precedence over the ones provided by the wheel. As a result, Homebrew's copy of freetype is used, which doesn't include Brotli support. I presume this wasn't an issue previously because an explicit DYLD_LIBRARY_PATH comes earlier in library resolution precedence; when Homebrew was being used implicitly, Homebrew's library resolve after Pillow's bundled dependencies.

This can be fixed by pointing the DYLD_LIBRARY_PATH at the cellar version of the vendored fribidi library (which is the only library we actually need from Homebrew). I've pushed that update (update: I'm looking at the linting error now...)

The update includes 2 other small changes:

The location of the "homebrew isolated" macOS build is slightly altered - the extra darwin folder became necessary when I started working on the iOS build, and the build folders for macOS started colliding with iOS.
I've added some safety checks against some uses that are valid cibuildwheel invocations, but will break if you try to use them. Specifically, CIBW_ARCHS="arm64 x86_64", and not specifying CIBW_ARCHS at all, are both valid cibuildwheel setup, but Pillow doesn't allow it because the before_all step requires a single explicit architecture. This bit me a couple of times before I worked out what was happening; I figured it couldn't hurt to raise an explicit error when we know it won't work.

.github/workflows/wheels-dependencies.sh

nulano · 2024-11-07T19:32:15Z

Thanks, looks good to me now.

Co-authored-by: Andrew Murray <[email protected]>

.github/workflows/wheels-dependencies.sh

Co-authored-by: Andrew Murray <[email protected]>

radarhere · 2024-11-12T11:32:29Z

setup.py

@@ -448,7 +454,7 @@ def _remove_extension(self, name: str) -> None:
    def get_macos_sdk_path(self) -> str | None:
        try:
            sdk_path = (
-                subprocess.check_output(["xcrun", "--show-sdk-path"])
+                subprocess.check_output(["xcrun", "--show-sdk-path", "--sdk", "macosx"])


What was the reason for this change?

Explicitness. Xcode can manage multiple SDKs (iOS being an obvious one); adding the explicit --sdk macosx SDK ensures that you're definitely getting the macOS one. That will be the default on almost every install of Xcode, but if you have a stray SDKROOT environment variable from some other activity, it could be inadvertently pointing at an iOS, tvOS, visionOS or MacCatalyst SDK.

radarhere · 2024-11-13T02:39:08Z

.github/workflows/wheels-test.sh

+    # installed copy of fribidi is cellared. This ensures we don't pick up the
+    # Homebrew version of any other library that we're dependent on (most notably,
+    # freetype).
+    export DYLD_LIBRARY_PATH=$(dirname $(realpath $HOMEBREW_HOME/lib/libfribidi.dylib))


Suggested change

export DYLD_LIBRARY_PATH=$(dirname $(realpath $HOMEBREW_HOME/lib/libfribidi.dylib))

if [[ "$(uname -m)" == "x86_64" ]]; then

FRIBIDI_DYLIB=$(find /usr/local/Cellar/fribidi -type f -name libfribidi.dylib)

else

FRIBIDI_DYLIB=$HOMEBREW_HOME/lib/libfribidi.dylib

fi

export DYLD_LIBRARY_PATH=$(dirname $FRIBIDI_DYLIB)

https://github.com/python-pillow/Pillow/actions/runs/11752543578/job/32744256384?pr=8497#step:5:9142

realpath: /usr/local/Homebrew/lib/libfribidi.dylib: No such file or directory

libfribidi.dylib isn't located there on the macOS Intel jobs.

Huh... at some point, I've clearly done something to mess up my x86 test machine's Homebrew install, because I've got a /usr/local/Homebrew/Cellar... but I now see I'm also getting a bunch of warnings because of the existence of that folder.

The find-based solution you've got here will fail (or be prone to failure) if there's more than one version of fribidi installed, which is possible on user machines; but if HOMEBREW_HOME is moved back to /usr/local, the realpath approach should work.

Disable platform guessing instead of adding dependencies-prefix

radarhere · 2024-11-15T09:32:28Z

.github/workflows/wheels-dependencies.sh

+    PLAT=$CIBW_ARCHS
+else
+    # Build prefix will default to /usr/local
+    WORKDIR=$(pwd)/build
+    PLAT=$CIBW_ARCHS
+    MB_ML_LIBC=${AUDITWHEEL_POLICY::9}
+    MB_ML_VER=${AUDITWHEEL_POLICY:9}
 fi


Suggested change

PLAT=$CIBW_ARCHS

else

# Build prefix will default to /usr/local

WORKDIR=$(pwd)/build

PLAT=$CIBW_ARCHS

MB_ML_LIBC=${AUDITWHEEL_POLICY::9}

MB_ML_VER=${AUDITWHEEL_POLICY:9}

fi

else

# Build prefix will default to /usr/local

WORKDIR=$(pwd)/build

MB_ML_LIBC=${AUDITWHEEL_POLICY::9}

MB_ML_VER=${AUDITWHEEL_POLICY:9}

fi

PLAT=$CIBW_ARCHS

radarhere · 2024-11-15T09:46:18Z

.github/workflows/wheels-dependencies.sh

+        # Custom tiff build to include jpeg; by default, configure won't include
+        # headers/libs in the custom macOS prefix. Explicitly disable webp and
+        # zstd, because on x86_64 macs, it will pick up the Homebrew versions of
+        # webp and zstd from /usr/local.


Should this comment be updated to also mention libdeflate being disabled?

Isolate macOS build from Homebrew.

0fe55d6

radarhere changed the title ~~Isolate macOS wheel builds from Homebrew.~~ Isolate macOS wheel builds from Homebrew Oct 25, 2024

radarhere reviewed Oct 25, 2024

View reviewed changes

.github/workflows/wheels-dependencies.sh Outdated Show resolved Hide resolved

radarhere reviewed Oct 25, 2024

View reviewed changes

.github/workflows/wheels-test.sh Outdated Show resolved Hide resolved

radarhere reviewed Oct 25, 2024

View reviewed changes

freakboy3742 added 3 commits October 25, 2024 10:55

Cleanups and typos identified by code review.

fc35fcc

Tweaks to ensure isolation from Homebrew on x86_64.

00809a2

Bump the multibuild version to fix jpeg-turbo issue.

06dbfed

freakboy3742 requested a review from radarhere October 25, 2024 05:53

Correct a dumb pip invocation error.

5a8373e

hugovk added macOS Build labels Oct 25, 2024

freakboy3742 added 3 commits October 25, 2024 16:04

Explicitly disable libdeflate on libtiff.

140a06e

Possible fix for linux build failures.

0961d3d

Copy manylinux lib64 files from the correct built prefix.

43c34fc

radarhere mentioned this pull request Oct 26, 2024

brew remove libdeflate #8502

Merged

radarhere mentioned this pull request Oct 28, 2024

Set LIBDIR for remaining cmake-built libraries. multi-build/multibuild#538

Merged

radarhere and others added 8 commits October 29, 2024 09:52

Merge branch 'main' into homebrew-isolation

3e4be4b

Revert fribidi/raqm changes for macOS builds.

0855468

Bump multibuild to include more cmake changes.

8308bf3

Correct paths used for Linux build.

c74a5bd

Simplify Linux config by correcting a logic error in macOS config.

72d81e2

Can't check IS_MACOS until common_utils is invoked.

ec214e4

Don't use multibuild variables before invoking multibuild.

d1a4f80

Remove stray debug.

6d13704

freakboy3742 and others added 2 commits October 30, 2024 05:43

Corrected typo in code comment.

c6912f8

Co-authored-by: Andrew Murray <[email protected]>

Merge branch 'main' into homebrew-isolation

96ae15c

radarhere reviewed Oct 30, 2024

View reviewed changes

freakboy3742 and others added 2 commits October 31, 2024 06:27

Use the intended entry point for the x86_64 brew binary.

01270b5

Co-authored-by: Andrew Murray <[email protected]>

Revert x86_64 homebrew location change (with explanation).

51e3623

freakboy3742 force-pushed the homebrew-isolation branch from 4e0312c to 51e3623 Compare October 31, 2024 01:54

Correct handling of vendored fribidi.

e82b539

Merge branch 'main' into homebrew-isolation

904416b

radarhere reviewed Nov 7, 2024

View reviewed changes

.github/workflows/wheels-dependencies.sh Outdated Show resolved Hide resolved

Correct typo in comment.

4e35852

Co-authored-by: Andrew Murray <[email protected]>

radarhere reviewed Nov 8, 2024

View reviewed changes

.github/workflows/wheels-dependencies.sh Outdated Show resolved Hide resolved

radarhere reviewed Nov 8, 2024

View reviewed changes

.github/workflows/wheels-dependencies.sh Outdated Show resolved Hide resolved

Apply suggestions from code review

681a03b

Co-authored-by: Andrew Murray <[email protected]>

radarhere reviewed Nov 12, 2024

View reviewed changes

Disable platform guessing instead of adding dependencies-prefix

378df7a

This was referenced Nov 12, 2024

Disable platform guessing instead of adding dependencies-prefix freakboy3742/Pillow#1

Merged

Having trouble installing libraqm #8232

Closed

radarhere reviewed Nov 13, 2024

View reviewed changes

freakboy3742 added 2 commits November 13, 2024 11:03

Correct the lookup of libfribidi on x86 macOS installs.

9dc6904

Merge pull request #1 from radarhere/homebrew-isolation

0e3eb70

Disable platform guessing instead of adding dependencies-prefix

radarhere reviewed Nov 15, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Isolate macOS wheel builds from Homebrew #8497

Isolate macOS wheel builds from Homebrew #8497

freakboy3742 commented Oct 25, 2024 •

edited

Loading

radarhere Oct 25, 2024

freakboy3742 Oct 25, 2024

radarhere Oct 25, 2024

freakboy3742 commented Oct 25, 2024

freakboy3742 commented Oct 25, 2024

nulano commented Oct 25, 2024

freakboy3742 commented Oct 28, 2024

hugovk commented Oct 28, 2024

freakboy3742 commented Oct 29, 2024

freakboy3742 commented Oct 29, 2024

radarhere Oct 30, 2024

freakboy3742 Oct 30, 2024 •

edited

Loading

freakboy3742 Oct 31, 2024 •

edited

Loading

hugovk commented Oct 30, 2024

nulano commented Nov 4, 2024 •

edited

Loading

freakboy3742 commented Nov 4, 2024

nulano commented Nov 4, 2024

freakboy3742 commented Nov 6, 2024

nulano commented Nov 7, 2024

radarhere Nov 12, 2024

freakboy3742 Nov 13, 2024

radarhere Nov 13, 2024

freakboy3742 Nov 13, 2024

radarhere Nov 15, 2024

radarhere Nov 15, 2024

-    export DYLD_LIBRARY_PATH=$(dirname $(realpath $HOMEBREW_HOME/lib/libfribidi.dylib))
+    if [[ "$(uname -m)" == "x86_64" ]]; then
+        FRIBIDI_DYLIB=$(find /usr/local/Cellar/fribidi -type f -name libfribidi.dylib)
+    else
+        FRIBIDI_DYLIB=$HOMEBREW_HOME/lib/libfribidi.dylib
+    fi
+    export DYLD_LIBRARY_PATH=$(dirname $FRIBIDI_DYLIB)

Isolate macOS wheel builds from Homebrew #8497

Are you sure you want to change the base?

Isolate macOS wheel builds from Homebrew #8497

Conversation

freakboy3742 commented Oct 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freakboy3742 commented Oct 25, 2024

freakboy3742 commented Oct 25, 2024

nulano commented Oct 25, 2024

freakboy3742 commented Oct 28, 2024

hugovk commented Oct 28, 2024

freakboy3742 commented Oct 29, 2024

freakboy3742 commented Oct 29, 2024

Choose a reason for hiding this comment

freakboy3742 Oct 30, 2024 • edited Loading

Choose a reason for hiding this comment

freakboy3742 Oct 31, 2024 • edited Loading

Choose a reason for hiding this comment

hugovk commented Oct 30, 2024

nulano commented Nov 4, 2024 • edited Loading

freakboy3742 commented Nov 4, 2024

nulano commented Nov 4, 2024

freakboy3742 commented Nov 6, 2024

nulano commented Nov 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freakboy3742 commented Oct 25, 2024 •

edited

Loading

freakboy3742 Oct 30, 2024 •

edited

Loading

freakboy3742 Oct 31, 2024 •

edited

Loading

nulano commented Nov 4, 2024 •

edited

Loading