Symbol relocation errors when used with more than 1 layer on squashfs or dwarfs filesystems

[ruanformigoni]

Greetings,

I've found an issue where if overlayfs is used with more than one layer it breaks system libraries when used with squashfs or dwarfs. This is shown by symbol not found errors:

Error relocating /usr/lib/libQt5Gui.so.5: glLoadIdentity: symbol not found
Error relocating /usr/lib/libQt5Gui.so.5: glMatrixMode: symbol not found
Error relocating /usr/lib/libQt5Gui.so.5: glLoadMatrixf: symbol not found
Error relocating /usr/lib/libQt5Gui.so.5: glOrtho: symbol not found
Error relocating /usr/lib/libxcb.so.1: XauGetBestAuthByAddr: symbol not found
Error relocating /usr/lib/libxcb.so.1: XauDisposeAuth: symbol not found

If I copy the libraries to the upper directory, it works again. Here is a script to replicate the issue, it quickly sets up two layers, one with an xorg base and another with krita and its dependencies, the issue arises when krita (layer-1) requires libraries from layer-0. If everything is on the same layer this issue does not happen.

#!/usr/bin/env bash

# Script to setup alpine base
# First layer contains an xorg setup
# Second layer contains krita and its dependencies

set -x

DIR_SCRIPT="$(cd -- "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

DIR_BUILD="$DIR_SCRIPT"/build

# Cleanup previous execution
fusermount -u "$DIR_BUILD/mount"
rm -rf "$DIR_BUILD/mount"
fusermount -u "$DIR_BUILD/root-0"
rm -rf "$DIR_BUILD/root-0"
fusermount -u "$DIR_BUILD/root-1"
rm -rf "$DIR_BUILD/root-1"

rm -rf "$DIR_BUILD"
mkdir -p "$DIR_BUILD"
cd "$DIR_BUILD"

# Bootstrap alpine
wget -O apk-tools.apk http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/apk-tools-static-2.10.8-r1.apk
tar zxf apk-tools.apk
./sbin/apk.static --arch x86_64 -X http://dl-cdn.alpinelinux.org/alpine/latest-stable/main/ -U --allow-untrusted --root ./root-0 --initdb add alpine-base
{ sed -E 's/^\s+://' | tee ./root-0/etc/apk/repositories; } <<-END
  :http://dl-cdn.alpinelinux.org/alpine/v3.16/main
  :http://dl-cdn.alpinelinux.org/alpine/v3.16/community
  :#http://dl-cdn.alpinelinux.org/alpine/edge/main
  :#http://dl-cdn.alpinelinux.org/alpine/edge/community
  :#http://dl-cdn.alpinelinux.org/alpine/edge/testing
END

# Configure base
proot -R ./root-0 /bin/sh -c 'apk update'
proot -R ./root-0 /bin/sh -c 'apk upgrade'
proot -R ./root-0 /bin/sh -c 'setup-xorg-base'

# Compress first layer
mksquashfs root-0 root-0.layer
# Remove original root & replace with mounted squashfs
rm -rf root-0 && mkdir root-0
# Mount root
squashfuse root-0.layer root-0
# Mount overlayfs and install krita
mkdir -p upper work mount
overlayfs \
  -o lowerdir="$DIR_BUILD"/root-0 \
  -o upperdir="$DIR_BUILD"/upper \
  -o workdir="$DIR_BUILD"/work \
  mount
proot -R ./mount /bin/sh -c 'apk add krita'
fusermount -u mount
# Compress second layer
mksquashfs upper root-1.layer
# Remove original root & replace with mounted squashfs
rm -rf upper && mkdir -p upper root-1
# Mount root
squashfuse root-1.layer root-1
overlayfs \
  -o lowerdir="$DIR_BUILD"/root-1:"$DIR_BUILD"/root-0 \
  -o upperdir="$DIR_BUILD"/upper \
  -o workdir="$DIR_BUILD"/work \
  mount
proot -R ./mount /bin/sh -c 'krita'

Note that this is not a proot issue, since this also breaks in bwrap. I tested with bwrap new --overlay options, and they do not present the issue found here.

[ruanformigoni]

The commit 6a0de4a is the culprit behind this issue, reverting to the one previous to it with git reset --hard 918e19c fixes the symbol issues and krita opens.