Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python linkage enhancements #419

Closed

Conversation

whitslack
Copy link
Contributor

  • On systems supporting Libtool (i.e., anything but Windows), skip compiling the "amalgamated" sources for the Python module and simply link in the just-built libwallycore.a and libsecp256k1.a.

  • setup.py: support optionally building and linking to libwally-core as a dynamic shared object if WALLY_ABI_PY_WHEEL_USE_DSO=1 is set. Ignored on Windows, where setup.py does not build libwally-core.

libwallycore.la is linked into the Python native extension library.
Previously this wasn't really being done at all; rather, all of the
libwally-core code was being recompiled into some "combined" objects
that would then be linked into the Python extension. This was a needless
duplication of work since the compiled objects already exist in
libwallycore.a and libsecp256k1.a. Simply link those libraries into the
extension.

Since Windows does not use libtool, continue to compile the "combined"
objects on Windows until a cleaner solution can be implemented.
Iff the environment contains WALLY_ABI_PY_WHEEL_USE_DSO=1, then:

 * setup.py will configure and build libwally-core as a shared library;
 * the Python native extension library will not contain any
   libwally-core or libsecp256k1 code; and
 * the dynamic linker will need to find and load libwallycore.so.1
   whenever the Python native extension is loaded.
@whitslack
Copy link
Contributor Author

whitslack commented Sep 4, 2023

It should be possible to determine what args setup.py was called with I think, can you give me the calls that are made (and the arguments) by the Gentoo machinery?

@jgriffiths: This is the command line that Gentoo's distutils-r1.eclass invokes to build the libwally-core Python wheel:

gpep517 build-wheel --backend setuptools.build_meta:__legacy__ --output-fd 3 --wheel-dir /var/tmp/portage/net-libs/libwally-core-1.0.0/work/libwally-core-release_1.0.0-python3_11/wheel

And when I set DISTUTILS_EXT=1 like I'm supposed to for Python modules that include a native extension library, then before the above command the machinery runs this command to pre-build the extension before building the wheel:

python3.11 setup.py build_ext -j 5

Ostensibly the reason for splitting it out like that is so that the native library build can be parallelized, although in the case of libwally-core there is only one C source file (src/swig_python/swig_python_wrap.c), so there's no advantage, but I'll set DISTUTILS_EXT=1 anyway just to suppress the QA warning.
EDIT: Gentoo no longer runs build_ext in advance but now rather sets DIST_EXTRA_CONFIG to the path of a config file that specifies the parallelism for the build_ext phase that runs as part of the normal wheel building process.

In either case, after building the wheel, the machinery runs this command to install it into a staging location:

gpep517 install-wheel --destdir=/var/tmp/portage/net-libs/libwally-core-1.0.0/work/libwally-core-release_1.0.0-python3_11/install --interpreter=/usr/bin/python3.11 --prefix=/usr --optimize=all /var/tmp/portage/net-libs/libwally-core-1.0.0/work/libwally-core-release_1.0.0-python3_11/wheel/wallycore-1.0.0-cp311-cp311-linux_x86_64.whl

I'm not sure if any of that helps you at all. (I rather doubt it.)

@jgriffiths jgriffiths deleted the branch ElementsProject:build_updates October 30, 2023 12:01
@jgriffiths jgriffiths closed this Oct 30, 2023
@whitslack
Copy link
Contributor Author

@jgriffiths: Did this slip through the cracks, or have you decided not to support this after all?

@jgriffiths
Copy link
Contributor

@whitslack yes it did. The python build machinery has been updated somewhat, so this will need re-working. Please submit under a new PR if you would.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants