Fix PyJulia on Python 3.12 #538

MilesCranmer · 2023-10-30T23:46:44Z

cc @mkitti do you think you could give this a look over? It still seems like there are some issues even though from julia import Main doesn't blow up... Any help appreciated.

This updates the importlib API according to what was noticed in #509 (which were giving deprecation warnings, we simply didn't update in time).

This is a lot of guess work about the importlib API: https://docs.python.org/3/library/importlib.html, so I'd appreciate if someone could give a close reading.

I found this stackoverflow answer to be particularly helpful: https://stackoverflow.com/a/43573798/2689923

cc @IlyaOrson this should finally fix the PySR issue you saw (the proper way!)

MilesCranmer · 2023-10-31T03:08:27Z

Seems like the issue now is that the returned module is not a JuliaModule object, and thus doesn't have access to the Julia properties of the symbol. Maybe the sys.modules.setdefault is not working? Or somehow Julia is turning it into a regular module type?

mkitti · 2023-10-31T11:12:16Z

I'm looking.

codecov · 2023-11-05T21:19:23Z

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (bdefc8b) 85.22% compared to head (13abc67) 81.43%.

Files	Patch %	Lines
src/julia/core.py	94.44%	1 Missing ⚠️

Additional details and impacted files

@@                 Coverage Diff                 @@
##           fix-ci-settings     #538      +/-   ##
===================================================
- Coverage            85.22%   81.43%   -3.79%     
===================================================
  Files                   39       39              
  Lines                 2342     2349       +7     
===================================================
- Hits                  1996     1913      -83     
- Misses                 346      436      +90

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

MilesCranmer · 2023-11-05T21:47:36Z

Okay it seems like it's mostly working now after adding a missing method:

1 failed, 135 passed, 9 skipped, 7 warnings in 65.80s (0:01:05)

Want to take a look?

MilesCranmer · 2023-11-13T02:25:48Z

@mkitti do you think you might be able to take a look at this? Seems like we're very close now.

mkitti · 2023-11-13T13:28:03Z

I will have a little more time on Thursday.

MilesCranmer · 2023-11-17T16:54:37Z

Did you get a chance to take a look? Anything I can help push on?

mkitti · 2023-11-17T17:08:43Z

I'm taking a look now.

mkitti · 2023-11-18T05:09:20Z

This is taking a while to review since I'm trying to figure out why CI is failing. If you have any insight on getting CI to succeed to disabling some tests, let me know.

MilesCranmer · 2023-12-15T08:00:03Z

I did a downstream integration test with the full PySR suite and with this PR, everything seems to be working.

I think the current CI failure is due to this:

In Python 3.12, distutils will no longer be installed by make install or any of the first-party distribution. Third-party redistributors should no longer include distutils in their bundles or repositories.

Seems like distutils is being used somewhere in the pipeline?

MilesCranmer · 2023-12-15T08:03:46Z

Which tool is running this command?

│ ProcessFailedException(Base.Process[Process(setenv(`/home/runner/work/pyjulia/pyjulia/.tox/py/bin/python -c "import distutils.sysconfig; print(distutils.sysconfig.get_config_var('VERSION'))"`,["GITHUB_ACTIONS=true", "PYTHONHASHSEED=3720144583", "PATH=/home/runner/work/pyjulia/pyjulia/.tox/py/bin:/opt/hostedtoolcache/julia/1.4.2/x64/bin:/opt/hostedtoolcache/Python/3.12.0/x64/bin:/opt/hostedtoolcache/Python/3.12.0/x64:/home/runner/.local/bin:/opt/pipx_bin:/home/runner/.cargo/bin:/home/runner/.config/composer/vendor/bin:/usr/local/.ghcup/bin:/home/runner/.dotnet/tools:/snap/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games", "TOX_ENV_NAME=py", "PYTHON=/home/runner/work/pyjulia/pyjulia/.tox/py/bin/python", "PYJULIA_TEST_REBUILD=yes", "TOX_PACKAGE=/home/runner/work/pyjulia/pyjulia/.tox/.tmp/package/1/julia-0.6.1.tar.gz", "LD_LIBRARY_PATH=/opt/hostedtoolcache/Python/3.12.0/x64/lib", "PIP_DISABLE_PIP_VERSION_CHECK=1", "CI=true", "LANG=C.UTF-8", "PKG_CONFIG_PATH=/opt/hostedtoolcache/Python/3.12.0/x64/lib/pkgconfig", "JULIA_LOAD_PATH=@:/tmp/jl_aivj83", "TOX_WORK_DIR=/home/runner/work/pyjulia/pyjulia/.tox", "HOME=/home/runner", "PYJULIA_TEST_PYTHON_JL_IS_INSTALLED=yes", "TOX_ENV_DIR=/home/runner/work/pyjulia/pyjulia/.tox/py", "VIRTUAL_ENV=/home/runner/work/pyjulia/pyjulia/.tox/py", "PYTHONIOENCODING=UTF-8", "OPENBLAS_MAIN_FREE=1"]), ProcessExited(1))])

Is this from Conda.jl or PyCall.jl maybe? (distutils has been removed in 3.12)

MilesCranmer · 2023-12-15T08:32:15Z

I think this is partially fixed by my PR JuliaPy/PyCall.jl#1068... let's see

MilesCranmer · 2023-12-16T14:38:51Z

@mkitti it seems like with JuliaPy/PyCall.jl#1068, many of the 3.12 CI runs are now passing! There are some remaining issues but those look to be weirdness about the CI cache (?).

I wonder if we can merge this now (after your review) to fix 3.12 and then fix those other GitHub action-specific issues later?

mkitti · 2023-12-16T14:48:46Z

Ok, let's try

MilesCranmer · 2023-12-16T15:00:28Z

Thanks!

Not sure why but it merged into #541 rather than master... I guess I had set that as the base or something?

* Update setup actions * Fix PyJulia on Python 3.12 (#538) * Fix importlib deprecated API * Test Python 3.12 * Test Julia 1.9 * Fix semantics of exec_module vs create_module * Ensure we can find spec from JuliaModule * Bump version with python 3.12 compat * Declare compat for 3.11 and 3.12 * Only test 1.6, 1.9, 1

Squashed commit of the following: commit 12cba08 Merge: 28af425 ce3ac83 Author: Miles Cranmer <[email protected]> Date: Sat Dec 30 02:23:21 2023 +0000 Merge pull request #544 from JuliaPy/update-to-0.6.2 Bump to 0.6.2 with Python 3.12 fix commit ce3ac83 Author: MilesCranmer <[email protected]> Date: Sat Dec 30 02:03:45 2023 +0000 Bump to 0.6.2 with Python 3.12 fix commit 28af425 Author: Miles Cranmer <[email protected]> Date: Fri Dec 29 16:55:50 2023 +0000 Update setup actions and get working on 3.12 (#541) * Update setup actions * Fix PyJulia on Python 3.12 (#538) * Fix importlib deprecated API * Test Python 3.12 * Test Julia 1.9 * Fix semantics of exec_module vs create_module * Ensure we can find spec from JuliaModule * Bump version with python 3.12 compat * Declare compat for 3.11 and 3.12 * Only test 1.6, 1.9, 1

Squashed commit of the following: commit 18589bb Author: MilesCranmer <[email protected]> Date: Sat Dec 30 02:25:30 2023 +0000 Bump to 0.6.2 Squashed commit of the following: commit 12cba08 Merge: 28af425 ce3ac83 Author: Miles Cranmer <[email protected]> Date: Sat Dec 30 02:23:21 2023 +0000 Merge pull request #544 from JuliaPy/update-to-0.6.2 Bump to 0.6.2 with Python 3.12 fix commit ce3ac83 Author: MilesCranmer <[email protected]> Date: Sat Dec 30 02:03:45 2023 +0000 Bump to 0.6.2 with Python 3.12 fix commit 28af425 Author: Miles Cranmer <[email protected]> Date: Fri Dec 29 16:55:50 2023 +0000 Update setup actions and get working on 3.12 (#541) * Update setup actions * Fix PyJulia on Python 3.12 (#538) * Fix importlib deprecated API * Test Python 3.12 * Test Julia 1.9 * Fix semantics of exec_module vs create_module * Ensure we can find spec from JuliaModule * Bump version with python 3.12 compat * Declare compat for 3.11 and 3.12 * Only test 1.6, 1.9, 1

commit 57d6b6a Author: MilesCranmer <[email protected]> Date: Sat Dec 30 02:45:53 2023 +0000 Ensure setuptools installed in test commit 18589bb Author: MilesCranmer <[email protected]> Date: Sat Dec 30 02:25:30 2023 +0000 Bump to 0.6.2 Squashed commit of the following: commit 12cba08 Merge: 28af425 ce3ac83 Author: Miles Cranmer <[email protected]> Date: Sat Dec 30 02:23:21 2023 +0000 Merge pull request #544 from JuliaPy/update-to-0.6.2 Bump to 0.6.2 with Python 3.12 fix commit ce3ac83 Author: MilesCranmer <[email protected]> Date: Sat Dec 30 02:03:45 2023 +0000 Bump to 0.6.2 with Python 3.12 fix commit 28af425 Author: Miles Cranmer <[email protected]> Date: Fri Dec 29 16:55:50 2023 +0000 Update setup actions and get working on 3.12 (#541) * Update setup actions * Fix PyJulia on Python 3.12 (#538) * Fix importlib deprecated API * Test Python 3.12 * Test Julia 1.9 * Fix semantics of exec_module vs create_module * Ensure we can find spec from JuliaModule * Bump version with python 3.12 compat * Declare compat for 3.11 and 3.12 * Only test 1.6, 1.9, 1

MilesCranmer mentioned this pull request Oct 30, 2023

Rebuild for python312 conda-forge/pysr-feedstock#106

Closed

MilesCranmer requested review from tkf and mkitti October 30, 2023 23:48

MilesCranmer mentioned this pull request Oct 30, 2023

[BUG]: PySR currently incompatible with Python 3.12 MilesCranmer/PySR#451

Closed

MilesCranmer mentioned this pull request Nov 5, 2023

Update setup actions and get working on 3.12 #541

Merged

MilesCranmer changed the base branch from master to fix-ci-settings November 5, 2023 21:45

MilesCranmer added 5 commits November 5, 2023 21:53

Fix importlib deprecated API

ca1adbc

Test Python 3.12

930c22a

Test Julia 1.9

546d2ee

Fix semantics of exec_module vs create_module

02a626b

Ensure we can find spec from JuliaModule

6feab5a

MilesCranmer force-pushed the import-issue branch from 764aa33 to 6feab5a Compare November 5, 2023 21:53

Bump version with python 3.12 compat

06cc12e

This was referenced Dec 15, 2023

buildutils.jl broken on Python 3.12 JuliaPy/PyCall.jl#1067

Closed

Rename distutils.sysconfig to sysconfig JuliaPy/PyCall.jl#1068

Merged

stevengj closed this in JuliaPy/PyCall.jl#1068 Dec 16, 2023

MilesCranmer reopened this Dec 16, 2023

Declare compat for 3.11 and 3.12

13abc67

mkitti merged commit 1f08cc4 into JuliaPy:fix-ci-settings Dec 16, 2023
36 of 38 checks passed

MilesCranmer deleted the import-issue branch December 16, 2023 14:51

MilesCranmer restored the import-issue branch December 16, 2023 14:51

MilesCranmer mentioned this pull request Jan 16, 2024

Python 3.11 needs find_spec implemented in JuliaImporter #509

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix PyJulia on Python 3.12 #538

Fix PyJulia on Python 3.12 #538

MilesCranmer commented Oct 30, 2023 •

edited

Loading

MilesCranmer commented Oct 31, 2023 •

edited

Loading

mkitti commented Oct 31, 2023

codecov bot commented Nov 5, 2023 •

edited

Loading

MilesCranmer commented Nov 5, 2023

MilesCranmer commented Nov 13, 2023

mkitti commented Nov 13, 2023 •

edited

Loading

MilesCranmer commented Nov 17, 2023

mkitti commented Nov 17, 2023

mkitti commented Nov 18, 2023

MilesCranmer commented Dec 15, 2023

MilesCranmer commented Dec 15, 2023

MilesCranmer commented Dec 15, 2023

MilesCranmer commented Dec 16, 2023 •

edited

Loading

mkitti commented Dec 16, 2023

MilesCranmer commented Dec 16, 2023

Fix PyJulia on Python 3.12 #538

Fix PyJulia on Python 3.12 #538

Conversation

MilesCranmer commented Oct 30, 2023 • edited Loading

MilesCranmer commented Oct 31, 2023 • edited Loading

mkitti commented Oct 31, 2023

codecov bot commented Nov 5, 2023 • edited Loading

Codecov Report

MilesCranmer commented Nov 5, 2023

MilesCranmer commented Nov 13, 2023

mkitti commented Nov 13, 2023 • edited Loading

MilesCranmer commented Nov 17, 2023

mkitti commented Nov 17, 2023

mkitti commented Nov 18, 2023

MilesCranmer commented Dec 15, 2023

MilesCranmer commented Dec 15, 2023

MilesCranmer commented Dec 15, 2023

MilesCranmer commented Dec 16, 2023 • edited Loading

mkitti commented Dec 16, 2023

MilesCranmer commented Dec 16, 2023

MilesCranmer commented Oct 30, 2023 •

edited

Loading

MilesCranmer commented Oct 31, 2023 •

edited

Loading

codecov bot commented Nov 5, 2023 •

edited

Loading

mkitti commented Nov 13, 2023 •

edited

Loading

MilesCranmer commented Dec 16, 2023 •

edited

Loading