Add scripting and documentation for pushing releases to PyPI. #519
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently downloads some wheels given a specified version. Has code to edit release versions and upload, taken from other similar scripts, but not tested e2e.
marbre
reviewed
Nov 15, 2024
Co-authored-by: Marius Brehler <[email protected]>
marbre
approved these changes
Nov 15, 2024
marbre
added a commit
that referenced
this pull request
Nov 18, 2024
Refactors the scripts so that a version suffix can be passed as an argument. This is used to generate the suffix only once in the workflow instead of computing it in multiple scripts and partly scattered across different steps (`write_requirements.py` runs in a different step). Further adds dependencies to IREE packages also for nightly shark-ai builds. Requires #519.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Progress on #400.
This scripting allows us to publish .whl files from https://github.com/nod-ai/SHARK-Platform/releases/tag/dev-wheels to PyPI. Here are the basic steps:
2.9.1rc20241114)rcYYYYMMDDsuffixshark-aimeta package using the versions in those whls (NOTE: currently this uses the versions in the source tree, see below)Logs of this running: https://gist.github.com/ScottTodd/9c7418d5bbbebc8aea72a39bc2ac37b0 (to push 2.9.1 to PyPI)
The new
pypi_deploy.shscript is based on the similar script we maintain in IREE: https://github.com/iree-org/iree/blob/main/build_tools/python_deploy/pypi_deploy.sh . TheREADME.mdfile is also forked from IREE.Known sharp edges with the publishing process:
shark-ai/build_tools/wheelhouse/to the working directory that gets sent totwine upload *. If there are preexisting files in that directory they will be published.pip downloadso it can leverage PyPI's version resolution logic, but I couldn't figure out a way to download 3.13t wheels without running a 3.13t python interpreter. I happen to have 3.13t installed on my system, but we shouldn't require that in the scripting for all release engineers. We could try using theghtool as in IREE, if we properly filter the download to just the version we want: https://github.com/iree-org/iree/blob/9eaa4ef7d6b439d8c444b533beddd82146578e25/build_tools/python_deploy/pypi_deploy.sh#L69-L72 (that has one release per nightly, while here we have a single release shared between all nightlies)