[Roadmap] Implement PEP 740

[woodruffw]

PEP 740 is in a final but not yet approved state. This issue is intended to lay out the dependencies/subcomponents of its implementing, including things that can be done in a preliminary manner.

Index side

• Support the attestations form field in the current legacy upload endpoint
  • Add support for uploading attestations in legacy API #15952
• Parse each received attestation from attestations and verify each against the uploading trusted publisher; fail the file upload if any attestations are invalid
  • This should be done in as generic a manner as possible, to ensure that future extension to email and key identities is possible.
  • Add support for uploading attestations in legacy API #15952
  • Dockerfile: put some XDG dirs under /tmp #16304
  • Store attestations for PEP740 #16302
• Persist verified attestations and expose them via the PEP 503 and Simple JSON API indices (via provenance objects, as specified in PEP 740)
  • Provide Trusted Publisher information pypi's API #14799
  • Project Details Sidebar Statistics behavior #14727
  • Reapply "Store attestations for PEP740 (#16302)" (#16545) #16546

Uploader/publish side

• Support attestation generation in gh-action-pypi-publish
  • Concretely, this means gh-action-pypi-publish should use sigstore-python to sign the attestation payload defined in PEP 740
  • The resulting bundle should be transformed into a PEP 740 attestation object
  • Expose PEP 740 attestations functionality pypa/gh-action-pypi-publish#236
  • requirements: re-compile requirements with latest twine pypa/gh-action-pypi-publish#245
• Support attestation uploading in twine: Proposal: preliminary support for PEP 740 pypa/twine#1094
  • twine/upload: attestations scaffolding pypa/twine#1095
  • upload: add attestations to PackageFile pypa/twine#1098
  • upload: prevent --attestations on non-PyPI indices pypa/twine#1099
  • upload: turn attestation error into a warning pypa/twine#1101
  • Update changelog for 5.1.0 pypa/twine#1107
• Once twine supports uploading, gh-action-pypi-publish should use that support to actually upload the attestation objects it generates above

Docs & UI

• Update all relevant docs (PyPI's docs, PyPUG)
  • PEP 740: initial attestation docs #16063
  • Initial PEP 740 documentation #16398
• Add a per-artifact view #16491
• PEP 740 blog post #17070

CC @di @webknjaz @facutuesca for visibility

[webknjaz]

I've gone ahead and filed a tracking issue in pip-tools as well: jazzband/pip-tools#2080.

[webknjaz]

• Concretely, this means gh-action-pypi-publish should use sigstore-python to sign the attestation payload defined in PEP 740

With the exciting changes suggested in pypa/gh-action-pypi-publish#230, it looks like we'll be able to leverage bits of composite actions and call the action even (if it makes sense to do so, as an alternative to using a Python lib within the container like the rest of stuff).

[webknjaz]

• Concretely, this means gh-action-pypi-publish should use sigstore-python to sign the attestation payload defined in PEP 740

Should this be done by the uploader, though? My initial feeling was that this should be done right after the job building the artifacts. Though, perhaps not by pypa/build (because of the privilege elevation concerns I often complain about). Are we going to treat the artifacts that the uploader gets as coming from trusted sources? Should we be emphasizing the security considerations and teaching the users more about the least privilege principle?

[woodruffw]

Should this be done by the uploader, though?

Hmm -- I suppose it doesn't have to be, although for targeting publish provenance (and getting provenance generation exposed to the largest number of users) I think it's the most straightforward path.

Long term, PEP 740 is designed to enable multiple attestations: the building step(s) can generate one, as can the publishing step. But I believe getting that into a workable state will require a decent bit more coordination, so I want to target an MVP of "the artifact is signed by the same identity that pushed it to the index" 😅

[webknjaz]

@woodruffw fair! I just wanted to make sure we're on the same page.

[sethmlarson]

@webknjaz Glad you're thinking about build provenance already, I'm interested in tackling build provenance for Python too 🚀

These initial attestations are similar to NPM's publish provenance so is best done by the uploader. If the attestation can show "this artifact was uploaded by this workflow which matches a Trusted Publisher configuration for the project" we can already build some interesting things like verified GitHub URLs, exposing which artifacts have Trusted Publishers configured to users, monitoring for changes in provenance, etc

[webknjaz]

@sethmlarson fair enough! I remember similar ideas being brought up during the initial private beta of trusted publishing. It's nice to have some more or less specified goals recorded and synchronized, though, which is why I posted my thoughts :)

Looking back, I'm glad that I encouraged implementing it in my action from the very beginning, months before it's gone GA so everybody could start using it right away with almost no changes. So I'm hoping that the Sigstore integration happens in a similar fashion, enabling any early adopters to start uploading signatures with zero changes to their workflows (if they already use trusted publishing, they have the OIDC access set up).

[di]

Should this be done by the uploader, though? My initial feeling was that this should be done right after the job building the artifacts.

I think your initial feeling is valid: since it's not guaranteed that the workflow using the publish action is also the workflow that built the artifacts, this isn't necessarily "build provenance".

That said, having an attestation ('publish provenance') from the uploader in the absence of any other attestation being provided to it would still be very valuable here I think, especially for projects that aren't using Trusted Publishing yet.

[facutuesca]

@woodruffw What behavior do we want when uploading attestations to a file already in PyPI? Currently, the behavior for duplicate files is either stop the upload process and return OK (if the new file is the same as the existing one), or fail (if the files don't match).
How do we want to deal with those cases when the request also includes attestations?

[woodruffw]

What behavior do we want when uploading attestations to a file already in PyPI? Currently, the behavior for duplicate files is either stop the upload process and return OK (if the new file is the same as the existing one), or fail (if the files don't match).

For the MVP, I think we want to skip if the distribution file already exists, for consistency with the existing upload API behavior.

Longer term, this probably falls under the "Upload 2.0 API" PEP -- we'll want a way to augment existing uploaded distributions with new attestations. But that's out of scope for now, IMO 🙂

[webknjaz]

FYI: there's some new developments within GitHub, it seems — pypa/gh-action-pypi-publish#236 (comment).

[webknjaz]

* [ ]  Update PyPUG and adjacent documentation to remove references to `gh-action-sigstore-python`

I think this one is somewhat separate — it signs built artifacts (in that snippet) while the pypi-publish integration signs whatever's uploaded (no matter if it was built within the workflow or downloaded).

[woodruffw]

Something that @haydentherapper raised: Trusted Publishing supports private repos.

This isn't a "disclosure" issue at the moment since we don't expose the Trusted Publisher metadata anywhere, but with PEP 740 we will.

Some points:

1. No matter what, we will continue supporting Trusted Publishing from private repos.
2. Conceptually, there's nothing stopping someone from generating a private attestation, although its semantics are unclear (since a publish or build attestation from a private repo can't be really analyzed by third parties, besides a basic "GitHub promises this is right" check.)
3. There's a (fair) PII concern about leaking a private repository name.

Some engineering details:

• Both GitLab and GitHub provide a visibility type claim, which either PyPI or the publishing workflow could filter on in terms of including attestations. It should probably be PyPI though, since PyPI is already handling the JWT claims.

[woodruffw]

Per NPM's docs, it looks like they require a public repo for provenance:

Ensure your package.json is configured with a public repository that matches (case-sensitive) where you are publishing with provenance from.

This doesn't quite mesh with PyPI's model since each distribution file has its own metadata, but in theory we could check them all for consistency. But that is potentially more brittle than just rejecting attestations if the accompanying OIDC JWT doesn't have the right visibility.

[woodruffw]

From discussion in the Sigstore Slack: one route forward here is to reject attestations from private repositories, unless the project explicitly is explicitly configured (either on PyPI or the project metadata) to allow them. This eliminates the risk of a user inadvertently leaking a private repository name (or similar) through their Trusted Publisher configuration and subsequent publish attestations.

If configured on PyPI, this could be something like a new checkbox under /manage/<project>/settings, e.g. "allow attestations from private Trusted Publishers."

If configured via project metadata, things get a little hairier (since the metadata is duplicated once per distribution, and there's no free-form metadata at the moment). So doing it via PyPI is probably easier.

[woodruffw]

@facutuesca and I had a conversation about the persistence side of this the other day; summarizing:

• For the time being, we should treat publish provenance as a baseline: in other words: PyPI can accept SLSA Provenance as well if provided, but every attestation set should come with a Publish Provenance attestation at the minimum (since they're all coming from Trusted Publishing)
• With Publish Provenance as a baseline, we can add a File <-> OIDCPublisher relationship that tracks the Trusted Publisher that was verified to have published the given File
• That relationship then allows us to bootstrap verified URLs, etc.

Some related thoughts:

• Having a File <-> OIDCPublisher relationship means that OIDCPublishers are never deleted once one or more files are published against them, even if no Project has that OIDCPublisher registered as an active publisher anymore. This may or may not be an issue; just highlighting as a "lifetime" change.
• We need to determine the appropriate "policy" for rendering things like provenance markers on the project-level page (i.e. /p/<project>): in addition to verified URLs, we might want a little checkmark similar to NPM's that says the project is provenanced. For that, it probably make senses to require all files in latest release to have publish attestations, even if the data layer underneath doesn't prohibit "heterogenous" uploads.

[facutuesca]

Having a File <-> OIDCPublisher relationship means that OIDCPublishers are never deleted once one or more files are published against them, even if no Project has that OIDCPublisher registered as an active publisher anymore

This might add some complexity, since right now there is no way of modelling an "inactive" publisher: if a publisher exists in the DB, then it's active. Do we want to add a new column to the table, an is_active field that becomes False if the user deletes the corresponding Trusted Publisher? (more specifically, it becomes False iff the user deletes it and there exists at least one file that was uploaded with a publish attestation signed by that publisher)

[woodruffw]

This might be problematic, since right now there is no way of modelling an "inactive" publisher: if a publisher exists in the DB, then it's active.

Hmm, I hadn't thought of this. A new is_active column makes sense to me, or alternatively we could define an is_active helper that checks whether len(projects) != 0. I think the latter would work approximately as well, without requiring a DB migration, although I might have missed something 🙂

[sethmlarson]

With Publish Provenance as a baseline, we can add a File <-> OIDCPublisher relationship that tracks the Trusted Publisher that was verified to have published the given File

I was imagining a separate model for a "verified attestation" that's separate from the publisher, due to the publisher potentially changing over time.

[webknjaz]

Oh, does that mean that workflow/project management tools can start implementing distribution verification against it?

[woodruffw]

Oh, does that mean that workflow/project management tools can start implementing distribution verification against it?

Unfortunately not quite -- we had to revert that last PR due to performance regressions on the simple index + some incorrect assumptions about being able to stream from PyPI's object storage 😅. We're reworking the implementation to use just the DB instead; I'll update this issue with the stack of PRs for that.

[woodruffw]

Tracking: the last two PRs for index support are now up:

• Provenance retrieval route #16778
• PEP 740: add provenance to simple API #16801

The latter is still drafted since it'll need to be rebased once the former is merged.

Once those are in, there are some follow-ons that we should pursue:

• Relax the predicate check: right now we reject anything besides PyPI "publish" attestations, but we should also accept SLSA attestations.

[woodruffw]

pypa/gh-action-pypi-publish#277 is the last moving piece on the publishing side: that'll make generating attestations the default.

[webknjaz]

We can cross that off the list now!

pypa/gh-action-pypi-publish#281

[webknjaz]

@woodruffw I filed a tracking issue for uploading attestations to GitHub: pypa/gh-action-pypi-publish#288.

[woodruffw]

The roadmap in this issue is fully complete, except for the UI elements (which are not part of PEP 740). As a result, I've opened the PEP acceptance PR: python/peps#4114