Update "install_requires vs. requirements files" discussion

[jeanas]

Use pyproject.toml metadata instead of referring to setuptools' install_requires setup() keyword.

---

📚 Documentation preview 📚: https://python-packaging-user-guide--1427.org.readthedocs.build/en/1427/

[webknjaz]

Hey @jeanas, thanks for taking a stab at this!

I posted a rant comment in pip-tools the other day, regarding what I think might be a disconnect in perception of lockfiles vs. loose direct project deps. Check it out @ jazzband/pip-tools#1326 (comment).

Do you think it might make sense to incorporate some of that explanation? What's the scope of this PR? I see it goes beyond just renaming the declaration field and file names...

[jeanas]

@webknjaz I agree with what you wrote in that comment, but I'm not sure what part of it you'd like to see incorporated here.

Would you mind making a PR yourself after this one is merged? The scope of this PR is basically updating the metadata format and install_requires terminology; the rest is just a couple nitpicking changes like i.e. → i.e., that I did “in passing”.

[webknjaz]

Would you mind making a PR yourself after this one is merged? The scope of this PR is basically updating the metadata format and install_requires terminology; the rest is just a couple nitpicking changes like i.e. → i.e., that I did “in passing”.

Yeah, I'm not sure about myself, just felt related so I shared. Trying to post the info in places that seem relevant first, so it's not overlooked when somebody gets to describing this.

[webknjaz]

Also, kinda dislike replacing one implementation detail (install_requires key name in a config file) with another (dependencies key name in a config file). That's also why I shared that post.
The problem is that the semantics of these configuration options tends to be ambiguous and I feel like we need to start talking about "a package runtime dependencies in metadata" in place of "a config setting name". And in that context, it's not just install_requires but also extras that is a public API for declaring what a project needs in runtime.

[webknjaz]

Feels like this guide is written with a setuptools-derived mental model in mind and it'd be better to reorient it towards "project metadata vs environment declaration" conceptually.

[webknjaz]

@dstufft since this document what heavily inspired by your blog post originally, would you mind sharing an opinion?

[webknjaz]

Although, I missed that you already implemented some of it. @jeanas do you think we could reduce repeat use of dependencies as an implementation detail towards saying "abstract/loose runtime dependencies"?

[webknjaz]

(readability idea)

[jeanas]

@webknjaz OK, I've changed “dependencies” to “metadata dependencies”.

[sinoroc]

One nitpick, otherwise looks good to me

[jeanas]

@webknjaz Is anything still needed here? I'd like to defer the file renaming until I get around to implementing #1408. From my point of view this is ready to be merged.

[webknjaz]

I think this is confusing. Pip installs concrete deps, but it derives them from the abstract ones. The first sentence in this paragraph makes it sounds like it skips the resolution and installs the abstract deps which is not something installable.

[jeanas]

I've changed “will automatically install the metadata dependencies” to “will automatically resolve the metadata dependencies and install them”. There's a later paragraph that goes into more detail (“Lastly, it's important to understand…”).

[webknjaz]

Honestly, it seems weird to read of this as "metadata dependencies". Abstract dependencies can be in requirements files, concrete pins can be in constraint files.

[jeanas]

I'm not sure I understand your comment, would you mind expanding please?

[webknjaz]

Well, if you introduce a concept of metadata requirements (which I feel weird about by itself for some reason), and later on the document suggests that the requirements are opposite because they can contain pip options. The reality is that the requirements aren't opposite. They can have the same properties and be abstract.

I argue that more places need to be updated to reduce such confusion. What you call "metadata" deps refers to the location of specification. Just like "requirements". But that's about it. "requirements" can have the both same property of being abstract and the same semantics, except for the tool-specific additions, while the package metadata is standardized.

Perhaps I should've started this thread in a different place in the diff, though.

[pfmoore]

I agree with @webknjaz - "Metadata dependencies" aren't a thing. "Dependencies" is the correct term, and remains the correct term regardless of where they are declared.

Requirements files hold requirements - because they are pip-specific, some of the subtleties of what a "requirement" is in this context are pip-dependent, but basically you can consider them as concrete requirements used to build an environment or application. There's a subtle but important difference between "dependencies" and concrete requirements, but this text isn't making it very clear.

Constraints are a whole different thing, very much pip specific. They add limits to specifiers (concrete requirements or abstract dependencies) when resolving them, but they aren't dependencies themselves. And if that last sentence seems hard to follow, that's because we don't have well-defined terms for everything we're discussing here 🙁

[jeanas]

I agree with @webknjaz - "Metadata dependencies" aren't a thing. "Dependencies" is the correct term, and remains the correct term regardless of where they are declared.

In the abstract, I agree. But I don't want to use just “dependencies” to explain the distinction “dependencies vs. requirements files”, because you can often hear that “the dependencies to do XXX are in this requirements files”, so a more precise term is needed to help Joe User understand that this isn't the same meaning of “dependency”. “Metadata dependencies” is the best I could come up with (I also considered “package dependencies”). Do you have another suggestion?

[pfmoore]

Not really. The terms really are confused. We don't have well-defined terminology here, and getting consensus on terminology isn't something we should be doing in a discussion on a single PR.

We had this debate before, a long time ago. The term "package" is horribly overloaded, so we coined the terms "distribution package" and "install package". No-one uses them. They use "package", project", and probably other terms as well, and work out what is meant from context. It sucks, but that's human communication for you 😉

And to make things worse here, requirement files aren't standardised. So how can we have standard terminology for something that's fundamentally tool-specific? Someone using hatch or PDM to manage their "dependencies for running the script that deploys to the internal test server" may be using something that's not a requirement file, but performs the same logical function. But their tool may not (for example) support using local filenames as requirements, unlike requirement files.

The existing text seems to come from a background of requirement-files-as-lockfiles. Which is certainly one valid use case, but by no means the only one.

[webknjaz]

I'd like to hear from more PyPA folks. This patch doesn't feel entirely accurate, so I welcome more inputs.

[pfmoore]

I know this is just re-wording the original text, but frankly I don't like it.

metadata dependencies define the dependencies for a single project

Well, obviously. It's project metadata. This doesn't seem to be saying anything useful. And it's not a useful contrast with requirement files (IMO).

requirements files are often used to define the requirements for a complete Python environment

Requirement files can be used for many things. They are a mechanism (and a pip-specific one, at that) and don't have any particular "intended use". People use requirement files to define the concrete dependencies of a standalone application, pinning exact versions to offer some level of reproducibility. That's arguably "the dependencies for the (application) project", but in a very different sense than the dependencies in pyproject.toml (if the application even has a pyproject.toml).

What's really being discussed here is the "abstract requirements vs concrete requirements" distinction, which is about concepts, not about how those concepts are implemented. The implementation is (and should be!) irrelevant, as long as the concepts are used correctly. But the implementation is important, because it's tricky to map concrete/abstract requirements onto the features available - whether we restrict ourselves to solely standards-defined features, or allow for tool-specific ones (which may be pip's requirement files, or whatever features hatch/pdm/poetry provide for "adding requirements to a project").

Things are made even harder because of the mess that is projects which are in fact applications, but which are built as if they were libraries (with entry point metadata providing a command line executable). That method of application deployment mixes up abstract and concrete requirements in a way that cannot be disentangled by a general document like this is intended to be.

I don't have a good answer here. In a very real sense, this is way too complex to be appropriate for an introductory document like this is supposed to be. But conversely, new users need some guidance on how to set up their dependencies, if for no other reason than many of the examples available to them are terrible, and won't teach them good practices 🙁

[jeanas]

What's really being discussed here is the "abstract requirements vs concrete requirements" distinction, which is about concepts, not about how those concepts are implemented.

Ok, I understand better where you're coming from now.

We have packages which can define metadata dependencies which are always abstract, and we have requirements files which define dependencies which can be either abstract or concrete. You're framing the document as “what are abstract vs concrete dependencies”. On the other hand, I am framing it as “what are metadata dependencies vs requirements files” (as in the title).

Yes, abstract vs concrete dependencies is important here, and we need to explain the link, but realistically, the question Joe Random Beginner is going to ask themselves is “I saw package names in pyproject.toml and also in requirements.txt, what's the difference?” and that's what they're going to Google for. There could be a separate document to explain abstract vs concrete dependencies or a new section, but it's not what I'm trying to explain here (and it's not what the page before this PR is explaining either).

The text you quoted said “are often used”, not “are used”, so it's correct, even though it doesn't give details. Those details IMHO don't belong here.

[pfmoore]

requirements files which define dependencies which can be either abstract or concrete

Nope. I don't agree. Requirement files define requirements. They may not be "dependencies" at all, in that there may be nothing (except in a very abstract sense) that actually depends on them. In pip, a "requirement" is a very real thing. Unfortunately, it's not something that's captured in any standard, so you can't talk about (pip's) requirements in a tool-agnostic context (which this is, at least in the sense that it's not the pip documentation).

realistically, the question Joe Random Beginner is going to ask themselves is “I saw package names in pyproject.toml and also in requirements.txt, what's the difference?”

Understood. But the real answer is "well, it's complicated..."

Do we want to give an over-simplified, and often wrong, answer here, and risk reinforcing an incorrect understanding? Personally, I don't think we do, but the original authors of this page thought we did. Maybe you prefer to just paper over the cracks and do (in effect) a search and replace on the idea install_requires, replacing it with "dependencies as specified in pyproject.toml". I don't think that's a worthwhile thing to spend time on, and I don't have any insight into how to make it "correct"¹.

The text you quoted said “are often used”, not “are used”, so it's correct, even though it doesn't give details. Those details IMHO don't belong here.

I have no stomach for nitpicking over this. I don't think² it's right to tie requirement files to the idea of being "the definition of an environment" like this. Take that opinion or leave it. It's not as if your PR added that text anyway, so you can legitimately say you don't intend to change the sense of the text, just the specific fact that install_requires is out of date.

Footnotes

1. Because to my mind, it simply isn't correct no matter how you word it. ↩

2. As a pip maintainer, someone who is fundamentally responsible for defining what requirement files are for. ↩

[jeanas]

It's not as if your PR added that text anyway, so you can legitimately say you don't intend to change the sense of the text, just the specific fact that install_requires is out of date.

🤷 It was not my intent to do more than that, but reviewers seemed to be against merging without more improvements.

Honestly, I'm having trouble understanding everything you write, probably because I lack context on pip's implementation and the terms used in it. For example, I'm not sure I completely get the distinction you make between dependencies and requirements. (I was using them in the same sense.)

If your problem is with excessive framing of requirements files as “defining environments”, then I'm happy to change that. But fundamentally, yes, I'm really just trying to get rid of setuptools-specific language and trying to do whatever it takes to get this merged.

I'm not as pessimistic as you on the possibility of getting the page to a useful state, though — and, importantly, I don't think we can just delete it without adding a redirect from it to some explanation somewhere (so it might as well be here).

[jeanas]

Also, this is a discussion, not a specification — the level we're trying to take people from is “I am writing a package for PyPI, where should I write my dependencies, pyproject.toml or requirements.txt”. There's a tradeoff we have to make with “little white lies” that are needed to keep the text understandable for laymen even if it occasionally hurts the eyes of a pip maintainer.

[pfmoore]

Honestly, I'm having trouble understanding everything you write, probably because I lack context on pip's implementation and the terms used in it.

To a large extent, that's my fault - because we don't have well-defined meanings for the various terms we're using, it's easy to misunderstand each other. I didn't have time to clearly define and explain the concepts I was using so I went with what I thought was "common usage". Clearly it's not (demonstrating nicely the problem here...).

Let me try to clarify. This is simply my view on what needs to be part of everyone's "shared understanding". I don't want to get sucked into the question of "how do we teach this", as long as whatever we do present doesn't lead to a different understanding, or reinforces misconceptions that are commonly held in the community.

For me, the distinction between "concrete requirements" and "abstract requirements" is crucial, and should be a core part of any vocabulary we use when talking about dependencies/requirements/etc. The caremad article linked from this page remains a good description of the concepts - I don't know if any better description has been published since.

Building on that, the next important concept I see is the idea of what pip calls "top level requirements" - the things that the user requests an installer actually installs into an environment. These can be concrete or abstract, the key point here is that the user is explicitly asking for them. Pip's requirement files are simply a way of bundling together a set of such top-level requirements for reusability. Whether you fully pin the contents of a requirement file, or whether you view what's in such a file as concrete or abstract requirements depends entirely on what you plan on using the requirement file for. Note that other tools can have different ideas of what counts as a "top level requirement", or how to enable reusability. One such idea is that of "adding a requirement to a project", which IMO isn't very well-defined, but is nevertheless a common thing for people to want to do¹. I can't comment on how the idea of "adding a requirement to a project" fits into this, as I don't work on a tool that supports such an idea, and as a user I've never really understood what tools that do offer such a concept actually mean by it...

At the next level, we have package dependencies, which are stored in the package metadata, and typically defined in the pyproject.toml file in the project source tree. These are abstract requirements, and say what else must be present for that package to function correctly. An installer will add these requirements to the top-level requirements requested by the user, as part of the process of dependency resolution, which turns an install request into an actionable list of packages and versions to install.

And that's it, IMO. With these concepts, a user stands a reasonable chance of making good decisions on how to express their own project's requirements and dependencies, and of understanding how other projects do so.

A note on terminology: In the above, I'm using the specific term "requirement" (which isn't standardised) in the way pip uses it - to mean "something that can be installed". Typically this is what is referred to in the standards as a "dependency specifier", but it's not always a dependency, and individual tools may allow extra options (pip allows local paths, for example). Yes, this is confusing. But I didn't decide on the terminology, and it's IMO too late to expect people to change at this point. So we live with it (and from a teaching point of view, making new users aware that the world isn't entirely perfect is a good learning experience!)

I hope this is a little clearer. As I said, it's just my view on how these ideas hang together. You're free to do whatever you want with it - I don't want to get sucked into editorial discussions on "how do we express this in the packaging guide" because I'll only end up burning out if I do.

Footnotes

1. Step one should be "clarify the purpose of the requirement" 🙂 ↩

[pfmoore]

I'd like to hear from more PyPA folks. This patch doesn't feel entirely accurate, so I welcome more inputs.

Well, I added a review comment which sort of turned into a comment on the whole page (to an extent, both before and after the PR). I agree that this patch doesn't feel accurate, although I can't be sure how much of that is because the original material is inaccurate and the changes simply fail to improve it.

While I don't want the perfect to be the enemy of the good here, I sort of feel that a better resolution would just be to delete this page altogether. Having just re-read the current version, I'm left with the feeling that I hope new users never actually find it - and that's more because of the advice it gives than because it refers to install_requires rather than pyproject.toml...

[pfmoore]

I should probably drop out of this conversation now. It's bringing back all of my reservations and frustrations with the PUG that I expressed when these pages were initially written, and which led to me burning out and deciding to leave the decisions to the people who were writing the text, even if that meant I didn't consider the result to be something I would recommend. I think I'm going the same way again, so for everyone's sake, I'll quit sooner rather than later this time.