Sounds very reasonable to me. When a user expects all links to a domain (or path) to be ignored, linkcheck should also ignore the redirections pointing to that domain.

I tried to understand why Wiley URLs have this problem but was not successful, so I asked for help upstream: psf/requests#6471

Meanwhile having the option to ignore specific redirect sites of DOI links would a good idea.

python -c "import requests; print(requests.head('https://doi.org/10.1002/jccs.200600142', allow_redirects=True)); import sphinx.util.requests; print(sphinx.util.requests.head('https://doi.org/10.1002/jccs.200600142', allow_redirects=True))"
<Response [403]>
<Response [403]>

import requests
with requests.Session() as s:
    print(s.get('https://doi.org/10.1002/jccs.200600142', allow_redirects=True, headers={'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/114.0'}))

Unsure what problem Wiley URLs have? They are always replying with a 403. The issue here is that developers can instruct linkcheck to ignore URLs matching a regexp pattern, but that URL is only ignored if it appears in the documentation, not if it comes as the result of a redirect.

So:

# conf.py
linkcheck_ignore = [
    "https://onlinelibrary.wiley.com",  # 403 Client Error: Forbidden for url
]

.. doc.rst
.. this link is ignored by linkcheck, it matches the pattern from linkcheck_ignore
`direct link <https://onlinelibrary.wiley.com/doi/10.1002/jemt.20597>`_
.. doi.org redirects to Wiley, but linkcheck does not check the linkcheck_ignore during the redirection chain, so the following shows as a broken link for linkcheck
`indirect link <https://doi.org/10.1002/jemt.20597>`_

I'm interested in implementing this feature, but there's a detail about the existing implementation that I think is important to consider first:

The Sphinx linkcheck builder currently uses the requests HTTP client to handle following of redirects, and so the logic to determine whether one-or-more-redirects will be followed is encapsulated by that library, and a naive application of linkcheck_ignore rules would only apply after a final response URL is resolved (or a timeout occurs, or an error code is returned, etc).

My intuition for a feature like this is that we'd ideally want to ignore redirections as soon as they suggest navigating through an ignored path -- that is, we'd follow the initial hyperlink, and if it tells us to go to a known-ignorable URL, we'd stop immediately and return the ignore status code for that link. I believe that'd be preferable because it'd imply that we'd generate less network traffic, we'd spend less time linkchecking, and we wouldn't attempt to initiate traffic to domains we've configured as ignored.

To do so, however, we'd probably want to adjust Sphinx's linkchecker so that requests doesn't follow redirects on its behalf. That's OK... but it potentially has other consequences; we'd want to enforce a maximum-redirect limit (similar to the long-standing requests.sessions.DEFAULT_REDIRECT_LIMIT), for example. And requests itself will have had a wealth of experience dealing with unusual behaviours and bugreports related to redirects. So... as a maintainer I'm initially a bit cautious (an understatement) about this.

We could perhaps apply the naive solution, and ignore URLs after requests has followed all redirects. Perhaps that's OK - I just don't love the idea that some traffic may in fact navigate through ignored URLs. Hiding those by marking them as ignored -- even though they weren't -- feels potentially worse than doing nothing.