Twine upload fails with large file, SSLEOFError, 'EOF occurred in violation of protocol' and ConnectionError: RemoteDisconnected

[akkana]

Problem description

I'm trying to make a test release of my PyTopo project (current git:
https://github.com/akkana/pytopo/tree/05262ed7484bd37efb7aaa4e76c401f523bd1814) but for months, I've been unable to. I try to upload with:
python -m twine upload --repository testpypi dist/*
and first it fails and retries:

WARNING  Retrying (Retry(total=9, connect=5, read=None, redirect=None,          
         status=None)) after connection broken by 'SSLEOFError(8, 'EOF occurred 
         in violation of protocol (_ssl.c:2406)')': /legacy/            

and then it goes to 100% but then sits there for 5-10 minutes with no output before finally failing with
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
(I'll attach the full error log).

I worked with SnoopJ on #python for a long time today, trying a Python-3.12.6 source build to make sure it wasn't a problem with Debian's 3.12.6, and a Python-3.11.2 build because that was the last build before a patch related to OP_IGNORE_UNEXPECTED_EOF went in, but always got the same result.

I wondered if it had to do with the size of the upload. I had one large data file (test/files/Surface_Ownership_10_14_2015.geojson, 36M) in the tarball, so I tried removing it (it's only used for unit tests) and the error went away.

I've had that file in the source tarball for several years, and it used to work fine (I've released several versions of the package during that time), so something has apparently changed that has made testpypi and/or twine less tolerant of large uploads, and maybe of OpenSSL 3.x protocol errors. And the error message gives no hint of what the real problem is.

This may be related to the following other issues:
pypa/twine#273
#696
python/cpython#87960

Full error log here;
twineerrs.txt

[SnoopJ]

a patch related to OP_IGNORE_UNEXPECTED_EOF went in

To expand on the OpenSSL parts of this a bit:

OpenSSL changed how they respond to an unexpected EOF between 1.1.1 and 3.x (the observed error is from OpenSSL 3.3.2 3 Sep 2024 specifically), the behavior used to be non-fatal and now it is fatal. See this discussion on the OpenSSL repo: openssl/openssl#22690 (comment)

I suspect this means that Test PyPI is running a TLS implementation that has some gear-grinding with OpenSSL 3.x. That is, it sometimes sends EOF without first sending close_notify. I'm not sure how the large file interacts with that.

I suggested that @akkana try a build of 3.11.2 because of an upstream change that was present in 3.11.2 but removed in 3.11.3 that enabled the SSL_OP_IGNORE_UNEXPECTED_EOF option, which the OpenSSL discussion seems to suggest would work around the new behavior. Debian sid (the distro at play here) switched from Python 3.11.2 to Python 3.11.4 on 25 June 2023, and she said on IRC that this lines up with the timeline when she started seeing the problem.

We confirmed that her build of 3.11.2 did set this flag in the default SSLContext, but it didn't seem to have any effect on the behavior of twine. Possibly some interaction between twine, requests, urllib3 means that the SSL context does not include this option even when the underlying Python's default context has it.

I think at this point someone from PyPA who knows their TLS better than we do should have a look.