[Vimeo] HTTP Error 406: Not Acceptable when downloading video

[salty-horse]

Checklist

• I'm reporting a broken site support
• I've verified that I'm running youtube-dl version 2021.12.17 (Also tried latest git)
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar issues including closed ones

Verbose log

$ youtube-dl -v -x 'https://vimeo.com/9010456'
[debug] System config: []
[debug] User config: ['--socket-timeout=2']
[debug] Custom config: []
[debug] Command-line args: ['-v', '-x', 'https://vimeo.com/9010456']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: a70dc03d2
[debug] Python 3.12.7 (CPython x86_64 64bit) - Linux-6.11.0-9-generic-x86_64-with-glibc2.40 - OpenSSL 3.3.1 4 Jun 2024 - glibc 2.40
[debug] exe versions: ffmpeg 7.0.2, ffprobe 7.0.2, rtmpdump 2.4
[debug] Proxy map: {}
[vimeo] 9010456: Downloading webpage
[vimeo] 9010456: Downloading JSON metadata
[vimeo] 9010456: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 406: Not Acceptable (caused by <HTTPError 406: 'Not Acceptable'>); please report this issue on https://github.com/ytdl-org/youtube-dl/issues , using the appropriate issue template. Make sure you are using the latest version; see  https://github.com/ytdl-org/youtube-dl/#user-content-installation  on how to update. Be sure to call youtube-dl with the --verbose option and include the complete output.
  File "/home/ori/devel/youtube-dl/youtube_dl/extractor/common.py", line 679, in _request_webpage
    return self._downloader.urlopen(url_or_request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ori/devel/youtube-dl/youtube_dl/YoutubeDL.py", line 2496, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/urllib/request.py", line 521, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/urllib/request.py", line 630, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/urllib/request.py", line 559, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/lib/python3.12/urllib/request.py", line 639, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

Description

I think Vimeo added some protections, and youtube-dl needs to adapt. For a while it returned 403 Forbidden and 429 Too Many Requests, and now it's returning 406.

I don't think it's related to #26421 from 2020, as, IIRC, I managed to download from Vimeo between then and now.

yt-dlp successfully downloaded the video, and they have made changes to the Vimeo extractor in the past year, particularly adding browser impersonation. (See yt-dlp/yt-dlp#10325)

[dirkf]

From recollection of yt-dlp issues, 406 might just mean that we're not asking for Content-Type: application/json, but it's going to be tricky to handle anything that needs impersonation:

1. a function like --external-requester ... would have to be implemented
2. even if that were done, the required dependencies for curl-ffi mean that users could use yt-dlp instead with a better UI.

Any further data points regarding Vimeo are welcome as I won't have popped the stack down to my Vimeo WIP for a while.

[salty-horse]

That's very reasonable.

I didn't know what to do about this issue for a while now. If you decide not to support it, would you consider improving the error message, suggesting other tools? It would really help lost users.

BTW, I have this PR #26636 about a improving different Vimeo error message that's been patiently waiting for attention :)

[dirkf]

Anyhow, yt-dlp/yt-dlp#9125 was the fix there.

[dirkf]

And then:

$ python -m youtube_dl -vF 'https://vimeo.com/9010456'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-vF', u'https://vimeo.com/9010456']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: f7dc45d3a
[debug] Python 2.7.15 (CPython i686 32bit) - Linux-6.1.0-25-686-pae-i686-with-debian-12.7 - OpenSSL 1.1.1a  20 Nov 2018 - glibc 2.1.3
[debug] exe versions: ffmpeg 5.1.6-0, ffprobe 5.1.6-0
[debug] Proxy map: {}
[vimeo] 9010456: Downloading webpage
[vimeo] 9010456: Downloading JSON metadata
[vimeo] 9010456: Downloading JSON metadata
[vimeo] 9010456: Downloading JSON metadata
[vimeo] 9010456: Downloading akfire_interconnect_quic m3u8 information
[vimeo] 9010456: Downloading fastly_skyfire m3u8 information
[vimeo] 9010456: Downloading akfire_interconnect_quic MPD information
WARNING: [vimeo] 9010456: Failed to parse XML not well-formed (invalid token): line 1, column 0
[vimeo] 9010456: Downloading fastly_skyfire MPD information
WARNING: [vimeo] 9010456: Failed to parse XML not well-formed (invalid token): line 1, column 0
[info] Available formats for 9010456:
format code                                       extension  resolution note
hls-akfire_interconnect_quic-audio-high-Original  mp4        audio only 
hls-fastly_skyfire-audio-high-Original            mp4        audio only 
hls-akfire_interconnect_quic-318                  mp4        360x272     318k , avc1.64001F, 30.0fps, video only
hls-fastly_skyfire-318                            mp4        360x272     318k , avc1.64001F, 30.0fps, video only
http-240p                                         mp4        360x272    30fps (best)
$

[salty-horse]

I fixed it myself just as you posted this :) salty-horse@9a45840