HTTP Error 429: Too Many Requests - youtube-dl stopped working

Question

tldr: up-to-date youtube-dl behaves oddly on 1 out of 4 devices, ignoring timeout between requests

I've been using youtube-dl to download videos for a while now. From time to time when something is changed server side it stops working, then a quick update fixes it. I have a desktop PC I've been using this, it stopped working yet again. The last successful video download was on 2019-10-08. I've waited for a while for an update, but none fixed this.

At the time my PC ran the previous LTS (16.04 I think), so I've upgraded it to 18.04, deleted youtube-dl package, cleaned my previous pip packages. Tried the youtube-dl from pip's repository, and directly from their site (with wget https://yt-dl.org/latest/youtube-dl -O ~/bin/ytdl2), neither works.

This PC sits behind my router with the same external IP address, along with a Raspberry Pi (1B+), and - when I'm at home - a laptop and an android phone. Both pip's youtube-dl and the one from the above URL work on all the other devices perfectly (inside Termux's python on the phone). But I'd like to run this on my PC.

The program works with different websites on my PC as well (checked bitchute), but not with youtube. Since the error code 429 suggests the server doesn't like the speed of my download, I've tried to restrict its download rate using the --socket-timeout, --sleep-interval, --limit-rate options (several seconds, and down to 50kB/s), nothing changed. I've noticed the user agent string is slightly different on every device, so I've tried to modify that as well. --geo-bypass is not a problem. I've tried about 30 different videos, tried just the sound. Even getting just the video title with --get-title fails somehow. Watching the same videos in a browser works. My IP is clearly not banned either.

This is the same even when I just want to get the title. One more thing: when running time ytdl2 --print-traffic --get-title --socket-timeout 5 --sleep-interval 5 --max-sleep-interval 10 https://www.youtube.com/watch?v=SuqRisgZfrU on my PC and my laptop, and comparing the results, I can clearly see that my laptop waits a bit longer between requests, but completes the process in about 2 seconds. But even when I use the --ignore-config, something causes different speeds, thus different results. (my PC is about 8 years older than my laptop, but my phone and the RPi are clearly slower)

So somehow even though the code is identical on all 4 devices, both computers running 18.04 Ubuntu ignore the timeouts, the config, but one of them somehow waits long enough between requests to succeed, and the other does not. It used to work.

Any suggestions what to check or how to fix?

Full output from my PC:

nyos@hex:~$ time ytdl2 --print-traffic --get-title --socket-timeout 5 --sleep-interval 5 --max-sleep-interval 10 --ignore-config https://www.youtube.com/watch?v=SuqRisgZfrU
send: u'GET /watch?has_verified=1&bpctr=9999999999&hl=en&v=SuqRisgZfrU&disable_polymer=true&gl=US HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.38 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: www.youtube.com\r\nCookie: PREF=f1=50000000&hl=en\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: no-cache
header: Strict-Transport-Security: max-age=31536000
header: Expires: Tue, 27 Apr 1971 19:44:06 EST
header: X-Frame-Options: SAMEORIGIN
header: X-Content-Type-Options: nosniff
header: Content-Encoding: gzip
header: Content-Type: text/html; charset=utf-8
header: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
header: Date: Thu, 17 Oct 2019 00:51:35 GMT
header: Server: YouTube Frontend Proxy
header: X-XSS-Protection: 0
header: Set-Cookie: VISITOR_INFO1_LIVE=sn_5wz6eikA; path=/; domain=.youtube.com; expires=Tue, 14-Apr-2020 00:51:35 GMT; httponly
header: Set-Cookie: GPS=1; path=/; domain=.youtube.com; expires=Thu, 17-Oct-2019 01:21:35 GMT
header: Set-Cookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; path=/; domain=.youtube.com
header: Set-Cookie: YSC=OGFieKP1qeE; path=/; domain=.youtube.com; httponly
header: Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
header: Connection: close
header: Transfer-Encoding: chunked
send: u'GET /get_video_info?ps=default&el=embedded&video_id=SuqRisgZfrU&hl=en&disable_polymer=true&gl=US&eurl= HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.38 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: www.youtube.com\r\nCookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; VISITOR_INFO1_LIVE=sn_5wz6eikA; YSC=OGFieKP1qeE; PREF=f1=50000000&hl=en; GPS=1\r\n\r\n'
reply: 'HTTP/1.1 429 Too Many Requests\r\n'
header: Expires: Tue, 27 Apr 1971 19:44:06 EST
header: X-Frame-Options: SAMEORIGIN
header: Cache-Control: no-cache
header: Content-Type: text/html; charset=utf-8
header: Strict-Transport-Security: max-age=31536000
header: X-Content-Type-Options: nosniff
header: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
header: Content-Length: 0
header: Date: Thu, 17 Oct 2019 00:51:35 GMT
header: Server: YouTube Frontend Proxy
header: X-XSS-Protection: 0
header: Set-Cookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; path=/; domain=.youtube.com
header: Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
header: Connection: close
WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests
send: u'GET /get_video_info?ps=default&el=detailpage&video_id=SuqRisgZfrU&hl=en&disable_polymer=true&gl=US&eurl= HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.38 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: www.youtube.com\r\nCookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; VISITOR_INFO1_LIVE=sn_5wz6eikA; YSC=OGFieKP1qeE; PREF=f1=50000000&hl=en; GPS=1\r\n\r\n'
reply: 'HTTP/1.1 429 Too Many Requests\r\n'
header: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
header: Strict-Transport-Security: max-age=31536000
header: Content-Type: text/html; charset=utf-8
header: Cache-Control: no-cache
header: Content-Length: 0
header: X-Frame-Options: SAMEORIGIN
header: X-Content-Type-Options: nosniff
header: Expires: Tue, 27 Apr 1971 19:44:06 EST
header: Date: Thu, 17 Oct 2019 00:51:35 GMT
header: Server: YouTube Frontend Proxy
header: X-XSS-Protection: 0
header: Set-Cookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; path=/; domain=.youtube.com
header: Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
header: Connection: close
WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests
send: u'GET /get_video_info?ps=default&el=vevo&video_id=SuqRisgZfrU&hl=en&disable_polymer=true&gl=US&eurl= HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.38 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: www.youtube.com\r\nCookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; VISITOR_INFO1_LIVE=sn_5wz6eikA; YSC=OGFieKP1qeE; PREF=f1=50000000&hl=en; GPS=1\r\n\r\n'
reply: 'HTTP/1.1 429 Too Many Requests\r\n'
header: Content-Length: 0
header: X-Content-Type-Options: nosniff
header: Strict-Transport-Security: max-age=31536000
header: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
header: Cache-Control: no-cache
header: Content-Type: text/html; charset=utf-8
header: Expires: Tue, 27 Apr 1971 19:44:06 EST
header: X-Frame-Options: SAMEORIGIN
header: Date: Thu, 17 Oct 2019 00:51:36 GMT
header: Server: YouTube Frontend Proxy
header: X-XSS-Protection: 0
header: Set-Cookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; path=/; domain=.youtube.com
header: Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
header: Connection: close
WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests
send: u'GET /get_video_info?ps=default&video_id=SuqRisgZfrU&hl=en&disable_polymer=true&gl=US&eurl= HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.38 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: www.youtube.com\r\nCookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; VISITOR_INFO1_LIVE=sn_5wz6eikA; YSC=OGFieKP1qeE; PREF=f1=50000000&hl=en; GPS=1\r\n\r\n'
reply: 'HTTP/1.1 429 Too Many Requests\r\n'
header: X-Frame-Options: SAMEORIGIN
header: Content-Type: text/html; charset=utf-8
header: Cache-Control: no-cache
header: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
header: Expires: Tue, 27 Apr 1971 19:44:06 EST
header: Content-Length: 0
header: Strict-Transport-Security: max-age=31536000
header: X-Content-Type-Options: nosniff
header: Date: Thu, 17 Oct 2019 00:51:36 GMT
header: Server: YouTube Frontend Proxy
header: X-XSS-Protection: 0
header: Set-Cookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; path=/; domain=.youtube.com
header: Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
header: Connection: close
WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests
ERROR: SuqRisgZfrU: YouTube said: Unable to extract video data

real    0m2,595s
user    0m1,494s
sys     0m0,054s

and my laptop:

nyos@shodan:~$ time ytdl2 --print-traffic --get-title --socket-timeout 5 --sleep-interval 5 --max-sleep-interval 10 --ignore-config https://www.youtube.com/watch?v=SuqRisgZfrU
send: u'GET /watch?has_verified=1&bpctr=9999999999&hl=en&v=SuqRisgZfrU&disable_polymer=true&gl=US HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: www.youtube.com\r\nCookie: PREF=f1=50000000&hl=en\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Encoding: gzip
header: Strict-Transport-Security: max-age=31536000
header: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: Content-Type: text/html; charset=utf-8
header: Cache-Control: no-cache
header: Expires: Tue, 27 Apr 1971 19:44:06 EST
header: Date: Thu, 17 Oct 2019 00:53:09 GMT
header: Server: YouTube Frontend Proxy
header: X-XSS-Protection: 0
header: Set-Cookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; path=/; domain=.youtube.com
header: Set-Cookie: VISITOR_INFO1_LIVE=HQNpuTQ4WDc; path=/; domain=.youtube.com; expires=Tue, 14-Apr-2020 00:53:09 GMT; httponly
header: Set-Cookie: YSC=F1swT5PPA-s; path=/; domain=.youtube.com; httponly
header: Set-Cookie: GPS=1; path=/; domain=.youtube.com; expires=Thu, 17-Oct-2019 01:23:09 GMT
header: Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
header: Connection: close
header: Transfer-Encoding: chunked
send: u'GET /get_video_info?ps=default&el=embedded&video_id=SuqRisgZfrU&hl=en&disable_polymer=true&gl=US&eurl= HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: www.youtube.com\r\nCookie: YSC=F1swT5PPA-s; VISITOR_INFO1_LIVE=HQNpuTQ4WDc; GPS=1; PREF=f1=50000000&hl=en; s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: X-Content-Type-Options: nosniff
header: Cache-Control: no-store
header: Content-Type: application/x-www-form-urlencoded
header: Content-Encoding: gzip
header: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
header: Strict-Transport-Security: max-age=31536000
header: Expires: Tue, 27 Apr 1971 19:44:06 EST
header: X-Frame-Options: SAMEORIGIN
header: Date: Thu, 17 Oct 2019 00:53:10 GMT
header: Server: YouTube Frontend Proxy
header: X-XSS-Protection: 0
header: Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
header: Connection: close
header: Transfer-Encoding: chunked
Itt a legcuccosabb lasagne a világon!

real    0m1,970s
user    0m1,028s
sys     0m0,044s

Answer 1

Had the same problem on my openwrt router, was working fine on my computer, so No IP ban problem.

curl, aria2c even netcat responded with 429.

For me the problem solved when I upgraded to kernel 4.14.150 ...

(both 4.14.148 and 4.14.149 had the same problem)

Answer 2

The site and/or proxy do not like your "User-Agent" header, I just told wget not to send it and it worked fine.

From the man page: Specifying empty user agent with --user-agent="" instructs Wget not to send the "User-Agent" header in HTTP requests.