2

tldr: up-to-date youtube-dl behaves oddly on 1 out of 4 devices, ignoring timeout between requests

I've been using youtube-dl to download videos for a while now. From time to time when something is changed server side it stops working, then a quick update fixes it. I have a desktop PC I've been using this, it stopped working yet again. The last successful video download was on 2019-10-08. I've waited for a while for an update, but none fixed this.

At the time my PC ran the previous LTS (16.04 I think), so I've upgraded it to 18.04, deleted youtube-dl package, cleaned my previous pip packages. Tried the youtube-dl from pip's repository, and directly from their site (with wget https://yt-dl.org/latest/youtube-dl -O ~/bin/ytdl2), neither works.

This PC sits behind my router with the same external IP address, along with a Raspberry Pi (1B+), and - when I'm at home - a laptop and an android phone. Both pip's youtube-dl and the one from the above URL work on all the other devices perfectly (inside Termux's python on the phone). But I'd like to run this on my PC.

The program works with different websites on my PC as well (checked bitchute), but not with youtube. Since the error code 429 suggests the server doesn't like the speed of my download, I've tried to restrict its download rate using the --socket-timeout, --sleep-interval, --limit-rate options (several seconds, and down to 50kB/s), nothing changed. I've noticed the user agent string is slightly different on every device, so I've tried to modify that as well. --geo-bypass is not a problem. I've tried about 30 different videos, tried just the sound. Even getting just the video title with --get-title fails somehow. Watching the same videos in a browser works. My IP is clearly not banned either.

This is the same even when I just want to get the title. One more thing: when running time ytdl2 --print-traffic --get-title --socket-timeout 5 --sleep-interval 5 --max-sleep-interval 10 https://www.youtube.com/watch?v=SuqRisgZfrU on my PC and my laptop, and comparing the results, I can clearly see that my laptop waits a bit longer between requests, but completes the process in about 2 seconds. But even when I use the --ignore-config, something causes different speeds, thus different results. (my PC is about 8 years older than my laptop, but my phone and the RPi are clearly slower)

So somehow even though the code is identical on all 4 devices, both computers running 18.04 Ubuntu ignore the timeouts, the config, but one of them somehow waits long enough between requests to succeed, and the other does not. It used to work.

Any suggestions what to check or how to fix?

Full output from my PC:

nyos@hex:~$ time ytdl2 --print-traffic --get-title --socket-timeout 5 --sleep-interval 5 --max-sleep-interval 10 --ignore-config https://www.youtube.com/watch?v=SuqRisgZfrU
send: u'GET /watch?has_verified=1&bpctr=9999999999&hl=en&v=SuqRisgZfrU&disable_polymer=true&gl=US HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.38 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: www.youtube.com\r\nCookie: PREF=f1=50000000&hl=en\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: no-cache
header: Strict-Transport-Security: max-age=31536000
header: Expires: Tue, 27 Apr 1971 19:44:06 EST
header: X-Frame-Options: SAMEORIGIN
header: X-Content-Type-Options: nosniff
header: Content-Encoding: gzip
header: Content-Type: text/html; charset=utf-8
header: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
header: Date: Thu, 17 Oct 2019 00:51:35 GMT
header: Server: YouTube Frontend Proxy
header: X-XSS-Protection: 0
header: Set-Cookie: VISITOR_INFO1_LIVE=sn_5wz6eikA; path=/; domain=.youtube.com; expires=Tue, 14-Apr-2020 00:51:35 GMT; httponly
header: Set-Cookie: GPS=1; path=/; domain=.youtube.com; expires=Thu, 17-Oct-2019 01:21:35 GMT
header: Set-Cookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; path=/; domain=.youtube.com
header: Set-Cookie: YSC=OGFieKP1qeE; path=/; domain=.youtube.com; httponly
header: Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
header: Connection: close
header: Transfer-Encoding: chunked
send: u'GET /get_video_info?ps=default&el=embedded&video_id=SuqRisgZfrU&hl=en&disable_polymer=true&gl=US&eurl= HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.38 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: www.youtube.com\r\nCookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; VISITOR_INFO1_LIVE=sn_5wz6eikA; YSC=OGFieKP1qeE; PREF=f1=50000000&hl=en; GPS=1\r\n\r\n'
reply: 'HTTP/1.1 429 Too Many Requests\r\n'
header: Expires: Tue, 27 Apr 1971 19:44:06 EST
header: X-Frame-Options: SAMEORIGIN
header: Cache-Control: no-cache
header: Content-Type: text/html; charset=utf-8
header: Strict-Transport-Security: max-age=31536000
header: X-Content-Type-Options: nosniff
header: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
header: Content-Length: 0
header: Date: Thu, 17 Oct 2019 00:51:35 GMT
header: Server: YouTube Frontend Proxy
header: X-XSS-Protection: 0
header: Set-Cookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; path=/; domain=.youtube.com
header: Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
header: Connection: close
WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests
send: u'GET /get_video_info?ps=default&el=detailpage&video_id=SuqRisgZfrU&hl=en&disable_polymer=true&gl=US&eurl= HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.38 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: www.youtube.com\r\nCookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; VISITOR_INFO1_LIVE=sn_5wz6eikA; YSC=OGFieKP1qeE; PREF=f1=50000000&hl=en; GPS=1\r\n\r\n'
reply: 'HTTP/1.1 429 Too Many Requests\r\n'
header: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
header: Strict-Transport-Security: max-age=31536000
header: Content-Type: text/html; charset=utf-8
header: Cache-Control: no-cache
header: Content-Length: 0
header: X-Frame-Options: SAMEORIGIN
header: X-Content-Type-Options: nosniff
header: Expires: Tue, 27 Apr 1971 19:44:06 EST
header: Date: Thu, 17 Oct 2019 00:51:35 GMT
header: Server: YouTube Frontend Proxy
header: X-XSS-Protection: 0
header: Set-Cookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; path=/; domain=.youtube.com
header: Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
header: Connection: close
WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests
send: u'GET /get_video_info?ps=default&el=vevo&video_id=SuqRisgZfrU&hl=en&disable_polymer=true&gl=US&eurl= HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.38 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: www.youtube.com\r\nCookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; VISITOR_INFO1_LIVE=sn_5wz6eikA; YSC=OGFieKP1qeE; PREF=f1=50000000&hl=en; GPS=1\r\n\r\n'
reply: 'HTTP/1.1 429 Too Many Requests\r\n'
header: Content-Length: 0
header: X-Content-Type-Options: nosniff
header: Strict-Transport-Security: max-age=31536000
header: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
header: Cache-Control: no-cache
header: Content-Type: text/html; charset=utf-8
header: Expires: Tue, 27 Apr 1971 19:44:06 EST
header: X-Frame-Options: SAMEORIGIN
header: Date: Thu, 17 Oct 2019 00:51:36 GMT
header: Server: YouTube Frontend Proxy
header: X-XSS-Protection: 0
header: Set-Cookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; path=/; domain=.youtube.com
header: Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
header: Connection: close
WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests
send: u'GET /get_video_info?ps=default&video_id=SuqRisgZfrU&hl=en&disable_polymer=true&gl=US&eurl= HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.38 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: www.youtube.com\r\nCookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; VISITOR_INFO1_LIVE=sn_5wz6eikA; YSC=OGFieKP1qeE; PREF=f1=50000000&hl=en; GPS=1\r\n\r\n'
reply: 'HTTP/1.1 429 Too Many Requests\r\n'
header: X-Frame-Options: SAMEORIGIN
header: Content-Type: text/html; charset=utf-8
header: Cache-Control: no-cache
header: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
header: Expires: Tue, 27 Apr 1971 19:44:06 EST
header: Content-Length: 0
header: Strict-Transport-Security: max-age=31536000
header: X-Content-Type-Options: nosniff
header: Date: Thu, 17 Oct 2019 00:51:36 GMT
header: Server: YouTube Frontend Proxy
header: X-XSS-Protection: 0
header: Set-Cookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; path=/; domain=.youtube.com
header: Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
header: Connection: close
WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests
ERROR: SuqRisgZfrU: YouTube said: Unable to extract video data

real    0m2,595s
user    0m1,494s
sys     0m0,054s

and my laptop:

nyos@shodan:~$ time ytdl2 --print-traffic --get-title --socket-timeout 5 --sleep-interval 5 --max-sleep-interval 10 --ignore-config https://www.youtube.com/watch?v=SuqRisgZfrU
send: u'GET /watch?has_verified=1&bpctr=9999999999&hl=en&v=SuqRisgZfrU&disable_polymer=true&gl=US HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: www.youtube.com\r\nCookie: PREF=f1=50000000&hl=en\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Encoding: gzip
header: Strict-Transport-Security: max-age=31536000
header: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: Content-Type: text/html; charset=utf-8
header: Cache-Control: no-cache
header: Expires: Tue, 27 Apr 1971 19:44:06 EST
header: Date: Thu, 17 Oct 2019 00:53:09 GMT
header: Server: YouTube Frontend Proxy
header: X-XSS-Protection: 0
header: Set-Cookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; path=/; domain=.youtube.com
header: Set-Cookie: VISITOR_INFO1_LIVE=HQNpuTQ4WDc; path=/; domain=.youtube.com; expires=Tue, 14-Apr-2020 00:53:09 GMT; httponly
header: Set-Cookie: YSC=F1swT5PPA-s; path=/; domain=.youtube.com; httponly
header: Set-Cookie: GPS=1; path=/; domain=.youtube.com; expires=Thu, 17-Oct-2019 01:23:09 GMT
header: Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
header: Connection: close
header: Transfer-Encoding: chunked
send: u'GET /get_video_info?ps=default&el=embedded&video_id=SuqRisgZfrU&hl=en&disable_polymer=true&gl=US&eurl= HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: www.youtube.com\r\nCookie: YSC=F1swT5PPA-s; VISITOR_INFO1_LIVE=HQNpuTQ4WDc; GPS=1; PREF=f1=50000000&hl=en; s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: X-Content-Type-Options: nosniff
header: Cache-Control: no-store
header: Content-Type: application/x-www-form-urlencoded
header: Content-Encoding: gzip
header: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
header: Strict-Transport-Security: max-age=31536000
header: Expires: Tue, 27 Apr 1971 19:44:06 EST
header: X-Frame-Options: SAMEORIGIN
header: Date: Thu, 17 Oct 2019 00:53:10 GMT
header: Server: YouTube Frontend Proxy
header: X-XSS-Protection: 0
header: Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
header: Connection: close
header: Transfer-Encoding: chunked
Itt a legcuccosabb lasagne a világon!

real    0m1,970s
user    0m1,028s
sys     0m0,044s
  • Have you tried the "external downloader" options to try and use something like aria2 or curl instead of the default (which I think is ffmpeg)? For example, you can use: youtube-dl --external-downloader aria2c --external-downloader-args "-x 1" 'http://www.youtube.com/watch?v=P9pzm5b6FFY' This will use only one thread to download, however you can change the value of -x to something like -x 4 to use 4 threads to download the file instead. Alternatively, you can use curl like this instead: youtube-dl --external-downloader curl 'http://www.youtube.com/watch?v=P9pzm5b6FFY' – mchid Oct 17 '19 at 1:14
  • I've tried external downloaders as well (curl to be specific), unsuccessful on my PC, but works on my laptop. – Nyos Oct 17 '19 at 1:17
  • Yeah, you might have to install the aria2 package to use aria2c but it has the option to do multithreaded downloads which may bypass the throttling that you are experiencing. I would suggest trying different numbers for the -x option to see if that helps. According to this reddit post, some people have had luck using -x 8: reddit.com/r/DataHoarder/comments/7qpuhn/… – mchid Oct 17 '19 at 1:55
  • 1
    No difference. Tried with -x 1 and -x 8, still HTTP Error 429: Too Many Requests – Nyos Oct 17 '19 at 1:58
  • Try completely disabling ipv6 and try running with the --force-ipv4 flag. Other than that, you may have some success using a proxy or a VPN for now until your IP is no longer flagged. github.com/ytdl-org/youtube-dl/issues/21729 – mchid Oct 17 '19 at 3:48
1

Had the same problem on my openwrt router, was working fine on my computer, so No IP ban problem.

curl, aria2c even netcat responded with 429.

For me the problem solved when I upgraded to kernel 4.14.150 ...

(both 4.14.148 and 4.14.149 had the same problem)

| improve this answer | |
  • I have the problem with 4.15.0-70-generic on two separate Ubutnu computers (but not on either my Mac mini or Macbook Air). I really don't want to upgrade ubuntu. I wonder if there's a Docker solution. – Sridhar Sarnobat Dec 5 '19 at 21:52
  • 1
    why don't you just update the kernel askubuntu.com/questions/860695/how-to-upgrade-just-the-kernel docker would be using exact same network stack -- so I think it won't solve the problem. – rifaqat Dec 6 '19 at 7:15
  • I guess I should. I hope the Ubuntu part won't change, other stuff will break I predict. But you're right about docker, that made no difference. Actually it still works on both my Mac computers. I wonder if a bsd vm in Ubuntu will work. – Sridhar Sarnobat Dec 13 '19 at 16:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.