0

Got a page full of downloads but all using https://www.google.com/url?q=http://www.$$$/*.pdf&....

I can download using http://www.$$$/*.pdf direct but there are 50+ files. Anyway to avoid this? Can wget do it? I tried but it only download the links as it is under www.google.com/

Any help would be appreciated.

0

1 Answer 1

0

You could use grep -P to filter out the real links and either add as input-file (-i) to wget using process substitution:

wget -i <(grep -Po '[?&]q=\K[^&]*' links)

However, I guess that the embedded URL is URL encoded, then you need a step to unquote:

urldecode() { [ $# -eq 0 ] && str=$(</dev/stdin) || str="$@";  : "${str//+/ }"; echo -e "${_//%/\\x}"; }
wget -i <(grep -Po '[?&]q=\K[^&]*' links | urldecode)

or with python's urllib.parse.unquote:

wget -i <(python -c '
import re
from urllib.parse import unquote
with open("links") as f:
  for line in f.readlines():
    url=re.search("([&?]q=)([^&]*)", line)
    print(unquote(url.group(2)))
')

(of course you could use python to replace the wget part also ...)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .