You could use grep -P
to filter out the real links and either add as input-file (-i
) to wget
using process substitution:
wget -i <(grep -Po '[?&]q=\K[^&]*' links)
However, I guess that the embedded URL is URL encoded, then you need a step to unquote:
urldecode() { [ $# -eq 0 ] && str=$(</dev/stdin) || str="$@"; : "${str//+/ }"; echo -e "${_//%/\\x}"; }
wget -i <(grep -Po '[?&]q=\K[^&]*' links | urldecode)
or with python
's urllib.parse.unquote
:
wget -i <(python -c '
import re
from urllib.parse import unquote
with open("links") as f:
for line in f.readlines():
url=re.search("([&?]q=)([^&]*)", line)
print(unquote(url.group(2)))
')
(of course you could use python
to replace the wget
part also ...)