Glutanimate posted a script that extracts all PDF links from a single page.
Any idea on how to change this so the script can grab all the PDF links from a list of URLs ?
Glutanimate posted a script that extracts all PDF links from a single page.
Any idea on how to change this so the script can grab all the PDF links from a list of URLs ?
I would create a separate (text) file with all url's listed in single lines:
www.url1
www.url2
Then change the line in the script to append the found pdf links to the produced pdflinks.txt
(tee -a pdflinks.txt | more
instead of tee pdflinks.txt
):
lynx -cache=0 -dump -listonly "$WEBSITE" | grep ".*\.pdf$" | awk '{print $2}' | tee -a pdflinks.txt | more
Then make the script executable and run it by another (python in this case) script:
#!/usr/bin/python3
import subprocess
url_list = "/path/to/url_list.txt"
script = "/path/to/script.sh"
with open(url_list) as sourcefile:
sourcefile = sourcefile.readlines()
for item in sourcefile:
subprocess.call([script, item.replace("\n", "")])
Paste the text above in an empty document, add appropriate paths and save it as run_pdflinkextractor.py and run it by the command
python3 /path/to/run_pdflinkextractor.py
More options
You didn't actually ask for it, but in case you would like to download the found pdf links, it is a pity to stop half way. The script below might be handy to do that. The procedure is the same: paste the text below in an empty file, save it as download_pdffiles.py
, add the path to the pdflinks.txt
that was created in the first step, the path to the folder you would like to download the files to, and run it by the command:
python3 /path/to/download_pdffiles.py
The script to actually download the files:
#!/usr/bin/python3
import subprocess
pdf_list = "/path/to/pdflinks.txt"
download_directory = "/path/to/downloadfolder"
with open(pdf_list) as sourcefile:
sourcefile = sourcefile.readlines()
for item in sourcefile:
subprocess.call(["wget", "-P", download_directory, item.replace("\n", "")])
You can of course add more options to the script, what to do in case of errors for example (errors are ignored in the script as it is). See man wget
for more options.
Answer depends on what you define as "list of URL's".
If you want to do it as multi-arg command line script, use code like this:
#!/bin/bash
for WEBSITE in "$*"
do
<scriptname> "$WEBSITE"
done
There is also a way to read a list of URL's from file, line by line:
#!/bin/bash
_file="$1"
while IFS= read -r line
do
<scriptname> "$line"
done < "$_file"
Improve my answer as you wish. I'm not a bash master :)
I don't like hardcoding file names into scripts. I prefer giving them as arguments. This can be done via a very minor modification of Glutanimate's script:
#!/usr/bin/env bash
# NAME: pdflinkextractor
# AUTHOR: Glutanimate (http://askubuntu.com/users/81372/), 2013
# LICENSE: GNU GPL v2
# DEPENDENCIES: wget lynx
# DESCRIPTION: extracts PDF links from websites and dumps them to the stdout and as a textfile
# only works for links pointing to files with the ".pdf" extension
#
# USAGE: pdflinkextractor "www.website.com" > output_file
echo "Getting link list..."
## Go through each URL given and find the PDFs it links to
for website in "$@"; do
lynx -cache=0 -dump -listonly "$website" | awk '/.pdf$/{print $2}'
done
You can save that as downloadpdfs
, make it executable (chmod +x downloadpdfs
) and then run it, giving it multiple addresses as arguments:
downloadpdfs "http://example.com" "http://example2.com" "http://example3.com" > pdflinks.txt
The above will create a file called pdflinks.txt
containing all the links extracted from each of the input URLs.
$@
be safer than $*
? (Or maybe I understood their usage wrong. Still a newbie with scripting.)
May 20, 2014 at 10:18
"$@"
is almost always better than using "$*"
. Thanks for catching it and thanks for the edit :).