2

Glutanimate posted a script that extracts all PDF links from a single page.

Any idea on how to change this so the script can grab all the PDF links from a list of URLs ?

3 Answers 3

3

I would create a separate (text) file with all url's listed in single lines:

www.url1
www.url2

Then change the line in the script to append the found pdf links to the produced pdflinks.txt (tee -a pdflinks.txt | more instead of tee pdflinks.txt):

lynx -cache=0 -dump -listonly "$WEBSITE" | grep ".*\.pdf$" | awk '{print $2}' | tee -a pdflinks.txt | more

Then make the script executable and run it by another (python in this case) script:

#!/usr/bin/python3
import subprocess

url_list = "/path/to/url_list.txt"
script = "/path/to/script.sh"

with open(url_list) as sourcefile:
    sourcefile = sourcefile.readlines()
for item in sourcefile:
    subprocess.call([script, item.replace("\n", "")])

Paste the text above in an empty document, add appropriate paths and save it as run_pdflinkextractor.py and run it by the command

python3 /path/to/run_pdflinkextractor.py

More options

You didn't actually ask for it, but in case you would like to download the found pdf links, it is a pity to stop half way. The script below might be handy to do that. The procedure is the same: paste the text below in an empty file, save it as download_pdffiles.py, add the path to the pdflinks.txt that was created in the first step, the path to the folder you would like to download the files to, and run it by the command:

 python3 /path/to/download_pdffiles.py

The script to actually download the files:

#!/usr/bin/python3

import subprocess

pdf_list = "/path/to/pdflinks.txt"
download_directory = "/path/to/downloadfolder"

with open(pdf_list) as sourcefile:
    sourcefile = sourcefile.readlines()
for item in sourcefile:
    subprocess.call(["wget", "-P", download_directory, item.replace("\n", "")])

You can of course add more options to the script, what to do in case of errors for example (errors are ignored in the script as it is). See man wget for more options.

1

Answer depends on what you define as "list of URL's".

If you want to do it as multi-arg command line script, use code like this:

#!/bin/bash
for WEBSITE in "$*"
do
    <scriptname> "$WEBSITE"
done

There is also a way to read a list of URL's from file, line by line:

#!/bin/bash
_file="$1"
while IFS= read -r line
do
    <scriptname> "$line"
done < "$_file"

Improve my answer as you wish. I'm not a bash master :)

1

I don't like hardcoding file names into scripts. I prefer giving them as arguments. This can be done via a very minor modification of Glutanimate's script:

#!/usr/bin/env bash

# NAME:         pdflinkextractor
# AUTHOR:       Glutanimate (http://askubuntu.com/users/81372/), 2013
# LICENSE:      GNU GPL v2
# DEPENDENCIES: wget lynx
# DESCRIPTION:  extracts PDF links from websites and dumps them to the stdout and as a textfile
#               only works for links pointing to files with the ".pdf" extension
#
# USAGE:        pdflinkextractor "www.website.com" > output_file


echo "Getting link list..."

## Go through each URL given and find the PDFs it links to
for website in "$@"; do
    lynx -cache=0 -dump -listonly "$website" | awk '/.pdf$/{print $2}'
done

You can save that as downloadpdfs, make it executable (chmod +x downloadpdfs) and then run it, giving it multiple addresses as arguments:

downloadpdfs "http://example.com" "http://example2.com" "http://example3.com" > pdflinks.txt

The above will create a file called pdflinks.txt containing all the links extracted from each of the input URLs.

2
  • While it's probably not as important in this case, wouldn't $@ be safer than $*? (Or maybe I understood their usage wrong. Still a newbie with scripting.) May 20, 2014 at 10:18
  • 1
    @Glutanimate yes indeed it would. It won't make any practical difference at all in this case but using "$@" is almost always better than using "$*". Thanks for catching it and thanks for the edit :).
    – terdon
    May 20, 2014 at 11:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .