so I have HTML files which have a specific portion I'd like to extract. These HTML addresses are in a text file. A sample HTML webpage, taken from this textfile would look like this, and I'd want to get part 009514HB.JPG which is different for every HTML file.

My .txt file would be something like this -


and if I click open one of those html pages using a text editor I can find the information I need..

   *** some code here***
         <a href="/imagelibrary/large/009514HB.JPG" target="_blank" class="preview-image"><img src="/imagelibrary/medium/009514HB.JPG" alt="acne keloidalis nuchae"/></a>

Now I would like to get these numbers from various HTML files, and then append these numbers to https://www.dermquest.com/imagelibrary/large. For example, I would want my final txt file to have urls which are like this, [slightly NSFW] https://www.dermquest.com/imagelibrary/large/009514HB.JPG This line would be easier for me to wget the images! I don't know a lot about SED or AWK so any sort of advice/help would be great.


tl;dr: The links point to a webpage not an image, so when I wget I'm downloading html pages rather than images I want. This is how I think I could do it but any better solutions would be helpful too!

  • Just wondering (haven't tried): Can't you just download these images automatically with a single wget --base=https://www.dermquest.com/ --mirror localfile.html or similar command, leaving all that "extracting" work you wish to do to wget? – egmont Sep 28 '18 at 21:28

Depending on the complexity of the input files I suggest not trying to parse HTML with awk, grep and such but to use an HTML parser. For similar tasks I use lynx, the text mode browser. To install it a simple sudo apt install lynx is sufficient. Then:

for file in *.html; do
    lynx -dump -listonly -nonumbers $file >> links.txt

For your sample snippet it creates the following output:


When done, the file:// part needs to be replaced with a proper base URL:

sed -i 's|file://|https://www.dermquest.com|' links.txt


| improve this answer | |
  • Thanks for the reply! In the above code, could you please tell me if I should be replacing 'file' with my filename listings.txt, which has my list of urls? If I type your command as it is I don't see anything.... – ai_boy Sep 29 '18 at 0:08
  • @ai_boy I don't get it. I thought you have a bunch of *.html files with <a href="/imagelibrary/large/009514HB.JPG" …><img src="…" alt="…"/></a> in them and wanted to extract those links to get a list of these URLs. When you do that via lynx … file.html then it prints file:///imagelibrary/large/009514HB.JPG because lynx doesn't know a better base url. Then apply the sed command to replace the text file:// with https://www.dermquest.com. – PerlDuck Sep 30 '18 at 13:20

Breaking it down into steps, you want to:

  1. Process a bunch of files (named *.html?).
  2. Extract lines like <a href="/imagelibrary/large/009514HB.JPG" ....
  3. Extract the filename part ("009514HB.JPG").
  4. Produce text using the filename part.

    find . -type f -name '*.html' -print0 | \
        xargs -0 -r grep --no-filename "a href=" | \
        grep -E -o '[0-9A-Z]+\.JPG'

Then, by wrapping the above inside a for $() construct, we get:

for i in $( find . -type f -name '*.html' -print0 | \
    xargs -0 -r grep --no-filename "a href=" | \
    grep -E -o '[0-9A-Z]+\.JPG'
          ) ; do
    echo "https://www.dermquest.com/imagelibrary/large/$i"

Of course, read man find, man xargs, man grep.

| improve this answer | |
  • It says Xargs and grep not found... Am I just supposed to type this out in the command line? – ai_boy Sep 28 '18 at 23:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.