1

so I have HTML files which have a specific portion I'd like to extract. These HTML addresses are in a text file. A sample HTML webpage, taken from this textfile would look like this, and I'd want to get part 009514HB.JPG which is different for every HTML file.

My .txt file would be something like this -

    https://www.dermquest.com/image-library/image/5044bfd0c97267166cd653c8
https://www.dermquest.com/image-library/image/5044bfcfc97267166cd61ff4
https://www.dermquest.com/image-library/image/5044bfd0c97267166cd6310a
https://www.dermquest.com/image-library/image/5044bfd0c97267166cd6310b
https://www.dermquest.com/image-library/image/5044bfcfc97267166cd61ff5
https://www.dermquest.com/image-library/image/5044bfd0c97267166cd64278
https://www.dermquest.com/image-library/image/5044bfcfc97267166cd61ff6
https://www.dermquest.com/image-library/image/5044bfcfc97267166cd61ff7
https://www.dermquest.com/image-library/image/5044bfd0c97267166cd63c1b
https://www.dermquest.com/image-library/image/5044bfcfc97267166cd619b1
https://www.dermquest.com/image-library/image/5044bfcfc97267166cd619b4
https://www.dermquest.com/image-library/image/5044bfcfc97267166cd625a3
https://www.dermquest.com/image-library/image/5044bfd0c97267166cd64844
https://www.dermquest.com/image-library/image/5044bfcfc97267166cd619ba
https://www.dermquest.com/image-library/image/5044bfd0c97267166cd63ce3
https://www.dermquest.com/image-library/image/5044bfd0c97267166cd64437
https://www.dermquest.com/image-library/image/5044bfd1c97267166cd67203

and if I click open one of those html pages using a text editor I can find the information I need..

   *** some code here***
<figure>
         <a href="/imagelibrary/large/009514HB.JPG" target="_blank" class="preview-image"><img src="/imagelibrary/medium/009514HB.JPG" alt="acne keloidalis nuchae"/></a>
</figure>
...

Now I would like to get these numbers from various HTML files, and then append these numbers to https://www.dermquest.com/imagelibrary/large. For example, I would want my final txt file to have urls which are like this, [slightly NSFW] https://www.dermquest.com/imagelibrary/large/009514HB.JPG This line would be easier for me to wget the images! I don't know a lot about SED or AWK so any sort of advice/help would be great.

Thanks!

tl;dr: The links point to a webpage not an image, so when I wget I'm downloading html pages rather than images I want. This is how I think I could do it but any better solutions would be helpful too!

  • Just wondering (haven't tried): Can't you just download these images automatically with a single wget --base=https://www.dermquest.com/ --mirror localfile.html or similar command, leaving all that "extracting" work you wish to do to wget? – egmont Sep 28 '18 at 21:28
1

Depending on the complexity of the input files I suggest not trying to parse HTML with awk, grep and such but to use an HTML parser. For similar tasks I use lynx, the text mode browser. To install it a simple sudo apt install lynx is sufficient. Then:

for file in *.html; do
    lynx -dump -listonly -nonumbers $file >> links.txt
done

For your sample snippet it creates the following output:

file:///imagelibrary/large/009514HB.JPG

When done, the file:// part needs to be replaced with a proper base URL:

sed -i 's|file://|https://www.dermquest.com|' links.txt

Result:

https://www.dermquest.com/imagelibrary/large/009514HB.JPG
  • Thanks for the reply! In the above code, could you please tell me if I should be replacing 'file' with my filename listings.txt, which has my list of urls? If I type your command as it is I don't see anything.... – ai_boy Sep 29 '18 at 0:08
  • @ai_boy I don't get it. I thought you have a bunch of *.html files with <a href="/imagelibrary/large/009514HB.JPG" …><img src="…" alt="…"/></a> in them and wanted to extract those links to get a list of these URLs. When you do that via lynx … file.html then it prints file:///imagelibrary/large/009514HB.JPG because lynx doesn't know a better base url. Then apply the sed command to replace the text file:// with https://www.dermquest.com. – PerlDuck Sep 30 '18 at 13:20
0

Breaking it down into steps, you want to:

  1. Process a bunch of files (named *.html?).
  2. Extract lines like <a href="/imagelibrary/large/009514HB.JPG" ....
  3. Extract the filename part ("009514HB.JPG").
  4. Produce text using the filename part.

    find . -type f -name '*.html' -print0 | \
        xargs -0 -r grep --no-filename "a href=" | \
        grep -E -o '[0-9A-Z]+\.JPG'
    

Then, by wrapping the above inside a for $() construct, we get:

for i in $( find . -type f -name '*.html' -print0 | \
    xargs -0 -r grep --no-filename "a href=" | \
    grep -E -o '[0-9A-Z]+\.JPG'
          ) ; do
    echo "https://www.dermquest.com/imagelibrary/large/$i"
done

Of course, read man find, man xargs, man grep.

  • It says Xargs and grep not found... Am I just supposed to type this out in the command line? – ai_boy Sep 28 '18 at 23:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.