How to parse through a text file which has urls and indexing, and download images from it?

Question

so I get that I can use wget -i text_file.txt to download images from urls in.txt but the file also has some weird indexing. I'm trying to download a dataset for machine learning and it has different classes of images.

It has something like

2598 98 https://www.dermquest.com/image-library/image/5044bfcfc97267166cd620a2
2599 99 https://www.dermquest.com/image-library/image/5044bfcfc97267166cd61a48
2600 00 https://www.dermquest.com/image-library/image/5044bfcfc97267166cd61a49
2601 01 https://www.dermquest.com/image-library/image/5044bfd0c97267166cd64e01
2602 02 https://www.dermquest.com/image-library/image/5044bfd0c97267166cd64e02
2603 03 https://www.dermquest.com/image-library/image/5044bfd0c97267166cd63c9c

and later...

6577 77 https://www.dermquest.com/image-library/image/5044bfd0c97267166cd64727
6578 78 https://www.dermquest.com/image-library/image/5044bfd1c97267166cd662c7
6579 79 https://www.dermquest.com/image-library/image/5044bfd0c97267166cd65888
6580 80 https://www.dermquest.com/image-library/image/5044bfd0c97267166cd635e0
6581 81 https://www.dermquest.com/image-library/image/5044bfd1c97267166cd6676d
6582 82 https://www.dermquest.com/image-library/image/5044bfd1c97267166cd6676e
6583 83 https://www.dermquest.com/image-library/image/5044bfd0c97267166cd6472b
6584 84 https://www.dermquest.com/image-library/image/5044bfd0c97267166cd6588d

Now I would like to use the indexing and download images with the same indexes to the same file... or something like that.

Thanks!

Answer 1

That looks like a simple job for cut(1):

cut -d ' ' -f 3 < url-listing.txt

You can pipe its output directly to wget and use the “special” file name - to read from standard input:

cut -d ' ' -f 3 < url-listing.txt | wget -i -

Answer 2

You can use sed to remove the numbers at the start of each line:

sed -r 's/^[0-9]+//g' urls.txt > urls_without_numbers.txt

Now you can use wget with the new url_without_numbers.txt

If for some reason you really need to do it without making a new file like above

sed -r 's/^[0-9]+//g' urls.txt | wget -i /dev/stdin

In the event that your lines contain number ranges like 1-100 try this:

sed -r 's/^[0-9\-]+//g' urls.txt > urls_without_numbers.txt

Answer 3

Seems easier to me to solve this by awk. Awk splits the by a string and then executes a command. With

for url in $(awk '{print $NF}' url1.txt | tr -d '\r'); do wget -L $url -O - | grep "preview-image"; done 2>&1 | grep "img src" | awk '{print $5}' | tr -d "\"" | awk -F'=' '{print $2}' &> real_urls.txt

you first print the last element, if the line is split by space (default). Then, you remove '\r' (which should not be in the URL) and then use the URL as argument for wget. Then, in the wget output the correct img-tag is search by grep. Afterwards, you'll need to somehow get what is after src. This is done by deleting the " (which needs to be escaped) and then by using Awk in order to get what is behind =. Then, everything is saved into real_urls.txt. Then, you can just download by:

for url in $(cat real_urls.txt); do wget "https://www.dermquest.com$url"; done