so I get that I can use wget -i text_file.txt to download images from urls in.txt but the file also has some weird indexing. I'm trying to download a dataset for machine learning and it has different classes of images.

It has something like

2598 98 https://www.dermquest.com/image-library/image/5044bfcfc97267166cd620a2
2599 99 https://www.dermquest.com/image-library/image/5044bfcfc97267166cd61a48
2600 00 https://www.dermquest.com/image-library/image/5044bfcfc97267166cd61a49
2601 01 https://www.dermquest.com/image-library/image/5044bfd0c97267166cd64e01
2602 02 https://www.dermquest.com/image-library/image/5044bfd0c97267166cd64e02
2603 03 https://www.dermquest.com/image-library/image/5044bfd0c97267166cd63c9c

and later...

6577 77 https://www.dermquest.com/image-library/image/5044bfd0c97267166cd64727
6578 78 https://www.dermquest.com/image-library/image/5044bfd1c97267166cd662c7
6579 79 https://www.dermquest.com/image-library/image/5044bfd0c97267166cd65888
6580 80 https://www.dermquest.com/image-library/image/5044bfd0c97267166cd635e0
6581 81 https://www.dermquest.com/image-library/image/5044bfd1c97267166cd6676d
6582 82 https://www.dermquest.com/image-library/image/5044bfd1c97267166cd6676e
6583 83 https://www.dermquest.com/image-library/image/5044bfd0c97267166cd6472b
6584 84 https://www.dermquest.com/image-library/image/5044bfd0c97267166cd6588d

Now I would like to use the indexing and download images with the same indexes to the same file... or something like that.


  • 1
    Please edit your post and add what you want exactly. You show the same "url.com" five times and write it is "something like that". We can only guess what your input and your desired output is. – PerlDuck Sep 24 '18 at 17:13
  • 1
    Question looks OK to me. OP wants to extract URLs from a text file and download all the URLs, but the text file has other stuff in it than just URLs, so it can't be fed directly to wget. – thomasrutter Sep 25 '18 at 0:57
  • Please format text files as code, not blockquote. It's impossible to tell what is really in the file with all the markdown rendering. I found the actual file from your link and copied something from it, but please clarify what you want to get from this file – Zanna Sep 25 '18 at 8:21
  • 1
    @thomasrutter I was especially referring to the part "and download images with the same indexes to the same file...". The current answer (and you, apparently) assumes all leading numbers can simply be dropped. Now that Zanna edited some real data into the question it gets clearer but still: what does the "same index, same file" mean? What is the index? The first number, the second, or both? And "same index, same file, or something like that" could also be interpreted as "download all URLs to a single file". That's as well "something like that"! – PerlDuck Sep 25 '18 at 11:05
  • 2
    @RoVo coding questions aren't off-topic here - text-processing and Bash scripting are 100% on topic – Zanna Sep 25 '18 at 18:30

That looks like a simple job for cut(1):

cut -d ' ' -f 3 < url-listing.txt

You can pipe its output directly to wget and use the “special” file name - to read from standard input:

cut -d ' ' -f 3 < url-listing.txt | wget -i -

You can use sed to remove the numbers at the start of each line:

sed -r 's/^[0-9]+//g' urls.txt > urls_without_numbers.txt

Now you can use wget with the new url_without_numbers.txt

If for some reason you really need to do it without making a new file like above

sed -r 's/^[0-9]+//g' urls.txt | wget -i /dev/stdin

In the event that your lines contain number ranges like 1-100 try this:

sed -r 's/^[0-9\-]+//g' urls.txt > urls_without_numbers.txt
  • thanks for the reply but it seems I've still got indexing left for some lines idk why! it is probably because for some entries there are numberings 1-100. For others the numbering is like 100 1, 101 2, (so two numbers with a space). Any chance you could tell me how to edit above line for this problem? thanks!! – ai_boy Sep 24 '18 at 17:46
  • @sasi I have updated my answer! – Kristopher Ives Sep 24 '18 at 17:49
  • Hey I'm sorry but it still doesn't work. Let me share the link of File I'm trying to download. I wish I could share the .txt directly but idk how to do it in stackoverflow. Anyways if you go to cv.nankai.edu.cn/projects/SD-198 and download the 198 dataset, there is the url1.txt file. If you could look into it that'd be great! thanks – ai_boy Sep 24 '18 at 17:55

Seems easier to me to solve this by awk. Awk splits the by a string and then executes a command. With

for url in $(awk '{print $NF}' url1.txt | tr -d '\r'); do wget -L $url -O - | grep "preview-image"; done 2>&1 | grep "img src" | awk '{print $5}' | tr -d "\"" | awk -F'=' '{print $2}' &> real_urls.txt

you first print the last element, if the line is split by space (default). Then, you remove '\r' (which should not be in the URL) and then use the URL as argument for wget. Then, in the wget output the correct img-tag is search by grep. Afterwards, you'll need to somehow get what is after src. This is done by deleting the " (which needs to be escaped) and then by using Awk in order to get what is behind =. Then, everything is saved into real_urls.txt. Then, you can just download by:

for url in $(cat real_urls.txt); do wget "https://www.dermquest.com$url"; done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.