Im trying to iterate over a list containing urls, and download the content. The content is piped in json_pp to beatify it.

But the problem is that it only generates loot_0.json! And keeps overwriting the content.

cat links.txt | while read line; do #links.txt
    PAGE=$(curl -s $line)
    echo $PAGE | json_pp
done > loot/loot_$((COUNTER++)).json

I have also tried

cat links.txt | while read line; do #links.txt
    PAGE=$(curl -s $line)
    echo $PAGE | json_pp > loot/loot_$((COUNTER++)).json

Expected behavior should be files

  • 1
    Your first example will not work since COUNTER is incremented outside of the while loop. However, it should contain all pages that were brought in by curl. The second loop should produce individual files, one for each page brought in by curl. That should be right. I tested it on my Ubuntu 18.04 system and it worked. I did not have the "| json_pp", but I had everything else. – Lewis M Nov 28 '18 at 19:19

The following shellscript works for me. I have problems with json_pp, it complains about malformed JSON string, neither array, object, number, string or atom.


while read line;
# wget "$line" -O "${line##*/}$((counter++))"
 page=$(curl -s "$line")
 echo "$page" > "loot_$((counter++))"
done < links.txt
  • I prefer lower case variables (to decrease the risk of conflict with already existing environment variables).

  • I redirect from the input file (and avoid calling cat)

  • I update counter inside the loop

  • I use local variables loot_... for testing, but you can keep the subdirectory in the path if you wish.

  • Please notice that I have quoted, "...", the variables when used in the script.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.