0

Im trying to iterate over a list containing urls, and download the content. The content is piped in json_pp to beatify it.

But the problem is that it only generates loot_0.json! And keeps overwriting the content.

#!/bin/bash
COUNTER=0
cat links.txt | while read line; do #links.txt
    PAGE=$(curl -s $line)
    echo $PAGE | json_pp
done > loot/loot_$((COUNTER++)).json

I have also tried

#!/bin/bash
COUNTER=0
cat links.txt | while read line; do #links.txt
    PAGE=$(curl -s $line)
    echo $PAGE | json_pp > loot/loot_$((COUNTER++)).json
done

Expected behavior should be files

loot_1.json
loot_2.json
loot_3.json
..
  • 1
    Your first example will not work since COUNTER is incremented outside of the while loop. However, it should contain all pages that were brought in by curl. The second loop should produce individual files, one for each page brought in by curl. That should be right. I tested it on my Ubuntu 18.04 system and it worked. I did not have the "| json_pp", but I had everything else. – Lewis M Nov 28 '18 at 19:19
0

The following shellscript works for me. I have problems with json_pp, it complains about malformed JSON string, neither array, object, number, string or atom.

#!/bin/bash

counter=0
while read line;
do
# wget "$line" -O "${line##*/}$((counter++))"
 page=$(curl -s "$line")
 echo "$page" > "loot_$((counter++))"
done < links.txt
  • I prefer lower case variables (to decrease the risk of conflict with already existing environment variables).

  • I redirect from the input file (and avoid calling cat)

  • I update counter inside the loop

  • I use local variables loot_... for testing, but you can keep the subdirectory in the path if you wish.

  • Please notice that I have quoted, "...", the variables when used in the script.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.