0

Trying to create a script using curl to probe multiple URLs for updated information. Currently, I'm using individual commands for each URL, which I am trying to shorten lines of codes by using increment variables in a for loop. What I have at the moment: (dummy urls as examples)

#!/bin/bash

url01="https://askubuntu1.com"
url02="https://askubuntu2.com"
url03="https://askubuntu3.com"
url04="https://askubuntu4.com"
url05="https://askubuntu5.com"

for i in {01..05}
    do
        url=$((url$i))
        curl $url 2>&1| grep 'askubuntu' >> output.txt
done &

Error Output:

line 10: https://askubuntu1.com: syntax error in expression (error token is "://askubuntu1.com")

Expected result: command input to bash shell for each of curl queries pipes to grep then outputs to "output.txt".

curl "https://askubuntu1.com"
curl "https://askubuntu2.com"
.
.
curl "https://askubuntu3.com"

Thank You

0

The assignment url=$((url$i)) attempts to evaluate url$i as an arithmetic expression - this is not appropriate here. What you could do instead is make use of variable indirection:

for url in url{01..05}
do
  curl "${!url}" 2>&1| grep 'askubuntu' >> output.txt
done

However it would probably cleaner to use an array:

#!/bin/bash

urls=(
  "https://askubuntu1.com"
  "https://askubuntu2.com"
  "https://askubuntu3.com"
  "https://askubuntu4.com"
  "https://askubuntu5.com"
)

for url in "${urls[@]}"
do
  curl "$url" 2>&1| grep 'askubuntu' >> output.txt
done

You could also loop over the indices of the array ${!urls[@]} - which would give you the desired incrementing integer variable from which to construct a sequence of numbered output files ex.:

for i in ${!urls[@]}
do
  printf -v output 'file%02d.txt' $((i+1))
  curl "${urls[i]}" 2>&1 | grep 'askubuntu' > "$output"
done

The $((i+1)) is because bash array indexing starts from 0 - if you don't mind the outputs being file00 through file04 then you can use plain $i.

For the version based on variable indirection, you could do either

for i in {01..05}
do
  url="url$i"
  curl "${!url}" 2>&1| grep 'askubuntu' > "file${i}.txt"
done

or

for url in url{01..05}
do
  curl "${!url}" 2>&1| grep 'askubuntu' > "${url/#url/file}.txt"
done
4
  • cheers, the first error is a typo... which I have corrected to "{ }" (thanks!) – danh Nov 29 '20 at 1:52
  • confirmed that both of your solutions worked! @steeldriver I would use this as the best answer to my question. additionally, what if I would like to output files to have increment variables per each url? example: output01=file01 output02=file02 ... output05=file05? – danh Nov 29 '20 at 5:23
  • @danh please see updated answer – steeldriver Nov 29 '20 at 13:41
  • thank you!@ steeldriver, this is very helpful and easy to understand. – danh Dec 1 '20 at 8:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.