2

I want to make a script so I can check if I can connect to several URL.

For example I do curl -i -L --silent URL | grep "HTTP/2 200" >> *.txt

I want to combine it with an if statement so I can do if grep "HTTP/" 200" >> *.txt", then: grep "Location" >> *.txt

So if it does grep "HTTP/2 200" then it should grep the "location" (URL) as well. I can't get it to work though.

Also I want to do echo "-------------------------------------" between every URL output so it shows like this in a file:

#URL1

HTTP(1)

URL(1)

#URL2

HTTP(2)

URL(2)

and so on.

thanks for every help.

4
  • Yep, possible ... curl -i -L --silent URL | if grep -q "HTTP/2 200"; then grep "Location"; fi ... But, please give examples to make your question clearer so we can help you better ... i.e. Be specific please.
    – Raffa
    Nov 24 at 15:56
  • did you think about curl -i -L --silent URL | grep "HTTP/2 200" | grep "Location" >> *.txt ? this may solve the fist part of your question.
    – dummyuser
    Nov 24 at 16:44
  • Check the bash man pages for the && construct. execute what's after if the previous succeeded.
    – ubfan1
    Nov 24 at 16:47
  • Please edit your question and show some example output of curl -i -L --silent URL and the corresponding expected result in your output file.
    – Bodo
    Nov 24 at 17:33

1 Answer 1

2

If you want the HTTP response/status and the URL saved to a file in the format you desire. then curl alone can do that ... It has the option -w or --write-out(Which defines what to display on stdout after a completed and successful operation. The format is a string that may contain plain text mixed with any number of variables ... e.g. -w "-----------\nStatus: %{http_code}\nURL: %{url}\n") and if you redirect the rest of the output to e.g. /dev/null then you end up with a clean result of just what you want and you can then redirect that result and append it to a file called e.g. url.txt like so:

curl -o /dev/null -s -w "-----------\nStatus: %{http_code}\nURL: %{url}\n" https://askubuntu.com >> url.txt

If you have a list of URLs(each on a new line) in a file called e.g. list.txt like so:

$ cat list.txt
https://askubuntu.com/
https://unix.stackexchange.com/
https://notworkingurlblabla.com/

Then, you can check all the URLs in that file, filter-out non 200 status URLs and append the result to url.txt all at once e.g. like so:

xargs -n 1 <list.txt curl -o /dev/null -s -w "%{http_code} %{url}\n" | \
awk '{if ($1 == "200") printf "------------\nStatus: "$1"\nURL: "$2"\n"}' >> url.txt

You can also, obviously, use awk to filter output with a single URL as well like so:

curl -o /dev/null -s -w "%{http_code} %{url}\n" https://askubuntu.com/ | \
awk '{if ($1 == "200") printf "------------\nStatus: HTTP "$1"\nURL: "$2"\n"}' >> url.txt
8
  • how can i output the HTTP 200 and Location to a file tho?
    – bahsnub
    Nov 25 at 7:47
  • @dummyuser sadly no, i dont want both the HTTP status + the Location to get saved into the file. i Only want them to be saved if there is a HTTP2/2 200, but with your command it would save the Location anyway even if there is no http 200
    – bahsnub
    Nov 25 at 7:48
  • @user2315 curl -o /dev/null -s -w "%{http_code} %{url}\n" https://askubuntu.com/ | awk '{if ($1 == "200") printf "------------\nStatus: HTTP "$1"\nURL: "$2"\n"}' >> url.txt
    – Raffa
    Nov 25 at 8:05
  • 1
    Oh okay thank you. I have my script like this now: xargs -n 1 <url.txt curl -o /dev/null -s -w "%{http_code} %{url_effective}\n" | awk '{if ($1 == "200") printf "------------\nStatus: "$1"\nURL: "$2"\n"}' >> test.txt it doenst work for me with "url" tho so i tried using "url_effective", it doenst copy the output to the test.txt file tho. My file with all the URL`s is in the file "url.txt" so i changed that off aswell
    – bahsnub
    Nov 25 at 8:28
  • 1
    I was missing the -i -L because i get 3 http status codes because of nginx but now it works thank you.
    – bahsnub
    Nov 25 at 8:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.