1

I have a file containing the URL of websites. I am trying to write a bash script that will take any file containing URL and write to another file - the URL and the related status code.
What I am trying to achieve, is here :
I have URL in a file as:

http://google.com
http://trafficinviter.com
http://stackoverflow.com   
............
............
Upto 30 crore lines of URL in a file.   

I am using the cURL command to get the Http Status of these URLs.
Hence, I am not understanding how I can get the URL and the corresponding HTTP status of that URL.
Say I need to have the output in a file like this:

    http://google.com
    200 ok    
    http://trafficinviter.com
    200 ok
    http://stackoverflow.com   
    200 ok
    http://example.com
    404 
    ............
    ............
    Upto 30 crore lines of URL in a file. 

Also I am looking for a bash that makes my work done in a jiffy and easily. Because there are many lines, I am not able to find a proper solution for this. Kindly, help me. Thank you very much.

3

Try:

while read url
do
    echo "$url"
    curl -sI "$url" | head -1
done < list-of-urls > output-file

This just reads each line of input from the list-of-urls file, outputs it back and then gets the first line of output from curl -I on that URL (which is the HTTP response status). The entire loop's output goes to output-file.

To test multiple URLs at once, use GNU parallel:

GNU parallel makes sure output from the commands is the same output as
you would get had you run the commands sequentially. This makes it
possible to use output from GNU parallel as input for other programs.

For example:

parallel -a list-of-urls 'echo {}; curl -sI {} | head -1' > output-file

No guarantees that the output will be in order, though:

$ cat output-file
http://example.com
HTTP/1.1 200 OK
http://google.com
HTTP/1.1 302 Moved Temporarily
http://stackoverflow.com
HTTP/1.1 200 OK
http://trafficinviter.com
HTTP/1.1 200 OK
  • I will try this. But is this approach helpful for multi threading. Means multiple request at time so it make my process faster? and is it a bash script. Because I don't want to make the script static. I want to give the file name at run time – Jaffer Wilson Jan 4 '17 at 7:02
  • @JafferWilson and how do you intend to ensure that when the output goes into a file, it doesn't get mixed up? – muru Jan 4 '17 at 7:03
  • well that's an excellent question. I just will try for few urls first whether it is helpful . If not then I need to drop that idea... :P – Jaffer Wilson Jan 4 '17 at 7:05
  • Please Muru .. help me with a solution having multi threading in it. So my process become faster, – Jaffer Wilson Jan 4 '17 at 7:07
  • Try the update. – muru Jan 4 '17 at 7:31
0

With GNU Parallel you would do something like:

cat input | parallel -k -j 100 --tag 'curl -sI {} | head -1'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.