Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I have a shell script to download some of my stuff over Internet. How can I know if a file exists over the Internet? Let's say I want to know if http://192.168.1.1/backup/01012011.zip exists or not? I have try using ping command, but it shows error, i guess this because / character.

Can anyone can help me? or is there another way?

share|improve this question
    
It should be noted that ping does not send HTTP requests at all. Rather, ping uses a protocol called 'ICMP' to determine whether a host is reachable and to check latency. –  Nathan Osman Jul 9 '11 at 7:31

3 Answers 3

up vote 6 down vote accepted

There certainly is another way - but this requires understanding what actually happens when a request is made over the Internet. When you visit a page in your web browser, data is transferred using a protocol called HTTP (yes, this is why you'll often see http:// at the beginning of URLs).

HTTP is a text-based protocol. Information is exchanged between the client and the server by sending headers followed by the body of the request. The headers contain a lot of status information about the request and the information being transferred. The header that you will be interested in to help you with your problem isn't really a header at all - it's the very first line transferred and contains a number called the status code. This number is 3 digits and conveys status information. If a request was successful, the result is usually 200 (not always - there are exceptions).

One thing is for sure - if the file you have requested does not exist on the web server, the server should reply with a status code of 404. This indicates that the resource could not be found. (For the curious, here is a list of HTTP status codes and their meaning.)

Well, enough theory. Let's see how we can do this on the terminal. A great tool for fetching requests using HTTP that also provides us with the ability to examine the status code is cURL, which is available in the Ubuntu repos. You can install it with:

sudo apt-get install curl

Once you have it installed, you can invoke it like so:

curl [website]

...and the content of the given URL will be printed to the terminal. This is the information that your web browser sees when it visits that URL. How does this help us? Well, take a close look at the flags for the curl command. If we pass the parameter --head, cURL will return only the headers from the request. Try it with a URL. You'll get a list of lines of the form:

header-name: header-value

Notice, of course, that the very first line looks nothing like this. Remember the status code that we talked about earlier? You'll notice it in the first line as the three-digit number. What we need to do now is extract it from the first line using Perl - and we can do it in the terminal using Perl's -e flag which let's us pass Perl code directly to the Perl interpreter. We'll also need to add an extra flag to cURL (--silent) in order to keep it from displaying a progress bar and messing up our Perl script.

Here is what we need... it's quite complicated due to the need to escape a lot of it from the shell:

perl -e "\$s=\`curl [URL] --head --silent\`; \$s=~m/(\\d{3})/;print \$1"

What this is basically doing is fetching the URL with cURL and running it through a Perl regular expression that extracts the status code and prints it out.

Now all you need to is put in the URL of the file you are checking for and compare it to '404'. If you get '404', you can assume the file does not exist.

Of course, this could be very difficult to manipulate in the terminal, so you can write a small script that makes this not only easier to understand, but also easier to execute:

#!/usr/bin/perl

# Get the URL
$url = $ARGV[0];

# Fetch the header
$header = `curl $url --head --silent`;

# Try to find the status code
$header =~ m/(\d{3})/;

# Return the result
exit(0) if $1 == 404;
exit(1);

Simply copy and paste that into a file. For this example, I'll call the file url_check. Then make the file executable with:

chmod 755 url_check

Then you can check any file with the following simple command:

./url_check [URL]

The return value will be '0' if the server returned a 404 and '1' otherwise. You can then chain this command in the shell just like you would any other command.

share|improve this answer
    
thanks so much about the theory and the solution,... but the perl part,.. i wonder to make it just with a simple shell script,.. on working,.... –  Warung Nasi 49 Jul 11 '11 at 10:11
    
@Warung: Well... a shell script will need to call an external command to not only query a remote URL, but also to parse the response. –  Nathan Osman Jul 11 '11 at 16:54
    
yups... and mybe i can try to parse response with cut commands... but still not work,.. for now, i just do it like what you did.. –  Warung Nasi 49 Jul 12 '11 at 6:04
    
@WarungNasi49: something like curl $url --head --silent | head -n 1 | cut -d ' ' -f 2? –  zpea Jul 13 '12 at 23:10
    
@GeorgeEdison: Nice answer! As you mentioned quoting the perl code from bash: You can get rid of lots of backslashes if you single quotes (') instead of double quotes (") around your perl expression. –  zpea Jul 13 '12 at 23:15

You can use the --spider option of wget, which does not actually download the file, but just checks if it's there. In your example:

wget --spider http://192.168.1.1/backup/01012011.zip

This will either return a message containing 200 OK if the file is there, or an error, e.g. 404 Not Found if it's not there, or 403 Forbidden if you don't have permission to get it.

share|improve this answer
wget http://192.168.1.1/backup/01012011.zip

Result code 0 means yes, something else - no.

You can check result code inside the script with $? variable.

share|improve this answer
1  
Hey Mikail! Interpreting the return values is a good idea. However this command would download the whole file, not just check if it is available. –  zpea Jul 13 '12 at 23:19

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.