Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I have a bash script called test.sh and it current is:

url=
lynx -dump $url > story.txt
echo "step one completed"

Except when I run ./test.sh http://www.fox.com or any website, story.txt says:

                            Forbidden
   You don't have permission to access / on this server.

How do I fix it so that when I open story.txt, it displays the actual dump of the url?

share|improve this question
1  
this would probably be better asked on stackoverflow. –  ImaginaryRobots Dec 6 '12 at 20:19
    
This won't work because $url has no value. Try setting it and seeing what you get. –  Scott Severance Dec 7 '12 at 3:37
add comment

2 Answers 2

up vote 4 down vote accepted

Your script does not work for me, but I guess it does for you so we might have a different setup. I changed the script to

lynx -dump $1 > story.txt
echo "step one completed"

I run it with

 ./test.sh http://www.techland.ro

and I get the same result.

Instead of running the script with fox.com try with techland.ro and you will see that the script actually works just fine. The problem is not with your script but with the website you are trying to access. I believe it blocks website crawlers, because lynx is a command line browser it actually acts exactly like a crawler, resulting in the message you get.

You can set different rules for websites, if a website blocks unknown crawlers then it will also probably block lynx (or any other command line browser). Your script is fine, the website is blocking you.

share|improve this answer
    
This worked!!!! Thank you so much! My script wasn't working which is why I asked in the first place! –  helloworld Dec 6 '12 at 20:46
add comment

That URL gives an error with curl too:

curl -i http://www.fox.com/
HTTP/1.1 403 Forbidden
Server: AkamaiGHost
Mime-Version: 1.0
Content-Type: text/html
Content-Length: 260
Cache-Control: max-age=3600
Date: Thu, 06 Dec 2012 20:17:17 GMT
Connection: close

<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>

You don't have permission to access "http&#58;&#47;&#47;www&#46;fox&#46;com&#47;" on this server.<P>
Reference&#32;&#35;18&#46;2fa2f5cc&#46;1354825037&#46;8323a81
</BODY>
</HTML>

Apparently the site is doing some weird user agent sniffing. Passing a Firefox user agent seems to work:

curl -i -A 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:17.0) Gecko/17.0 Firefox/17.0' http://www.fox.com/
HTTP/1.1 200 OK
Server: Apache
Content-Length: 70647
Content-Type: text/html; charset=utf-8
X-FarmName: www.fox.com
X-FarmAddr: 10.96.57.103
Cache-Control: max-age=3600
Date: Thu, 06 Dec 2012 20:22:58 GMT
Connection: keep-alive

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:tp="http://player.theplatform.com/" xmlns:og="http://ogp.me/ns#" xmlns:fb="http://www.facebook.com/2008/fbml" >
<head>
        <title>FOX Broadcasting Company - FOX Television Shows</title>
share|improve this answer
    
It's Fox after all - they fear what they don't understand (wget in this case), so they ban it. Sorry, could not resist. –  Marcin Kaminski Dec 6 '12 at 20:34
    
That works fine on it's own. But when I run in terminal, ./test.sh fox.com it doesn't work where test.sh is now: url=<br> curl -i -A 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:17.0) Gecko/17.0 Firefox/17.0' $url > story.txt<br> echo "step one completed" –  helloworld Dec 6 '12 at 20:36
    
This did not work with the test.sh but thank you nevertheless! –  helloworld Dec 6 '12 at 20:56
    
You have to set $url to something. Instead of url=, try something like url="http://www.fox.com/". –  Scott Severance Dec 7 '12 at 3:41
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.