wget -m inserts robots.txt?

Question

I was mirroring a server Im working on before revising code. To do this I used the command

wget -m <address>

Looks like everything is there. but what is this? a robots.txt file? I double checked, robots.txt doesn't exist on the server. So then I looked at the content of the robots.txt file, and it contains the 404 page from the server.

As in on the server:

/var/www$ ls robots.txt
ls: cannot access robots.txt: No such file or directory

but in the wget mirrored folder, there is a robots.txt file that is created.

Ok, So I thought wget -m was a means of mirroring. Why is it inserting things that aren't there?

Answer 1

You can use the option -e robots=off to ignore the robots.txt file. If the webserver is not yours, however, ignoring the robots.txt file is considered rude.

Answer 2

The Problem isn't with wget inserting a robots.txt file where it doesn't belong, it is with the server you are mirroring content from. The server is evidently miss-configured to handle type 200 requests, and thus, when wget looks for a wget page, it recreates a 404'd request.

I was looking at the wget source code a while ago. I think what it came down to was that wget always looks for a robots.txt file. But, in most cases, if it is left blank wget deletes it. In your case, it is filed by the 404 page it gets when wget tries to access it.

Unfortunately, I do not know how to fix a type 200 HTTP request error, for that, we would need more info about the server.