I was mirroring a server Im working on before revising code. To do this I used the command
wget -m <address>
Looks like everything is there. but what is this? a robots.txt file? I double checked, robots.txt doesn't exist on the server. So then I looked at the content of the robots.txt file, and it contains the 404 page from the server.
As in on the server:
/var/www$ ls robots.txt ls: cannot access robots.txt: No such file or directory
but in the wget mirrored folder, there is a robots.txt file that is created.
Ok, So I thought wget -m was a means of mirroring. Why is it inserting things that aren't there?