I made a ~/.bashrc
function to save some web directories into my local disk. It works well except some unwanted index files that is not present in the website.
I use it like
crwl http://ioccc.org/2013/cable3/
but it also retrieves some files such as index.html?C=D;O=A index.html?C=D;O=D index.html?C=M;O=A index.html?C=M;O=D index.html?C=N;O=A index.html?C=N;O=D index.html?C=S;O=A index.html?C=S;O=D
Complete file list:
kenn@kenn:~/experiment/crwl/ioccc.org/2013/cable3$ ls
bios index.html?C=D;O=A index.html?C=S;O=A screenshot_flightsim4.png
cable3.c index.html?C=D;O=D index.html?C=S;O=D screenshot_lotus123.png
fd.img index.html?C=M;O=A Makefile screenshot_qbasic.png
hint.html index.html?C=M;O=D runme screenshot_simcity.png
hint.text index.html?C=N;O=A sc-ioccc.terminal screenshot_win3_on_macosx.png
index.html index.html?C=N;O=D screenshot_autocad.png
I want to exclude those files while cloning that directory with wget
Is there any wget
switch or trick to clone a web directory as it is?
My script function in .bashrc
:
crwl() {
wget --tries=inf --timestamping --recursive --level=inf --convert-links --page-requisites --no-parent "$@"
}
EDIT: I found two possible workarounds
1) Adding -R index.html?*
flag
2) Adding -R =A,=D
flag
which rejects index.html?C=D;O=A
files except index.html
I don't know which one is proper but both of them seem unsafe.
I don't know which one is proper but both of them seem unsafe.
index.html?*
which is essential part of retrieved files. I don't understand usage ofindex.html?C=D;O=A
files. There might be more precise rule to exclude those files.