3

I made a ~/.bashrc function to save some web directories into my local disk. It works well except some unwanted index files that is not present in the website. I use it like

crwl http://ioccc.org/2013/cable3/  

but it also retrieves some files such as index.html?C=D;O=A index.html?C=D;O=D index.html?C=M;O=A index.html?C=M;O=D index.html?C=N;O=A index.html?C=N;O=D index.html?C=S;O=A index.html?C=S;O=D

Complete file list:

 kenn@kenn:~/experiment/crwl/ioccc.org/2013/cable3$ ls
 bios        index.html?C=D;O=A  index.html?C=S;O=A           screenshot_flightsim4.png
 cable3.c    index.html?C=D;O=D  index.html?C=S;O=D           screenshot_lotus123.png
 fd.img      index.html?C=M;O=A  Makefile                     screenshot_qbasic.png
 hint.html   index.html?C=M;O=D  runme                        screenshot_simcity.png
 hint.text   index.html?C=N;O=A  sc-ioccc.terminal            screenshot_win3_on_macosx.png
 index.html  index.html?C=N;O=D  screenshot_autocad.png

I want to exclude those files while cloning that directory with wget Is there any wget switch or trick to clone a web directory as it is?

My script function in .bashrc:

crwl() {
wget --tries=inf --timestamping --recursive --level=inf --convert-links --page-requisites --no-parent "$@"

}

EDIT: I found two possible workarounds

1) Adding -R index.html?* flag

2) Adding -R =A,=D flag which rejects index.html?C=D;O=A files except index.html

I don't know which one is proper but both of them seem unsafe.

  • can you explain what you mean by safety, in: I don't know which one is proper but both of them seem unsafe. – clarity123 Jan 12 '16 at 18:20
  • @user454038 I mean it may reject some files with index.html?* which is essential part of retrieved files. I don't understand usage of index.html?C=D;O=A files. There might be more precise rule to exclude those files. – kenn Jan 12 '16 at 18:31
2

To exclude index-sort files such as those with URL index.html?C=... without excluding any other kind of index.html* files, there is indeed a more precise specification possible. Try: -R '\?C='

Quick Demo

Set up an different empty directory, for example

$ mkdir ~/experiment2
$ cd ~/experiment2

Then a shorter version of your command, without the recursion and levels in order to do a quick one page test:

$ wget --tries=inf --timestamping --convert-links --page-requisites --no-parent -R '\?C=' http://ioccc.org/2013/cable3/

After wget is done, ~/experiment2, will have no index.html?C=... files:

.
└── ioccc.org
    ├── 2013
    │   └── cable3
    │       └── index.html
    ├── icons
    │   ├── back.gif
    │   ├── blank.gif
    │   ├── image2.gif
    │   ├── text.gif
    │   └── unknown.gif
    └── robots.txt

4 directories, 7 files

So it has indeed excluded those redundant index-sort index.html?C=... directories while keeping all other index.html directories, in this case just index.html

Implement

So just implement the -R '\?C=' , by updating your shell function in ~/.bashrc:

crwl() {
  wget --tries=inf --timestamping --recursive --level=inf --convert-links --page-requisites --no-parent -R '\?C=' "$@"
}

Then remember to either test in a new terminal, or re-source bash to make it effective:

$ . ~/.bashrc

Then try it in a new directory, for comparison:

$ mkdir ~/experiment3
$ cd ~/experiment3
$ crwl http://ioccc.org/2013/cable3/

Warranty

  • wget 1.14 and up only. So if your wget -V says it is 1.13 this may not work and you have need to actually delete those pesky index.html?C=... yourself, or try to get a more recent version of wget.
  • works by specifying you want to -R or reject a pattern, in this case pages with ?C= pattern that is typical of the index.html?C=... versions of index.html.
  • however ? happens to be a wget wildcard, thus to match a literal ? you need to escape it as \?
  • don't interrupt wget. Because it seems the way wget works with browse-able web pages is to actually download first, delete later, as if it needs to check in case those pages have further links to crawl. So if you cancel this halfway you are still going to end up with index.html?C= files. Only if you let wget finish, then wget will follow your -R specification and delete any temporarily downloaded index.html?C=... files for you
1

Try this after download, if you do not want to use wget's removal mechanism or are on a system not suporting this option.

FIND=$($WHICH find)
PWD2=$($WHICH pwd)
SH=$($WHICH sh)
ECHO=$($WHICH echo)
LESS=$($WHICH less)

Command:

$FIND "$($PWD2)" -regextype posix-egrep -type f -regex '^(.*?html\?C=[DNSM];O=[AD])$' -exec "$SH" -c 'o="{}";$ECHO -f -v "${o}"' \; | $LESS

When you are satisfied with the output, do the following:

  1. Issue the following command (see below box)
  2. Replace $ECHO with $RM in the above command.
  3. Remove the pipe (|) and the $LESS, to get actual output.

(I'm not responsible for when you delete your whole file system, therefore this way.)

RM=$($WHICH rm);export RM
$FIND "$($PWD2)" -regextype ... ;$RM -f -v "${xox}"' \; 

Hope this helps.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.