1

I've hunted high and low and have not been able to find an answer to my problem. I'm not sure if this is the proper forum, but I hope someone here might be able to help.

I have been using wget (FTP) to backup my website to my local HD, using the mirror option, with no problem, until recently. Recently, a directory that holds images reached the limit, set by the hosting service, for how many items can be in a FTP listing. I do not have SSL access to the account.

I can run a script, on the remote server, and get a full directory listing (ls), in a file, and download it to my local machine.

Is it possible to feed this file to wget and have it mirror those files to my local drive, as it did prior to hitting the listing limit? I would like wget to read the generated list and check for changed and added files. Here are the questions I have been unable to find answers to:

1) What would be the proper ls command to generate the needed information? Currently, I'm only getting the actual filenames, but I'm sure that more is needed.

2) Will wget with mirror option still work with an input file, or will it just download all files in the input file without checking for changed status? I have found absolutely no information on this combination - only on one or the other. I don't want to test and wind up getting a full backup of the directory - waste of bandwidth and time to pull all that data down.

1
  • Do you have ssh access or are you limited to FTP commands?
    – terdon
    Jun 16, 2015 at 15:21

1 Answer 1

0

You can do this, the only problem is getting an output from ls that is the same on the remote and on your local machine. My ls has an option --time-style=+%s that shows the last modified time of the file in seconds since the epoch, which should match on both machines. I'm assuming you only want to do this for one directory, otherwise find would be more appropriate if you have that command.

On the remote do the ls in the required directory into a file:

cd ... # cd to the directory to list
ls -l --time-style=+%s . >/tmp/listfile

On the local machine, copy the remote /tmp/listfile to local /tmp/listfile, then use awk to suppress the first 4 columns of output and sort the list of file size, date and filename. (I'm assuming you might have filenames with spaces hence the bizarre awk).

awk '{$1="";$2="";$3="";$4="";print}' </tmp/listfile | sort >/tmp/a

In the mirrored directory get the same listing:

cd ... # cd to same directory we listed remotely
ls -l --time-style=+%s . | awk '{$1="";$2="";$3="";$4="";print}' | sort >/tmp/b

Compare the two sorted lists and drop the lines that are the same with comm. Drop the size and date columns (and incidentally the tab at the start of lines in /tmp/b not in /tmp/a, and 2 inter-column spaces) and drop duplicate filenames with uniq. (You dont need to sort again as the duplicate lines are adjacent).

comm -3 /tmp/a /tmp/b | awk '{$1="";$2="";print substr($0,3)}' | uniq >/tmp/c

The list in /tmp/c is all files that are new, of different size or timestamp, or have been removed. Pass each one to wget to fetch (still in the appropriate directory).

while read file
do  rm -f "$file"
    wget -N --no-directories ..../"$file" # non recursive single file to .
done </tmp/c

You should configure your wget mirror to no longer fetch this directory.

If you can do find ... -printf '%9s %T+ %p\n', you could use this sort of technique, starting at the top of the mirror directory, to effectively do the mirroring yourself.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .