How to Write a Shell Script to Move Files from FTP Server to Network Share Drive

Question

I have customers who upload large amounts of data to my FTP server (an Ubuntu 9.10 machine). Once the files are uploaded, I am faced with transferring them to a local Droboshare network drive so our technical support staff can retrieve them and analyze the issue.

After I mount the network share at /media/<sharename> I run this command manually to get these files:

sudo mv /home/ftp/<data_dir>/<file_name> /media/<network_share>/<ftp_user_data>

but this takes forever and if there are tens or hundreds of files, I can't realistically do this over and over again. I thought it would be nice to have a script that I could run periodaclly to transfer a group of these files to the network share. I can sort them as the technical support staff uses them.

I am still new to Linux and writing shell scripts. Anyway to do this easily? Just to add to the description here, the files could be ZIP files, FASTA files, TAR.GZ files, and/or TXT files. Also, if the ZIP files are large, certain zipping programs convert these to ZIP.001, ZIP.002, ZIP.003, etc... So the file type in this FTP directory could be quite varied.

I was thinking the way I can identify these files is by simply transferring ALL files in the directory (although this may take a while) or to somewhat designate a subset of these, according to time completed or something like this.

I'm open to ideas. Thanks in advance.

Answer 1

Why not just make the FTP folder the actual mount point for the network drive? I run into a similar issue where I'm constantly backing up VirtualMachines on a Linux machine to a Windows network share (since the majority of the company infrastructure is Windows). This is my structure:

/media/windows-share is my mount point

For continuity I've created a symlink in my backup application:

/opt/backup/mnt so if the mount point changes in the future my program doesn't need an update.

Lastly I employ a cool little tool called autofs (sudo apt-get install autofs) That guide is decent (and up to date) - though I've employed this with a slightly different approach on my blog.

In your case I would create a symlink where my FTP dropbox is (/home/ftp/<data_dir>/) to the appropriate network folder in /media/<network_share>/<ftp_user_data> that way there's no wasted time in transferring across disks and the files are available immediately.

Answer 2

For duplicating files from here to there (or there to here), rsync is a swiss army chainsaw which will likely do what you want.

Answer 3

You could try to process files as soon as they're uploaded to the FTP, using a technology sucha s inotify. inoticoming is a tool you can use for that, see http://manpages.ubuntu.com/manpages/lucid/man1/inoticoming.1.html.