1

I am developing an API in which I need the paths of multiple files from a remote server. The number of files varies from 100 - 500. The files are in different folders.

So, I am looping over and over again, like 10-50(depending on number of files) times, and then getting paths using ssh in my python api.

But I want an optimized solution for this problem. Right now, I am making ssh connection as many times as loop goes, which is slow, and also not the best thing to do.

I was thinking of copying the /var/lib/mlocate/mlocate.db of remote server daily in my local machine and then find the path using locate command using this db, If that is possible. OR Like storing the remote directory index in my local machine which I can query more fastly.

What are other better ways to achieve this?

  • I would suggest looking at tree in general. There is also this perl script over at UNIX SE that gets a whole directory tree in a parse-able file. Maybe that is a good starting point – Robert Riedl Feb 23 '18 at 17:08
  • tree -if /your/path/here seems a good way to achieve this – Robert Riedl Feb 23 '18 at 17:09
  • Jut type locate --help and read about the -d option. :-) – pa4080 Feb 23 '18 at 17:37
  • locate might not be a good solution, since typically it is outdated since it is run via cron-job once a day – Robert Riedl Feb 23 '18 at 18:27
  • Are NFS or samba available options? They are meant for remote file access which is what you are doing. – rtaft Feb 23 '18 at 19:48
2

tree

It seems tree is very fast at producing a list of all file paths. With SSH, and in the context of your script, a command could look like this

ssh user@host "tree -if --noreport /your/base/directory" > /tmp/tree.output

-i to have this batch-like output instead of the neat graphical tree
-f to print the full path prefix for each line
--noreport to cut off the last two lines since tree appends an empty line and a summary of files/directories found

The output would look like this

/var
/var/backups
/var/backups/alternatives.tar.0
/var/backups/alternatives.tar.1.gz
/var/backups/apt.extended_states.0
/var/backups/apt.extended_states.1.gz
/var/backups/apt.extended_states.2.gz
/var/backups/aptitude.pkgstates.0
/var/backups/dpkg.arch.0
/var/backups/dpkg.arch.1.gz

You might want to have a look at the manpage here, to see some options on how symbolic links are displayed with -l, etc


find

find could also be used for the same purpose

ssh user@host "find /your/base/directory" > /tmp/find.output

again, you might want to check how you deal with symbolic links in the man page.
With find you can also granulate in regards to types of files, patterns, depth etc...


  • Is tree better than find for this purpose? – wjandrea Feb 23 '18 at 18:29
  • @wjandrea, tree seems faster, but find is certainly also an option - I will update my answer – Robert Riedl Feb 23 '18 at 18:35
  • 1
    One important advantage of find over tree (which is a great tool though, no doubt) is that find should be preinstalled pretty much on every system, whereas tree usually needs to be installed first, which often restricts its usage to machines where one has admin rights. – Byte Commander Feb 23 '18 at 20:18
  • @bytecommander, true...true.. but I guess since locate is an option, installing software is not an issue for OP – Robert Riedl Feb 23 '18 at 20:20

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.