How to copy files to multiple hosts across the network?

Question

I have more than 200 Ubuntu systems to which I want to copy a file.

How can I copy files to multiple hosts across the network? What possibilities do I have?

Note: This question is related to How to find the version of an Ubuntu machine remotely?

Answer 1

I realise you probably only want something simple but there are dozens of ways of doing this. But first we want to be able to control all the computers at once.

That might seem a strange direction to go but it'll save you time later on when you might want to send a shutdown, reboot, upgrade to all the computers.

clusterssh

Let's install clusterssh and fire off a command to connect to all our servers at once (this might not be viable for 200 at once - I don't have 200 computers to test it on!)

sudo apt-get install clusterssh
cssh user@first_computer:port user@second_computer:port ...

That will spawn little output windows for each connection (why I say 200 might be a stretch) and one global input dialogue. From there you can send the same command to all your servers.

If you want to do it from the command line, and this might support 200 connections, you can look at the -a flag for cssh. You can throw each server a command (or string of commands). It'll connect, execute and close the connection.

cssh -a 'cd directory; ls'  user@first_computer:port user@second_computer:port 

You can simplify your connections by using clusterssh's powerful configuration. Open man cssh and head down to the FILES section. It explains how you can create an /etc/clusters file to allow you to quickly create groups (called tags) like this:

clusters = <tag1> <tag2> <tag3>
<tag1> = host1 host2 host3
<tag2> = user@host4 user@host5 host6
<tag3> = <tag1> <tag2>

Then you can do:

cssh -T 'tag2' -a 'cd directory; ls'

Use a central repository

I don't mean a deb repo (but that's possible too, I guess) rather something like git or brz. It could be something as simple as a web server or a network share. If network bandwidth has bottlenecks in places, something like a torrent server might even be the best course (that's how Facebook updates all their nodes)

Tell your network to download the files

From here it's just connecting the dots. Using cssh you'd just tell all your computers to fetch the file and do whatever needs doing with it.

I suggest you start small. Get the file shared and then try a simple cluster of a couple of computer and see how it goes. Expand that out.

I suggest that even once you're done you keep a cluster of a couple of computers for testing before you push things to the main group.

Answer 2

Alternatively you can keep it push-based. The easiest way of that is writing a script that:

1. Iterates through a list of servers

2. Connects to the server and copies the files (two options here):

   1. Uses sshfs to mount the remote filesystem locally, cp/rsync the files, fusermount -u the directory you mounted it on (to unmount the remote system). Continue.

   2. Just use rsync's network abilities to copy the files over.

      rsync /localdir/ user@remote:/remotedir
      

Answer 3

[Oli's answer came on while I was typing]

An alternative to clusterss is parallel-scp. It is part of the pssh package

You need to provide the list of hosts in a text file (e.g. hosts.txt) and then you issue:

parallel-scp -h hosts.txt file2copy.dat /target/directory

Consult the man page to suite your particular needs. This shouldn't hit limits for too many connections since it doesn't require to open live connections to all as cssh does it will try this sequentially.

PS: Having key-based authorization set up will of course make this a lot easier but I suspect having your key distributed to 200 systems in the 1st place might be why you're asking this question ...