How to run a job in several machines that have Ubuntu as operating system

Question

I need to run a job in several virtual machines exists in a specific cloud and I have the IP address for all of them and I configured the hostname for all of them. How to run a command in a network of virtual machine ? note that the script of this job divides the job into 4 steps to run in parallel since each step needs 2 thread that run in parallel on one machine, now I need to use more than one machine to run the same script in which to let a threads to perform the parallelism using all theses machines by using this command:

./script.py '100 digits number' slaves.hostnames=hostname1,hostname2,hostname3 --slaves 4 --client-threads 2

The above number is number that I need to factor using all these machines to decrease the overall execution time.

Answer 1

There are several tools for having a kind of "parallel ssh". As you use Python, pssh should be fine. Install it with pip:

$ sudo pip install pssh  

Create a pssh hosts file like this:

192.168.0.10:22
192.168.0.11:22

and then invoke:

$ pssh -h pssh-hosts -l root -A echo "Hi!"

You will asked for your password, but you can use certificates to avoid manual entering of passwords.

See more:

http://www.tecmint.com/execute-commands-on-multiple-linux-servers-using-pssh/

One important note: If you want to solve a problem in parallel, you may need MPI (Message Passing Interface). I suggest MPICH. MPI provides a standard means of communication to solve a problem collaboratively with multiple machines in parallel. Although, writing fast applications to find prime numbers/factors is nothing new.

If you have a multicore machine, use multithreading. But if you are going to use multiple computers, you will need MPI.