2

I am trying to install hadoop via script on my small cluster [introduced in the book: Apache Hadoop Yarn]. I have set up ssh connect without password, and pdsh&pdcp are all fine if I use it alone. The problem is when I put the command in script, pdsh and pdcp fail. I really don't understand where goes wrong.

works when use it standalone:

pdcp -R ssh -w ^all_hosts hadoop-2.5.1.tar.gz /opt

here is part of the script that doesn't work: Whole script click here

#!/bin/bash
HADOOP_VERSION=2.5.1
install()
{
    echo "Copying Hadoop $HADOOP_VERSION to all hosts..."
    pdcp -R ssh -w ^all_hosts hadoop-"$HADOOP_VERSION".tar.gz /opt
....

all_hosts file is like this (hostname):

dave

micheal

jose

the error is:

micheal: Host key verification failed.
pdcp@dave: micheal: ssh exited with exit code 255
jose: Host key verification failed.
pdcp@dave: jose: ssh exited with exit code 255
dave: Host key verification failed.
pdcp@dave: dave: ssh exited with exit code 255

2 Answers 2

1

The "Host key verification failed" is from the underlying SSH process. See pdcp(1):

When using ssh for remote execution, stderr of ssh to be folded in with that of the remote command. When invoked by pdcp, it is not possible for ssh to prompt for confirmation if a host key changes, prompt for passwords if RSA keys are not configured properly, etc..

A solution would be to connect to all hosts in all_hosts once before and to accept their host keys or to disable (!) StrictHostKeyChecking for said hosts:

$ cat ~/.ssh/config 
Host dave micheal jose
        StrictHostKeyChecking   no

$ pdcp -R ssh -w ^all_hosts mate-terminal /opt
sid0: Warning: Permanently added 'micheal,192.168.0.130' (ED25519) to the list of known hosts.
[...]
1
  • You can also use Host * to set it for all hosts
    – Qin Heyang
    Commented Aug 24, 2022 at 17:26
1

Since this error comes from the underlying SSH process, we can actually pass an option to it to either automatically add the host to the known hosts or ignore the host key checking this one time. You can use the PDSH_SSH_ARGS_APPEND environment variable to pass these options to the underlying process.

PDSH_SSH_ARGS_APPEND="-o StrictHostKeyChecking=no" pdcp -R ssh -w ^all_hosts hadoop-2.5.1.tar.gz /opt

This environment variable works for straight pdsh as well.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .