I was trying to configure Hadoop by using the public IPs of amazon instances instead of their intra-network IPs as my aim is to create a hybrid cluster i.e. Cloud + Local machine cluster. Although all the ssh settings are perfectly fine, Hadoop is still not able to connect when amazon public IPs are used (the datanodes cannot find the namenode). I had used amazon instance's public IP in the zookeeper configuration of hbase and it properly connected to it. So why does HBase connect and Hadoop doesn't?
The same problem was with kafka too.