4

So I am running Hadoop in pseudo distributed mode on my laptop, to learn about these things.

I am setting the thing up as explained in the manual.

I can start hadoop, no problem, but the launch process is inconvenient. See, to launch the thing, one does:

start-dfs.sh

Then, one is prompted for password 3 times (in my case all three are the same):

Starting namenodes on [localhost]
me@localhost's password: 
localhost: starting namenode, logging to /home/me/hadoop-2.7.3/logs/hadoop-me-namenode-me-HP-ENVY-15-Notebook-PC.out
me@localhost's password: 
localhost: starting datanode, logging to /home/me/hadoop-2.7.3/logs/hadoop-me-datanode-me-HP-ENVY-15-Notebook-PC.out
Starting secondary namenodes [0.0.0.0]
me@0.0.0.0's password: 
0.0.0.0: starting secondarynamenode, logging to /home/me/hadoop-2.7.3/logs/hadoop-me-secondarynamenode-me-HP-ENVY-15-Notebook-PC.out

(and same when one does stop-dfs.sh).

This is a pain. Is there a way to have ubuntu remember that password so that one doesn't have to type it at all?

2

Solved my problem using the steps described in this S.O. answer. Basically, do:

ssh-keygen -t rsa -P ""
cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
ssh localhost

then

start-dfs.sh

will not ask for passwords.

2
+50

I would also try the following (if you have an hdfs user):

sudo -u hdfs start-dfs.sh  service-name

This way you would enter password once, but consequent calls should be cached.

0

One idea is to launch the process in the background, redirecting its stdout (stderr?) to a file, and have your script wait and monitor that file in a loop. Whenever the process prints the password prompt, the script feeds the password into the process via its stdin.

Achieving this is actually tricky. You can use two scripts, one for monitoring the output and feeding the passwords, and another one for launching them first, applying the necessary redirects.

So the launcher script is:

#!/bin/bash
TEMP_FILE=/tmp/hapboon.stdout
start-dfs.sh  <(./feed-passwords.py "$TEMP_FILE")  > "$TEMP_FILE"

For whatever reason, I'll use Python for the other one:

#!/usr/bin/python3
PASSWORD="""TYPE_YOUR_PASSWORD_HERE"""

from sys import argv
temp_file = argv[1]  # Raises exception if argument is missing

counter = 0
maximum = 3
previous_length = 0

while True:
    with open(temp_file) as f:
        output = f.read()
    if len(output) == previous_length:
        continue
    previous_length = len(output)
    last_line = output.strip().split('\n')[-1]
    if last_line.endswith(' password:'):
        print(PASSWORD)  # Our stdout goes to `start-dfs.sh`
        counter += 1
    if counter == maximum:
        break

I haven't tested any of this. One assumption this approach relies on is that the one who recieves the password via its stdin is start-dfs.sh itself, which has a pretty good possibility to be untrue. Start scripts often just invoke other processes, so the password likely needs to be sent to the other processes' stdin.

You might have to tinker with the start-dfs.sh script itself, and have it redirect the stdin of the processes that will recieve the password. This also implies that our feed-passwords.py script must print the passwords into the appropriate file each time, which likely means the architecture of the whole approach would change.

Also, while we're still not there yet, one more thing about this script that is is inflexible; it only works with a preset number of passwords (defined by the maximum variable). It might be desirable to modify it such that it keeps monitoring for password prompts until a certain timeout (say, 15 seconds) has passed, thus accomodating to any number of password prompts without the need to change the number every time. (I can help you with that in case it is needed.)

Another note: If you don't want to store your password, just replace the PASSWORD=... line with: PASSWORD=input("Type your password: "). This will have you input your password only once every time you launch the script, and will not store your password anywhere.

  • I have to go now but will check this asap! Thanks, sorry, seasonal greetings! – user2413 Dec 23 '16 at 12:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.