0

Can anyone help me in installing hadoop on my ubuntu vm . I am struck in formatting the namenode step I have tried all possible steps but I'm not able to understand where I have made the mistake. The following is the error I am getting while formatting .

user@ubuntu:/lib/hadoop/hadoop-2.7.1/bin$ hadoop namenode -format

DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

/lib/hadoop/hadoop-2.7.1/bin/hdfs: line 304: /lib/jvm/jdk1.8.0_111/bin/java: Permission denied
/lib/hadoop/hadoop-2.7.1/bin/hdfs: line 304: exec: /lib/jvm/jdk1.8.0_111/bin/java: cannot execute: Permission denied

And when i am going to start the namenodes it is telling me the following error and repetedly asking me the password.

user@ubuntu:/lib/hadoop/hadoop-2.7.1/sbin$ start-all.sh

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
/lib/hadoop/hadoop-2.7.1/bin/hdfs: line 304: /lib/jvm/jdk1.8.0_111/bin/java: Permission denied
/lib/hadoop/hadoop-2.7.1/bin/hdfs: line 304: exec: /lib/jvm/jdk1.8.0_111/bin/java: cannot execute: Permission denied
Starting namenodes on []
user@localhost's password: 
localhost: starting namenode, logging to /lib/hadoop/hadoop-2.7.1/logs/hadoop-user-namenode-ubuntu.out
localhost: /lib/hadoop/hadoop-2.7.1/bin/hdfs: line 304: /lib/jvm/jdk1.8.0_111/bin/java: Permission denied
localhost: /lib/hadoop/hadoop-2.7.1/bin/hdfs: line 304: exec: /lib/jvm/jdk1.8.0_111/bin/java: cannot execute: Permission denied
user@localhost's password: 
localhost: starting datanode, logging to /lib/hadoop/hadoop-2.7.1/logs/hadoop-user-datanode-ubuntu.out
localhost: /lib/hadoop/hadoop-2.7.1/bin/hdfs: line 304: /lib/jvm/jdk1.8.0_111/bin/java: Permission denied
localhost: /lib/hadoop/hadoop-2.7.1/bin/hdfs: line 304: exec: /lib/jvm/jdk1.8.0_111/bin/java: cannot execute: Permission denied
/lib/hadoop/hadoop-2.7.1/bin/hdfs: line 304: /lib/jvm/jdk1.8.0_111/bin/java: Permission denied
/lib/hadoop/hadoop-2.7.1/bin/hdfs: line 304: exec: /lib/jvm/jdk1.8.0_111/bin/java: cannot execute: Permission denied
starting yarn daemons
starting resourcemanager, logging to /lib/hadoop/hadoop-2.7.1/logs/yarn-user-resourcemanager-ubuntu.out
/lib/hadoop/hadoop-2.7.1/bin/yarn: line 330: /lib/jvm/jdk1.8.0_111/bin/java: Permission denied
/lib/hadoop/hadoop-2.7.1/bin/yarn: line 330: exec: /lib/jvm/jdk1.8.0_111/bin/java: cannot execute: Permission denied
user@localhost's password: 
  • 1
    What is your ubuntu version and it appears the script your using is deprecated, where did you get it ? – George Udosen Jan 12 '17 at 17:05
0

I just ran this on a VM (Kubuntu 16.04) with hadoop version 2.7.3.

Prerequisites:

  1. Ubuntu OS
  2. Non-root user with sudo privileges
  3. Installed java

Steps:

  1. Download hadoop version from here by

    • Clicking the binary option under the tarball column for desired version
    • Click on the link under We suggest the following mirror site for your download: or right click and save link

      • if the link was saved then download using:

        wget http://www-us.apache.org/dist/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz

        enter image description here

  2. Check the integrity of the download:

    • Download .md5 file from download page enter image description here enter image description here enter image description here

    • Run shasum -a 256 hadoop-2.7.3.tar.gz and compare result with cat hadoop-2.7.3.tar.gz.mds, at this line

      ...
       hadoop-2.7.3.tar.gz: SHA256 = D489DF38 08244B90 6EB38F4D 081BA49E 50C4603D B03EFD5E 594A1E98 B09259C2
      ...
      
  3. Install hadoop:

    • Decompress and move:

      tar -xzvf hadoop-2.7.3.tar.gz && sudo mv hadoop-2.7.3 /usr/local/hadoop
      
    • locate java on your system with:

      readlink -f /usr/bin/java | sed "s:bin/java::"
      # result
      /usr/lib/jvm/java-8-openjdk-amd64/jre/
      
    • edit hadoop-env.sh file sudo nano /usr/local/hadoop/etc/hadoop/hadoop-env.sh:

      • Option 1 (static value):

        #export JAVA_HOME=${JAVA_HOME}
        export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/
        
      • Option 2 (dynamic value changes with change in java on system):

        #export JAVA_HOME=${JAVA_HOME}
        export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::")
        
  4. Run hadoop:

    • /usr/local/hadoop/bin/hadoop

      • Output:

        Usage: hadoop [--config confdir] [COMMAND | CLASSNAME]
        CLASSNAME            run the class named CLASSNAME
        or
        where COMMAND is one of:
        fs                   run a generic filesystem user client
        version              print the version
        jar <jar>            run a jar file
                             note: please use "yarn jar" to launch
                                  YARN applications, not this command.
        checknative [-a|-h]  check native hadoop and compression libraries availability
        distcp <srcurl> <desturl> copy file or directories recursively
        archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive
        classpath            prints the class path needed to get the
        credential           interact with credential providers
                             Hadoop jar and the required libraries
        daemonlog            get/set the log level for each daemon
        
  5. Test:

    mkdir ~/input
    cp /usr/local/hadoop/etc/hadoop/*.xml ~/input
    
    /usr/local/hadoop/bin/hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar grep ~/input ~/grep_example 'principal[.]*';
    
    • Check for errors in the result:

       File System Counters
              FILE: Number of bytes read=1247674
              FILE: Number of bytes written=2324248
              FILE: Number of read operations=0
              FILE: Number of large read operations=0
              FILE: Number of write operations=0
       Map-Reduce Framework
              Map input records=2
              Map output records=2
              Map output bytes=37
              Map output materialized bytes=47
              Input split bytes=114
              Combine input records=0
              Combine output records=0
              Reduce input groups=2
              Reduce shuffle bytes=47
              Reduce input records=2
              Reduce output records=2
              Spilled Records=4
              Shuffled Maps =1
              Failed Shuffles=0
              Merged Map outputs=1
              GC time elapsed (ms)=61
              Total committed heap usage (bytes)=263520256
        Shuffle Errors
              BAD_ID=0
              CONNECTION=0
              IO_ERROR=0
              WRONG_LENGTH=0
              WRONG_MAP=0
              WRONG_REDUCE=0
       File Input Format Counters
              Bytes Read=151
       File Output Format Counters
              Bytes Written=37
      

Source:

https://www.digitalocean.com/community/tutorials/how-to-install-hadoop-in-stand-alone-mode-on-ubuntu-16-04

Add path to .bashrc:

# hadoop executable
export PATH="/usr/local/hadoop/bin:$PATH"
| improve this answer | |
0

It seems you have missed some steps related to over riding the permission setting in hadoop configuration. But it shouldn't be a problem. It will ask for password for each component in Hadoop, so from your screenshot, 3times paswword have been asked, one for namenode, then the datanode and then the resource manager. It will ask another 2 times, one for secondary namenode and the other for Node Manager. Once you are done with this, the rest should follow smoothly.

Please refer the below link:

Detailed Step-by-Step Installation of Hadoop on Ubuntu, with detailed screenshots and Explanation:

https://drive.google.com/file/d/0B0YcHWv8cfAjdzAxbjlReUNLWUU/view

This document has been created by me after installing several versions of Hadoop on several versions of Ubuntu. So, most of the common problems/errors/queries that a person faces while installing Hadoop on Ubuntu has been discussed in the above document.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.