Hot answers tagged

29

First you need to have a running Zookeeper server-instance. Open a terminal, and start the Zookeeper server: /usr/share/zookeeper/bin/zkServer.sh start-foreground Note: Zookeeper will not start unless it can write to it's datadir and logfile. Open another terminal, and replace zkserver1 with the hostname\ip eg. 127.0.0.1 : echo ruok | nc zkserver1 2181 ...


8

If you have done a ssh-copy-id like: remote='user@machine' ssh-copy-id -i $remote So you can access this remote machine without using a password: ssh $remote To undo it programmatically, you can script something like: idssh=$(awk '{print $2}' ~/.ssh/id_rsa.pub) ssh $remote "sed -i '\#$idssh#d' .ssh/authorized_keys" I use it in scripts I need to ...


7

I was able to successfully change my ulimit real time priority and memlock by doing this: Open /etc/security/limits.conf and add these lines (replace 'tyler' with your Ubuntu user name) tyler hard rtprio 95 tyler soft rtprio 95 tyler hard memlock unlimited tyler soft memlock unlimited Open /etc/pam.d/su and make sure the following line is present, and not ...


7

You are using the wrong command. To check hadoop version, type hadoop version in the terminal. Here is the sample output $ hadoop version Hadoop 2.4.1 Subversion http://svn.apache.org/repos/asf/hadoop/common -r 1604318 Compiled by jenkins on 2014-06-21T05:43Z Compiled with protoc 2.5.0 From source with checksum bb7ac0a3c73dc131f4844b873c74b630 This command ...


6

Open the /etc/ssh/ssh_config and remove all the listed options. They are supposed to be part of the sshd_config (the server configuration), not the client one.


5

Actually I just delete the folder hadoop which given permission(shown below) to my HDFS user. $ sudo rm -r -f /usr/local/hadoop/ Previously I extracted hadoop-2.4.0 and given permission for my HDFS user(hadoopuser - HDFS user), $ sudo chown -R hadoopuser:hadoop hadoop After that I removed user(hadoopuser) and group(hadoop) root@ARUL-PC:~# deluser ...


5

for putting files on hadoop use hadoop fs -put /<local machime path> /<hdfs path> and to get files from hadoop use hadoop fs -get /<hdfs path> /<local machime path> for more information see this


5

You can use hdfs fsck utility to locate the name of the block and then you can manually find it in the local filesystem: $ echo "Hello world" >> test.txt $ hdfs dfs -put test.txt /tmp/ $ hdfs fsck /tmp/test.txt -files -blocks /tmp/test.txt 12 bytes, 1 block(s): OK 0. BP-1186293916-10.25.5.169-1427746975858:blk_1075191146_1451047 len=12 repl=1 ...


5

The reason for connection refused is simple - the port 9000 is NOT being used (not open). Use the command => lsof -i :9000 to see what app is using the port. If the result is empty (return value 1) then it is not open. You can even test further with netcat. List on port 9000 in terminal session 1 nc -l -p 9000 In another session, connect to it $ lsof -...


4

Run following command on terminal to find it : find hadoop/ -name hadoop-env.sh Replace hadoop/ with your hadoop folder name


4

You cannot directly browse HDFS from terminal using cat or similar commands. HDFS is a logical file system and does not directly map to Unix file system. You should have an HDFS client and your Hadoop cluster should be running. When you browse HDFS, you are getting your directory structure from namenode and actual data from datanodes. Although you cannot ...


4

OP posted: I've resolved that in this way: after to add the new environment i run these commands: juju sync-tools -e maassvr02 juju bootstrap -e maassvr02 --to node4.maas --debug


4

It is not a problem with your installation. Hadoop 2 uses YARN framework i.e. MapReduce 2.0 (MRv2). Refer Hadoop Yarn for more details. So, start-mapred.sh is not required. To start Hadoop, use start-all.sh or start-dfs.sh and start-yarn.sh. YARN’s original purpose was to split up the two major responsibilities of the JobTracker/TaskTracker into separate ...


4

In your ~/.bashrc, instead of writing export HADOOP_INSTALL=/usr/local/hadoop change it to export HADOOP_INSTALL=/usr/local/hadoop/bin/hadoop I was facing the same issue for a long time. This worked for me. Or try this: /usr/local/hadoop/bin/hadoop namenode -format


4

A number of companies now produce small "laptop in a box" PCs that can be used as small scale internal servers running Ubuntu, etc. A quick check reveals that the Intel NUC, Gigabyte BRIX, ASUS Vivo, and Zotac ZBOX are all pretty strong contenders. These boxes all come with Intel i5 or i7 multi-core CPUs and can have up to 32GB of RAM and support M.2 and 2....


3

Check this log file: hadoop-hduser-namenode-ratan-Inspiron-N5110.log If it says namenode is not formatted, format it bin/hadoop namenode -format


3

To add a user to the sudoers the correct syntax is: hduser ALL=(ALL:ALL) ALL If you want to add a group is: %hdgroup ALL=(ALL:ALL) ALL


3

The usual location for the configuration and environment is in /etc/hadoop. Example: hadoop-2.5.1//etc/hadoop/hadoop-env.sh


3

Derived from @Luis Alvarado's answer, here is my version for Ubuntu 14.04 and Hadoop 2.5.1 In brief Install Java Prepare an executive user for Hadoop hduser Switch to hduser from now on Allow hduser to remote via ssh with pass-phrase-less Disable IPv6 Download and config Hadoop package Prepare system path $HADOOP_HOME and $JAVA_HOME Config Hadoop's ...


3

The tutorial step 2 asked to create a new user: hduser. So if you decided to adopt this username for the Hadoop user, the path of start-all.sh should be: cd /home/hduser/hadoop/bin/ start-all.sh Or if installed system-wide: /usr/lib/hadoop/bin/


3

On reading this blog, I came to know this was not issue more of broken packages. To resolve this broken package error: Uninstall previous Zookeeper: $ sudo apt-get remove zookeeper $ sudo apt-get purge zookeeper Fresh install Zookeeper $ sudo apt-get install zookeeper Install Zookeeper Server $ sudo apt-get install zookeeper-server


3

did you clean up zookeeper data directory from previous trials ? by default zookeeper.znode.parent is /hbase , options you can try if you are changing it intentionally then changing zookeeper data directory when changing it. In /home/hduser/zookeeper try changing znode.parent to /hbase


3

You need to rename the mapred-site.xml.template file to mapred-site.xml. (in hadoop-2.7.0/etc/hadoop) For this, you can use: mv mapred-site.xml.template mapred-site.xml and then make changes in the configuration file mapred-site.xml as required.


3

Probably your .bashrc or .profile is messedup. Your gedit will not start if it is not in path and since your path is empty it is not in the path. You should first establish a good enough path. export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin And then you should be able to start gedit. Alternativelly you could just run gedit with ...


3

Source and destination users are different, and that should be the reason of this. Would suggest you trying to start Hadoop daemons without sudo. ssh is actually from one user@host to another user@host pair. When you skip target user in your ssh command it is assumed to be same as the source user. But here your source user becomes root as you are using sudo ...


3

Executive summary: It seems like you are trying to copy the files in non-existed HDFS folder Detailed Answer: There is a huge difference between HDFS file-system and regular file-system. HDFS file-system - Hadoop Distributed File System (HDFS) is designed to reliably store very large files across machines in a large cluster. The file system is ...


3

You should take a look at this answer which describes configuring ssh for your user (editing ~/.ssh/config) and other details. The steps are: Generate your ssh key. Add the Host to your ~/.ssh/config file. Add your Public (.pub) key to the remote user's ~/.ssh/authorized_keys This is most easily done with ssh-copy-id command. ssh is very particular about ...


2

DataNode attempts to start but then shuts down. It can be checked by hadoop datanode -start. The problem is due to Incompatible namespaceID.So, remove tmp directory using commands sudo rm -Rf /app/hadoop/tmp Then follow the steps from: sudo mkdir -p /app/hadoop/tmp http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-...


2

Add entry 127.0.0.1 for localhost in /etc/hosts file. 127.0.0.1 localhost


2

You have to add the user hadoopz to the sudoers list. sudo adduser hadoopz sudo Logout and login back to take effect. Reference


Only top voted, non community-wiki answers of a minimum length are eligible