Questions tagged [hadoop]

Apache Hadoop is a software framework that supports data-intensive distributed applications under a free license. It enables applications to work with thousands of computational independent computers and petabytes of data. Hadoop was derived from Google's MapReduce and Google File System (GFS) papers.

Filter by
Sorted by
Tagged with
1 vote
1 answer
2k views

hadoop command is not working

I am trying to install single node hadoop on my ubuntu 14.04 as per the documentation available at bogotobogo.com. after following all the steps when I tried to format the hadoop file system it shows ...
Mohd Maaz's user avatar
  • 113
0 votes
1 answer
278 views

Error Installing Bigtop - E: Sub-process /usr/bin/dpkg returned an error code (1)

I'm facing the following error when trying to install the Bigtop distribution of Hadoop. Errors were encountered while processing: /var/cache/apt/archives/oozie_4.3.0-1_all.deb E: Sub-process /usr/...
mastelaro's user avatar
0 votes
1 answer
4k views

hadoop-datanode is not running

I am new to hadoop and did installation hadoop-2.7.3.Also completed all the steps for installation.however my datanode is not running after ran the command start-all.sh. So my doubt is what action ...
user721514's user avatar
1 vote
1 answer
2k views

Shell script does not work on crontab but works otherwise

I have a script which merges all the small files present at hourly directories at hdfs into one single large file. The script works OK when executed through CLI. I then set the script to run at 01:30 ...
earl's user avatar
  • 113
4 votes
1 answer
10k views

"ssh localhost " command is showing 20 bad configuration options

hduser1@archit-HP-Notebook:~$ ssh localhost /etc/ssh/ssh_config: line 11: Bad configuration option: hostkey /etc/ssh/ssh_config: line 12: Bad configuration option: hostkey /etc/ssh/ssh_config: line 13:...
Archit Jain's user avatar
0 votes
1 answer
149 views

hadoop fs -ls /usr/local is dealing my Hadoop installation folder as file

I am very much new to Hadoop. Recently I installed Hadoop-2.8.0 with the help of the blogs on internet. I thought everything is running fine, but today when I tried running hadoop fs -ls /usr/local ...
chandrapvs's user avatar
2 votes
0 answers
2k views

SLF4J: Class path contains multiple SLF4J bindings

I am trying to stream real time twitter data into HDFS using Apache flume. And when i run the command ./flume-ng agent -c /usr/local/apache-flume-1.4.0-bin/conf/ -f /usr/local/apache-flume-1.4.0-bin/...
user642283's user avatar
1 vote
1 answer
1k views

On Hadoop installation, bash: gives an 'is a directory error'

When I run the command: $ $HADOOP_INSTALL/etc/hadoop fs -mkdir /tmp it gives me the following error: bash: /etc/hadoop-2.7.1/etc/hadoop: Is a directory These are the contents of my .bashrc file: ...
user642283's user avatar
1 vote
0 answers
2k views

SSH asking password each time i run start-all.sh for hadoop

I have installed Hadoop on following path: /opt/hadoop Also installed openssh-server with following commands: sudo apt-get install openssh-server openssh-client After this I created RSA keys with ...
Girish Khole's user avatar
1 vote
1 answer
9k views

root@localhost's password: localhost: Permission denied, please try again

I am trying to setup Hadoop in "Pseudo-distributed Mode". I have already performed the below steps: I have installed Openssh-server. I have generated the rsa type key pair. Copied *rsa.pub to ...
Kashif Khan's user avatar
1 vote
0 answers
161 views

Installing OpenTSDB on Ubuntu 14

Can you please guide me for how to install OpenTSDB on ubuntu 14.04? And is hadoop Prerequisite of OpenTSDB?
sohaib1015's user avatar
4 votes
3 answers
5k views

Avoid entering password when launching Hadoop in Pseudo-distributed mode

So I am running Hadoop in pseudo distributed mode on my laptop, to learn about these things. I am setting the thing up as explained in the manual. I can start hadoop, no problem, but the launch ...
user2413's user avatar
  • 14.3k
1 vote
1 answer
1k views

How to access pyspark in anaconda installed in another user?

I have two user user1 and user2 with latter one having root privilege.I have installed spark and hadoop in user2. Now i have installed Anaconda2 on user1 and when iam trying to run my job on jupyter ...
bob's user avatar
  • 31
0 votes
1 answer
2k views

Unable to upload directory to hdfs. `/usr/local/tmp/': No such file or directory

I'm trying to upload files to a directory i made on HDFS but I'm not able to do so as ubuntu is not able to locate the files on my machine. The directory 'states'(one i'm trying to upload) contains 50 ...
Shradha Shiwani's user avatar
0 votes
0 answers
281 views

kylin.sh command

I have been trying to install Kylin and after I tried to execute the command Kylin.sh start, I get the following error. I can successfully run hadoop, hbase and hive from the root. Any kind of ...
harish's user avatar
  • 1
1 vote
0 answers
317 views

Error: java.io.IOException: Communications link failure

I am doing a small project to achieve auto-completion on ubuntu. I run the code in Docker with hadoop. I also use Ampps downloaded from http://ampps.com/download. But I have a problem that I can't ...
wangzishan's user avatar
3 votes
1 answer
9k views

echo $HADOOP_HOME is blank

I installed hadoop but when i am using echo $HADOOP_HOME .its showing blank.please help how to check hadoop location.
Veeresh Kumar's user avatar
1 vote
0 answers
253 views

How do I change the IP address of ubuntu(16.x) machine on oracle virtual box(5.0.20)

I'm trying to setup a hadoop multi node cluster on Ubuntu 16.x with oracle virtual box(5.0.20). As part of the setup I want to change the IP address of one Ubuntu machine. But the problem is, there ...
Jayant Mishra's user avatar
1 vote
0 answers
711 views

Installation process of Apache Nutch in Ubuntu 16.04 , hadoop 2.6 already installed

What is the correct compatible format of Apache Nutch for Ubuntu 16.04 and hadoop 2.6 ? What is the installation process ?
Aditya2956's user avatar
0 votes
1 answer
14k views

Editing the ~/.bashrc file for hadoop installation

I have been trying to install hadoop-2.6.0 in my Ubuntu 16.04 machine. I have been logged into the hduser from another user krishna by: sudo su hduser and the hduser then gets into its use. After ...
Krishna Chalise's user avatar
2 votes
0 answers
599 views

Ubuntu 16.04 /bin/kill bug?

I recently installed Ubuntu 16.04 on the virtual machine using VMware. To test my MapReduce code, I also installed hadoop 2.7.3 and setup a single node cluster. When I execute MapReduce algorithm, ...
Yoonjae Park's user avatar
2 votes
1 answer
2k views

namenode not running... i have already tried sudo chown -R username /usr/local/hadoop/ please help

$ /usr/local/hadoop/sbin/start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh Starting namenodes on [localhost] divyeshlad@localhost's password: localhost: starting ...
curious's user avatar
  • 33
1 vote
0 answers
504 views

Help needed for Hadoop to execute sbin/start-dfs.sh without being asked root login

I have followed some other answers but was unable to find something that works When I do sudo sbin/start-dfs.sh I get prompted for the root passwd I set in /etc/ssh/sshd_config The following ...
user6556673's user avatar
0 votes
1 answer
535 views

Type '--2016-08-04' is not known on line 1 in source list /etc/apt/sources.list [duplicate]

I want to install hadoop in my Ubuntu 14.04 system. Following the instruction of a blog, I need to update the "source list". But when I input the command: sudo apt-get update I get errors: E:...
zhangwp's user avatar
2 votes
1 answer
2k views

start-all.sh giving error of not resolving hostname and name or servce not known

i am setting up hadoop-2.7.2 on ubuntu 14.04 through VMware. After steps described in michael g. noll's tutorials , when i tried this, /usr/local/hadoop/sbin/start-all.sh i am getting this stuff, ...
bob's user avatar
  • 31
1 vote
1 answer
476 views

Why my .bashrc has error?

I added this path to .bashrc file . export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 export HADOOP_INSTALL=/usr/local/hadoop export $PATH=$PATH:$HADOOP_INSTALL/bin export $PATH=$PATH:$...
hersh's user avatar
  • 53
2 votes
1 answer
2k views

How to resolve UnsatisfiedLinkError installing Spark on Ubuntu 16.04?

I am trying to install Spark on Ubuntu 16.04 but it won't. What I have done is I have tried to install Spark with ./Spark but it is showing me the following error: Caused by: java.lang....
santosh kale's user avatar
-1 votes
1 answer
168 views

I tried installing the ssh_package into my ubuntu 16.04 but it is giving me this error. I am running ubuntu on VMWare workstation pro [closed]

rmehra@ubuntu:~/Desktop/ssh_package$ sudo dpkg -i openssh-client_1%3a5.3p1-3ubuntu7_i386.deb (Reading database ... 172254 files and directories currently installed.) Preparing to unpack openssh-...
user3620890's user avatar
2 votes
3 answers
3k views

move files in a hadoop dir with specified number of files

I wish to move files in a hadoop dir in a timely manner. The hadoop dir contains 1000 files with the same extension. I wish to move 100 of them every 10 minutes. I can setup a cron job to move the ...
GulshanS's user avatar
1 vote
1 answer
25k views

-bash: export: not a valid identifier [duplicate]

Whenever I open a terminal, I always end up with this same error. -bash: export: `/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/usr/...
James Wong's user avatar
4 votes
1 answer
9k views

Ubuntu 16.04 kills session when resource usage is extremely high

I'm working on project in hadoop on my computer. When I start some job consuming most of the available resources (100% CPU, high RAM usage) 'something' kills my user's session and all its processes. ...
michalo2882's user avatar
3 votes
0 answers
3k views

Install Cloudera and Hadoop on ubuntu 16

I am trying to find online materials to find the steps or guideline on installing cloudera and Hadoop. I found this and this page but none of them helped. If you know please help me on this as I am ...
Nirupa's user avatar
  • 151
0 votes
1 answer
2k views

HADOOP commands not working any where in ubuntu 16.04

Hadoop commands not working any where commands working hadoop/bin directory only once abserve first I have tried outside hadoop/bin second time I had done with in bin
Naveen Reddy's user avatar
0 votes
1 answer
3k views

Hadoop and Eclipse - Server IPC version 9 cannot communicate with client version 4

I'm working on Ubuntu and using Eclipse to work with Hadoop. After setting the Run configuration, a message says Incomplete HDFS URI, no host UPDATE: found the solution! but after running the program,...
Arwa's user avatar
  • 13
0 votes
1 answer
1k views

Problem when starting a single node of Hadoop cluster in Ubuntu

I am new to Hadoop and I was going through the installation process mentioned here. I have installed hadoop-1.2.1 on my system. After configuring everything when I tried to start a single node cluster ...
asad_hussain's user avatar
0 votes
1 answer
984 views

Hadoop - Error on binding to port 9000

I've been attempting to setup Hadoop on AWS for a few days now. I am on fresh EC2 Ubuntu instances with no other software installed apart from Java. I run these commands: ubuntu@ip-172-30-0-117:/...
BOMEz's user avatar
  • 115
1 vote
2 answers
10k views

Issues with disabling iPv6 on Ubuntu 14.0.4

To disable iPv6 on Ubuntu 14.0.4 (LTS) I added these lines in sysctl.conf file: net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1 Since I ...
Mr. Hobo's user avatar
  • 113
1 vote
1 answer
46 views

adduser fails when relating hadoop-plugin to hbase-master

I'm trying to set up Hbase in juju using the following charm: https://jujucharms.com/u/bigdata-dev/apache-hbase/trusty/19 I receive an error when I try to add a relation between the hadoop-plugin and ...
user3499430's user avatar
1 vote
2 answers
1k views

Hadoop 2.6 start-dfs.sh errors on Centos 6.7 [closed]

I use this tutorial to install Hadoop 2.6 on Centos 6.7 with Java 1.8.0_72 and everything goes well before execution of start-dfs.sh from Hadoop-home/sbin/srart-dfs.sh. Below is the output: [hadoop@10 ...
Espanta's user avatar
  • 129
0 votes
1 answer
897 views

Jobtracker and tasktracker not showing

I am using hadoop 2.7.1 on Ubuntu 14.04, when I run hadoop and use jps, the following processes are shown: Resourcemanager Namenode Datanode Jps SecondaryNameNode NodeManager Here job tracker and ...
Indresh Rajwade's user avatar
0 votes
1 answer
178 views

How to use amazon instances' public IPs in hadoop configuration?

I was trying to configure Hadoop by using the public IPs of amazon instances instead of their intra-network IPs as my aim is to create a hybrid cluster i.e. Cloud + Local machine cluster. Although all ...
Punit Naik's user avatar
1 vote
2 answers
15k views

Permission denied on start-dfs.sh

I am trying to create a 2-node Hadoop cluster, following this guide. The one node is my Ubuntu laptop and the slave node, the 2nd one, is a virtualbox that also runs Ubuntu. I am doing in the master ...
gsamaras's user avatar
  • 605
-1 votes
1 answer
37 views

Ubuntu and Windows [duplicate]

Currently I am working in Ubuntu. Big Data. I do not have Windows OS in my system. i am going to install hadoop in Ubuntu. But I don't know whether windows is compulsory to have along with ubuntu. Can ...
Nikitha JV's user avatar
0 votes
1 answer
864 views

Problem with cluster installation with cloudera manager

I am installing Cloudera Manager in my system(14.04/64b). While installing at the final step, before finish installation I got some ERRORs in validation as shown below, errors in above page are, ...
A J's user avatar
  • 11.4k
0 votes
1 answer
869 views

Hadoop Installation Problem

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh 15/11/07 16:01:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes ...
Nikhil V S's user avatar
1 vote
1 answer
499 views

Hadoop & Spark - why no Ubuntu packages?

I want to install Hadoop & Spark using aptitude, not the lengthy and complicated process described in How to install Hadoop?. hadoop-ubuntu appears to have been dead for almost 2 years. I wonder ...
sds's user avatar
  • 2,543
3 votes
1 answer
5k views

gedit - cannot open display

I am using Ubuntu 14.04 LTS, and for installing Hadoop I created hadoop user as hduser. When I try to open ~/.bashrc using gedit am getting an error as follows, hduser@arul-java ~ $ gedit ~/.bashrc (...
A J's user avatar
  • 11.4k
2 votes
1 answer
580 views

Moving hadoop folder into /usr/local/hadoop

I want to move the Hadoop installation to the /usr/local/hadoop directory, but it says- mv: cannot move ‘/home/pg/Desktop/hadoop-2.7.1’ to ‘/user/local/hadoop’: No such file or directory
pratibha's user avatar
0 votes
2 answers
2k views

Which IP to use for localhost in Ubuntu virtual box- installing hadoop

I am trying to install hadoop single node on a Ubuntu 14 Virtual machine( on VMWare) on a windows 7 host. One of the steps says : Get IP address using: ifconfig You get xxx.xxx.xxx.xxx Edit /...
Shuvadeep Chakraborty's user avatar
1 vote
1 answer
4k views

I forgot my localhost root password

I installed Hadoop on my machine. To start it, I was logging in as the user named hduser. I connected to the ssh port using ssh localhost command. Then I went to the bin folder of hadoop to start ...
Coolsam's user avatar
  • 11