HADOOP - Installation setup

Prerequisites

Hadoop requires a working Java 1.5+ (aka Java 5) installation. However, using Java 1.6/1.7 (aka Java 6/7) is recommended for running Hadoop. Please refer to jdk installation instructions here.

Dedicated user for Hadoop system

A dedicated Hadoop user will help Hadoop installation from other software applications and user accounts running on the same machine.

 
umasarath@ubuntu:~$ sudo addgroup hadoop
umasarath@ubuntu:~$ sudo adduser --ingroup hadoop hduser
The above two commands will add "hduser" user and "hadoop" group.

Hadoop Installation

Download Hadoop from the Apache Download Mirrors and extract the contents of the Hadoop package to a location of your choice. The folder I chosen was the hduser home folder. Extract the downloaded file in /home/hduser folder and make sure the file should be extracted in hduser login.

 
hduser@ubuntu:~$ sudo tar xzf hadoop-1.2.1.tar.gz
hduser@ubuntu:~$ sudo mv hadoop-1.2.1 hadoop


Configuration of Hadoop

Once the Hadoop installation is completed, Hadoop environment file should be configured with Java by adding the Java path to its file. File modifications are done as shown below.


hduser@ubuntu:~$ cd hadoop/conf
hduser@ubuntu:~$ vi hadoop-env.sh
# The java implementation to use.  Add the below line to the existing file.
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/


Now Hadoop installation and configuration was done!

Let’s run it without any arguments to see its usage documentation.
hduser@ubuntu:~$ cd hadoop
hduser@ubuntu:/home/hduser/hadoop$ bin/hadoop
Usage: hadoop [--config confdir] COMMAND
where COMMAND is one of:
namenode -format  format the DFS filesystem
secondarynamenode  run the DFS secondary namenode
namenode   run the DFS namenode
datanode   run a DFS datanode
dfsadmin   run a DFS admin client
fsck    run a DFS filesystem checking utility
fs    run a generic filesystem user client
balancer   run a cluster balancing utility
jobtracker   run the MapReduce job Tracker node
pipes    run a Pipes job
tasktracker   run a MapReduce task Tracker node
job    manipulate MapReduce jobs
version   print the version
jar <jar>   run a jar file
distcp <srcurl> <desturl> copy file or directories recursively
archive -archiveName NAME <src>* <dest> create a hadoop archive
daemonlog   get/set the log level for each daemon
or
CLASSNAME   run the class named CLASSNAME
Most commands print help when invoked w/o parameters.

 Bingo! Your installation setup is done. Now you are ready for running your first example program in Hadoop.

Comments

Post a Comment

Popular posts from this blog

[SOLVED] - RSYNC not executing via CRON

The below error was faced and tried for online answers, made my head to heat for couple of days and finally i was able to crack the solution for the below issue. umasarath@ubuntu:~$ tail -50 cron_alc.log Cronjob started for back-up files + rsync -v umasarath@xx.xx.xx.xx:/tmp/compressed_logfiles/*.zip /home/umasarath/archive/ Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(601) [Receiver=3.0.7] umasarath@ubuntu:~$ Please follow the below steps inorder to avoid the above error. My script contains the below code. umasarath@ubuntu:~$ cat transfer_files.sh #!/bin/sh set -xv echo "Cronjob started for back-up files" `date` /usr/bin/rsync -vv umasarath@xx.xx.xx.xx:/tmp/compressed_logfiles/*.zip /home/umasarath/archive/ echo "Cronjob ended for back-u
Read more

RSYNC command without authentication - 8 simple steps

Initially, when i try to sync the files from remotehost to localhost, i have to execute manually by executing in command prompt. I thought of executing the command by shell script and then call this script from cronjob. When i do this, the script was asking for authentication which is seems no difference between manual effort and an automated one. In order to avoid this, the following steps will explain how to perform RSYNC from localhost to remote host without entering password. umasarath@localhost:~$ rsync -v umasarath@remotehost:/home/umasarath/test/* test/ umasarath@remotehost's password: test11.txt umasarath.txt sent 61 bytes received 144 bytes 82.00 bytes/sec total size is 18 speedup is 0.09 umasarath@localhost:~$ Verify that localhost and remotehost is running openSSH umasarath@local:~$ ssh -V OpenSSH_5.3p1 Debian-3ubuntu7, OpenSSL 0.9.8k 25 Mar 2009 umasarath@remote:~$ ssh -V OpenSSH_5.3p1 Debian-3ubuntu7, OpenSSL 0.9.8k 25 Mar 2009 Generate key-pair on the lo
Read more

Install JDK on Ubuntu

Installing Open JDK from Command Prompt Issue command apt-get install openjdk-7-jdk to install JDK7. Ubuntu will auto download JDK and start the installation, wait a few minutes for the downloading process. umasarath @ ubuntu:~ $ sudo apt-get install openjdk- 7 -jdk Verifying Java after installation Ubuntu installs JDK at /usr/lib/jvm/jdk-folder, for example /usr/lib/jvm/java-7-openjdk-amd64/. In additional, Ubuntu also puts the JDK bin folder in the system path, via symbolic link.  For example, /usr/bin/java. To verify if JDK is installed properly, type java -version in the command prompt. umasarath@ubuntu:~$java-version java version "1.7.0_25" OpenJDK Runtime Environment (IcedTea 2.3.10) (7u25-2.3.10-1ubuntu0.12.04.2) OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode) umasarath@ubuntu:/usr/lib/jvm/java-7-openjdk-amd64/bin$ Post-Installation Setup To configure JAVA_HOME in system path each time the terminal is started, you can append the expor
Read more