Do You Make These Mistakes?

Prerequisites Hadoop requires a working Java 1.5+ (aka Java 5) installation. However, using Java 1.6/1.7 (aka Java 6/7) is recommended for running Hadoop. Please refer to jdk installation instructions here. Dedicated user for Hadoop system A dedicated Hadoop user will help Hadoop installation from other software applications and user accounts running on the same machine. umasarath@ubuntu:~$ sudo addgroup hadoop umasarath@ubuntu:~$ sudo adduser --ingroup hadoop hduser The above two commands will add "hduser" user and "hadoop" group. Hadoop Installation Download Hadoop from the Apache Download Mirrors and extract the contents of the Hadoop package to a location of your choice. The folder I chosen was the hduser home folder. Extract the downloaded file in /home/hduser folder and make sure the file should be extracted in hduser login. hduser@ubuntu:~$ sudo tar xzf hadoop-1.2.1.tar.gz hduser@ubuntu:~$ sudo mv hadoop-1.2.1 hadoop Configuratio...

Installing Open JDK from Command Prompt Issue command apt-get install openjdk-7-jdk to install JDK7. Ubuntu will auto download JDK and start the installation, wait a few minutes for the downloading process. umasarath @ ubuntu:~ $ sudo apt-get install openjdk- 7 -jdk Verifying Java after installation Ubuntu installs JDK at /usr/lib/jvm/jdk-folder, for example /usr/lib/jvm/java-7-openjdk-amd64/. In additional, Ubuntu also puts the JDK bin folder in the system path, via symbolic link.  For example, /usr/bin/java. To verify if JDK is installed properly, type java -version in the command prompt. umasarath@ubuntu:~$java-version java version "1.7.0_25" OpenJDK Runtime Environment (IcedTea 2.3.10) (7u25-2.3.10-1ubuntu0.12.04.2) OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode) umasarath@ubuntu:/usr/lib/jvm/java-7-openjdk-amd64/bin$ Post-Installation Setup To configure JAVA_HOME in system path each time the terminal is started, you can append the expor...

The below error was faced and tried for online answers, made my head to heat for couple of days and finally i was able to crack the solution for the below issue. umasarath@ubuntu:~$ tail -50 cron_alc.log Cronjob started for back-up files + rsync -v umasarath@xx.xx.xx.xx:/tmp/compressed_logfiles/*.zip /home/umasarath/archive/ Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(601) [Receiver=3.0.7] umasarath@ubuntu:~$ Please follow the below steps inorder to avoid the above error. My script contains the below code. umasarath@ubuntu:~$ cat transfer_files.sh #!/bin/sh set -xv echo "Cronjob started for back-up files" `date` /usr/bin/rsync -vv umasarath@xx.xx.xx.xx:/tmp/compressed_logfiles/*.zip /home/umasarath/archive/ echo "Cronjob ended for back-u...

You might come across the below command like  chmod 755 filename and of course you will be wondering what this is.  The thing is, that you can change the entire permission pattern of a file in one go using one number like the one in this example. Every mode has a corresponding code number, and as we shall see there is a very simple way to figure out what number corresponds to any mode.  Every one of the three digits on the mode number corresponds to one of the three permission triplets. (u, g and o) Every permission bit in a triplet corresponds to a value:  4 for r,  2 for w,  1 for x.  If the permission bit you add this value to the number of the permission triplet. If it is cleared, then you add nothing. So if a file has rwxr-xr-x permissions we do the following calculation:  Triplet for u: rwx => 4 + 2 + 1 = 7 Triplet for g: r-x => 4 + 0 + 1 = 5 Triplet for o: r-x => 4 + 0 + 1 = 5 ...

Permissions Every file on the system has associated with it a set of permissions. Permissions tell UNIX what can be done with that file and by whom. There are three things you can (or can't) do with a given file:   read it,   write (modify) it and   execute it. For any given ownership relation, we need three letters to specify access permissions: the first denotes read (r) access, the second denotes write (w) access and the third denotes execute (x) access. We have three ownership relations: 'owner', 'group' and 'all' so we need a triplet for each, resulting in nine letters. Lets try something in our command prompt with ls -l command. umasarath@ubuntu:~$ ls -l total 53780 -rw-r--r-- 1 umasarath group 27455157 2012-11-16 10:28 sample.xml drwxr-xr-x 2 umasarath group      4096 2013-01-23 10:17 files -rw-r--r-- 1 umasarath group      1338 2013-09-14 08:32 keystore drwxr-xr-x 2 umasarath grou...

The following code can run ssh-agent automatically when you open bash by adding the following to ~/.profile or ~/.bashrc file: Please note after inserting the code in your profile or bashrc, passphrase will be asked for the first time. From the next login, you can enjoy password less login. # Note: ~/.ssh/environment should not be used, as it # already has a different purpose in SSH. env=~/.ssh/agent.env # Note: Don't bother checking SSH_AGENT_PID. It's not used # by SSH itself, and it might even be incorrect # (for example, when using agent-forwarding over SSH). agent_is_running() { if [ "$SSH_AUTH_SOCK" ]; then # ssh-add returns: # 0 = agent running, has keys # 1 = agent running, no keys # 2 = agent not running ssh-add -l >/dev/null 2>&1 || [ $? -eq 1 ] else false fi } agent_has_keys() { ssh-add -l >/dev/null 2>&1 } agent_load_env() { . "...

TAR is archiving utility in Linux. With tar command, multiple files and directories can be archived into a single file, with extension of that file is ".tar". "gzip" and "bzip" compression techniques can be used with tar command. Syntax: $ tar [options] [desired tar file name] [list of file and/or directories] Examples for your reference I need to create a tar file named umasarath from the directory_name folder. Below is the basic command to create a tar archive. $ tar cvf umasarath.tar directory_name/ Here cvf stands for  c - create new archive file v - verbosely list files f - following is archive file name. The above example tar cvf option, does not provide any compression. Inorder to use a gzip compression on the tar archive, use the z option as shown below. $ tar cvzf umasarath.tar directory_name/ Here z stands for  z - filter the archive through gzip Please Note: .tgz is same as .tar.gz