The Hadoop Distributed Filesystem

Design of HDFS


HDFS is a filesystem designed for

  • Very large files - Files that are of hundereds of MB, GB or TB. Hadoop clusters running today stores petabytes of data.
  • Streaming data access - write once, read many times pattern
  • Commodity hardware - Hadoop doesn’t require expensive, highly reliable hardware to run on.

The applications for which using HDFS does not work so well. While this may change in the future, these are areas where HDFS is not a good fit today
  • Low-latency data access
  • Lots of small files
  • Multiple writers, arbitrary file modifications

Blocks


 HDFS has the concept of a block, but it is a much larger unit—64 MB by default. HDFS blocks are large compared to disk blocks, and the reason is to minimize the cost of seeks. By making a block large enough, the time to transfer the data from the disk can be made to be significantly larger than the time to seek to the start of the block. Thus the time to transfer a large file made of multiple blocks operates at the disk transfer rate.

 HDFS’s fsck command understands blocks. For example
% hadoop fsck / -files -blocks
will list the blocks that make up each file in the filesystem.

Namenodes and Datanodes


 An HDFS cluster has two types of node operating in a master-worker pattern: a namenode (the master) and a number of datanodes (workers). The namenode manages the filesystem namespace. It maintains the filesystem tree and the metadata for all the files and directories in the tree. This information is stored persistently on the local disk in the form of two files: the namespace image and the edit log. The namenode also knows the datanodes on which all the blocks for a given file are located, however, it does not store block locations persistently, since this information is reconstructed from datanodes when the system starts.

 Datanodes are the workhorses of the filesystem. They store and retrieve blocks when they are told to (by clients or the namenode), and they report back to the namenode periodically with lists of blocks that they are storing.

 The file system cannot be used without namenode. If namenode goes down, there would be no way of knowing how to reconstruct the files from the blocks on the datanodes. 

 There are 2 ways to recover

  • Run a secondary name node
  • Backup the files that make up the persistant state of the filesystem matadata and ensure that the writes are synchronous and atomic. The usual configuration choice is to write to local disk as well as a remote NFS mount.



Comments

Post a Comment

Popular posts from this blog

[SOLVED] - RSYNC not executing via CRON

The below error was faced and tried for online answers, made my head to heat for couple of days and finally i was able to crack the solution for the below issue. umasarath@ubuntu:~$ tail -50 cron_alc.log Cronjob started for back-up files + rsync -v umasarath@xx.xx.xx.xx:/tmp/compressed_logfiles/*.zip /home/umasarath/archive/ Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(601) [Receiver=3.0.7] umasarath@ubuntu:~$ Please follow the below steps inorder to avoid the above error. My script contains the below code. umasarath@ubuntu:~$ cat transfer_files.sh #!/bin/sh set -xv echo "Cronjob started for back-up files" `date` /usr/bin/rsync -vv umasarath@xx.xx.xx.xx:/tmp/compressed_logfiles/*.zip /home/umasarath/archive/ echo "Cronjob ended for back-u
Read more

RSYNC command without authentication - 8 simple steps

Initially, when i try to sync the files from remotehost to localhost, i have to execute manually by executing in command prompt. I thought of executing the command by shell script and then call this script from cronjob. When i do this, the script was asking for authentication which is seems no difference between manual effort and an automated one. In order to avoid this, the following steps will explain how to perform RSYNC from localhost to remote host without entering password. umasarath@localhost:~$ rsync -v umasarath@remotehost:/home/umasarath/test/* test/ umasarath@remotehost's password: test11.txt umasarath.txt sent 61 bytes received 144 bytes 82.00 bytes/sec total size is 18 speedup is 0.09 umasarath@localhost:~$ Verify that localhost and remotehost is running openSSH umasarath@local:~$ ssh -V OpenSSH_5.3p1 Debian-3ubuntu7, OpenSSL 0.9.8k 25 Mar 2009 umasarath@remote:~$ ssh -V OpenSSH_5.3p1 Debian-3ubuntu7, OpenSSL 0.9.8k 25 Mar 2009 Generate key-pair on the lo
Read more

Install JDK on Ubuntu

Installing Open JDK from Command Prompt Issue command apt-get install openjdk-7-jdk to install JDK7. Ubuntu will auto download JDK and start the installation, wait a few minutes for the downloading process. umasarath @ ubuntu:~ $ sudo apt-get install openjdk- 7 -jdk Verifying Java after installation Ubuntu installs JDK at /usr/lib/jvm/jdk-folder, for example /usr/lib/jvm/java-7-openjdk-amd64/. In additional, Ubuntu also puts the JDK bin folder in the system path, via symbolic link.  For example, /usr/bin/java. To verify if JDK is installed properly, type java -version in the command prompt. umasarath@ubuntu:~$java-version java version "1.7.0_25" OpenJDK Runtime Environment (IcedTea 2.3.10) (7u25-2.3.10-1ubuntu0.12.04.2) OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode) umasarath@ubuntu:/usr/lib/jvm/java-7-openjdk-amd64/bin$ Post-Installation Setup To configure JAVA_HOME in system path each time the terminal is started, you can append the expor
Read more