Compression in Hadoop


File compression brings two major benefits: it reduces the space needed to store files, and it speeds up data transfer across the network, or to or from disk. When dealing with large volumes of data, both of these savings can be significant, so it pays to carefully consider how to use compression in Hadoop.

Some of the compression formats used in Hadoop

Compression Format
Tool
Algorithm
Filename Extension
Splittable
DEFLATE
NA
DEFLATE
.deflate
No
gzip
gzip
DEFLATE
.gz
No
bzip2
bzip2
bzip2
.bz2
Yes
LZO
lzop
LZO
.lzo
No
Snappy
NA
Snappy
.snappy
No

Codecs

A codec is the implementation of a compression-decompression algorithm and in Hadoop, it is represented by an implementation of the CompressionCodec interface.


Compression Format
Hadoop CompressionCodec
DEFLATE
org.apache.hadoop.io.compress.DefaultCodec
gzip
org.apache.hadoop.io.compress.GzipCodec
bzip2
org.apache.hadoop.io.compress.BZip2Codec
LZO
com.hadoop.compression.lzo.LzopCodec
Snappy
org.apache.hadoop.io.compress.SnappyCodec









Comments

Post a Comment

Popular posts from this blog

[SOLVED] - RSYNC not executing via CRON

The below error was faced and tried for online answers, made my head to heat for couple of days and finally i was able to crack the solution for the below issue. umasarath@ubuntu:~$ tail -50 cron_alc.log Cronjob started for back-up files + rsync -v umasarath@xx.xx.xx.xx:/tmp/compressed_logfiles/*.zip /home/umasarath/archive/ Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(601) [Receiver=3.0.7] umasarath@ubuntu:~$ Please follow the below steps inorder to avoid the above error. My script contains the below code. umasarath@ubuntu:~$ cat transfer_files.sh #!/bin/sh set -xv echo "Cronjob started for back-up files" `date` /usr/bin/rsync -vv umasarath@xx.xx.xx.xx:/tmp/compressed_logfiles/*.zip /home/umasarath/archive/ echo "Cronjob ended for back-u
Read more

RSYNC command without authentication - 8 simple steps

Initially, when i try to sync the files from remotehost to localhost, i have to execute manually by executing in command prompt. I thought of executing the command by shell script and then call this script from cronjob. When i do this, the script was asking for authentication which is seems no difference between manual effort and an automated one. In order to avoid this, the following steps will explain how to perform RSYNC from localhost to remote host without entering password. umasarath@localhost:~$ rsync -v umasarath@remotehost:/home/umasarath/test/* test/ umasarath@remotehost's password: test11.txt umasarath.txt sent 61 bytes received 144 bytes 82.00 bytes/sec total size is 18 speedup is 0.09 umasarath@localhost:~$ Verify that localhost and remotehost is running openSSH umasarath@local:~$ ssh -V OpenSSH_5.3p1 Debian-3ubuntu7, OpenSSL 0.9.8k 25 Mar 2009 umasarath@remote:~$ ssh -V OpenSSH_5.3p1 Debian-3ubuntu7, OpenSSL 0.9.8k 25 Mar 2009 Generate key-pair on the lo
Read more

Install JDK on Ubuntu

Installing Open JDK from Command Prompt Issue command apt-get install openjdk-7-jdk to install JDK7. Ubuntu will auto download JDK and start the installation, wait a few minutes for the downloading process. umasarath @ ubuntu:~ $ sudo apt-get install openjdk- 7 -jdk Verifying Java after installation Ubuntu installs JDK at /usr/lib/jvm/jdk-folder, for example /usr/lib/jvm/java-7-openjdk-amd64/. In additional, Ubuntu also puts the JDK bin folder in the system path, via symbolic link.  For example, /usr/bin/java. To verify if JDK is installed properly, type java -version in the command prompt. umasarath@ubuntu:~$java-version java version "1.7.0_25" OpenJDK Runtime Environment (IcedTea 2.3.10) (7u25-2.3.10-1ubuntu0.12.04.2) OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode) umasarath@ubuntu:/usr/lib/jvm/java-7-openjdk-amd64/bin$ Post-Installation Setup To configure JAVA_HOME in system path each time the terminal is started, you can append the expor
Read more