Compression in Hadoop


File compression brings two major benefits: it reduces the space needed to store files, and it speeds up data transfer across the network, or to or from disk. When dealing with large volumes of data, both of these savings can be significant, so it pays to carefully consider how to use compression in Hadoop.

Some of the compression formats used in Hadoop

Compression Format
Tool
Algorithm
Filename Extension
Splittable
DEFLATE
NA
DEFLATE
.deflate
No
gzip
gzip
DEFLATE
.gz
No
bzip2
bzip2
bzip2
.bz2
Yes
LZO
lzop
LZO
.lzo
No
Snappy
NA
Snappy
.snappy
No

Codecs

A codec is the implementation of a compression-decompression algorithm and in Hadoop, it is represented by an implementation of the CompressionCodec interface.


Compression Format
Hadoop CompressionCodec
DEFLATE
org.apache.hadoop.io.compress.DefaultCodec
gzip
org.apache.hadoop.io.compress.GzipCodec
bzip2
org.apache.hadoop.io.compress.BZip2Codec
LZO
com.hadoop.compression.lzo.LzopCodec
Snappy
org.apache.hadoop.io.compress.SnappyCodec









Comments

Post a Comment

Popular posts from this blog

[SOLVED] - RSYNC not executing via CRON

The below error was faced and tried for online answers, made my head to heat for couple of days and finally i was able to crack the solution for the below issue. umasarath@ubuntu:~$ tail -50 cron_alc.log Cronjob started for back-up files + rsync -v umasarath@xx.xx.xx.xx:/tmp/compressed_logfiles/*.zip /home/umasarath/archive/ Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(601) [Receiver=3.0.7] umasarath@ubuntu:~$ Please follow the below steps inorder to avoid the above error. My script contains the below code. umasarath@ubuntu:~$ cat transfer_files.sh #!/bin/sh set -xv echo "Cronjob started for back-up files" `date` /usr/bin/rsync -vv umasarath@xx.xx.xx.xx:/tmp/compressed_logfiles/*.zip /home/umasarath/archive/ echo "Cronjob ended for back-u...
Read more

RSYNC command without authentication - 8 simple steps

Initially, when i try to sync the files from remotehost to localhost, i have to execute manually by executing in command prompt. I thought of executing the command by shell script and then call this script from cronjob. When i do this, the script was asking for authentication which is seems no difference between manual effort and an automated one. In order to avoid this, the following steps will explain how to perform RSYNC from localhost to remote host without entering password. umasarath@localhost:~$ rsync -v umasarath@remotehost:/home/umasarath/test/* test/ umasarath@remotehost's password: test11.txt umasarath.txt sent 61 bytes received 144 bytes 82.00 bytes/sec total size is 18 speedup is 0.09 umasarath@localhost:~$ Verify that localhost and remotehost is running openSSH umasarath@local:~$ ssh -V OpenSSH_5.3p1 Debian-3ubuntu7, OpenSSL 0.9.8k 25 Mar 2009 umasarath@remote:~$ ssh -V OpenSSH_5.3p1 Debian-3ubuntu7, OpenSSL 0.9.8k 25 Mar 2009 Generate key-pair on the lo...
Read more

The javax.servlet Package

The javax.servlet package contains a number of interfaces and classes that establish the framework in which servlets operate. The most significant of these is Servlet. All servlets must implement this interface or extend a class that implements the interface Interface Description Servlet Declares life cycle methods for a servlet. ServletConfig Allows servlets to get initialization parameters ServletContext Enables servlets to log events and access information about their environment. ServletRequest Used to read data from a client request. ServletResponse Used to write data to a client response. The following table summarizes the core classes that are provided in the javax.servlet package: Class Description GenericServlet Implements the Servlet and ServletConfig interfaces. ServletInputStream Provides an input stream for reading requests from a client. ...
Read more