1,安装和更新gcc、ant(系统已经安装的话,略去下面步骤)
yum -y install gcc gcc-c++ autoconf automake
wget
tar -jxvf apache-ant-1.8.2-bin.tar.bz2
export ANT_HOME=/usr/local/apache-ant-1.8.2
vi /etc/profile
source /etc/profile
export PATH=$PATH:$ANT_HOME/bin
2在各个节点安装lzo
wget
tar -zxvf lzo-2.04.tar.gzcd lzo-2.04
./configure --enable-shared
make && make install
拷贝/usr/local/lib目录下的lzo库文件到/usr/lib(32位平台),或/usr/lib64(64位平台)
3,安装lzo编码/解码器
wget https://download.github.com/kevinweil-Hadoop-lzo-2ad6654.tar.gz
tar -zxvf kevinweil-hadoop-lzo-2ad6654.tar.gz
cd kevinweil-hadoop-lzo-2ad6654
ant compile-native tar
安装编译过程中有错误,请考虑安装下面两个rpm包
wget
wget
rpm -ivh lzo-2.04-1.el5.rf.i386.rpm
rpm -ivh lzo-devel-2.04-1.el5.rf.i386.rpm
(根据系统选择)编译成功后,拷贝kevinweil-hadoop-lzo-2ad6654/bulid/hadoop-lzo-0.4.14.jar到hadoop/lib下
拷贝build/native/Linux-amd64-64/lib下文件到hadoop/lib/native/Linux-amd64-64下
4,修改配置文件,并同步各节点的配置文件
core-site.xml中增加
<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,com.hadoop.compression.lzo.LzoCodec,co
m.hadoop.compression.lzo.LzopCodec,org.apache.hadoop.io.compress.BZip2Codec</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
mapred-site.xml中增加
<property>
<name>mapred.compress.map.output</name>
<value>true</value>
</property>
<property>
<name>mapred.map.output.compression.codec</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
5,安装lzop压缩软件,到的下面download部分下载,可以选择选择Precompiled binaries are available for下载,直接配置可用,或者下载source,./configure make&makeinstall即可这是就可以通过lzop文件名,来生成lzo压缩文件,传到hadoop平台,平台支持文件分割。