Hadoop CDH4伪分布式安装

对于一名Hadoop开发人员来说,第一件事就是需要安装一个自己的Hadoop环境供日常的开发及学习使用,本文主要介绍如何安装Hadoop CDH4的伪分布式环境。

由于Hadoop是由Java编写,因此需要运行在JDK 1.6以上平台,所以第一步需要安装相应的JDK。相应的JDK可以通过下面这个连接下载。

安装JDK

[kevin@localhost ~]$ sudo ./jdk-6u34-linux-x64-rpm.bin

[kevin@localhost ~]$ sudo cp jdk-6u34-linux-amd64.rpm /usr/local
[kevin@localhost ~]$ cd /usr/local/
[kevin@localhost local]$ sudo chmod +x jdk-6u34-linux-amd64.rpm
[kevin@localhost local]$ sudo rpm -ivh jdk-6u34-linux-amd64.rpm

配置环境变量

[kevin@localhost jdk1.6.0_34]$ sudo vi /etc/profile

添加JDK的安装目录,具体添加内容如下:

export JAVA_HOME=/usr/java/jdk1.6.0_34

export PATH=$PATH:$JAVA_HOME/bin

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

[kevin@localhost jdk1.6.0_34]$ java -version (验证JDK是否安装成功)
java version "1.6.0_34"
Java(TM) SE Runtime Environment (build 1.6.0_34-b04)
Java HotSpot(TM) 64-Bit Server VM (build 20.9-b04, mixed mode)

Hadoop安装(Yarn)

Hadoop CHD4是基本Apache Hadoop 2.0,Map/Reduce框架使用Yarn。可以通用yum来安装(sudo yum install hadoop-conf-pseudo)。为了了解Hadoop的依赖关系,这里介绍使用rpm来安装,手动解决相应的依赖关系。具体的rpm包下载地址(Hadoop CDH4 RPM)

[kevin@localhost ~]$ chmod +x hadoop-conf-pseudo-2.0.0+545-1.cdh4.1.1.p0.5.el5.x86_64.rpm
[kevin@localhost ~]$ sudo rpm -ivh hadoop-conf-pseudo-2.0.0+545-1.cdh4.1.1.p0.5.el5.x86_64.rpm
[sudo] password for kevin:
warning: hadoop-conf-pseudo-2.0.0+545-1.cdh4.1.1.p0.5.el5.x86_64.rpm: Header V4 DSA signature: NOKEY, key ID e8f86acd
error: Failed dependencies:
hadoop = 2.0.0+545-1.cdh4.1.1.p0.5.el5 is needed by hadoop-conf-pseudo-2.0.0+545-1.cdh4.1.1.p0.5.el5.x86_64
hadoop-hdfs-namenode = 2.0.0+545-1.cdh4.1.1.p0.5.el5 is needed by hadoop-conf-pseudo-2.0.0+545-1.cdh4.1.1.p0.5.el5.x86_64
hadoop-hdfs-datanode = 2.0.0+545-1.cdh4.1.1.p0.5.el5 is needed by hadoop-conf-pseudo-2.0.0+545-1.cdh4.1.1.p0.5.el5.x86_64
hadoop-hdfs-secondarynamenode = 2.0.0+545-1.cdh4.1.1.p0.5.el5 is needed by hadoop-conf-pseudo-2.0.0+545-1.cdh4.1.1.p0.5.el5.x86_64
hadoop-yarn-resourcemanager = 2.0.0+545-1.cdh4.1.1.p0.5.el5 is needed by hadoop-conf-pseudo-2.0.0+545-1.cdh4.1.1.p0.5.el5.x86_64
hadoop-yarn-nodemanager = 2.0.0+545-1.cdh4.1.1.p0.5.el5 is needed by hadoop-conf-pseudo-2.0.0+545-1.cdh4.1.1.p0.5.el5.x86_64
hadoop-mapreduce-historyserver = 2.0.0+545-1.cdh4.1.1.p0.5.el5 is needed by hadoop-conf-pseudo-2.0.0+545-1.cdh4.1.1.p0.5.el5.x86_64

报如上依赖关系错误,可以清楚的看到hadoop-conf-pseudo依赖hadoop, hadoop-hdfs-namenode, hadoop-hdfs-datanode, hadoop-hdfs-secondarynamenode, hadoop-yarn-resourcemanager, hadoop-yarn-nodemanager, hadoop-mapreduce-historyserver,由于伪分布所有的组件都运行在同一台机器上,所以需要安装上面这些依赖包。

具体的hadoop环境参数如下:

/etc/hadoop/conf (具体的配置文件)

/etc/init.d (所有相关的启动脚本)

/usr/lib (具体安装目录包括Hadoop, hdfs, yarn, map/reduce)

安装完成之后,会默认创建几个用户(zookeeper, hdfs, yarn, mapred),及相应的组(zookeeper, hadoop, hdfs, yarn, mapred)

Format NameNode

Hadoop已经安装成功,在启动hadoop之前,需要对namenode进行format,format namenode必须使用hdfs这个用户,如下:

[kevin@localhost init.d]$ sudo -u hdfs hdfsnamenode -format

启动Hadoop HDFS

$ for service in /etc/init.d/hadoop-hdfs-*
> do
> sudo $service start
> done

可以通过sudo jps查看启来的java进程

Hadoop CDH4伪分布式安装

启动正常,可以通过hadoop fs -ls / 来访问HDFS文件系统,通过:50070访问web console的时候不能访问,查看防火墙是启来的,应该防火墙的导致,暂时先把防火墙关闭之后,重新访问正常。

Hadoop CDH4伪分布式安装

创建Hadoop HDFS初始目录

内容版权声明:除非注明,否则皆为本站原创文章。

转载注明出处:http://www.heiqu.com/6264aecbec7fef773d007d0579341436.html