对于一名Hadoop开发人员来说,第一件事就是需要安装一个自己的Hadoop环境供日常的开发及学习使用,本文主要介绍如何安装Hadoop CDH4的伪分布式环境。
由于Hadoop是由Java编写,因此需要运行在JDK 1.6以上平台,所以第一步需要安装相应的JDK。相应的JDK可以通过下面这个连接下载。
安装JDK
[kevin@localhost ~]$ sudo ./jdk-6u34-linux-x64-rpm.bin
[kevin@localhost ~]$ sudo cp jdk-6u34-linux-amd64.rpm /usr/local
[kevin@localhost ~]$ cd /usr/local/
[kevin@localhost local]$ sudo chmod +x jdk-6u34-linux-amd64.rpm
[kevin@localhost local]$ sudo rpm -ivh jdk-6u34-linux-amd64.rpm
配置环境变量
[kevin@localhost jdk1.6.0_34]$ sudo vi /etc/profile
添加JDK的安装目录,具体添加内容如下:
export JAVA_HOME=/usr/java/jdk1.6.0_34
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
[kevin@localhost jdk1.6.0_34]$ java -version (验证JDK是否安装成功)java version "1.6.0_34"
Java(TM) SE Runtime Environment (build 1.6.0_34-b04)
Java HotSpot(TM) 64-Bit Server VM (build 20.9-b04, mixed mode)
Hadoop安装(Yarn)
Hadoop CHD4是基本Apache Hadoop 2.0,Map/Reduce框架使用Yarn。可以通用yum来安装(sudo yum install hadoop-conf-pseudo)。为了了解Hadoop的依赖关系,这里介绍使用rpm来安装,手动解决相应的依赖关系。具体的rpm包下载地址(Hadoop CDH4 RPM)
[kevin@localhost ~]$ chmod +x hadoop-conf-pseudo-2.0.0+545-1.cdh4.1.1.p0.5.el5.x86_64.rpm
[kevin@localhost ~]$ sudo rpm -ivh hadoop-conf-pseudo-2.0.0+545-1.cdh4.1.1.p0.5.el5.x86_64.rpm
[sudo] password for kevin:
warning: hadoop-conf-pseudo-2.0.0+545-1.cdh4.1.1.p0.5.el5.x86_64.rpm: Header V4 DSA signature: NOKEY, key ID e8f86acd
error: Failed dependencies:
hadoop = 2.0.0+545-1.cdh4.1.1.p0.5.el5 is needed by hadoop-conf-pseudo-2.0.0+545-1.cdh4.1.1.p0.5.el5.x86_64
hadoop-hdfs-namenode = 2.0.0+545-1.cdh4.1.1.p0.5.el5 is needed by hadoop-conf-pseudo-2.0.0+545-1.cdh4.1.1.p0.5.el5.x86_64
hadoop-hdfs-datanode = 2.0.0+545-1.cdh4.1.1.p0.5.el5 is needed by hadoop-conf-pseudo-2.0.0+545-1.cdh4.1.1.p0.5.el5.x86_64
hadoop-hdfs-secondarynamenode = 2.0.0+545-1.cdh4.1.1.p0.5.el5 is needed by hadoop-conf-pseudo-2.0.0+545-1.cdh4.1.1.p0.5.el5.x86_64
hadoop-yarn-resourcemanager = 2.0.0+545-1.cdh4.1.1.p0.5.el5 is needed by hadoop-conf-pseudo-2.0.0+545-1.cdh4.1.1.p0.5.el5.x86_64
hadoop-yarn-nodemanager = 2.0.0+545-1.cdh4.1.1.p0.5.el5 is needed by hadoop-conf-pseudo-2.0.0+545-1.cdh4.1.1.p0.5.el5.x86_64
hadoop-mapreduce-historyserver = 2.0.0+545-1.cdh4.1.1.p0.5.el5 is needed by hadoop-conf-pseudo-2.0.0+545-1.cdh4.1.1.p0.5.el5.x86_64
报如上依赖关系错误,可以清楚的看到hadoop-conf-pseudo依赖hadoop, hadoop-hdfs-namenode, hadoop-hdfs-datanode, hadoop-hdfs-secondarynamenode, hadoop-yarn-resourcemanager, hadoop-yarn-nodemanager, hadoop-mapreduce-historyserver,由于伪分布所有的组件都运行在同一台机器上,所以需要安装上面这些依赖包。
具体的hadoop环境参数如下:
/etc/hadoop/conf (具体的配置文件)
/etc/init.d (所有相关的启动脚本)
/usr/lib (具体安装目录包括Hadoop, hdfs, yarn, map/reduce)
安装完成之后,会默认创建几个用户(zookeeper, hdfs, yarn, mapred),及相应的组(zookeeper, hadoop, hdfs, yarn, mapred)
Format NameNode
Hadoop已经安装成功,在启动hadoop之前,需要对namenode进行format,format namenode必须使用hdfs这个用户,如下:
[kevin@localhost init.d]$ sudo -u hdfs hdfsnamenode -format
启动Hadoop HDFS
$ for service in /etc/init.d/hadoop-hdfs-*
> do
> sudo $service start
> done
可以通过sudo jps查看启来的java进程
启动正常,可以通过hadoop fs -ls / 来访问HDFS文件系统,通过:50070访问web console的时候不能访问,查看防火墙是启来的,应该防火墙的导致,暂时先把防火墙关闭之后,重新访问正常。
创建Hadoop HDFS初始目录