Hadoop2.4.1尝鲜部署+完整版配置文件

转眼间,Hadoop的stable版本已经升级到2.4.1了,社区的力量真是强大!3.0啥时候release呢?

今天做了个调研,尝鲜了一下2.4.1版本的分布式部署,包括NN HA(目前已经部署好了2.2.0的NN HA,ZK和ZKFC用现成的),顺便也结合官方文档   梳理、补全了关键的配置文件属性,将同类属性归类,方便以后阅读修改,及作为模板使用。

下面记录参照官方文档及过去经验部署2.4.1的过程。

--------------------------------------------------------------------------------

注意
1.本文只记录配置文件,不记录其余部署过程,其余过程和2.2.0相同,参见

2.配置中所有的路径、IP、hostname均需根据实际情况修改。

--------------------------------------------------------------------------------

Ubuntu 13.04上搭建Hadoop环境

Ubuntu 12.10 +Hadoop 1.2.1版本集群配置

Ubuntu上搭建Hadoop环境(单机模式+伪分布模式)

Ubuntu下Hadoop环境的配置

单机版搭建Hadoop环境图文教程详解

搭建Hadoop环境(在Winodws环境下用虚拟机虚拟两个Ubuntu系统进行搭建)

--------------------------------------------------------------------------------

1.实验环境:
4节点集群,ZK节点3个,hosts文件和各节点角色分配如下:
hosts:
192.168.66.91 master
192.168.66.92 slave1
192.168.66.93 slave2
192.168.66.94 slave3


角色分配:
  Active NN Standby NN DN JournalNode Zookeeper FailoverController
master V    V V V
slave1  V V V V V
slave2    V V V 
slave3    V 

--------------------------------------------------------------------------------

2.hadoop-env.sh  修改以下三处即可
# The java implementation to use.
export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_07


# The directory where pid files are stored. /tmp by default.
# NOTE: this should be set to a directory that can only be written to by the user that will run the hadoop daemons.  Otherwise there is the potential for a symlink attack.
export HADOOP_PID_DIR=/home/yarn/Hadoop/hadoop-2.4.1/hadoop_pid_dir
export HADOOP_SECURE_DN_PID_DIR=/home/yarn/Hadoop/hadoop-2.4.1/hadoop_pid_dir

--------------------------------------------------------------------------------

3.core-site.xml 完整文件

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="https://www.linuxidc.com/configuration.xsl"?>
<!-- Licensed under the Apache License, Version 2.0 (the "License"); you
    may not use this file except in compliance with the License. You may obtain
    a copy of the License at Unless
    required by applicable law or agreed to in writing, software distributed
    under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
    OR CONDITIONS OF ANY KIND, either express or implied. See the License for
    the specific language governing permissions and limitations under the License.
    See accompanying LICENSE file. -->
<!-- Put site-specific property overrides in this file. -->
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://myhadoop</value>
        <description>NameNode UR,格式是hdfs://host:port/,如果开启了NN
            HA特性,则配置集群的逻辑名,具体参见我的
        </description>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/home/yarn/Hadoop/hadoop-2.4.1/tmp</value>
    </property>
    <property>
        <name>io.file.buffer.size</name>
        <value>131072</value>
        <description>Size of read/write buffer used in SequenceFiles.
        </description>
    </property>
    <property>
        <name>ha.zookeeper.quorum</name>
        <value>master:2181,slave1:2181,slave2:2181</value>
        <description>注意,配置了ZK以后,在格式化、启动NameNode之前必须先启动ZK,否则会报连接错误
        </description>
    </property>
</configuration> 


--------------------------------------------------------------------------------

4.hdfs-site.xml  完整文件

内容版权声明:除非注明,否则皆为本站原创文章。

转载注明出处:https://www.heiqu.com/472397fdc932c0be00760ba66a59c48b.html