Hadoop 2.2.0版本HDFS的HA配置

注:以下配置描述的是HDFS的QJM方式的HA配置。

1.1 zookeeper集群配置

这里我使用了三台机器(在笔记本上使用vmware创建了三个虚拟机来实现)部署zookeeper集群,机器IP分别是:

l 192.168.111.130(hostname:hd0)

l 192.168.111.131(hostname:hd1)

l 192.168.111.132(hostname:hd2)

首先建立zookeeper的数据目录,比如:

mkdir -p /opt/Hadoop/data/zookeeper

同时建立日志存放目录:

mkdir –p /opt/hadoop/logs/zookeeper

然后修改环境变量(比如修改~/.profile),增加如下变量:

export ZOO_HOME=/opt/hadoop/apps/zookeeper-3.4.5

export ZOO_LOG_DIR=/opt/hadoop/logs/zookeeper

在$ZOO_HOME/conf下创建配置文件:

touch zoo.cfg

在zoo.cfg文件是加入以下配置:

tickTime=2000

dataDir=/opt/hadoop/data/zookeeper

clientPort=31315

initLimit=5

syncLimit=2

server.1=192.168.111.130:31316:31317

server.2=192.168.111.131:31316:31317

server.3=192.168.111.132:31316:31317

在三台服务器的目录/opt/hadoop/data/zookeeper下分别创建一个叫myid的文件,内容分别是1,2,3,如:

#在192.168.111.130上执行如下命令

echo 1 >/opt/hadoop/data/zookeeper/myid

#在192.168.111.131上执行如下命令

echo 2 >/opt/hadoop/data/zookeeper/myid

#在192.168.111.132上执行如下命令

echo 3 >/opt/hadoop/data/zookeeper/myid

最后就是分别启动zookeeper服务了:

cd $ZOO_HOME

./bin/zkServer.sh start

通过jps命令可以检查是否启动成功:

hd@hd0:/opt/hadoop/apps/zookeeper-3.4.5$ jps

1239 QuorumPeerMain

看到QuorumPeerMain进程就表示zookeeper启动成功了。

测试zookeeper集群是否建立成功,在$ZOO_HOME目录下执行以下命令即可,如无报错表示集群创建成功:

./bin/zkCli.sh -server localhost:31315

1.2 HDFS2.2.0HA配置 1.2.1 core-site.xml

<configuration>

<property>

<name>fs.defaultFS</name>

<value>hdfs://mycluster</value>

</property>

<property>

<name>io.file.buffer.size</name>

<value>131072</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>/opt/hadoop/data/hadoop/temp</value>

<description>Abase for other temporarydirectories.</description>

</property>

<property>

<name>hadoop.proxyuser.hadoop.hosts</name>

<value>*</value>

</property>

<property>

<name>hadoop.proxyuser.hadoop.groups</name>

<value>*</value>

</property>

<property>

<name>ha.zookeeper.quorum</name>

<value>192.168.111.130:31315,192.168.111.131:31315,192.168.111.132:31315</value>

</property>

<property>

<name>ha.zookeeper.session-timeout.ms</name>

<value>1000</value>

<description>ms</description>

</property>

</configuration>

1.2.2 hdfs-site.xml

<?xml version="1.0"encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl"href="configuration.xsl"?>

<configuration>

<property>

<name>dfs.namenode.name.dir</name>

<value>/opt/hadoop/data/hadoop/dfs/name</value>

</property>

<property>

<name>dfs.datanode.data.dir</name>

<value>/opt/hadoop/data/hadoop/dfs/data</value>

</property>

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

<property>

<name>dfs.webhdfs.enabled</name>

<value>true</value>

</property>

<property>

<name>dfs.permissions</name>

<value>false</value>

</property>

<property>

<name>dfs.permissions.enabled</name>

<value>false</value>

</property>

<property>

<name>dfs.nameservices</name>

<value>mycluster</value>

<description>Logical name for this newnameservice</description>

</property>

<property>

<name>dfs.ha.namenodes.mycluster</name>

<value>nn1,nn2</value>

<description>Unique identifiers for each NameNode in thenameservice</description>

</property>

<property>

<name>dfs.namenode.rpc-address.mycluster.nn1</name>

<value>192.168.111.130:8020</value>

</property>

<property>

<name>dfs.namenode.rpc-address.mycluster.nn2</name>

<value>192.168.111.132:8020</value>

</property>

<property>

<name>dfs.namenode.servicerpc-address.mycluster.nn1</name>

<value>192.168.111.130:53310</value>

</property>

<property>

<name>dfs.namenode.servicerpc-address.mycluster.nn2</name>

<value>192.168.111.132:53310</value>

</property>

<property>

<name>dfs.namenode.http-address.mycluster.nn1</name>

<value>192.168.111.130:50070</value>

</property>

<property>

<name>dfs.namenode.http-address.mycluster.nn2</name>

<value>192.168.111.132:50070</value>

</property>

<property>

<name>dfs.namenode.shared.edits.dir</name>

<value>qjournal://192.168.111.130:8485;192.168.111.131:8485;192.168.111.132:8485/mycluster</value>

</property>

<property>

<name>dfs.client.failover.proxy.provider.mycluster</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

<property>

<name>dfs.ha.fencing.methods</name>

<value>sshfence</value>

</property>

<property>

<name>dfs.ha.fencing.ssh.private-key-files</name>

<value>/home/hadoop/.ssh/id_rsa_nn1</value>

</property>

<property>

<name>dfs.ha.fencing.ssh.connect-timeout</name>

<value>30000</value>

</property>

<property>

<name>dfs.journalnode.edits.dir</name>

<value>/opt/hadoop/data/hadoop/journaldata</value>

</property>

<property>

<name>dfs.ha.automatic-failover.enabled</name>

<value>true</value>

</property>

<property>

<name>ha.failover-controller.cli-check.rpc-timeout.ms</name>

<value>60000</value>

</property>

<property>

<name>ipc.client.connect.timeout</name>

<value>60000</value>

</property>

<property>

<name>dfs.image.transfer.bandwidthPerSec</name>

<value>4194304</value>

</property>

</configuration>

内容版权声明:除非注明,否则皆为本站原创文章。

转载注明出处:http://www.heiqu.com/b2e44d28f03c0ac95de3e118f6fd0148.html