Hadoop2.7.1 HA集群部署(3)

日期：2020-06-16 栏目：程序人生浏览：次

hadoop@Ubuntu:~$ sudo tar -zxvf zookeeper-3.4.8.tar.gz

hadoop@ubuntu:~$ chown -R hadoop:hadoop zookeeper-3.4.8

mv zookeeper-3.4.8 zookeeper

②　设置环境变量 vi /etc/profile

#set zookeeper

export ZOOKEEPER_HOME=/opt/hadoop/zookeeper

export PATH=$PATH:$ZOOKEEPER_HOME/bin

Vi ~/.bash_profile

#set zookeeper

export ZOOKEEPER_HOME=/opt/hadoop/zookeeper

export PATH=$PATH:$ZOOKEEPER_HOME/bin

[root@masternode hadoop]# source /etc/profile

[root@masternode hadoop]# cd zookeeper/conf/

[root@masternode conf]# pwd

/opt/hadoop/zookeeper/conf

③　配置zoo.cfg文件

配置文件存放在$ZOOKEEPER_HOME/conf/目录下，将zoo_sample.cfd文件名称改为zoo.cfg, 缺省的配置内容如下：

[root@masternode conf]# mv zoo_sample.cfg zoo.cfg

vi zoo.cfg

clientPort=2181

dataDir=/opt/hadoop/zookeeper/data/

dataLogDir=/opt/hadoop/zookeeper/log/

server.1=slavenode1:2888:3888

server.2=masternode:2888:3888

server.3=slavenode2:2888:3888

server.4=slavenode3:2888:3888

配置说明：
tickTime：这个时间是作为 Zookeeper 服务器之间或客户端与服务器之间维持心跳的时间间隔，也就是每个 tickTime 时间就会发送一个心跳。
dataDir：顾名思义就是 Zookeeper 保存数据的目录，默认情况下，Zookeeper 将写数据的日志文件也保存在这个目录里。
clientPort：这个端口就是客户端连接 Zookeeper 服务器的端口，Zookeeper 会监听这个端口，接受客户端的访问请求。

④　新建两个目录

mkdir /opt/hadoop/zookeeper/data/

mkdir /opt/hadoop/zookeeper/log/

在 /opt/hadoop/zookeeper/data/目录下创建一个文件：myid

touch /opt/hadoop/zookeeper/data/myid

[root@masternode data]# echo ‘1’> myid

⑤　将zookeeper目录和环境变量设置文件拷贝到其他机器上

[root@masternode hadoop]# scp -r zookeeper root@slavenode1:/opt/hadoop/

[root@masternode hadoop]# scp -r zookeeper root@slavenode3:/opt/hadoop/

[root@masternode hadoop]# scp -r zookeeper root@slavenode2:/opt/hadoop/

[root@masternode conf]# scp ~/.bash_profile root@slavenode2:~/.bash_profile

[root@masternode conf]# scp ~/.bash_profile root@slavenode3:~/.bash_profile

[root@masternode conf]# scp ~/.bash_profile root@slavenode1:~/.bash_profile

⑥　修改其他机器该文件

[root@masternode centos]# cat /opt/hadoop/zookeeper/data/myid

[root@slavenode3 centos]# cat /opt/hadoop/zookeeper/data/myid

⑦　启动zookeeper (所配的机器都要启动)

cd /opt/hadoop/zookeeper/bin

./zkServer.sh start

⑧　验证

[root@masternode zookeeper]# jps

16159 HQuorumPeer

14438 DataNode

16397 Jps

14549 NodeManager

查看状态

[root@slavenode3 centos]# /opt/hadoop/zookeeper/bin/zkServer.sh status

ZooKeeper JMX enabled by default

Using config: /opt/hadoop/zookeeper/bin/../conf/zoo.cfg

Mode: leader

客户端链接zookeeper

[root@slavenode3 centos]# /opt/hadoop/zookeeper/bin/zkCli.sh -server slavenode3:2181

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

[zk: slavenode3:2181(CONNECTED) 0] ls /

[hbase, zookeeper]

[zk: slavenode3:2181(CONNECTED) 1] ls /hbase

[meta-region-server, backup-masters, region-in-transition, table, draining, table-lock, running, master, namespace, hbaseid, online-snapshot, replication, splitWAL, recovering-regions, rs, flush-table-proc]

安装hadoop

1.安装hadoop软件包

①　解压hadoop软件包

Cd /opt/hadoop/

tar -xvf hadoop-2.7.2.tar.gz

[root@masternode hadoop]# ls

hadoop-2.7.2 jdk1.7.0_79 lost+found

hadoop-2.7.2.tar.gz jdk-7u79-linux-x64.tar.gz

[root@masternode hadoop]# cd hadoop-2.7.2

[root@masternode hadoop-2.7.2]# ls

bin include libexec NOTICE.txt sbin

etc lib LICENSE.txt README.txt share

②　设置环境变量

并把Hadoop的安装路径添加到"/etc/profile"中，修改"/etc/profile"文件，将以下语句添加到末尾，并使其生效(. /etc/profile)：

#set hadoop path

export HADOOP_HOME=/opt/hadoop/hadoop-2.7.2

export PATH=$PATH:$HADOOP_HOME/bin

[root@masternode conf]# hadoop version

Hadoop 2.7.2

Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r b165c4fe8a74265c792ce23f546c64604acf0e41

Compiled by jenkins on 2016-01-26T00:08Z

Compiled with protoc 2.5.0

From source with checksum d0fda26633fa762bff87ec759ebe689c

This command was run using /opt/hadoop/hadoop-2.7.2/share/hadoop/common/hadoop-common-2.7.2.jar

③　创建临时目录

[root@masternode hadoop-2.7.2]# mkdir -p /opt/hadoop/tmp

[root@masternode hadoop-2.7.2]# mkdir -p /opt/hadoop/hdfs/data

[root@masternode hadoop-2.7.2]# mkdir -p /opt/hadoop/hdfs/name

mkdir /opt/hadoop/hadoop-2.7.2/pids

mkdir /opt/hadoop/tmp/journal

chmod 755 /opt/hadoop/hdfs/data

chmod 755 /opt/hadoop/hdfs/name

chmod 755 /opt/hadoop/tmp

chmod 755 /opt/hadoop/tmp/journal

④　配置hadoop

修改配置文件

需要修改的配置文件主要有下面几个：

./core-site.xml

./hdfs-site.xml

./mapred-site.xml（开始的安装包中是没有这个文件的，需要将文

件./mapred-site.xml.template复制并且重名为mapred-site.xml）。

./yarn-site.xml

./yarn-env.sh

./hadoop-env.sh

./slave

[root@masternode hadoop-2.7.2]# pwd

/opt/hadoop/hadoop-2.7.2

[root@masternode hadoop-2.7.2]# ls

bin include libexec NOTICE.txt sbin

etc lib LICENSE.txt README.txt share

[root@masternode hadoop-2.7.2]# cd etc/hadoop/

/opt/hadoop/hadoop-2.7.2/etc/hadoop

（1）配置hadoop-env.sh

配置 hadoop-env.sh文件-->修改JAVA_HOME

export JAVA_HOME=/usr/java/jdk1.7.0_79/

export HADOOP_PID_DIR=$HADOOP_HOME/pids

export HADOOP_HOME=/opt/hadoop/hadoop-2.7.2

export HADOOP_PID_DIR=$HADOOP_HOME/pids

export PATH=$PATH:$HADOOP_HOME/bin

(2）配置yarn-env.sh文件

配置 yarn-env.sh 文件-->>修改JAVA_HOME

# some Java parameters

export JAVA_HOME=/usr/java/jdk1.7.0_79/

export YARN_PID_DIR=/opt/hadoop/hadoop-2.7.2/pids

（3）配置slaves

配置slaves文件-->>增加slave节点

slavenode2

slavenode3

(4)配置 core-site.xml文件

<name>fs.trash.interval</name>

</property>

<name>fs.defaultFS</name>

<value>hdfs://cluster-ha</value>

</property>

<name>dfs.journalnode.edits.dir</name>

<value>/opt/hadoop/tmp/journal</value>

</property>

<name>io.file.buffer.size</name>

</property>

<name>hadoop.tmp.dir</name>

<value>/opt/hadoop/tmp</value>

</property>

<name>ha.zookeeper.quorum</name>

<value>slavenode1:2181,slavenode3:2181,slavenode2:2181</value>

</property>

<name>ha.zookeeper.session-timeout.ms</name>

</property>

</configuration>

(5)配置hdfs-site.xml 文件-->>增加hdfs配置信息（namenode、datanode端口和目录位置）

<name>dfs.nameservices</name>

<value>cluster-ha</value>

</property>

<name>dfs.ha.namenodes.cluster-ha</name>

</property>

<name>dfs.namenode.rpc-address.cluster-ha.nn</name>

<value>slavenode1:9000</value>

</property>

<name>dfs.namenode.http-address.cluster-ha.nn</name>

<value>slavenode1:50070</value>

</property>

<name>dfs.namenode.rpc-address.cluster-ha.snn</name>

<value>masternode:9000</value>

</property>

<name>dfs.namenode.http-address.cluster-ha.snn</name>

<value>masternode:50070</value>

</property>

<name>dfs.namenode.shared.edits.dir</name>

<value>qjournal://slavenode1:8485;slavenode2:8485;slavenode3:8485;masternode:8485/cluster-ha</value>

</property>

<name>dfs.client.failover.proxy.provider.cluster-ha</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

<name>dfs.ha.fencing.methods</name>

<value>sshfence</value>

</property>

<name>dfs.journalnode.edits.dir</name>

<value>/opt/hadoop/tmp/journal</value>

</property>

<name>dfs.ha.automatic-failover.enabled</name>

</property>

<name>dfs.namenode.name.dir</name>

<value>file:/opt/hadoop/hdfs/name</value>

</property>

<name>dfs.datanode.data.dir</name>

<value>file:/opt/hadoop/hdfs/data</value>

</property>

<name>dfs.replication</name>

</property>

<name>dfs.webhdfs.enabled</name>

</property>

<name>ha.zookeeper.quorum</name>

<value>slavenode1:2181,slavenode3:2181,slavenode2:2181</value>

</property>

<name>dfs.namenode.handler.count</name>

<description>The number of server threads for the namenode.</description>

</property>

<name>dfs.datanode.handler.count</name>

<description>The number of server threads for the datanode.</description>

</property>

<name>dfs.client.socket-timeout</name>

</property>

<name>dfs.datanode.max.transfer.threads</name>

</property>

转载注明出处：https://www.heiqu.com/bc7a682ecf910654a6c6b40e9c9c78c8.html

Hadoop2.7.1 HA集群部署(3)

相关推荐