Hadoop动态添加删除节点datanode及恢复

修改/etc/sysconfig/network

然后执行命令
hostname 主机名
这个时候可以注销一下系统,再重登录之后就行了

[root@localhost ~]# hostname
localhost.localdomain
[root@localhost ~]# hostname -i
::1 127.0.0.1
[root@localhost ~]#
[root@localhost ~]# cat /etc/sysconfig/network
# Created by anaconda
NETWORKING=yes
HOSTNAME=slave2
GATEWAY=192.168.48.2
# Oracle-rdbms-server-11gR2-preinstall : Add NOZEROCONF=yes
NOZEROCONF=yes
[root@localhost ~]# hostname slave2
[root@localhost ~]# hostname
slave2
[root@localhost ~]# su - hadoop
Last login: Sat Feb 24 14:25:48 CST 2018 on pts/1
[hadoop@slave2 ~]$ su - root

建datanode目录并改所有者

(此处的具体路径值,请参照namenode中/usr/hadoop/hadoop-2.7.5/etc/hadoop/hdfs-site.xml,core-site.xml中的dfs.name.dir,dfs.data.dir,dfs.tmp.dir等)

Su - root

# mkdir -p /usr/local/hadoop-2.7.5/tmp/dfs/data

# chmod -R 777 /usr/local/hadoop-2.7.5/tmp

# chown -R hadoop:hadoop /usr/local/hadoop-2.7.5

[root@slave2 ~]# mkdir -p /usr/local/hadoop-2.7.5/tmp/dfs/data
[root@slave2 ~]# chmod -R 777 /usr/local/hadoop-2.7.5/tmp
 [root@slave2 ~]# chown -R hadoop:hadoop /usr/local/hadoop-2.7.5
 [root@slave2 ~]# pwd
/root
[root@slave2 ~]# cd /usr/local/
[root@slave2 local]# ll
total 0
drwxr-xr-x. 2 root  root  46 Mar 21  2017 bin
drwxr-xr-x. 2 root  root    6 Jun 10  2014 etc
drwxr-xr-x. 2 root  root    6 Jun 10  2014 games
drwxr-xr-x  3 hadoop hadoop 16 Feb 24 18:18 hadoop-2.7.5
drwxr-xr-x. 2 root  root    6 Jun 10  2014 include
drwxr-xr-x. 2 root  root    6 Jun 10  2014 lib
drwxr-xr-x. 2 root  root    6 Jun 10  2014 lib64
drwxr-xr-x. 2 root  root    6 Jun 10  2014 libexec
drwxr-xr-x. 2 root  root    6 Jun 10  2014 sbin
drwxr-xr-x. 5 root  root  46 Dec 17  2015 share
drwxr-xr-x. 2 root  root    6 Jun 10  2014 src
[root@slave2 local]#

ssh互信,即实现 master-->slave2免密码

master:

[root@hadoop-master ~]# cat /etc/hosts

127.0.0.1  localhost localhost.localdomain localhost4 localhost4.localdomain4

::1        localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.48.129    hadoop-master

192.168.48.132    slave1

192.168.48.131    slave2

[hadoop@hadoop-master ~]$ scp /usr/hadoop/.ssh/authorized_keys hadoop@slave2:/usr/hadoop/.ssh

The authenticity of host 'slave2 (192.168.48.131)' can't be established.

ECDSA key fingerprint is 1e:cd:d1:3d:b0:5b:62:45:a3:63:df:c7:7a:0f:b8:7c.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'slave2,192.168.48.131' (ECDSA) to the list of known hosts.

hadoop@slave2's password:

authorized_keys       

[hadoop@hadoop-master ~]$ ssh hadoop@slave2

Last login: Sat Feb 24 18:27:33 2018

[hadoop@slave2 ~]$

[hadoop@slave2 ~]$ exit

logout

Connection to slave2 closed.

[hadoop@hadoop-master ~]$

2. 修改namenode节点的slave文件,增加新节点信息

[hadoop@hadoop-master hadoop]$ pwd

/usr/hadoop/hadoop-2.7.5/etc/hadoop

[hadoop@hadoop-master hadoop]$ vi slaves

slave1

slave2

3. 在namenode节点上,将hadoop-2.7.3复制到新节点上,并在新节点上删除data和logs目录中的文件

Master

[hadoop@hadoop-master ~]$ scp -R hadoop-2.7.5 hadoop@slave2:/usr/hadoop

Slave2

[hadoop@slave2 hadoop-2.7.5]$ ll

total 124

drwxr-xr-x 2 hadoop hadoop  4096 Feb 24 14:29 bin

drwxr-xr-x 3 hadoop hadoop    19 Feb 24 14:30 etc

drwxr-xr-x 2 hadoop hadoop  101 Feb 24 14:30 include

drwxr-xr-x 3 hadoop hadoop    19 Feb 24 14:29 lib

drwxr-xr-x 2 hadoop hadoop  4096 Feb 24 14:29 libexec

-rw-r--r-- 1 hadoop hadoop 86424 Feb 24 18:44 LICENSE.txt

drwxrwxr-x 2 hadoop hadoop  4096 Feb 24 14:30 logs

-rw-r--r-- 1 hadoop hadoop 14978 Feb 24 18:44 NOTICE.txt

-rw-r--r-- 1 hadoop hadoop  1366 Feb 24 18:44 README.txt

drwxr-xr-x 2 hadoop hadoop  4096 Feb 24 14:29 sbin

drwxr-xr-x 4 hadoop hadoop    29 Feb 24 14:30 share

[hadoop@slave2 hadoop-2.7.5]$ pwd

/usr/hadoop/hadoop-2.7.5

[hadoop@slave2 hadoop-2.7.5]$ rm -R logs/*

4. 启动新datanode的datanode和nodemanger进程

先确认namenode和当前的datanode中,etc/hoadoop/excludes文件中无待加入的主机,再进行下面操作

[hadoop@slave2 hadoop-2.7.5]$ sbin/hadoop-daemon.sh start datanode
starting datanode, logging to /usr/hadoop/hadoop-2.7.5/logs/hadoop-hadoop-datanode-slave2.out
[hadoop@slave2 hadoop-2.7.5]$ sbin/yarn-daemon.sh start nodemanager
starting datanode, logging to /usr/hadoop/hadoop-2.7.5/logs/yarn-hadoop-datanode-slave2.out
[hadoop@slave2 hadoop-2.7.5]$
[hadoop@slave2 hadoop-2.7.5]$ jps
3897 DataNode
6772 NodeManager
8189 Jps
[hadoop@slave2 ~]$

5、在NameNode上刷新节点

[hadoop@hadoop-master ~]$ hdfs dfsadmin -refreshNodes
Refresh nodes successful
[hadoop@hadoop-master ~]$sbin/start-balancer.sh

6. 在namenode查看当前集群情况,

确认信节点已经正常加入

内容版权声明:除非注明,否则皆为本站原创文章。

转载注明出处:https://www.heiqu.com/700929ed1029e37a69854c0838f3b472.html