CentOS 6.4下Hadoop 0.20.2安装实录

CentOS 6.4下Hadoop 0.20.2安装实录

下载网址：

资料网址

节点规划：
10.10.1.131  hadoop1
10.10.1.132  hadoop2
10.10.1.133  hadoop3
10.10.1.134  dog
10.10.1.135  cat
10.10.1.136  gangster
一、解压安装
在主节点上做
验证JDK是否安装
[root@hadoop1 ~]# java -version
java version "1.7.0_09-icedtea"
OpenJDK Runtime Environment (rhel-2.3.4.1.el6_3-x86_64)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)
验证SSH是否安装
[root@hadoop1 ~]# ssh -version
OpenSSH_5.3p1, OpenSSL 1.0.0-fips 29 Mar 2010
Bad escape character 'rsion'.
各节点上都做
[root@hadoop1 ~]#vi /etc/hosts
10.10.1.131  hadoop1
10.10.1.132  hadoop2
10.10.1.133  hadoop3
10.10.1.134  dog
10.10.1.135  cat
10.10.1.136  gangster
[root@hadoop1 ~]# useradd hadoop
[root@hadoop1 ~]# passwd hadoop
[root@hadoop1 ~]# vi /etc/sysconfig/iptables
-A INPUT -s 10.10.1.131 -j ACCEPT
-A INPUT -s 10.10.1.132 -j ACCEPT
-A INPUT -s 10.10.1.133 -j ACCEPT
-A INPUT -s 10.10.1.170 -j ACCEPT
-A INPUT -s 10.10.1.171 -j ACCEPT
-A INPUT -s 10.10.1.172 -j ACCEPT
二、安装hadoop
把下载的hadoop-0.20.2.tar.gz文件上传到/home/hadoop目录下
[hadoop@hadoop1 ~]$ tar xzvf hadoop-0.20.2.tar.gz
-rw-r--r--.  1 hadoop hadoop 44575568 Feb 16 21:34 hadoop-0.20.2.tar.gz
[hadoop@hadoop1 ~]$ cd hadoop-0.20.2
[hadoop@hadoop1 hadoop-0.20.2]$ ll
total 4872
drwxr-xr-x.  2 hadoop hadoop    4096 Feb 16 21:37 bin
-rw-rw-r--.  1 hadoop hadoop  74035 Feb 19  2010 build.xml
drwxr-xr-x.  4 hadoop hadoop    4096 Feb 19  2010 c++
-rw-rw-r--.  1 hadoop hadoop  348624 Feb 19  2010 CHANGES.txt
drwxr-xr-x.  2 hadoop hadoop    4096 Feb 16 21:37 conf
drwxr-xr-x. 13 hadoop hadoop    4096 Feb 19  2010 contrib
drwxr-xr-x.  7 hadoop hadoop    4096 Feb 16 21:37 docs
-rw-rw-r--.  1 hadoop hadoop    6839 Feb 19  2010 hadoop-0.20.2-ant.jar
-rw-rw-r--.  1 hadoop hadoop 2689741 Feb 19  2010 hadoop-0.20.2-core.jar
-rw-rw-r--.  1 hadoop hadoop  142466 Feb 19  2010 hadoop-0.20.2-examples.jar
-rw-rw-r--.  1 hadoop hadoop 1563859 Feb 19  2010 hadoop-0.20.2-test.jar
-rw-rw-r--.  1 hadoop hadoop  69940 Feb 19  2010 hadoop-0.20.2-tools.jar
drwxr-xr-x.  2 hadoop hadoop    4096 Feb 16 21:37 ivy
-rw-rw-r--.  1 hadoop hadoop    8852 Feb 19  2010 ivy.xml
drwxr-xr-x.  5 hadoop hadoop    4096 Feb 16 21:37 lib
drwxr-xr-x.  2 hadoop hadoop    4096 Feb 16 21:37 librecordio
-rw-rw-r--.  1 hadoop hadoop  13366 Feb 19  2010 LICENSE.txt
-rw-rw-r--.  1 hadoop hadoop    101 Feb 19  2010 NOTICE.txt
-rw-rw-r--.  1 hadoop hadoop    1366 Feb 19  2010 README.txt
drwxr-xr-x. 15 hadoop hadoop    4096 Feb 16 21:37 src
drwxr-xr-x.  8 hadoop hadoop    4096 Feb 19  2010 webap
三、配置主节点hadoop用户无密码访问从节点
在MASTER节点上生成MASTER节点的ssh密钥对
[root@hadoop1 ~]#su - hadoop
[hadoop@hadoop1 ~]$ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
b9:5c:48:74:25:33:ac:9f:11:c9:77:5e:02:43:3b:ba hadoop@hadoop1.cfzq.com
The key's randomart image is:
+--[ RSA 2048]----+
|        .o=+=.  |
|      . .=+.oo .|
|        .. ooo o |
|      ..o.. ..  |
|        S.oo    |
|      . oo.    |
|        o E      |
|                |
|                |
+-----------------+
把公钥拷贝到各SLAVE节点上
[hadoop@hadoop1 .ssh]$ scp id_rsa.pub hadoop@hadoop2:~/master-key
The authenticity of host 'hadoop2 (10.10.1.132)' can't be established.
RSA key fingerprint is f9:47:3e:59:39:10:cd:7d:a4:5c:0d:ab:df:1f:14:21.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hadoop2,10.10.1.132' (RSA) to the list of known hosts.
hadoop@hadoop2's password:
id_rsa.pub                                                                                                                100%  405    0.4KB/s  00:00   
[hadoop@hadoop1 .ssh]$ scp id_rsa.pub hadoop@hadoop3:~/master-key
The authenticity of host 'hadoop3 (10.10.1.133)' can't be established.
RSA key fingerprint is f9:47:3e:59:39:10:cd:7d:a4:5c:0d:ab:df:1f:14:21.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hadoop3,10.10.1.133' (RSA) to the list of known hosts.
hadoop@hadoop3's password:
id_rsa.pub                                                                                                                100%  405    0.4KB/s  00:00   
[hadoop@hadoop1 .ssh]$ scp id_rsa.pub hadoop@cat:~/master-key
The authenticity of host 'cat (10.10.1.171)' can't be established.
RSA key fingerprint is f9:47:3e:59:39:10:cd:7d:a4:5c:0d:ab:df:1f:14:21.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'cat,10.10.1.171' (RSA) to the list of known hosts.
hadoop@cat's password:
id_rsa.pub                                                                                                                100%  405    0.4KB/s  00:00   
[hadoop@hadoop1 .ssh]$ scp id_rsa.pub hadoop@dog:~/master-key
The authenticity of host 'dog (10.10.1.170)' can't be established.
RSA key fingerprint is f9:47:3e:59:39:10:cd:7d:a4:5c:0d:ab:df:1f:14:21.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'dog,10.10.1.170' (RSA) to the list of known hosts.
hadoop@dog's password:
id_rsa.pub                                                                                                                100%  405    0.4KB/s  00:00   
[hadoop@hadoop1 .ssh]$ scp id_rsa.pub hadoop@gangster:~/master-key
The authenticity of host 'gangster (10.10.1.172)' can't be established.
RSA key fingerprint is f9:47:3e:59:39:10:cd:7d:a4:5c:0d:ab:df:1f:14:21.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'gangster,10.10.1.172' (RSA) to the list of known hosts.
hadoop@gangster's password:
id_rsa.pub                                                                                                                100%  405    0.4KB/s  00:00   
在各SLAVE节点上：
[hadoop@hadoop2 ~]$ mkdir .ssh
[hadoop@hadoop2 ~]$ chmod 700 .ssh/
[hadoop@hadoop2 ~]$ mv master-key .ssh/authorized_keys
[hadoop@hadoop2 ~]$ cd .ssh/
[hadoop@hadoop2 .ssh]$ chmod 600 authorized_keys
[hadoop@hadoop3 ~]$ mkdir .ssh
[hadoop@hadoop3 ~]$ chmod 700 .ssh/
[hadoop@hadoop3 ~]$ mv master-key .ssh/authorized_keys
[hadoop@hadoop3 ~]$ cd .ssh/
[hadoop@hadoop3 .ssh]$ chmod 600 authorized_keys
[hadoop@dog ~]$ mkdir .ssh
[hadoop@dog ~]$ chmod 700 .ssh/
[hadoop@dog ~]$ mv master-key .ssh/authorized_keys
[hadoop@dog ~]$ cd .ssh/
[hadoop@dog .ssh]$ chmod 600 authorized_keys
[hadoop@cat ~]$ mkdir .ssh
[hadoop@cat ~]$ chmod 700 .ssh/
[hadoop@cat ~]$ mv master-key .ssh/authorized_keys
[hadoop@cat ~]$ cd .ssh/
[hadoop@cat .ssh]$ chmod 600 authorized_keys
[hadoop@gangster ~]$ mkdir .ssh
[hadoop@gangster ~]$ chmod 700 .ssh/
[hadoop@gangster ~]$  mv master-key .ssh/authorized_keys
[hadoop@gangster ~]$ cd .ssh/
[hadoop@gangster .ssh]$ chmod 600 authorized_keys
在MASTER节点上测试主节点无口令访问各从节点
[hadoop@hadoop1 .ssh]$ ssh hadoop2
Last login: Sat Feb 15 19:35:21 2014 from hadoop1
[hadoop@hadoop2 ~]$ exit
logout
Connection to hadoop2 closed.
[hadoop@hadoop1 .ssh]$ ssh hadoop3
Last login: Sat Feb 15 21:57:38 2014 from hadoop1
[hadoop@hadoop3 ~]$ exit
logout
Connection to hadoop3 closed.
[hadoop@hadoop1 .ssh]$ ssh cat
Last login: Sat Feb 15 14:33:50 2014 from hadoop1
[hadoop@cat ~]$ exit
logout
Connection to cat closed.
[hadoop@hadoop1 .ssh]$ ssh dog
Last login: Sun Feb 16 20:41:19 2014 from hadoop1
[hadoop@dog ~]$ exit
logout
Connection to dog closed.
[hadoop@hadoop1 .ssh]$ ssh gangster
Last login: Sat Feb 15 18:03:45 2014 from hadoop1
[hadoop@gangster ~]$ exit
logout
Connection to gangster closed.