三、对master与slave之间做双机互信
1、对四台主机名与IP都需要按上面规划修改好。后面三台做相同修改，关闭selinux与iptables。
#vim /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=master
#hostname master
2、双机互信
SSH无密码原理
Master（NameNode | JobTracker）作为客户端，要实现无密码公钥认证，连接到服务器Salve（DataNode | Tasktracker）上时，需要在Master上生成一个密钥对，包括一个公钥和一个私钥，而后将公钥复制到所有的Slave上。当Master通过SSH连接Salve时，Salve就会生成一个随机数并用Master的公钥对随机数进行加密，并发送给Master。Master收到加密数之后再用私钥解密，并将解密数回传给Slave，Slave确认解密数无误之后就允许Master进行连接了。这就是一个公钥认证过程，其间不需要用户手工输入密码。重要过程是将客户端Master复制到Slave上
2.1、在master，和slave上的Hadoop用户下生成密钥对都执行下面命令
[hadoop@master ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
f8:09:0f:a1:63:72:c6:ed:2a:eb:18:83:d5:18:c7:b5 hadoop@master
The key's randomart image is:
[hadoop@master ~]$ ssh-copy-id -i 192.168.1.21
[hadoop@master ~]$ ssh-copy-id -i 192.168.1.22
[hadoop@master ~]$ ssh-copy-id -i 192.168.1.23
[hadoop@master ~]$ ssh-copy-id -i 192.168.1.24
此时master连接slave不再需要密码
[hadoop@slave1 ~]$ ssh-copy-id -i 192.168.1.22
[hadoop@slave2 ~]$ ssh-copy-id -i 192.168.1.22
[hadoop@slave3 ~]$ ssh-copy-id -i 192.168.1.22
此时master与slave连接不需要密码
四、启动hadoop集群
1、格式化名称节点
与普通文件系统一样，HDFS文件系统必须要先格式化，创建元数据数据结构以后才能使用。
[hadoop@master ~]$ hadoop namenode -format
13/07/25 19:02:41 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = master/192.168.1.21
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.0.3.16
STARTUP_MSG: build = git://localhost.localdomain/ on branch hdp-hadoop-1.0.3.16 -r 6cbf3791d67a1c726ce81927045e54eb1f2e3a95; compiled by 'jenkins' on Mon Oct 1 06:49:26 PDT 2012
************************************************************/
13/07/25 19:02:41 INFO util.GSet: VM type = 64-bit
13/07/25 19:02:41 INFO util.GSet: 2% max memory = 19.33375 MB
13/07/25 19:02:41 INFO util.GSet: capacity = 2^21 = 2097152 entries
13/07/25 19:02:41 INFO util.GSet: recommended=2097152, actual=2097152
13/07/25 19:02:41 INFO namenode.FSNamesystem: fsOwner=hadoop
13/07/25 19:02:42 INFO namenode.FSNamesystem: supergroup=supergroup
13/07/25 19:02:42 INFO namenode.FSNamesystem: isPermissionEnabled=true
13/07/25 19:02:42 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
13/07/25 19:02:42 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
13/07/25 19:02:42 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = -1
13/07/25 19:02:42 INFO namenode.NameNode: Caching file names occuring more than 10 times
13/07/25 19:02:42 INFO common.Storage: Image file of size 112 saved in 0 seconds.
13/07/25 19:02:42 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/usr/local/hadoop/tmp/dfs/name/current/edits
13/07/25 19:02:42 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/usr/local/hadoop/tmp/dfs/name/current/edits
13/07/25 19:02:42 INFO common.Storage: Storage directory /usr/local/hadoop/tmp/dfs/name has been successfully formatted.
13/07/25 19:02:42 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/192.168.1.21
************************************************************/
如果提示如下错误：
[hadoop@master ~]$ hadoop namenode -format
13/07/25 18:27:54 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = Java.net.UnknownHostException: master: master: Name or service not known
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.0.3.16
STARTUP_MSG: build = git://localhost.localdomain/ on branch hdp-hadoop-1.0.3.16 -r 6cbf3791d67a1c726ce81927045e54eb1f2e3a95; compiled by 'jenkins' on Mon Oct 1 06:49:26 PDT 2012
************************************************************/
13/07/25 18:27:54 INFO util.GSet: VM type = 64-bit
13/07/25 18:27:54 INFO util.GSet: 2% max memory = 19.33375 MB
13/07/25 18:27:54 INFO util.GSet: capacity = 2^21 = 2097152 entries
13/07/25 18:27:54 INFO util.GSet: recommended=2097152, actual=2097152
13/07/25 18:27:55 INFO namenode.FSNamesystem: fsOwner=hadoop
13/07/25 18:27:55 INFO namenode.FSNamesystem: supergroup=supergroup
13/07/25 18:27:55 INFO namenode.FSNamesystem: isPermissionEnabled=true
13/07/25 18:27:55 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
13/07/25 18:27:55 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
13/07/25 18:27:55 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = -1
13/07/25 18:27:55 INFO namenode.NameNode: Caching file names occuring more than 10 times
13/07/25 18:27:57 INFO common.Storage: Image file of size 112 saved in 0 seconds.
13/07/25 18:27:57 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/usr/local/hadoop/tmp/dfs/name/current/edits
13/07/25 18:27:57 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/usr/local/hadoop/tmp/dfs/name/current/edits
13/07/25 18:27:57 INFO common.Storage: Storage directory /usr/local/hadoop/tmp/dfs/name has been successfully formatted.
13/07/25 18:27:57 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at java.net.UnknownHostException: master: master: Name or service not known
则需要检查/etc/hosts文件中与/etc/sysconfig/network中的主机名设置是否对，并执行ping master是否正常。并删除rm -fr /usr/local/hadoop/tmp/* ,再次执行format，可以得到正常的的格式化。
2、启动hadoop
Hadoop提供了2个脚本start-dfs.sh和start-mapred.sh，分别用于启动hdfs相关的进程和mapred相关的进程。事实上，为了使用的便捷性，在NameNode和JobTracker运行于同一主机的场景中，Hadoop还专门提供了脚本start-all.sh脚本来自动执行前述两个脚本。不过，需要注意的是，使用这类脚本启动Hadoop集群，其需要当前主机的当前用户能够通过ssh服务以基于密钥认证的方式访问任何一个从节点。
除此之外，管理员也可以使用hadoop-daemon.sh脚本手动启动某一个Hadoop进程，或使用脚本hadoop-daemons.sh脚本手动启动某些进程。
[hadoop@master ~]$ start-all.sh
chown: changing ownership of `/usr/local/hadoop-1.0.3.16/libexec/../logs': Operation not permitted
starting namenode, logging to /usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-namenode-master.out
/usr/local/hadoop/bin/hadoop-daemon.sh: line 136: /usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-namenode-master.out: Permission denied
head: cannot open `/usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-namenode-master.out' for reading: No such file or directory
192.168.1.22: starting datanode, logging to /usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-datanode-slave1.out
192.168.1.23: starting datanode, logging to /usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-datanode-slave2.out
192.168.1.24: starting datanode, logging to /usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-datanode-slave3.out
192.168.1.21: chown: changing ownership of `/usr/local/hadoop-1.0.3.16/libexec/../logs': Operation not permitted
192.168.1.21: starting secondarynamenode, logging to /usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-secondarynamenode-master.out
192.168.1.21: /usr/local/hadoop/bin/hadoop-daemon.sh: line 136: /usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-secondarynamenode-master.out: Permission denied
192.168.1.21: head: cannot open `/usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-secondarynamenode-master.out' for reading: No such file or directory
chown: changing ownership of `/usr/local/hadoop-1.0.3.16/libexec/../logs': Operation not permitted
starting jobtracker, logging to /usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-jobtracker-master.out
/usr/local/hadoop/bin/hadoop-daemon.sh: line 136: /usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-jobtracker-master.out: Permission denied
head: cannot open `/usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-jobtracker-master.out' for reading: No such file or directory
192.168.1.23: starting tasktracker, logging to /usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-tasktracker-slave2.out
192.168.1.24: starting tasktracker, logging to /usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-tasktracker-slave3.out
192.168.1.22: starting tasktracker, logging to /usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-tasktracker-slave1.out
启动的过程中提示没有权限，那么需要检查hadoop下的所有文件的属主，属组是否都是hadoop。如果不是则切换到root进行修改。
[root@master ~]# chown -R hadoop:hadoop /usr/local/hadoop/*
[hadoop@master hadoop]$ stop-all.sh 先停止，然后再启动
no jobtracker to stop
192.168.1.23: stopping tasktracker
192.168.1.22: stopping tasktracker
192.168.1.24: stopping tasktracker
no namenode to stop
192.168.1.23: stopping datanode
192.168.1.24: stopping datanode
192.168.1.22: stopping datanode
192.168.1.21: no secondarynamenode to stop
[hadoop@master hadoop]$ start-all.sh 这次没有再报错
starting namenode, logging to /usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-namenode-master.out
192.168.1.23: starting datanode, logging to /usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-datanode-slave2.out
192.168.1.22: starting datanode, logging to /usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-datanode-slave1.out
192.168.1.24: starting datanode, logging to /usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-datanode-slave3.out
192.168.1.21: starting secondarynamenode, logging to /usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-secondarynamenode-master.out
starting jobtracker, logging to /usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-jobtracker-master.out
192.168.1.23: starting tasktracker, logging to /usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-tasktracker-slave2.out
192.168.1.24: starting tasktracker, logging to /usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-tasktracker-slave3.out
192.168.1.22: starting tasktracker, logging to /usr/local/hadoop-1.0.3.16/libexec/../logs/hadoop-hadoop-tasktracker-slave1.out