Hadoop 单节点 伪分布 安装札记(2)

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:/usr/local/hadoop-2.6.0/tmp</value>
        <description>Abase for other temporary directories.</description>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

同样的,修改配置文件 hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/usr/local/hadoop-2.6.0/tmp/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/usr/local/hadoop-2.6.0/tmp/dfs/data</value>
    </property>
</configuration>

关于Hadoop配置项的一点说明

虽 然只需要配置 fs.defaultFS 和 dfs.replication 就可以运行(官方教程如此),不过若没有配置 hadoop.tmp.dir 参数,则默认使用的临时目录为 /tmp/hadoo-hadoop,而这个目录在重启时有可能被系统清理掉,导致必须重新执行 format 才行。所以我们进行了设置,同时也指定 dfs.namenode.name.dir 和 dfs.datanode.data.dir,否则在接下来的步骤中可能会出错。

免密码ssh设置
现在确认能否不输入口令就用ssh登录localhost:
# ssh localhost date

如果不输入口令就无法用ssh登陆localhost,执行下面的命令:
# ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
# cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
#chmod  600 ~/.ssh/authorized_keys

格式化一个新的分布式文件系统:
$ bin/hadoop namenode -format
15/12/23 11:30:20 INFO util.GSet: VM type      = 64-bit
15/12/23 11:30:20 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
15/12/23 11:30:20 INFO util.GSet: capacity      = 2^15 = 32768 entries
15/12/23 11:30:20 INFO namenode.NNConf: ACLs enabled? false
15/12/23 11:30:20 INFO namenode.NNConf: XAttrs enabled? true
15/12/23 11:30:20 INFO namenode.NNConf: Maximum size of an xattr: 16384
15/12/23 11:30:20 INFO namenode.FSImage: Allocated new BlockPoolId: BP-823870322-192.168.9.70-1450841420347
15/12/23 11:30:20 INFO common.Storage: Storage directory /usr/local/hadoop-2.6.0/tmp/dfs/name has been successfully formatted.
15/12/23 11:30:20 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
15/12/23 11:30:20 INFO util.ExitUtil: Exiting with status 0
15/12/23 11:30:20 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ocean-lab.ocean.org/192.168.9.70
************************************************************/
成功的话,会看到 “successfully formatted” 和 “Exitting with status 0″ 的提示

注意
下次启动 hadoop 时,无需进行 NameNode 的初始化,只需要运行 ./sbin/start-dfs.sh 就可以!

启动 NameNode 和  DataNode

$  ./sbin/start-dfs.sh
15/12/23 11:37:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-hadoop-namenode-ocean-lab.ocean.org.out
localhost: starting datanode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-hadoop-datanode-ocean-lab.ocean.org.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
RSA key fingerprint is a5:26:42:a0:5f:da:a2:88:52:04:9c:7f:8d:6a:98:9b.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-hadoop-secondarynamenode-ocean-lab.ocean.org.out
15/12/23 11:37:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable


[13:57:08][hadoop@ocean-lab hadoop-2.6.0]$ jps
27686 SecondaryNameNode
28455 Jps
27501 DataNode
27405 NameNode
27006 GetConf

如果没有进程则说明启动失败 查看日志bebug

成功启动后,可以访问 Web 界面  [ip,fqdn]:/50070  查看 NameNode 和 Datanode 信息,还可以在线查看 HDFS 中的文件。

Hadoop 单节点 & 伪分布 安装札记


运行Hadoop伪分布式实例


上面的单机模式,grep 例子读取的是本地数据,伪分布式读取的则是 HDFS 上的数据。

内容版权声明:除非注明,否则皆为本站原创文章。

转载注明出处:https://www.heiqu.com/39c8e2e98c866d4346d70f7164d84143.html