创建三台虚拟机,IP地址为:192.168.169.101,192.168.169.102,192.168.169.103
将192.168.169.102为namenode,192.168.169.101,192.168.169.103为datanode
关闭防火墙,安装JDK1.8,设置SSH无密码登录,下载Hadoop-2.8.2.tar.gz到/hadoop目录下。
1 安装namenode结点
将hadoop-2.8.2.tar.gz解压到192.168.169.102的hadoop用户的home目录/hadoop下
[hadoop@hadoop02 ~]$ pwd
/hadoop
[hadoop@hadoop02 ~]$ tar zxvf hadoop-2.8.2.tar.gz
... ...
[hadoop@hadoop02 ~]$ cd hadoop-2.8.2/
[hadoop@hadoop02 hadoop-2.8.2]$ pwd
/hadoop/hadoop-2.8.2
[hadoop@hadoop02 hadoop-2.8.2]$ ls -l
总用量 132
drwxr-xr-x 2 hadoop hadoop 4096 10月 20 05:11 bin
drwxr-xr-x 3 hadoop hadoop 19 10月 20 05:11 etc
drwxr-xr-x 2 hadoop hadoop 101 10月 20 05:11 include
drwxr-xr-x 3 hadoop hadoop 19 10月 20 05:11 lib
drwxr-xr-x 2 hadoop hadoop 4096 10月 20 05:11 libexec
-rw-r--r-- 1 hadoop hadoop 99253 10月 20 05:11 LICENSE.txt
-rw-r--r-- 1 hadoop hadoop 15915 10月 20 05:11 NOTICE.txt
-rw-r--r-- 1 hadoop hadoop 1366 10月 20 05:11 README.txt
drwxr-xr-x 2 hadoop hadoop 4096 10月 20 05:11 sbin
drwxr-xr-x 4 hadoop hadoop 29 10月 20 05:11 share
[hadoop@hadoop02 hadoop-2.8.2]$
2 配置Hadoop环境变量
[hadoop@hadoop02 bin]$ vi /hadoop/.bash_profile
export HADOOP_HOME=/hadoop/hadoop-2.8.2
export PATH=$PATH:$HADOOP_HOME/bin
注意:另两台虚拟机也要同样配置。
执行source ~./.bash_profile使配置生效,并验证:
[hadoop@hadoop02 bin]$ source ~/.bash_profile
[hadoop@hadoop02 bin]$ echo $HADOOP_HOME
/hadoop/hadoop-2.8.2
[hadoop@hadoop02 bin]$ echo $PATH
/usr/java/jdk1.8.0_151/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/hadoop/.local/bin:/hadoop/bin:/hadoop/.local/bin:/hadoop/bin:/hadoop/hadoop-2.8.2/bin
[hadoop@hadoop02 bin]$
3 创建hadoop工作目录
[hadoop@hadoop02 bin]$ mkdir -p /hadoop/hadoop/dfs/name /hadoop/hadoop/dfs/data /hadoop/hadoop/tmp
4 修改hadoop配制文件
共修改7个配制文件:
hadoop-env.sh: java环境变量
yarn-env.sh: 制定yarn框架的java运行环境,yarn它将资源管理和处理组件分开。基于yarn的架构不受MapReduce约束。
slaves: 指定datanode数据存储服务器
core-site.xml: 指定访问hadoop web界面的路径
hdfs-site.xml: 文件系统的配置文件
mapred-site.xml: MapReducer任务配置文件
yarn-site.xml: yarn框架配置,主要是一些任务的启动位置
4.1 /hadoop/hadoop-2.8.2/etc/hadoop/hadoop-env.sh
[hadoop@hadoop02 hadoop]$ vi /hadoop/hadoop-2.8.2/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_151/
4.2 /hadoop/hadoop-2.8.2/etc/hadoop/yarn-env.sh
[hadoop@hadoop02 hadoop]$ vi /hadoop/hadoop-2.8.2/etc/hadoop/yarn-env.sh
JAVA_HOME=/usr/java/jdk1.8.0_151/
4.3 /hadoop/hadoop-2.8.2/etc/hadoop/slaves
[hadoop@hadoop02 hadoop]$ vi /hadoop/hadoop-2.8.2/etc/hadoop/slaves
hadoop01
hadoop03
4.4 /hadoop/hadoop-2.8.2/etc/hadoop/core-site.xml
[hadoop@hadoop02 hadoop]$ vi /hadoop/hadoop-2.8.2/etc/hadoop/core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoop/hadoop/tmp</value> //手工创建的
<final>true</final>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.169.102:9000</value>
<final>true</final>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
</configuration>
4.5 /hadoop/hadoop-2.8.2/etc/hadoop/hdfs-site.xml
[hadoop@hadoop02 hadoop]$ vi /hadoop/hadoop-2.8.2/etc/hadoop/hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/hadoop/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/hadoop/hadoop/dfs/data</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop02:9001</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
4.6 /hadoop/hadoop-2.8.2/etc/hadoop/mapred-queues.xml