修改mapred-site.xml文件
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
</configuration>
修改yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>1024</value>
</property>
</configuration>
在slaves文件中添加
slave1
slave2
slave3
似乎一切都好像搞定了,少年,别急,吓死你!
ldd /home/hadoop/hadoop-2.6.0/lib/native/libhadoop.so.1.0.0
然后你会看到:
/home/hadoop/hadoop-2.6.0/lib/native/libhadoop.so.1.0.0: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /home/hadoop/hadoop-2.6.0/lib/native/libhadoop.so.1.0.0)
linux-vdso.so.1 => (0x00007fff24dbc000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007ff8c6371000)
libc.so.6 => /lib64/libc.so.6 (0x00007ff8c5fdc000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff8c679b000)
人生是这样的无情,人生是这样的冷酷,之前有个小朋友问过我这个问题......我没有理,现在,然我亲手灭了这个问题!
不过大家可能明白了为什么我一上来就装个gcc了吧.
yum install -y wget
wget
tar zxvf glibc-2.14.tar.gz
cd glibc-2.14
mkdir build
cd build
../configure --prefix=/usr/local/glibc-2.14
make
make install
ln -sf /usr/local/glibc-2.14/lib/libc-2.14.so /lib64/libc.so.6
此时,ldd /home/hadoop/hadoop-2.6.0/lib/native/libhadoop.so.1.0.0
就没有任何问题了
linux-vdso.so.1 => (0x00007fff72b7c000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fb996ce9000)
libc.so.6 => /lib64/libc.so.6 (0x00007fb99695c000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb997113000
这样,我们的镜像就可以commit了
docker commit master songfy/hadoop
我们可以用docker images来查看镜像.
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
songfy/hadoop latest 311318c0a407 42 seconds ago 1.781 GB
insaneworks/centos latest 9d29fe7b2e52 9 days ago 121.1 MB
下面我们来启动hadoop集群
三.启动hadoop集群
docker rm master
sudo docker run -it -p 50070:50070 -p 19888:19888 -p 8088:8088 -h master --name master songfy/hadoop /bin/bash
sudo docker run -it -h slave1 --name slave1 songfy/hadoop /bin/bash
sudo docker run -it -h slave2 --name slave2 songfy/hadoop /bin/bash
sudo docker run -it -h slave3 --name slave3 songfy/hadoop /bin/bash
attach到每个节点上执行
source /etc/profile
service sshd start
接下来我们还要给每台机器配host
docker inspect --format='{{.NetworkSettings.IPAddress}}' master
这条语句可以查看ip
172.17.0.4 master
172.17.0.5 slave1
172.17.0.6 slave2
172.17.0.7 slave3
用scp将hosts文件分发到各个node中.
好了,我们终于要启动hadoop了.
hadoop namenode -format
/home/hadoop/hadoop-2.6.0/sbin/start-dfs.sh
/home/hadoop/hadoop-2.6.0/sbin/start-yarn.sh
用jps查看,发现都起来了.
下面我们简单来对hdfs操作一下.
hadoop fs -mkdir /input
hadoop fs -ls /
drwxr-xr-x - root supergroup 0 2015-08-09 09:09 /input
下面我们来运行一下大名鼎鼎的wordcount程序来看看.
hadoop fs -put /home/hadoop/hadoop-2.6.0/etc/hadoop/* /input/
hadoop jar /home/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount /input/ /output/wordcount/
不要以为一下就成功了.我们发现事实上,程序并没有跑出来,查了下日志,看到: