Ubuntu下Hadoop单机部署及分布式集群部署(2)

<configuration>
<property>
  <name>mapred.job.tracker</name>
  <value>localhost:54311</value>
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
</property>
</configuration>

hdfs-site.xml如下:
<configuration>
<property>
  <name>dfs.replication</name>
  <value>1</value>
  <description>Default block replication.
  The actual number of replications can be specified when the file is created.
  The default is used if replication is not specified in create time.
  </description>
</property>
</configuration>

vim hadoop-env.sh
export JAVA_HOME=/usr/lib/jdk1.6.0_33

ok,配置完毕
格式化HDFS:
/opt/hadoop/bin/hadoop namenode -format
输出
root@Master:/opt/hadoop# /opt/hadoop/bin/hadoop namenode -format
12/07/13 14:27:29 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = Master.Hadoop/127.0.1.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
Re-format filesystem in /opt/hadoop-datastore/dfs/name ? (Y or N) y
Format aborted in /opt/hadoop-datastore/dfs/name
12/07/13 14:27:35 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Master.Hadoop/127.0.1.1
************************************************************/

启动HDFS和MapReduce
改为hadoop用户
/opt/hadoop/bin/start-all.sh

输出
starting namenode, logging to /opt/hadoop/bin/../logs/hadoop-root-namenode-Master.Hadoop.out
The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is 3e:55:d8:be:47:46:21:95:29:9b:9e:c5:fb:02:f4:d2.
Are you sure you want to continue connecting (yes/no)? yes
localhost: Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
root@localhost's password:
localhost: starting datanode, logging to /opt/hadoop/bin/../logs/hadoop-root-datanode-Master.Hadoop.out
root@localhost's password:
localhost: starting secondarynamenode, logging to /opt/hadoop/bin/../logs/hadoop-root-secondarynamenode-Master.Hadoop.out
starting jobtracker, logging to /opt/hadoop/bin/../logs/hadoop-root-jobtracker-Master.Hadoop.out
root@localhost's password:
localhost: starting tasktracker, logging to /opt/hadoop/bin/../logs/hadoop-root-tasktracker-Master.Hadoop.out

9、停止服务的脚本是:
/opt/hadoop/bin/stop-all.sh


10.启动成功后,用jps查看下。


5.运行wordcount.java

在Hadoop所在目录里有几个jar文件,其中hadoop-examples-0.20.203.0.jar就是我们需要的,它里面含有wordcount,咱们使用命令建立测试的文件

(1)先在本地磁盘建立两个输入文件file01 和file02:
$ echo “Hello World Bye World” > file01
$ echo “Hello Hadoop Goodbye Hadoop” > file02

./hadoop fs -ls /

(2)在hdfs 中建立一个input 目录:./hadoop fs -mkdir input

删除./hadoop dfs -rmr input
(3)将file01 和file02 拷贝到hdfs 中:
./hadoop fs -copyFromLocal /home/linuxidc/file0* input

./hadoop fs -ls /user/root/input
Found 2 items
-rw-r--r--   1 root supergroup         22 2012-07-13 15:07 /user/root/input/file01
-rw-r--r--   1 root supergroup         28 2012-07-13 15:07 /user/root/input/file02

root@Master:/opt/hadoop/bin# ./hadoop fs -cat /user/root/input/file01/
Hello World Bye World

(4)执行wordcount:
$ Hadoop jar hadoop-0.20.1-examples.jar wordcount input output

$ bin/hadoop jar hadoop-0.20.1-examples.jar wordcount input output


Exceptioninthread"m

ain" java.io.IOException: Error openingjobjar: hadoop-0.20.2-examples.jar

at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
Caused by: java.util.zip.ZipException: error in opening zip file
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:114)
        at java.util.jar.JarFile.<init>(JarFile.java:135)
        at java.util.jar.JarFile.<init>(JarFile.java:72)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:88)


解决办法:
注意路径:
./hadoop jar /opt/hadoop/hadoop-0.20.2-examples.jar wordcount input output

输出

内容版权声明:除非注明,否则皆为本站原创文章。

转载注明出处:http://www.heiqu.com/65caa80911f473a1a9bbf3ece3e096e7.html