将测试文件上传到HDFS
首先我们要在HDFS上再创建两个文件,一个 input(输入) / ouput(输出)的文件夹。
调用Hadoop自带的WordCount方法
[root@localhost ~]# hadoop jar /opt/hadoop/hadoop-2.6.0-cdh5.8.2/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.8.2.jar wordcount /input /ouput/test 16/10/26 04:49:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/10/26 04:49:38 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032 16/10/26 04:49:42 INFO input.FileInputFormat: Total input paths to process : 1 16/10/26 04:49:43 INFO mapreduce.JobSubmitter: number of splits:1 16/10/26 04:49:44 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1477471653063_0001 16/10/26 04:49:46 INFO impl.YarnClientImpl: Submitted application application_1477471653063_0001 16/10/26 04:49:47 INFO mapreduce.Job: The url to track the job: :8088/proxy/application_1477471653063_0001/ 16/10/26 04:49:47 INFO mapreduce.Job: Running job: job_1477471653063_0001 16/10/26 04:50:21 INFO mapreduce.Job: Job job_1477471653063_0001 running in uber mode : false 16/10/26 04:50:21 INFO mapreduce.Job: map 0% reduce 0% 16/10/26 04:50:44 INFO mapreduce.Job: map 100% reduce 0% 16/10/26 04:51:04 INFO mapreduce.Job: map 100% reduce 100% 16/10/26 04:51:06 INFO mapreduce.Job: Job job_1477471653063_0001 completed successfully 16/10/26 04:51:06 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=116 FILE: Number of bytes written=232107 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=177 HDFS: Number of bytes written=78 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=18128 Total time spent by all reduces in occupied slots (ms)=17756 Total time spent by all map tasks (ms)=18128 Total time spent by all reduce tasks (ms)=17756 Total vcore-seconds taken by all map tasks=18128 Total vcore-seconds taken by all reduce tasks=17756 Total megabyte-seconds taken by all map tasks=18563072 Total megabyte-seconds taken by all reduce tasks=18182144 Map-Reduce Framework Map input records=10 Map output records=10 Map output bytes=116 Map output materialized bytes=116 Input split bytes=101 Combine input records=10 Combine output records=8 Reduce input groups=8 Reduce shuffle bytes=116 Reduce input records=8 Reduce output records=8 Spilled Records=16 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=454 CPU time spent (ms)=3450 Physical memory (bytes) snapshot=306806784 Virtual memory (bytes) snapshot=3017633792 Total committed heap usage (bytes)=163450880 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=76 File Output Format Counters Bytes Written=78 运行完成我们看看计算出来的结果: [root@localhost ~]# hdfs dfs -ls /ouput/test 16/10/26 04:53:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 2 items -rw-r--r-- 1 root supergroup 0 2016-10-26 04:51 /ouput/test/_SUCCESS -rw-r--r-- 1 root supergroup 78 2016-10-26 04:51 /ouput/test/part-r-00000 [root@localhost ~]# hdfs dfs -cat /ouput/test/part-r-00000 16/10/26 04:53:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 天天顺利 1 小张 1 张张 2 果哥 1 泽安 2 祖渊 1 跨越 1 麒麟 1 错误记录 [root@localhost ~]# hadoop jar /opt/hadoop/hadoop-2.6.0-cdh5.8.2/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.8.2.jar wordcount /input /ouput 16/10/26 04:40:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/10/26 04:40:33 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 16/10/26 04:40:34 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to localhost:9001 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to localhost:9001 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731) at org.apache.hadoop.ipc.Client.call(Client.java:1475) at org.apache.hadoop.ipc.Client.call(Client.java:1408) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:762) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2121) at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215) at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412) at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:145) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:270) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:143) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1325) at org.apache.hadoop.examples.WordCount.main(WordCount.java:87) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71) at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:713) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1524) at org.apache.hadoop.ipc.Client.call(Client.java:1447) ... 41 more [root@localhost ~]# 问题:没有关闭防火墙 service iptables stop 关闭 chkconfig iptables off 永久关闭下面关于Hadoop的文章您也可能喜欢，不妨看看：