Hadoop运行wordcount出现异常解决(2)

at org.apache.hadoop.ipc.Client.call(Client.java:1107)
 at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:230)
 at com.sun.proxy.$Proxy2.getFileInfo(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
 at com.sun.proxy.$Proxy2.getFileInfo(Unknown Source)
 at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1028)
 ... 12 more

原因:wordcount程序使用windows7的账户登录集群hadoop,我的系统账户名是guilin,而hadoop集群账户是hadoop,并且集群hadoop目录权限设置的是仅hadoop用户有读、写、执行权限。

解决办法:第一种是修改windows管理员(Administrator)账户名为hadoop账户名;第二种是在集群上创建一个账户名称与windows管理员账户名相同,并设置对hadoop目录有读、写、执行权限。推荐使用第一种,

异常3:

14/10/18 09:57:19 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
14/10/18 09:57:19 INFO input.FileInputFormat: Total input paths to process : 5
14/10/18 09:57:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/10/18 09:57:19 WARN snappy.LoadSnappy: Snappy native library not loaded
14/10/18 09:57:20 INFO mapred.JobClient: Running job: job_201410181754_0001
14/10/18 09:57:21 INFO mapred.JobClient:  map 0% reduce 0%
14/10/18 09:57:29 INFO mapred.JobClient: Task Id : attempt_201410181754_0001_m_000004_0, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: com.guilin.hadoop.mapreduce.WordCount$TokenizerMapper
 at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:849)
 at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
 at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
 at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
 at org.apache.hadoop.mapred.Child.main(Child.java:249)

原因:hadoop集群上运行mapreduce程序需要jar包。

解决办法:添加conf.set("mapred.jar","hadoop-test.jar");

把项目打包为jar文件hadoop-test.jar,放置在项目根目录下。

wordcount完整代码

package com.guilin.hadoop.mapreduce;

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class WordCount {

内容版权声明:除非注明,否则皆为本站原创文章。

转载注明出处:https://www.heiqu.com/6b188a5921e852252cd70c5e266f9697.html