Hadoop运行wordcount出现异常解决

近学习Hadoop,在Windows+Eclipse+虚拟机Hadoop集群环境下运行Mapreduce程序遇到了很多问题。上网查了查,并经过自己的分析,最终解决,在此分享一下,给遇到同样问题的人提供参考。

我的Hadoop集群环境:

虚拟机上4台机器:192.168.137.111(master)、192.168.137.112(slave1)、192.168.137.113(slave2)、192.168.137.114(slave3)

Hadoop集群用户名:hadoop

Hadoop版本:hadoop-1.1.2

开发环境:Windows7+Eclipse+Hadoop插件

异常1:

14/10/18 08:23:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/10/18 08:23:47 ERROR security.UserGroupInformation: PriviledgedActionException as:guilin cause:java.io.IOException: Failed to set permissions of path: \tmp\hadoop-guilin\mapred\staging\guilin1651756173\.staging to 0700
Exception in thread "main" java.io.IOException: Failed to set permissions of path: \tmp\hadoop-guilin\mapred\staging\guilin1651756173\.staging to 0700
 at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:689)
 at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:662)
 at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
 at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
 at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
 at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)
 at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:918)
 at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:1)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
 at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:912)
 at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
 at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
 at com.guilin.hadoop.mapreduce.WordCount.main(WordCount.java:75)

原因:wordcount程序连的是本地windows上的hadoop,需添加conf.set("mapred.job.tracker", "master:9001"),连接集群。

异常2:

14/10/18 08:37:14 ERROR security.UserGroupInformation: PriviledgedActionException as:guilin cause:org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=guilin, access=EXECUTE, inode="hadoop":hadoop:supergroup:rwx------
Exception in thread "main" org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=guilin, access=EXECUTE, inode="hadoop":hadoop:supergroup:rwx------
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
 at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
 at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
 at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1030)
 at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:524)
 at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:768)
 at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:103)
 at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:918)
 at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:1)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
 at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:912)
 at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
 at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
 at com.guilin.hadoop.mapreduce.WordCount.main(WordCount.java:75)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.security.AccessControlException: Permission denied: user=guilin, access=EXECUTE, inode="hadoop":hadoop:supergroup:rwx------
 at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:199)
 at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:155)
 at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:125)
 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5468)
 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkTraverse(FSNamesystem.java:5447)
 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:2168)
 at org.apache.hadoop.hdfs.server.namenode.NameNode.getFileInfo(NameNode.java:888)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)

内容版权声明:除非注明,否则皆为本站原创文章。

转载注明出处:https://www.heiqu.com/6b188a5921e852252cd70c5e266f9697.html