Hadoop集群(第7期)(8)

4、常见问题FAQ 4.1 "error: failure to login"问题

  下面以网上找的"Hadoop-0.20.203.0"为例,我在使用"V1.0"时也出现这样的情况,原因就是那个"hadoop-eclipse-plugin-1.0.0_V1.0.jar",是直接把源码编译而成,故而缺少相应的Jar包。具体情况如下

  详细地址:

  在我实践尝试中,发现hadoop-0.20.203.0版本的该包如果直接复制到eclipse的插件目录中,在连接DFS时会出现错误,提示信息为: "error: failure to login"。

  弹出的错误提示框内容为"An internal error occurred during: "Connecting to DFS hadoop".org/apache/commons/configuration/Configuration". 经过察看Eclipse的log,发现是缺少jar包导致的。进一步查找资料后,发现直接复制hadoop-eclipse-plugin-0.20.203.0.jar,该包中lib目录下缺少了jar包。

  经过网上资料搜集,此处给出正确的安装方法:

  首先要对hadoop-eclipse-plugin-0.20.203.0.jar进行修改。用归档管理器打开该包,发现只有commons-cli-1.2.jar 和hadoop-core.jar两个包。将hadoop/lib目录下的:

commons-configuration-1.6.jar ,

commons-httpclient-3.0.1.jar ,

commons-lang-2.4.jar ,

jackson-core-asl-1.0.1.jar

jackson-mapper-asl-1.0.1.jar

一共5个包复制到hadoop-eclipse-plugin-0.20.203.0.jar的lib目录下,如下图:

Hadoop集群(第7期)

  然后,修改该包META-INF目录下的MANIFEST.MF,将classpath修改为一下内容:

Bundle-ClassPath:classes/,lib/hadoop-core.jar,lib/commons-cli-1.2.jar,lib/commons-httpclient-3.0.1.jar,lib/jackson-core-asl-1.0.1.jar,lib/jackson-mapper-asl-1.0.1.jar,lib/commons-configuration-1.6.jar,lib/commons-lang-2.4.jar

Hadoop集群(第7期)

  这样就完成了对hadoop-eclipse-plugin-0.20.203.0.jar的修改。

  最后,将hadoop-eclipse-plugin-0.20.203.0.jar复制到Eclipse的plugins目录下。

  备注:上面的操作对"hadoop-1.0.0"一样适用。

4.2 "Permission denied"问题

  网上试了很多,有提到"hadoop fs -chmod 777 /user/hadoop ",有提到"dfs.permissions 的配置项,将value值改为 false",有提到"hadoop.job.ugi",但是通通没有效果。

   错误类型:org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security .AccessControlException: Permission denied: user=*********, access=WRITE, inode="hadoop": hadoop:supergroup:rwxr-xr-x

   解决方案:

    我的解决方案直接把系统管理员的名字改成你的Hadoop集群运行hadoop的那个用户。

4.3 "Failed to set permissions of path"问题

   参考文献:https://issues.apache.org/jira/browse/HADOOP-8089

   错误信息如下:

    ERROR security.UserGroupInformation: PriviledgedActionException as: hadoop cause:Java.io.IOException Failed to set permissions of path:\usr\hadoop\tmp\mapred\staging\hadoop753422487\.staging to 0700 Exception in thread "main" java.io.IOException: Failed to set permissions of path: \usr\hadoop\tmp \mapred\staging\hadoop753422487\.staging to 0700

   解决方法:

Configuration conf = new Configuration();

conf.set("mapred.job.tracker", "[server]:9001");

   "[server]:9001"中的"[server]"为Hadoop集群Master的IP地址。

4.4 "hadoop mapred执行目录文件权"限问题

   参考文献:

   错误信息如下:

   job Submission failed with exception 'java.io.IOException(The ownership/permissions on the staging directory /tmp/hadoop-hadoop-user1/mapred/staging/hadoop-user1/.staging is not as expected. It is owned by hadoop-user1 and permissions are rwxrwxrwx. The directory must be owned by the submitter hadoop-user1 or by hadoop-user1 and permissions must be rwx------)

   修改权限:

  这样就能解决问题。

内容版权声明:除非注明,否则皆为本站原创文章。

转载注明出处:http://www.heiqu.com/99ed94e37d28a5a2cd7dd7fe21f23c8c.html