Ubuntu环境下搭建Hadoop开发环境

实践环境:Ubuntu
机器名           ip             作用                 机器配置
主要过程
1.修改hosts
cat /etc/hosts
在master和slave上都修改,添加机器名和对应ip
2.ssh无密码登录本机和其他机器
3.JDK环境
4.安装Hadoop
5.配置hadoop
6.运行hadoop
在master上启动./start-all.sh
通过jps查看,
13136 SecondaryNameNode
13227 JobTracker
13530 Jps
在slave上,jps
6654 TaskTracker
6707 Jps
6463 DataNode
master上jps看不到namenode,可能有问题,查看logs下namenode日志,如下示

2012-05-31 22:04:31,872 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = guxu-Lenovo-B460/127.0.1.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.3
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1335192; compiled by 'hortonfo' on Tue May 8 20:31:25 UTC 2012
************************************************************/
2012-05-31 22:04:32,034 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2012-05-31 22:04:32,046 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2012-05-31 22:04:32,048 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2012-05-31 22:04:32,048 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2012-05-31 22:04:32,237 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2012-05-31 22:04:32,242 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2012-05-31 22:04:32,258 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
2012-05-31 22:04:32,259 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source NameNode registered.
2012-05-31 22:04:32,298 INFO org.apache.hadoop.hdfs.util.GSet: VM type = 32-bit
2012-05-31 22:04:32,298 INFO org.apache.hadoop.hdfs.util.GSet: 2% max memory = 17.77875 MB
2012-05-31 22:04:32,298 INFO org.apache.hadoop.hdfs.util.GSet: capacity = 2^22 = 4194304 entries
2012-05-31 22:04:32,298 INFO org.apache.hadoop.hdfs.util.GSet: recommended=4194304, actual=4194304
2012-05-31 22:04:32,335 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=guxu
2012-05-31 22:04:32,335 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup
2012-05-31 22:04:32,335 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true
2012-05-31 22:04:32,342 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.block.invalidate.limit=100
2012-05-31 22:04:32,342 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
2012-05-31 22:04:32,674 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStateMBean and NameNodeMXBean
2012-05-31 22:04:32,695 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2012-05-31 22:04:32,700 INFO org.apache.hadoop.hdfs.server.common.Storage: Storage directory /home/guxu/hadoop_tmp_dir/dfs/name does not exist.
2012-05-31 22:04:32,702 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /home/guxu/hadoop_tmp_dir/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible.
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:303)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:388)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:362)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288)
2012-05-31 22:04:32,704 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /home/guxu/hadoop_tmp_dir/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible.
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:303)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:388)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:362)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288)

2012-05-31 22:04:32,705 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at guxu-Lenovo-B460/127.0.1.1
************************************************************/

内容版权声明:除非注明,否则皆为本站原创文章。

转载注明出处:http://www.heiqu.com/e07b549679ea684ca90555c1b6ff6066.html