wget Hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz
然后解压:
tar zxvf hadoop-2.2.0-src.tar.gz
cd hadoop-2.2.0-src
运行下面的命令开始编译:
~/code/hadoop-2.2.0-src$ mvn package -Pdist,native -DskipTests -Dtar
下载了很多maven的东东后,编译报错:
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/hduser/code/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[88,11] error: cannot access AbstractLifeCycle
[ERROR] class file for org.mortbay.component.AbstractLifeCycle not found
/home/hduser/code/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[96,29] error: cannot access LifeCycle
[ERROR] class file for org.mortbay.component.LifeCycle not found
编辑hadoop-common-project/hadoop-auth/pom.xml文件,添加依赖:
<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty-util</artifactId>
<scope>test</scope>
</dependency>
再次编译,这个错误解决了。
之后遇到了没有安装protocol buffer库的错误,安装一下:
[INFO] --- hadoop-maven-plugins:2.2.0:protoc (compile-protoc) @ hadoop-common ---
[WARNING] [protoc, --version] failed: java.io.IOException: Cannot run program "protoc": error=2, No such file or directory
sudo apt-get install libprotobuf-dev
然后再次运行maven命令编译。发现新的错误,原来Ubuntu的安装包里面没有带上protoc编译器。算了,下源代码自己编译。
hduser@hm1:/usr/src$ sudo wget https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
...
$ sudo ./configure
$ sudo make
$ sudo make check
$ sudo make install
$ sudo ldconfig
$ protoc --version
hduser@hm1:~$ start-dfs.sh
Starting namenodes on [localhost]
localhost: namenode running as process 983. Stop it first.
localhost: datanode running as process 1101. Stop it first.
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 1346. Stop it first.
再来编译。等...., 说明hadoop不准备64bit的库浪费了大家多少时间。
还有错:
Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
[exec] system variable OPENSSL_ROOT_DIR (missing: OPENSSL_LIBRARIES
安装openssl库
sudo apt-get install libssl-dev
再来一次,编译成功了。在目录/home/hduser/code/hadoop-2.2.0-src/hadoop-dist/target 下有文件:
hadoop-2.2.0.tar.gz
解压后,进入目录,然后复制native 目录里的东西到制定位置,覆盖32bit文件
sudo cp -r ./hadoop-2.2.0/lib/native/* /usr/local/hadoop/lib/native/
现在回到~目录,运行下面的命令,成功了。
hduser@hm1:~$ start-dfs.sh
Starting namenodes on [localhost]
localhost: namenode running as process 983. Stop it first.
localhost: datanode running as process 1101. Stop it first.
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 1346. Stop it first.
hduser@hm1:~$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hduser-resourcemanager-hm1.out
localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hduser-nodemanager-hm1.out
hduser@hm1:~$ jps
32417 Jps
1101 DataNode
1346 SecondaryNameNode
983 NameNode
现在运行例子程序, 注意/etc/hosts中我配置了
127.0.0.1 hm1
hduser@hm1:~/code/hadoop-2.2.0-src$ cd /usr/local/hadoop/
hduser@hm1:/usr/local/hadoop$ hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5
Number of Maps = 2
Samples per Map = 5
Wrote input for Map #0
Wrote input for Map #1
Starting Job
14/02/21 18:07:29 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/02/21 18:07:30 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumC\
fountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/02/21 18:07:31 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumC\
ountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/02/21 18:07:32 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumC\
ountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
需要在yarn-site.xml中添加写配置:
<property>
<name>yarn.resourcemanager.address</name>
<value>127.0.0.1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>127.0.0.1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>127.0.0.1:8031</value>
</property>
重新启动虚拟机,再次启动服务,现在连接问题解决了。
hduser@hm1:/usr/local/hadoop$ hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5
Number of Maps = 2
Samples per Map = 5
Wrote input for Map #0
Wrote input for Map #1
Starting Job
14/02/21 18:15:53 INFO client.RMProxy: Connecting to ResourceManager at /127.0.0.1:8032
14/02/21 18:15:54 INFO input.FileInputFormat: Total input paths to process : 2
14/02/21 18:15:54 INFO mapreduce.JobSubmitter: number of splits:2
14/02/21 18:15:54 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.specul\
ative
14/02/21 18:15:54 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/02/21 18:15:54 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/02/21 18:15:54 INFO Configuration.deprecation: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/02/21 18:15:54 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1393006528872_0001
14/02/21 18:15:54 INFO impl.YarnClientImpl: Submitted application application_1393006528872_0001 to ResourceManager at /127.0.0.1:8032
14/02/21 18:15:55 INFO mapreduce.Job: The url to track the job: :8088/proxy/application_1393006528872_0001/
14/02/21 18:15:55 INFO mapreduce.Job: Running job: job_1393006528872_0001
如果重来一次的话,顺序应该是先编译64bit版本,然后用这个版本进行配置安装。