Hadoop MapReduce 下一代

你需从发布页面获得MapReduce tar包。若不能,你要将源码打成tar包。

$ mvn clean install -DskipTests $ cd Hadoop-mapreduce-project $ mvn clean install assembly:assembly -Pnative

 

注意:你需要安装有protoc 2.5.0。

忽略本地建立mapreduce,你可以在maven中省略-Pnative参数。tar包应该在target/directory。

相关阅读

配置环境

假设你已经安装hadoop-common/hadoop-hdfs,并且输出了$HADOOP_COMMON_HOME/$HADOOP_HDFS_HOME,解压hadoop mapreduce 包,配置环境变量$HADOOP_MAPRED_HOME到要安装的目录。$HADOOP_YARN_HOME的配置和 $HADOOP_MAPRED_HOME一样.

注意:下面的操作假设你已经运行了hdfs。

设置配置信息

要启动ResourceManager and NodeManager, 你必须升级配置。假设你的 $HADOOP_CONF_DIR是配置目录,并且已经安装了HDFS和core-site.xml。还有2个配置文件你必须设置 mapred-site.xml 和yarn-site.xml.

设置 mapred-site.xml

添加下面的配置到你的mapred-site.xml.

<property> <name>mapreduce.cluster.temp.dir</name> <value></value> <description>No description</description> <final>true</final> </property> <property> <name>mapreduce.cluster.local.dir</name> <value></value> <description>No description</description> <final>true</final> </property>

 
设置 yarn-site.xml

添加下面的配置到你的yarn-site.xml.

<property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>host:port</value> <description>host is the hostname of the resource manager and port is the port on which the NodeManagers contact the Resource Manager. </description> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>host:port</value> <description>host is the hostname of the resourcemanager and port is the port on which the Applications in the cluster talk to the Resource Manager. </description> </property> <property> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value> <description>In case you do not want to use the default scheduler</description> </property> <property> <name>yarn.resourcemanager.address</name> <value>host:port</value> <description>the host is the hostname of the ResourceManager and the port is the port on which the clients can talk to the Resource Manager. </description> </property> <property> <name>yarn.nodemanager.local-dirs</name> <value></value> <description>the local directories used by the nodemanager</description> </property> <property> <name>yarn.nodemanager.address</name> <value>0.0.0.0:port</value> <description>the nodemanagers bind to this port</description> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>10240</value> <description>the amount of memory on the NodeManager in GB</description> </property> <property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/app-logs</value> <description>directory on hdfs where the application logs are moved to </description> </property> <property> <name>yarn.nodemanager.log-dirs</name> <value></value> <description>the directories used by Nodemanagers as log directories</description> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> <description>shuffle service that needs to be set for Map Reduce to run </description> </property>  

 

设置 capacity-scheduler.xml

确保你放置根队列到capacity-scheduler.xml.

<property> <name>yarn.scheduler.capacity.root.queues</name> <value>unfunded,default</value> </property> <property> <name>yarn.scheduler.capacity.root.capacity</name> <value>100</value> </property> <property> <name>yarn.scheduler.capacity.root.unfunded.capacity</name> <value>50</value> </property> <property> <name>yarn.scheduler.capacity.root.default.capacity</name> <value>50</value> </property>

 
运行守护进程

内容版权声明:除非注明,否则皆为本站原创文章。

转载注明出处:https://www.heiqu.com/e1512ee47111c56c302983608fb4ec14.html