ResourceManager High Availability (RM高可用)
(简介)
(架构)
(RM 故障切换)
(恢复之前活动的RM的状态)
(部署)
(配置)
(管理命令)
(RM Web UI服务)
(Web 服务)
IntroductionThis guide provides an overview of High Availability of YARN’s ResourceManager, and details how to configure and use this feature. The ResourceManager (RM) is responsible for tracking the resources in a cluster, and scheduling applications (e.g., MapReduce jobs). Prior to Hadoop 2.4, the ResourceManager is the single point of failure in a YARN cluster. The High Availability feature adds redundancy in the form of an Active/Standby ResourceManager pair to remove this otherwise single point of failure.
这个知道提供YARN的ResourceManager的高可用综述,和如何配置和使用这个特性的细节。RM负责跟踪集群中的资源和调度应用(例如 MapReduce作业)。在Hadoop2.4之前,RM是YARN集群中的一个单点故障。这个高可用特性以活动/备用 RM对的形式增加了冗余来移除这个潜在的单点故障。
Architecture(架构) RM Failover(RM故障切换)ResourceManager HA is realized through an Active/Standby architecture - at any point of time, one of the RMs is Active, and one or more RMs are in Standby mode waiting to take over should anything happen to the Active. The trigger to transition-to-active comes from either the admin (through CLI) or through the integrated failover-controller when automatic-failover is enabled.
RM的高可用特性通过任何时间点的主/备架构来实现的,一个RM作为活动,而其他RMs进入备用模式随时等待接管出事的活动的RM。备用转活跃的触发可以通过管理员用命令行或者通过集成的故障切换控制器配置允许自动故障切换。
Manual transitions and failover(手动切换和故障切换)When automatic failover is not enabled, admins have to manually transition one of the RMs to Active. To failover from one RM to the other, they are expected to first transition the Active-RM to Standby and transition a Standby-RM to Active. All this can be done using the “yarn rmadmin” CLI.
当自动故障切换没有被激活时,管理员必须手动地转换RMs中的一个为活跃。RM的故障切换时首先将活跃的RM切换为备用然后将一个备用的RM切换为活跃状态。这些都可以用“yarn rmadmin”命令行来实现。
Automatic failover(自动故障切换)The RMs have an option to embed the Zookeeper-based ActiveStandbyElector to decide which RM should be the Active. When the Active goes down or becomes unresponsive, another RM is automatically elected to be the Active which then takes over. Note that, there is no need to run a separate ZKFC daemon as is the case for HDFS because ActiveStandbyElector embedded in RMs acts as a failure detector and a leader elector instead of a separate ZKFC deamon.
RMa有个选项来嵌入基于Zookeepper的主备选举机制来决定哪个RM是活跃的。当活跃的RM失效或者反应迟钝,另一个RM会被自动选举为主用然后接管工作。需要注意的是,没必要为HDFS运行一个单独的ZKFC进程因为主备选举机制内嵌到RMs作为一个失效检查器和选举器来替代一个单独的ZKFC进程。
Client, ApplicationMaster and NodeManager on RM failover(客户端、应用控制器和节点管理器在RM的故障切换下的转移)When there are multiple RMs, the configuration (yarn-site.xml) used by clients and nodes is expected to list all the RMs. Clients, ApplicationMasters (AMs) and NodeManagers (NMs) try connecting to the RMs in a round-robin fashion until they hit the Active RM. If the Active goes down, they resume the round-robin polling until they hit the “new” Active. This default retry logic is implemented as org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider. You can override the logic by implementing org.apache.hadoop.yarn.client.RMFailoverProxyProvider and setting the value of yarn.client.failover-proxy-provider to the class name.
当有多个RM,客户端和节点可以通过配置(yarn-site.xml)来获得RM的列表。客户端、应用控制器和节点管理器采用循环的方式来试图连上RM直到他们连上活跃RM。如果活跃的RM失效了,它们重新开始以循环的方式去连接RM直到他们连上新的活跃RM。这个默认的重试逻辑是org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider实现的。你可以通过实现 org.apache.hadoop.yarn.client.RMFailoverProxyProvider 来覆盖这个逻辑并将yarn.client.failover-proxy-provider的值设为该类名。
Recovering prevous active-RM’s state(恢复到之前活跃RM的状态)