1) 先关闭HA主节点的heartbeat
[root@ha-master ~]# /etc/init.d/heartbeat stop
Stopping High-Availability services: Done.
[root@ha-master ~]# ps -ef|grep heartbeat
root 21625 17616 0 16:03 pts/0 00:00:00 grep heartbeat
发现关闭heartbeat服务后, 主节点的ipvsadm 和 ldirectord都会被自带关闭, VIP资源也被转移走了, 即当前master节点不提供lvs转发服务
[root@ha-master ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:50:56:ac:50:9b brd ff:ff:ff:ff:ff:ff
inet 172.16.60.206/24 brd 172.16.60.255 scope global eth0
inet6 fe80::250:56ff:feac:509b/64 scope link
valid_lft forever preferred_lft forever
[root@ha-master ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
[root@ha-master ~]# ps -ef|grep ldirectord
root 21630 17616 0 16:03 pts/0 00:00:00 grep ldirectord
查看此时HA主节点的heartbeat日志
[root@ha-master ~]# tail -1000 /var/log/ha-log
........
Dec 25 16:02:38 ha-master heartbeat: [20886]: info: Heartbeat shutdown in progress. (20886)
Dec 25 16:02:38 ha-master heartbeat: [21454]: info: Giving up all HA resources.
ResourceManager(default)[21467]: 2018/12/25_16:02:38 info: Releasing resource group: ha-master IPaddr::172.16.60.111 ipvsadm ldirectord
ResourceManager(default)[21467]: 2018/12/25_16:02:38 info: Running /etc/init.d/ldirectord stop
ResourceManager(default)[21467]: 2018/12/25_16:02:38 info: Running /etc/init.d/ipvsadm stop
ResourceManager(default)[21467]: 2018/12/25_16:02:38 info: Running /etc/ha.d/resource.d/IPaddr 172.16.60.111 stop
IPaddr(IPaddr_172.16.60.111)[21563]: 2018/12/25_16:02:38 INFO: IP status = ok, IP_CIP=
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_172.16.60.111)[21549]: 2018/12/25_16:02:38 INFO: Success
接着查看HA备份节点的情况, 发现VIP已将已经切换到备份节点这边了, 说明当前备份节点提供lvs转发服务, 则备份节点的ipvsadm 和 ldirectord也被自带启动了
[root@ha-slave ha.d]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:50:56:ac:05:b5 brd ff:ff:ff:ff:ff:ff
inet 172.16.60.207/24 brd 172.16.60.255 scope global eth0
inet 172.16.60.111/24 brd 172.16.60.255 scope global secondary eth0
inet6 fe80::250:56ff:feac:5b5/64 scope link
valid_lft forever preferred_lft forever
[root@ha-slave ha.d]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.60.111:80 wlc persistent 600
-> 172.16.60.204:80 Route 1 0 0
-> 172.16.60.205:80 Route 1 0 0
[root@ha-slave ha.d]# ps -ef|grep ldirectord
root 22203 1 0 16:02 ? 00:00:01 /usr/bin/perl -w /usr/sbin/ldirectord start
root 22261 19163 0 16:07 pts/0 00:00:00 grep ldirectord
查看此时HA备份节点的heartbeat日志
[root@ha-slave ha.d]# tail -1000 /var/log/ha-log
...........
harc(default)[21887]: 2018/12/25_16:02:39 info: Running /etc/ha.d//rc.d/status status
mach_down(default)[21904]: 2018/12/25_16:02:39 info: Taking over resource group IPaddr::172.16.60.111
ResourceManager(default)[21931]: 2018/12/25_16:02:39 info: Acquiring resource group: ha-master IPaddr::172.16.60.111 ipvsadm ldirectord
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_172.16.60.111)[21959]: 2018/12/25_16:02:39 INFO: Resource is stopped
ResourceManager(default)[21931]: 2018/12/25_16:02:39 info: Running /etc/ha.d/resource.d/IPaddr 172.16.60.111 start
IPaddr(IPaddr_172.16.60.111)[22055]: 2018/12/25_16:02:39 INFO: Adding inet address 172.16.60.111/24 with broadcast address 172.16.60.255 to device eth0
IPaddr(IPaddr_172.16.60.111)[22055]: 2018/12/25_16:02:39 INFO: Bringing device eth0 up
IPaddr(IPaddr_172.16.60.111)[22055]: 2018/12/25_16:02:39 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-172.16.60.111 eth0 172.16.60.111 auto not_used not_used
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_172.16.60.111)[22041]: 2018/12/25_16:02:39 INFO: Success
ResourceManager(default)[21931]: 2018/12/25_16:02:39 info: Running /etc/init.d/ipvsadm start
ResourceManager(default)[21931]: 2018/12/25_16:02:39 info: Running /etc/init.d/ldirectord start
mach_down(default)[21904]: 2018/12/25_16:02:39 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down(default)[21904]: 2018/12/25_16:02:39 info: mach_down takeover complete for node ha-master.
2) 然后在重新启动HA主节点的heartbeat服务
由于在ha.cf文件中配置了"auto_failback on "参数, 所以当主节点恢复后, 会将VIP资源自动抢占回来并替换备份节点重新接管lvs转发服务.
主节点的heartbeat恢复后, ipvsadm 和 ldirectord也会被重新启动
[root@ha-master ~]# /etc/init.d/heartbeat start
Starting High-Availability services: INFO: Resource is stopped
Done.
[root@ha-master ~]# ps -ef|grep heartbeat
root 21778 1 0 16:12 ? 00:00:00 heartbeat: master control process
root 21783 21778 0 16:12 ? 00:00:00 heartbeat: FIFO reader
root 21784 21778 0 16:12 ? 00:00:00 heartbeat: write: bcast eth0
root 21785 21778 0 16:12 ? 00:00:00 heartbeat: read: bcast eth0
root 21786 21778 0 16:12 ? 00:00:00 heartbeat: write: ucast eth0
root 21787 21778 0 16:12 ? 00:00:00 heartbeat: read: ucast eth0
root 21788 21778 0 16:12 ? 00:00:00 heartbeat: write: ping_group group1
root 21789 21778 0 16:12 ? 00:00:00 heartbeat: read: ping_group group1
root 21809 21778 0 16:12 ? 00:00:00 /usr/lib64/heartbeat/ipfail
root 21812 21778 0 16:12 ? 00:00:00 heartbeat: master control process
root 21825 21812 0 16:12 ? 00:00:00 /bin/sh /usr/share/heartbeat/ResourceManager takegroup IPaddr::172.16.60.111 ipvsadm ldirectord
root 21949 21935 0 16:12 ? 00:00:00 /bin/sh /usr/lib/ocf/resource.d//heartbeat/IPaddr start
root 21956 17616 0 16:12 pts/0 00:00:00 grep heartbeat
[root@ha-master ~]# lsof -i:694
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
heartbeat 21784 root 7u IPv4 46306 0t0 UDP *:ha-cluster
heartbeat 21785 root 7u IPv4 46306 0t0 UDP *:ha-cluster
heartbeat 21786 root 7u IPv4 46312 0t0 UDP *:ha-cluster
heartbeat 21787 root 7u IPv4 46312 0t0 UDP *:ha-cluster
[root@ha-master ~]# ps -ef|grep ldirectord
root 22099 1 1 16:12 ? 00:00:00 /usr/bin/perl -w /usr/sbin/ldirectord start
root 22130 17616 0 16:12 pts/0 00:00:00 grep ldirectord
[root@ha-master ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:50:56:ac:50:9b brd ff:ff:ff:ff:ff:ff
inet 172.16.60.206/24 brd 172.16.60.255 scope global eth0
inet 172.16.60.111/24 brd 172.16.60.255 scope global secondary eth0
inet6 fe80::250:56ff:feac:509b/64 scope link
valid_lft forever preferred_lft forever
[root@ha-master ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.60.111:80 wlc persistent 600
-> 172.16.60.204:80 Route 1 0 0
-> 172.16.60.205:80 Route 1 1 0
查看此时HA主节点的heartbeat日志
[root@ha-master ~]# tail -1000 /var/log/ha-log
........
ResourceManager(default)[21825]: 2018/12/25_16:12:12 info: Acquiring resource group: ha-master IPaddr::172.16.60.111 ipvsadm ldirectord
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_172.16.60.111)[21853]: 2018/12/25_16:12:13 INFO: Resource is stopped
ResourceManager(default)[21825]: 2018/12/25_16:12:13 info: Running /etc/ha.d/resource.d/IPaddr 172.16.60.111 start
IPaddr(IPaddr_172.16.60.111)[21949]: 2018/12/25_16:12:13 INFO: Adding inet address 172.16.60.111/24 with broadcast address 172.16.60.255 to device eth0
IPaddr(IPaddr_172.16.60.111)[21949]: 2018/12/25_16:12:13 INFO: Bringing device eth0 up
IPaddr(IPaddr_172.16.60.111)[21949]: 2018/12/25_16:12:13 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-172.16.60.111 eth0 172.16.60.111 auto not_used not_used
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_172.16.60.111)[21935]: 2018/12/25_16:12:13 INFO: Success
ResourceManager(default)[21825]: 2018/12/25_16:12:13 info: Running /etc/init.d/ipvsadm start
ResourceManager(default)[21825]: 2018/12/25_16:12:13 info: Running /etc/init.d/ldirectord start
再观察此时HA备份节点的情况, 发现VIP资源在主节点的heartbeat恢复后就被主节点抢占回去了, 即此时备份节点没有vip资源, 也就不提供lvs转发服务了,
则备份节点的ipvsadm 和 ldirectord服务也会被关闭
[root@ha-slave ha.d]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:50:56:ac:05:b5 brd ff:ff:ff:ff:ff:ff
inet 172.16.60.207/24 brd 172.16.60.255 scope global eth0
inet6 fe80::250:56ff:feac:5b5/64 scope link
valid_lft forever preferred_lft forever
[root@ha-slave ha.d]# ps -ef|grep ldirectord
root 22516 19163 0 16:14 pts/0 00:00:00 grep ldirectord
[root@ha-slave ha.d]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
查看此时HA备份节点的heartbeat日志
[root@ha-slave ha.d]# tail -1000 /var/log/ha-log
.......
ResourceManager(default)[22342]: 2018/12/25_16:12:12 info: Releasing resource group: ha-master IPaddr::172.16.60.111 ipvsadm ldirectord
ResourceManager(default)[22342]: 2018/12/25_16:12:12 info: Running /etc/init.d/ldirectord stop
ResourceManager(default)[22342]: 2018/12/25_16:12:12 info: Running /etc/init.d/ipvsadm stop
ResourceManager(default)[22342]: 2018/12/25_16:12:12 info: Running /etc/ha.d/resource.d/IPaddr 172.16.60.111 stop
IPaddr(IPaddr_172.16.60.111)[22438]: 2018/12/25_16:12:12 INFO: IP status = ok, IP_CIP=
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_172.16.60.111)[22424]: 2018/12/25_16:12:12 INFO: Success
Dec 25 16:12:12 ha-slave heartbeat: [22329]: info: foreign HA resource release completed (standby).
在上面HA主备节点故障切换的过程中, 客户端访问都是不受影响的, 即对客户端访问来说是无感知的故障切换, 实现了lvs代理层的高可用!
3) 先后关闭两台realserver节点中的nginx, 然后观察lvs的转发情况
[root@ha-master ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.60.111:80 wlc persistent 600
-> 172.16.60.204:80 Route 1 0 0
-> 172.16.60.205:80 Route 1 0 2
先关闭rs-204的nginx服务
[root@rs-204 ~]# /etc/init.d/nginx stop
Stopping nginx: [ OK ]
[root@rs-204 ~]# lsof -i:80
[root@rs-204 ~]#
rs-205的nginx保留
[root@rs-205 ~]# ps -ef|grep nginx
root 5211 1 0 15:45 ? 00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx 5212 5211 0 15:45 ? 00:00:00 nginx: worker process
root 5313 4852 0 16:19 pts/0 00:00:00 grep nginx
查看lvs转发情况
[root@ha-master ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.60.111:80 wlc persistent 600
-> 172.16.60.205:80 Route 1 0 2
这时候访问, 结果是"this is test page of realserver02:172.16.60.205"
接着启动rs-204的nginx, 关闭rs-205的nginx
[root@rs-204 ~]# /etc/init.d/nginx start
Starting nginx: [ OK ]
[root@rs-204 ~]# lsof -i:80
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nginx 4883 root 6u IPv4 143621 0t0 TCP *:http (LISTEN)
nginx 4884 nginx 6u IPv4 143621 0t0 TCP *:http (LISTEN)
关闭rs-205的nginx
[root@rs-205 ~]# /etc/init.d/nginx stop
Stopping nginx: [ OK ]
[root@rs-205 ~]# lsof -i:80
[root@rs-205 ~]#
查看lvs转发情况
[root@ha-master ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.60.111:80 wlc persistent 600
-> 172.16.60.204:80 Route 1 0 0
这时候访问, 结果是"this is test page of realserver01:172.16.60.204"
然后把rs-204 和 rs-205两个节点的nginx都关闭
[root@rs-204 ~]# /etc/init.d/nginx stop
Stopping nginx: [ OK ]
[root@rs-205 ~]# /etc/init.d/nginx stop
Stopping nginx: [ OK ]
查看lvs转发情况
[root@ha-master ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.60.111:80 wlc persistent 600
-> 127.0.0.1:80 Local 1 0 0
这时候访问, 结果是"Sorry, the access is in maintenance for the time being. Please wait a moment."
上面可知, 在realserver节点发生故障后, 会从lvs集群中踢出来, 待realserver节点恢复后会再次重新加入到lvs集群中
这是因为在ldirectord.cf文件中配置了"quiescent=no "参数 , 这样就实现了代理节点的高可用!
Linux公社的RSS地址：https://www.linuxidc.com/rssFeed.aspx