在nginx01和nginx02节点上执行
Nginx是一个主流Web服务和反向代理服务器，这里用四层实现对apiserver实现负载均衡。
Keepalived是一个主流高可用软件，基于VIP绑定实现服务器双机热备，在上述拓扑中，Keepalived主要根据Nginx运行状态判断是否需要故障转移（偏移VIP），例如当Nginx主节点挂掉，VIP会自动绑定在Nginx备节点，从而保证VIP一直可用，实现Nginx高可用。
1、nginx01和nginx02节点上安装软件包（以nginx01为例）
[root@nginx01 ~]# yum -y install epel-release
[root@nginx01 ~]# yum -y install nginx keepalived
2、nginx01和nginx02节点上配置nginx（nginx主备一样）
[root@nginx01 ~]# mv /etc/nginx/nginx.conf{,.bak}
[root@nginx01 ~]# vim /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
# 四层负载均衡，为两台Master apiserver组件提供负载均衡
# stream_mudule:
stream {
log_format main \'$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent\';
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {
server 192.168.1.21:6443;
# Master1 APISERVER IP:PORT
server 192.168.1.24:6443;
# Master2 APISERVER IP:PORT
}
server {
listen 6443;
proxy_pass k8s-apiserver;
}
}
http {
log_format main \'$remote_addr - $remote_user [$time_local] "$request" \'
\'$status $body_bytes_sent "$http_referer" \'
\'"$http_user_agent" "$http_x_forwarded_for"\';
access_log /var/log/nginx/access.log main;
sendfile
on;
tcp_nopush
on;
tcp_nodelay
on;
keepalive_timeout 65;
types_hash_max_size 2048;
include
/etc/nginx/mime.types;
default_type
application/octet-stream;
server {
listen
80 default_server;
server_name _;
location / {
}
}
}
3、nginx01节点上配置keepalived
[root@nginx01 ~]# mv /etc/keepalived/keepalived.conf{,.bak}
[root@nginx01 ~]# vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
priority 100
# 优先级，备服务器设置 90
advert_int 1
# 指定VRRP 心跳包通告间隔时间，默认1秒
authentication {
auth_type PASS
auth_pass 1111
}
# 虚拟IP
virtual_ipaddress {
192.168.1.27/24
}
track_script {
check_nginx
}
}
释义：
vrrp_script：指定检查nginx工作状态脚本（根据nginx状态判断是否故障转移）
virtual_ipaddress：虚拟IP（VIP）
检查nginx状态脚本：
[root@nginx01 ~]# cat > /etc/keepalived/check_nginx.sh << EOF
#!/bin/bash
count=\$(ps -ef | grep nginx | egrep -cv "grep | \$\$")
if [ "\$count" -eq 0 ];then
exit 1
else
exit 0
fi
EOF
[root@nginx01 ~]# chmod +x /etc/keepalived/check_nginx.sh
nginx01节点启动nginx和keepalived并设置开机启动
systemctl daemon-reload
systemctl enable nginx
systemctl enable keepalived
systemctl start nginx
systemctl start keepalived
注：如启动时nginx报stream的错，则需看 https://www.cnblogs.com/smlile-you-me/p/14986179.html
[root@nginx01 ~]# systemctl status nginx keepalived
● nginx.service - The nginx HTTP and reverse proxy server
Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor preset: disabled)
Active: active (running) since 四 2021-07-08 21:37:15 CST; 28min ago
Process: 24693 ExecStart=http://www.likecs.com/usr/sbin/nginx (code=exited, status=0/SUCCESS)
Process: 24690 ExecStartPre=http://www.likecs.com/usr/sbin/nginx -t (code=exited, status=0/SUCCESS)
Process: 24689 ExecStartPre=http://www.likecs.com/usr/bin/rm -f /run/nginx.pid (code=exited, status=0/SUCCESS)
Main PID: 24695 (nginx)
CGroup: /system.slice/nginx.service
├─24695 nginx: master process /usr/sbin/nginx
└─24696 nginx: worker process
7月 08 21:37:15 nginx01 systemd[1]: Starting The nginx HTTP and reverse proxy server...
7月 08 21:37:15 nginx01 nginx[24690]: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
7月 08 21:37:15 nginx01 nginx[24690]: nginx: configuration file /etc/nginx/nginx.conf test is successful
7月 08 21:37:15 nginx01 systemd[1]: Failed to parse PID from file /run/nginx.pid: Invalid argument
7月 08 21:37:15 nginx01 systemd[1]: Started The nginx HTTP and reverse proxy server.
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
Active: active (running) since 四 2021-07-08 22:05:48 CST; 15s ago
Process: 26111 ExecStart=http://www.likecs.com/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 26112 (keepalived)
CGroup: /system.slice/keepalived.service
├─26112 /usr/sbin/keepalived -D
├─26113 /usr/sbin/keepalived -D
└─26114 /usr/sbin/keepalived -D
7月 08 22:05:55 nginx01 Keepalived_vrrp[26114]: Sending gratuitous ARP on ens33 for 192.168.1.27
7月 08 22:05:55 nginx01 Keepalived_vrrp[26114]: Sending gratuitous ARP on ens33 for 192.168.1.27
7月 08 22:05:55 nginx01 Keepalived_vrrp[26114]: Sending gratuitous ARP on ens33 for 192.168.1.27
7月 08 22:05:55 nginx01 Keepalived_vrrp[26114]: Sending gratuitous ARP on ens33 for 192.168.1.27
7月 08 22:06:00 nginx01 Keepalived_vrrp[26114]: Sending gratuitous ARP on ens33 for 192.168.1.27
7月 08 22:06:00 nginx01 Keepalived_vrrp[26114]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.1.27
7月 08 22:06:00 nginx01 Keepalived_vrrp[26114]: Sending gratuitous ARP on ens33 for 192.168.1.27
7月 08 22:06:00 nginx01 Keepalived_vrrp[26114]: Sending gratuitous ARP on ens33 for 192.168.1.27
7月 08 22:06:00 nginx01 Keepalived_vrrp[26114]: Sending gratuitous ARP on ens33 for 192.168.1.27
7月 08 22:06:00 nginx01 Keepalived_vrrp[26114]: Sending gratuitous ARP on ens33 for 192.168.1.27
[root@nginx01 ~]# ip a show ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:89:e3:dd brd ff:ff:ff:ff:ff:ff
inet 192.168.1.25/24 brd 192.168.1.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet 192.168.1.27/24 scope global secondary ens33
valid_lft forever preferred_lft forever
inet6 fe80::b77e:df59:787b:674b/64 scope link noprefixroute
valid_lft forever preferred_lft forever
4、nginx02节点上配置keepalived
[root@nginx02 ~]# mv /etc/keepalived/keepalived.conf{,.bak}
[root@nginx02 ~]# vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_BACKUP
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.1.27/24
}
track_script {
check_nginx
}
}
检查nginx状态脚本：
[root@nginx02 ~]# cat > /etc/keepalived/check_nginx.sh << EOF
#!/bin/bash
count=\$(ps -ef | grep nginx | egrep -cv "grep | \$\$")
if [ "\$count" -eq 0 ];then
exit 1
else
exit 0
fi
EOF
[root@nginx02 ~]# chmod +x /etc/keepalived/check_nginx.sh
注：keepalived根据脚本返回状态码（0为工作正常，非0不正常）判断是否故障转移。
nginx0节点启动nginx和keepalived并设置开机启动
systemctl daemon-reload
systemctl enable nginx
systemctl enable keepalived
systemctl start nginx
systemctl start keepalived
注：如启动时nginx报stream的错，则需看 https://www.cnblogs.com/smlile-you-me/p/14986179.html
[root@nginx02 ~]# systemctl status nginx keepalived
● nginx.service - The nginx HTTP and reverse proxy server
Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor preset: disabled)
Active: active (running) since 四 2021-07-08 22:21:21 CST; 12s ago
Process: 15716 ExecStart=http://www.likecs.com/usr/sbin/nginx (code=exited, status=0/SUCCESS)
Process: 15713 ExecStartPre=http://www.likecs.com/usr/sbin/nginx -t (code=exited, status=0/SUCCESS)
Process: 15711 ExecStartPre=http://www.likecs.com/usr/bin/rm -f /run/nginx.pid (code=exited, status=0/SUCCESS)
Main PID: 15718 (nginx)
Tasks: 2
Memory: 1.5M
CGroup: /system.slice/nginx.service
├─15718 nginx: master process /usr/sbin/nginx
└─15719 nginx: worker process
7月 08 22:21:21 nginx02 systemd[1]: Starting The nginx HTTP and reverse proxy server...
7月 08 22:21:21 nginx02 nginx[15713]: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
7月 08 22:21:21 nginx02 nginx[15713]: nginx: configuration file /etc/nginx/nginx.conf test is successful
7月 08 22:21:21 nginx02 systemd[1]: Failed to parse PID from file /run/nginx.pid: Invalid argument
7月 08 22:21:21 nginx02 systemd[1]: Started The nginx HTTP and reverse proxy server.
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
Active: active (running) since 四 2021-07-08 22:21:22 CST; 12s ago
Process: 15726 ExecStart=http://www.likecs.com/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 15727 (keepalived)
Tasks: 3
Memory: 5.5M
CGroup: /system.slice/keepalived.service
├─15727 /usr/sbin/keepalived -D
├─15728 /usr/sbin/keepalived -D
└─15729 /usr/sbin/keepalived -D
7月 08 22:21:22 nginx02 Keepalived_vrrp[15729]: Registering gratuitous ARP shared channel
7月 08 22:21:22 nginx02 Keepalived_vrrp[15729]: Opening file \'/etc/keepalived/keepalived.conf\'.
7月 08 22:21:22 nginx02 Keepalived_vrrp[15729]: WARNING - default user \'keepalived_script\' for script execution...eate.
7月 08 22:21:22 nginx02 Keepalived_vrrp[15729]: SECURITY VIOLATION - scripts are being executed but script_secu...bled.
7月 08 22:21:22 nginx02 Keepalived_vrrp[15729]: VRRP_Instance(VI_1) removing protocol VIPs.
7月 08 22:21:22 nginx02 Keepalived_vrrp[15729]: Using LinkWatch kernel netlink reflector...
7月 08 22:21:22 nginx02 Keepalived_vrrp[15729]: VRRP_Instance(VI_1) Entering BACKUP STATE
7月 08 22:21:22 nginx02 Keepalived_vrrp[15729]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
7月 08 22:21:22 nginx02 Keepalived_vrrp[15729]: VRRP_Script(check_nginx) succeeded
7月 08 22:21:22 nginx02 Keepalived_healthcheckers[15728]: Opening file \'/etc/keepalived/keepalived.conf\'.
Hint: Some lines were ellipsized, use -l to show in full.
[root@nginx02 ~]# ip a show ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:b4:34:4f brd ff:ff:ff:ff:ff:ff
inet 192.168.1.26/24 brd 192.168.1.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::d588:e427:12e7:ce21/64 scope link noprefixroute
valid_lft forever preferred_lft forever
## Nginx+Keepalived高可用测试
关闭主节点Nginx，测试VIP是否漂移到备节点服务器。
在Nginx Master执行 pkill nginx
在Nginx Backup，ip addr命令查看已成功绑定VIP。
[root@nginx01 ~]# ip a show ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:89:e3:dd brd ff:ff:ff:ff:ff:ff
inet 192.168.1.25/24 brd 192.168.1.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet 192.168.1.27/24 scope global secondary ens33
valid_lft forever preferred_lft forever
inet6 fe80::b77e:df59:787b:674b/64 scope link noprefixroute
valid_lft forever preferred_lft forever
[root@nginx01 ~]# pkill nginx
[root@nginx01 ~]# ip a show ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:89:e3:dd brd ff:ff:ff:ff:ff:ff
inet 192.168.1.25/24 brd 192.168.1.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::b77e:df59:787b:674b/64 scope link noprefixroute
valid_lft forever preferred_lft forever
[root@nginx02 ~]# ip a show ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:b4:34:4f brd ff:ff:ff:ff:ff:ff
inet 192.168.1.26/24 brd 192.168.1.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet 192.168.1.27/24 scope global secondary ens33
valid_lft forever preferred_lft forever
inet6 fe80::d588:e427:12e7:ce21/64 scope link noprefixroute
valid_lft forever preferred_lft forever
访问负载均衡器测试
找K8s集群中任意一个节点，使用curl查看K8s版本测试，使用VIP访问：
[root@nginx02 ~]# curl -k https://192.168.1.27:6443/version
{
"major": "1",
"minor": "18",
"gitVersion": "v1.18.19",
"gitCommit": "ac0cc736d0018d817c763083945e4db863168d12",
"gitTreeState": "clean",
"buildDate": "2021-05-12T11:21:54Z",
"goVersion": "go1.13.15",
"compiler": "gc",
"platform": "linux/amd64"
可以正确获取到K8s版本信息，说明负载均衡器搭建正常。
该请求数据流程：curl -> vip(nginx) -> apiserver
修改所有Worker Node(非2个master节点)连接LB VIP
