后端 Pod 中 nginx 的日志格式为
log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"';另起一个终端进行访问
[root@centos7-a ~]# curl -H 'Host:ng.5179.top' :36459 -I HTTP/1.1 200 OK Server: nginx/1.19.0 Date: Sat, 20 Jun 2020 12:58:27 GMT Content-Type: text/html Content-Length: 612 Connection: keep-alive Vary: Accept-Encoding Last-Modified: Tue, 26 May 2020 15:00:20 GMT ETag: "5ecd2f04-264" Accept-Ranges: bytes可以看到日志10.244.3.123 - - [20/Jun/2020:12:58:20 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "10.244.4.0"
然后我们可以配置前端的 LB
[root@centos7-nginx conf.d]# vim ng.conf [root@centos7-nginx conf.d]# cat ng.conf upstream nginx-dns{ ip_hash; server 10.10.10.131:36459 ; server 10.10.10.132:36459; } server { listen 80; server_name ng.5179.top; #access_log logs/host.access.log main; location / { root html; proxy_pass ; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $remote_addr; index index.html index.htm; } } # 添加内部解析 [root@centos7-nginx conf.d]# vim /etc/hosts [root@centos7-nginx conf.d]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.10.10.127 centos7-nginx lb.5179.top ng.5179.top 10.10.10.128 centos7-a 10.10.10.129 centos7-b 10.10.10.130 centos7-c 10.10.10.131 centos7-d 10.10.10.132 centos7-e # 重启 nginx [root@centos7-nginx conf.d]# nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful [root@centos7-nginx conf.d]# nginx -s reload访问该域名
[root@centos7-nginx conf.d]# curl -I HTTP/1.1 200 OK Server: nginx/1.16.1 Date: Sat, 20 Jun 2020 13:07:38 GMT Content-Type: text/html Content-Length: 612 Connection: keep-alive Vary: Accept-Encoding Last-Modified: Tue, 26 May 2020 15:00:20 GMT ETag: "5ecd2f04-264" Accept-Ranges: bytes后端也能正常收到日志
10.244.4.17 - - [20/Jun/2020:13:22:11 +0000] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.29.0" "10.244.4.1
$remote_addr ---> 10.244.4.17：为某一台 ingress-nginx 的 nginx_IP
$http_x_forwarded_for ---> 10.244.4.1：为节点上的 cni0 网卡 IP
[root@centos7-nginx conf.d]# kubectl get pods -n ingress-nginx -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES ingress-nginx-admission-create-tqp5w 0/1 Completed 0 112m 10.244.3.119 centos7-d <none> <none> ingress-nginx-admission-patch-78jmf 0/1 Completed 0 112m 10.244.3.120 centos7-d <none> <none> ingress-nginx-controller-5946fd499c-6cx4x 1/1 Running 0 11m 10.244.3.125 centos7-d <none> <none> ingress-nginx-controller-5946fd499c-khjdn 1/1 Running 0 11m 10.244.4.17 centos7-e <none> <none>修改 ingress-nginx-controller 的 svc
[root@centos7-nginx conf.d]# kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-controller NodePort 10.101.121.120 <none> 80:36459/TCP,443:33171/TCP 97m ingress-nginx-controller-admission ClusterIP 10.111.108.89 <none> 443/TCP 97m [root@centos7-nginx conf.d]# kubectl edit svc ingress-nginx-controller -n ingress-nginx ... spec: clusterIP: 10.101.121.120 externalTrafficPolicy: Cluster #---> 修改为 Local ... service/ingress-nginx-controller edited再次访问
[root@centos7-nginx conf.d]# curl -I HTTP/1.1 200 OK Server: nginx/1.16.1 Date: Sat, 20 Jun 2020 13:28:05 GMT Content-Type: text/html Content-Length: 612 Connection: keep-alive Vary: Accept-Encoding Last-Modified: Tue, 26 May 2020 15:00:20 GMT ETag: "5ecd2f04-264" Accept-Ranges: bytes # 查看本机网卡 IP [root@centos7-nginx conf.d]# ip addr show ens33 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:38:d4:e3 brd ff:ff:ff:ff:ff:ff inet 10.10.10.127/24 brd 10.10.10.255 scope global ens33 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fe38:d4e3/64 scope link valid_lft forever preferred_lft forevernginx的日志($http_x_forwarded_for)已经记录了客户端的真实IP
10.244.4.17 - - [20/Jun/2020:13:28:05 +0000] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.29.0" "10.10.10.127" 3.16.4 运行多个 ingress