在 Istio 中实现 Redis 集群的数据分片、读写分离和流量镜像 (2)

部署 Redis 服务器的 Statefulset 和 Configmap。

$ kubectl apply -f k8s/redis-cluster.yaml -n redis configmap/redis-cluster created statefulset.apps/redis-cluster created service/redis-cluster created 验证 Redis 部署

确认 Redis 节点已经启动并正常运行：

$ kubectl get pod -n redis NAME READY STATUS RESTARTS AGE redis-cluster-0 2/2 Running 0 4m25s redis-cluster-1 2/2 Running 0 3m56s redis-cluster-2 2/2 Running 0 3m28s redis-cluster-3 2/2 Running 0 2m58s redis-cluster-4 2/2 Running 0 2m27s redis-cluster-5 2/2 Running 0 117s 创建 Redis Cluster

在上面的步骤中，我们采用 Statefulset 部署了6个 Redis 节点，但目前这6个节点还是相互独立的，并未形成一个集群。下面我们采用 Redis 的 cluster create 命令将这些节点组成一个 Redis Cluster。

$ kubectl exec -it redis-cluster-0 -n redis -- redis-cli --cluster create --cluster-replicas 1 $(kubectl get pods -l app=redis-cluster -o jsonpath='{range.items[*]}{.status.podIP}:6379 ' -n redis) Defaulting container name to redis. Use 'kubectl describe pod/redis-cluster-0 -n redis' to see all of the containers in this pod. >>> Performing hash slots allocation on 6 nodes... Master[0] -> Slots 0 - 5460 Master[1] -> Slots 5461 - 10922 Master[2] -> Slots 10923 - 16383 Adding replica 172.16.0.72:6379 to 172.16.0.138:6379 Adding replica 172.16.0.201:6379 to 172.16.1.52:6379 Adding replica 172.16.0.139:6379 to 172.16.1.53:6379 M: 8fdc7aa28a6217b049a2265b87bff9723f202af0 172.16.0.138:6379 slots:[0-5460] (5461 slots) master M: 4dd6c1fecbbe4527e7d0de61b655e8b74b411e4c 172.16.1.52:6379 slots:[5461-10922] (5462 slots) master M: 0b86a0fbe76cdd4b48434b616b759936ca99d71c 172.16.1.53:6379 slots:[10923-16383] (5461 slots) master S: 94b139d247e9274b553c82fbbc6897bfd6d7f693 172.16.0.139:6379 replicates 0b86a0fbe76cdd4b48434b616b759936ca99d71c S: e293d25881c3cf6db86034cd9c26a1af29bc585a 172.16.0.72:6379 replicates 8fdc7aa28a6217b049a2265b87bff9723f202af0 S: ab897de0eca1376558e006c5b0a49f5004252eb6 172.16.0.201:6379 replicates 4dd6c1fecbbe4527e7d0de61b655e8b74b411e4c Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join . >>> Performing Cluster Check (using node 172.16.0.138:6379) M: 8fdc7aa28a6217b049a2265b87bff9723f202af0 172.16.0.138:6379 slots:[0-5460] (5461 slots) master 1 additional replica(s) M: 4dd6c1fecbbe4527e7d0de61b655e8b74b411e4c 172.16.1.52:6379 slots:[5461-10922] (5462 slots) master 1 additional replica(s) S: 94b139d247e9274b553c82fbbc6897bfd6d7f693 172.16.0.139:6379 slots: (0 slots) slave replicates 0b86a0fbe76cdd4b48434b616b759936ca99d71c M: 0b86a0fbe76cdd4b48434b616b759936ca99d71c 172.16.1.53:6379 slots:[10923-16383] (5461 slots) master 1 additional replica(s) S: ab897de0eca1376558e006c5b0a49f5004252eb6 172.16.0.201:6379 slots: (0 slots) slave replicates 4dd6c1fecbbe4527e7d0de61b655e8b74b411e4c S: e293d25881c3cf6db86034cd9c26a1af29bc585a 172.16.0.72:6379 slots: (0 slots) slave replicates 8fdc7aa28a6217b049a2265b87bff9723f202af0 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. 验证 Redis Cluster

我们可以采用 cluster info 命令查看 Redis Cluster 的配置信息和 Cluster 中的成员节点，以验证集群是否创建成功。

$ kubectl exec -it redis-cluster-0 -c redis -n redis -- redis-cli cluster info cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:6 cluster_size:3 cluster_current_epoch:6 cluster_my_epoch:1 cluster_stats_messages_ping_sent:206 cluster_stats_messages_pong_sent:210 cluster_stats_messages_sent:416 cluster_stats_messages_ping_received:205 cluster_stats_messages_pong_received:206 cluster_stats_messages_meet_received:5 cluster_stats_messages_received:416 部署测试用客户端

我们部署一个客户端，以用于发送测试命令:

$ kubectl apply -f k8s/redis-client.yaml -n redis deployment.apps/redis-client created 通过 Istio 下发 Redis Cluster 相关的 Envoy 配置

在下面的步骤中，我们将通过 Istio 向 Envoy Sidecar 下发 Redis Cluster 相关配置，以在无需改动客户端的情况下启用 Redis Cluster 的高级功能，包括数据分片、读写分离和流量镜像。

创建 Envoy Redis Cluster

Envoy 提供了 "envoy.clusters.redis" 类型的 Envoy Cluster 来连接后端的 Redis Cluster，Envoy 会通过该 Cluster 获取后端 Redis Cluster 的拓扑结构，包括有多少个分片（shard），每个分片负责哪些 slot，以及分片中包含哪些节点，以将来自客户端的请求分发到正确的 Redis 节点上。

采用 EnvoyFilter 来创建所需的 Envoy Redis Cluster：

$ kubectl apply -f istio/envoyfilter-custom-redis-cluster.yaml envoyfilter.networking.istio.io/custom-redis-cluster created 创建 Envoy Redis Proxy

Istio 缺省下发的 LDS 中配置的是 TCP proxy filter，我们需要将其替换为 Redis Proxy filter。