资讯详情

深入了解 · 专业解读

【云原生实战】K8s 集群高可用与中间件灾备平台全流程实践

发布时间：2026/8/6 17:06:07
【云原生实战】K8s 集群高可用与中间件灾备平台全流程实践 创作灵感记录云原生运维项目完整实践复盘整理 K8s 高可用、中间件灾备、故障切换技术笔记将个人原创实操项目文档搬运发布于此 目录一、项目概述二、整体架构设计三、环境规划四、入口层高可用HAProxy Keepalived4.1 ingress-nginx NodePort 入口4.2 HAProxy 负载均衡4.3 Keepalived VIP 漂移4.4 VIP 漂移演练五、RBAC 最小权限控制六、存储准备PV/PVC七、MySQL 主从复制与备份恢复7.1 MySQL 主从部署7.2 GTID 主从复制配置7.3 数据同步验证7.4 备份与误删恢复7.5 PVC 持久化验证八、Redis 主从 Sentinel 故障切换8.1 Redis 主从部署8.2 Sentinel 部署8.3 主从同步验证8.4 故障切换演练九、监控体系建设十、容量压测十一、故障演练与 Runbook十二、可选扩展12.1 MetalLB 负载均衡12.2 NetworkPolicy 网络策略十三、项目总结一、项目概述本项目聚焦 Kubernetes 集群基础设施与中间件灾备方向构建一套完整的高可用与灾备运维平台。覆盖入口层高可用、中间件主从复制与故障切换、数据备份恢复、权限控制、监控告警和故障演练全链路。核心能力✅ 入口层高可用HAProxy Keepalived VIP 自动漂移✅ MySQL 主从复制GTID 备份恢复 PVC 持久化✅ Redis 主从 Sentinel 故障自动切换✅ RBAC 最小权限控制命名空间级只读✅ Prometheus/Grafana 监控体系✅ 8 大故障场景演练 标准化 Runbook✅ 可选扩展MetalLB、NetworkPolicy技术栈Kubernetes、HAProxy、Keepalived、MySQL 主从、GTID、Redis Sentinel、PV/PVC、RBAC、Prometheus、Grafana、MetalLB二、整体架构设计2.1 入口层高可用链路用户请求 ↓ VIP (192.168.140.100) ↓ Keepalived 主备漂移 (VRRP) ↓ HAProxy 负载均衡 (roundrobin) ↓ ingress-nginx NodePort (30080) ↓ Ingress 七层转发 ↓ Service → Pod2.2 中间件灾备架构MySQL层 mysql-master (写) ←GTID同步→ mysql-slave (读) ↓ PVC持久化 ↓ PVC持久化 节点本地磁盘 节点本地磁盘 Redis层 redis-master ←复制→ redis-slave ↑ 监控/切换 Sentinel × 3 (选举仲裁)2.3 项目资源清单模块资源说明入口层HAProxy × 2 Keepalived × 2VIP 高可用入口K8s 入口ingress-nginx (NodePort)集群统一入口测试业务sre-ha-demo (Nginx × 2)验证入口链路数据库mysql-master / mysql-slaveGTID 主从复制缓存redis-master / redis-slave / sentinel × 3主从 故障切换存储4 组 PV/PVCMySQL/Redis 数据持久化权限ServiceAccount Role RoleBinding命名空间只读监控kube-prometheus-stack指标采集与可视化三、环境规划3.1 节点规划节点IP角色master192.168.140.206K8s 控制平面node1 / lb1192.168.140.207Worker 负载均衡主节点node2 / lb2192.168.140.208Worker 负载均衡备节点VIP192.168.140.100虚拟漂移 IP设计思路两个 Worker 节点同时兼任负载均衡节点无需额外虚拟机即可完整演示 HAProxy 转发和 Keepalived VIP 漂移。3.2 集群健康检查# 节点状态 kubectl get nodes -o wide # DNS测试 kubectl run dns-test --imagebusybox:1.36 --restartNever -- sleep 3600 kubectl exec -it dns-test -- nslookup kubernetes.default kubectl delete pod dns-test四、入口层高可用HAProxy Keepalived4.1 ingress-nginx NodePort 入口使用 Helm 安装 ingress-nginx固定 NodePort 端口helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm repo update helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \ -n ingress-nginx \ --create-namespace \ --set controller.service.typeNodePort \ --set controller.service.nodePorts.http30080 \ --set controller.service.nodePorts.https30443验证 NodePort 可达curl -I http://192.168.140.207:30080 curl -I http://192.168.140.208:300804.2 HAProxy 负载均衡安装# Rocky/CentOS dnf install -y haproxy keepalived # Ubuntu apt install -y haproxy keepalived配置lb1 和 lb2 相同global log /dev/log local0 log /dev/log local1 notice maxconn 4096 daemon defaults log global mode http option httplog option dontlognull timeout connect 5s timeout client 50s timeout server 50s frontend sre_ha_http bind *:80 default_backend ingress_nginx_nodes backend ingress_nginx_nodes balance roundrobin option tcp-check server node1 192.168.140.207:30080 check server node2 192.168.140.208:30080 check listen stats bind *:8404 stats enable stats uri /stats stats refresh 10s启动与验证# 配置语法检查 haproxy -c -f /etc/haproxy/haproxy.cfg # 启动 systemctl enable --now haproxy # 测试 curl -H Host: sre-ha.local http://127.0.0.1 # 监控页面http://lb1:8404/stats4.3 Keepalived VIP 漂移主节点lb1配置global_defs { router_id lb1 } vrrp_script check_haproxy { script /bin/bash -c systemctl is-active --quiet haproxy interval 2 weight -20 fall 2 rise 2 } vrrp_instance VI_1 { state MASTER interface ens33 virtual_router_id 51 priority 120 advert_int 1 authentication { auth_type PASS auth_pass SreHa123 } virtual_ipaddress { 192.168.140.100/24 } track_script { check_haproxy } }备节点lb2配置global_defs { router_id lb2 } vrrp_script check_haproxy { script /bin/bash -c systemctl is-active --quiet haproxy interval 2 weight -20 fall 2 rise 2 } vrrp_instance VI_1 { state BACKUP interface ens33 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass SreHa123 } virtual_ipaddress { 192.168.140.100/24 } track_script { check_haproxy } }关键设计vrrp_script检测 HAProxy 服务状态当 HAProxy 异常时自动降低优先级触发 VIP 漂移到备节点实现入口层服务级别的高可用。启动与验证# 两台都启动 systemctl enable --now keepalived # 查看VIP正常在lb1上 ip addr | grep 192.168.140.1004.4 VIP 漂移演练场景 1主节点 Keepalived 故障# lb1上停止Keepalived systemctl stop keepalived # lb2上查看VIP是否漂移过来 ip addr | grep 192.168.140.100 # VIP访问验证 curl -H Host: sre-ha.local http://192.168.140.100 # 恢复lb1 systemctl start keepalived场景 2HAProxy 服务异常HAProxy 停止 → vrrp_script 检测失败 → 主节点优先级降低 → VIP 漂移到备节点实现服务级别的故障自动切换五、RBAC 最小权限控制目标创建只能查看sre-ha命名空间资源的只读账号不能删除资源也不能访问其他命名空间。apiVersion: v1 kind: ServiceAccount metadata: name: sre-ha-readonly namespace: sre-ha --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: sre-ha-readonly-role namespace: sre-ha rules: - apiGroups: [] resources: [pods, services, endpoints, configmaps, events] verbs: [get, list, watch] - apiGroups: [apps] resources: [deployments, replicasets, statefulsets] verbs: [get, list, watch] - apiGroups: [networking.k8s.io] resources: [ingresses] verbs: [get, list, watch] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: sre-ha-readonly-binding namespace: sre-ha subjects: - kind: ServiceAccount name: sre-ha-readonly namespace: sre-ha roleRef: kind: Role name: sre-ha-readonly-role apiGroup: rbac.authorization.k8s.io权限验证# 允许的操作返回yes kubectl auth can-i get pods -n sre-ha --assystem:serviceaccount:sre-ha:sre-ha-readonly kubectl auth can-i list svc -n sre-ha --assystem:serviceaccount:sre-ha:sre-ha-readonly # 禁止的操作返回no kubectl auth can-i delete pods -n sre-ha --assystem:serviceaccount:sre-ha:sre-ha-readonly kubectl auth can-i get pods -n kube-system --assystem:serviceaccount:sre-ha:sre-ha-readonly六、存储准备PV/PVC为 MySQL 和 Redis 准备本地持久化存储PV 名称容量挂载路径节点用途mysql-master-pv5Gi/data/sre-ha/mysql-masternode1MySQL 主库数据mysql-slave-pv5Gi/data/sre-ha/mysql-slavenode2MySQL 从库数据redis-master-pv2Gi/data/sre-ha/redis-masternode1Redis 主数据redis-slave-pv2Gi/data/sre-ha/redis-slavenode2Redis 从数据创建本地目录# node1 mkdir -p /data/sre-ha/mysql-master /data/sre-ha/redis-master chmod -R 777 /data/sre-ha # node2 mkdir -p /data/sre-ha/mysql-slave /data/sre-ha/redis-slave chmod -R 777 /data/sre-haPV/PVC YAML节选 MySQL MasterapiVersion: v1 kind: PersistentVolume metadata: name: mysql-master-pv spec: capacity: storage: 5Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: manual local: path: /data/sre-ha/mysql-master nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node1 --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-master-pvc namespace: sre-ha spec: accessModes: - ReadWriteOnce storageClassName: manual resources: requests: storage: 5Gi七、MySQL 主从复制与备份恢复7.1 MySQL 主从部署Secret 管理密码apiVersion: v1 kind: Secret metadata: name: mysql-secret namespace: sre-ha type: Opaque stringData: MYSQL_ROOT_PASSWORD: Root123456 MYSQL_REPL_PASSWORD: Repl123456Master 配置开启 GTID、binlog[mysqld] server-id1 log-binmysql-bin binlog_formatROW gtid_modeON enforce_gtid_consistencyONSlave 配置只读、relay log[mysqld] server-id2 relay-logmysql-relay-bin log-binmysql-bin binlog_formatROW gtid_modeON enforce_gtid_consistencyON read_onlyON7.2 GTID 主从复制配置进入 Slave MySQL 执行CHANGE REPLICATION SOURCE TO SOURCE_HOSTmysql-master-svc, SOURCE_PORT3306, SOURCE_USERrepl, SOURCE_PASSWORDRepl123456, SOURCE_AUTO_POSITION1; START REPLICA; SHOW REPLICA STATUS\G关键状态验证Replica_IO_Running: Yes Replica_SQL_Running: Yes7.3 数据同步验证# Master写入测试数据 kubectl exec -it -n sre-ha $MYSQL_MASTER_POD -- mysql -uroot -pRoot123456 \ -e USE sre_dr; INSERT INTO service_data(name) VALUES (replication-test); # Slave查询验证 kubectl exec -it -n sre-ha $MYSQL_SLAVE_POD -- mysql -uroot -pRoot123456 \ -e USE sre_dr; SELECT * FROM service_data;7.4 备份与误删恢复备份# mysqldump备份 kubectl exec -n sre-ha $MYSQL_MASTER_POD -- sh -c \ mysqldump -uroot -pRoot123456 --databases sre_dr /tmp/sre_dr_backup.sql # 复制到本地 kubectl cp sre-ha/$MYSQL_MASTER_POD:/tmp/sre_dr_backup.sql ./sre_dr_backup.sql模拟误删与恢复# 模拟误删 kubectl exec -it -n sre-ha $MYSQL_MASTER_POD -- mysql -uroot -pRoot123456 \ -e USE sre_dr; DELETE FROM service_data; # 恢复备份 kubectl cp ./sre_dr_backup.sql sre-ha/$MYSQL_MASTER_POD:/tmp/sre_dr_backup.sql kubectl exec -n sre-ha $MYSQL_MASTER_POD -- sh -c \ mysql -uroot -pRoot123456 /tmp/sre_dr_backup.sql # 验证恢复 kubectl exec -it -n sre-ha $MYSQL_MASTER_POD -- mysql -uroot -pRoot123456 \ -e USE sre_dr; SELECT * FROM service_data;7.5 PVC 持久化验证# 删除MySQL Master Pod kubectl delete pod -n sre-ha $MYSQL_MASTER_POD # 等待重建 kubectl get pods -n sre-ha -l appmysql-master -w # 验证数据仍在 kubectl exec -it -n sre-ha $NEW_MASTER_POD -- mysql -uroot -pRoot123456 \ -e USE sre_dr; SELECT * FROM service_data;八、Redis 主从 Sentinel 故障切换8.1 Redis 主从部署Master 配置port 6379 bind 0.0.0.0 protected-mode no appendonly yes dir /dataSlave 配置port 6379 bind 0.0.0.0 protected-mode no appendonly yes dir /data replicaof redis-master-svc 63798.2 Sentinel 部署3 个 Sentinel 实例组成仲裁集群quorum2至少 2 个 Sentinel 同意才触发切换port 26379 bind 0.0.0.0 protected-mode no sentinel monitor mymaster redis-master-svc 6379 2 sentinel down-after-milliseconds mymaster 5000 sentinel failover-timeout mymaster 10000 sentinel parallel-syncs mymaster 1Sentinel 工作机制每个 Sentinel 每秒向 master、slave 和其他 Sentinel 发送 pingmaster 超过 down-after-milliseconds 未响应 → 标记为主观下线sdown足够数量 Sentinelquorum都认为 sdown → 标记为客观下线odownSentinel 选举领头者 → 发起 failover → 从 slave 中选举新 master8.3 主从同步验证# Master写入 kubectl exec -it -n sre-ha $REDIS_MASTER_POD -- redis-cli set project sre-ha-dr # Slave读取验证 kubectl exec -it -n sre-ha $REDIS_SLAVE_POD -- redis-cli get project # 查看复制状态 kubectl exec -it -n sre-ha $REDIS_SLAVE_POD -- redis-cli info replication # role:slave # master_link_status:up8.4 故障切换演练模拟 Master 故障# 缩容master为0模拟宕机 kubectl scale deploy redis-master -n sre-ha --replicas0观察切换过程# 查看Sentinel识别的当前master SENTINEL_POD$(kubectl get pod -n sre-ha -l appredis-sentinel -o jsonpath{.items[0].metadata.name}) kubectl exec -it -n sre-ha $SENTINEL_POD -- redis-cli -p 26379 sentinel get-master-addr-by-name mymaster # 查看slave是否被提升为master kubectl exec -it -n sre-ha $REDIS_SLAVE_POD -- redis-cli info replication # role:master恢复原 Masterkubectl scale deploy redis-master -n sre-ha --replicas1 # 恢复后原master会成为新master的slave九、监控体系建设使用 Helm 部署 kube-prometheus-stack复用集群监控能力helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update helm upgrade --install kube-prometheus-stack prometheus-community/kube-prometheus-stack \ -n monitoring \ --create-namespace重点观测指标监控对象关键指标NodeCPU / 内存 / 磁盘使用率PodCPU / 内存、重启次数、Ready 状态Deployment可用副本数、期望副本数PVC存储使用率MySQL Pod运行状态、重启次数Redis Pod运行状态、复制状态ingress-nginx请求量、响应状态码十、容量压测使用ab工具对 VIP 入口进行压测评估入口链路承载能力# 安装压测工具 dnf install -y httpd-tools # Rocky/CentOS apt install -y apache2-utils # Ubuntu # 压测VIP入口 ab -n 5000 -c 100 -H Host: sre-ha.local http://192.168.140.100/关注指标Requests per secondQPSTime per request平均响应时间Failed requests失败请求数Node CPU 峰值、Pod CPU 峰值同时观测kubectl top nodes kubectl top pods -n sre-ha十一、故障演练与 Runbook8 大故障演练场景序号故障场景验证能力1入口主节点 Keepalived 故障VIP 自动漂移到备节点2HAProxy 服务异常vrrp_script 检测触发 VIP 漂移3业务 Pod 被删除Deployment 自动重建服务不中断4Service 无 EndpointService selector 与 Pod label 不匹配5MySQL 主库 Pod 重建PVC 持久化数据不丢失6MySQL 数据误删mysqldump 备份恢复7Redis Master 故障Sentinel 自动主从切换8RBAC 越权操作只读账号被拒绝最小权限生效Runbook 通用模板每个故障场景整理标准化文档# 故障名称 ## 1. 故障现象 ## 2. 影响范围 ## 3. 可能原因 ## 4. 排查命令 ## 5. 恢复步骤 ## 6. 验证方法 ## 7. 复盘总结十二、可选扩展12.1 MetalLB 负载均衡裸金属 K8s 集群默认没有云厂商 LoadBalancer使用 MetalLB 为 LoadBalancer 类型 Service 分配外部 IPapiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: sre-ha-pool namespace: metallb-system spec: addresses: - 192.168.140.110-192.168.140.120 --- apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: sre-ha-l2 namespace: metallb-system spec: ipAddressPools: - sre-ha-pool创建 LoadBalancer 类型 Service 后自动从 IP 池中分配 EXTERNAL-IP模拟云环境服务暴露能力。12.2 NetworkPolicy 网络策略通过 NetworkPolicy 控制 Pod 间的访问关系需 Calico/Cilium 等支持 NetworkPolicy 的 CNIapiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-backend-to-mysql namespace: sre-ha spec: podSelector: matchLabels: app: mysql-master policyTypes: - Ingress ingress: - from: - podSelector: matchLabels: app: sre-ha-demo ports: - protocol: TCP port: 3306十三、项目总结完整能力闭环入口层高可用 → VIP Keepalived 主备漂移 → HAProxy 负载均衡 → ingress-nginx 统一入口 中间件灾备 → MySQL 主从复制 GTID → MySQL 备份恢复 → Redis Sentinel 故障切换 → PV/PVC 数据持久化 运维治理 → RBAC 最小权限 → Prometheus/Grafana 监控 → 故障演练 Runbook → 容量压测评估核心收获入口高可用设计掌握 HAProxy Keepalived 构建 VIP 高可用入口的完整方案理解 VRRP 协议和 VIP 漂移机制中间件灾备能力MySQL 主从复制、备份恢复Redis Sentinel 故障切换数据层具备基础灾备能力数据持久化通过 PV/PVC 实现有状态服务的数据持久化Pod 重建数据不丢失权限治理基于 RBAC 实现命名空间级最小权限控制遵循安全运维原则故障思维通过 8 大故障场景演练建立从现象→排查→恢复→复盘的完整故障处理流程后续扩展方向MySQL 高可用方案升级MGR / OrchestratorRedis Cluster 集群模式EFK / Loki 日志采集Alertmanager 告警通知邮件 / 钉钉 / 飞书Velero 集群备份与恢复 如果这篇文章对你有帮助欢迎点赞、收藏、关注有任何问题欢迎评论区交流

相关新闻