Kubernetes集群的高可用性设计与实践：从理论到落地

Kubernetes集群的高可用性设计与实践从理论到落地 硬核开场各位技术老铁们今天咱们来聊聊Kubernetes集群的高可用性设计。别跟我说你的K8s集群就一个master节点那都不叫生产环境在生产环境中高可用是底线是生命线。今天susu就带你们从理论到实践一步步构建高可用的Kubernetes集群从多master部署到etcd集群配置从网络规划到故障转移全给你整明白 核心内容1. Kubernetes高可用的核心概念什么是高可用系统在面对各种故障时仍能保持正常运行的能力Kubernetes的高可用目标99.95%的可用性每年 downtime 不超过4小时高可用的关键组件多master节点、etcd集群、负载均衡、网络冗余2. 高可用集群架构设计2.1 多Master架构--------------------- | Load Balancer | --------------------- | --------------------- | Master Node 1 | | kube-apiserver | | kube-controller-manager | | kube-scheduler | --------------------- | --------------------- | Master Node 2 | | kube-apiserver | | kube-controller-manager | | kube-scheduler | --------------------- | --------------------- | Master Node 3 | | kube-apiserver | | kube-controller-manager | | kube-scheduler | --------------------- | --------------------- | etcd Cluster | | (3/5 nodes) | --------------------- | --------------------- | Worker Nodes | | (multiple) | ---------------------2.2 etcd集群配置etcd是Kubernetes的核心存储必须高可用。# 安装etcd ETCD_VERv3.5.4 DOWNLOAD_URLhttps://github.com/etcd-io/etcd/releases/download curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o etcd.tar.gz tar xzvf etcd.tar.gz cd etcd-${ETCD_VER}-linux-amd64 cp etcd etcdctl /usr/local/bin/ # 配置etcd集群 # 在node1上执行 etcd --name etcd1 --initial-advertise-peer-urls http://192.168.1.101:2380 \ --listen-peer-urls http://192.168.1.101:2380 \ --listen-client-urls http://192.168.1.101:2379,http://127.0.0.1:2379 \ --advertise-client-urls http://192.168.1.101:2379 \ --initial-cluster-token etcd-cluster-1 \ --initial-cluster etcd1http://192.168.1.101:2380,etcd2http://192.168.1.102:2380,etcd3http://192.168.1.103:2380 \ --initial-cluster-state new # 在node2上执行 etcd --name etcd2 --initial-advertise-peer-urls http://192.168.1.102:2380 \ --listen-peer-urls http://192.168.1.102:2380 \ --listen-client-urls http://192.168.1.102:2379,http://127.0.0.1:2379 \ --advertise-client-urls http://192.168.1.102:2379 \ --initial-cluster-token etcd-cluster-1 \ --initial-cluster etcd1http://192.168.1.101:2380,etcd2http://192.168.1.102:2380,etcd3http://192.168.1.103:2380 \ --initial-cluster-state new # 在node3上执行 etcd --name etcd3 --initial-advertise-peer-urls http://192.168.1.103:2380 \ --listen-peer-urls http://192.168.1.103:2380 \ --listen-client-urls http://192.168.1.103:2379,http://127.0.0.1:2379 \ --advertise-client-urls http://192.168.1.103:2379 \ --initial-cluster-token etcd-cluster-1 \ --initial-cluster etcd1http://192.168.1.101:2380,etcd2http://192.168.1.102:2380,etcd3http://192.168.1.103:2380 \ --initial-cluster-state new # 验证etcd集群状态 etcdctl endpoint health --endpointshttp://192.168.1.101:2379,http://192.168.1.102:2379,http://192.168.1.103:23793. 部署多Master Kubernetes集群3.1 安装kubeadm# 安装依赖 apt-get update apt-get install -y apt-transport-https curl # 添加Kubernetes GPG key curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - # 添加Kubernetes apt源 cat EOF /etc/apt/sources.list.d/kubernetes.list deb https://apt.kubernetes.io/ kubernetes-xenial main EOF # 安装kubeadm, kubelet, kubectl apt-get update apt-get install -y kubelet kubeadm kubectl # 锁定版本 apt-mark hold kubelet kubeadm kubectl3.2 初始化第一个Master节点# 初始化Master节点 kubeadm init --control-plane-endpoint 192.168.1.200:6443 --upload-certs \ --pod-network-cidr10.244.0.0/16 \ --service-cidr10.96.0.0/12 \ --etcd-servershttps://192.168.1.101:2379,https://192.168.1.102:2379,https://192.168.1.103:2379 \ --etcd-cafile/etc/kubernetes/pki/etcd/ca.crt \ --etcd-certfile/etc/kubernetes/pki/apiserver-etcd-client.crt \ --etcd-keyfile/etc/kubernetes/pki/apiserver-etcd-client.key # 配置kubectl mkdir -p $HOME/.kube cp -i /etc/kubernetes/admin.conf $HOME/.kube/config chown $(id -u):$(id -g) $HOME/.kube/config # 安装网络插件 kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml3.3 添加其他Master节点# 获取加入命令 kubeadm token create --print-join-command # 在其他Master节点上执行包含--control-plane参数 kubeadm join 192.168.1.200:6443 --token token --discovery-token-ca-cert-hash sha256:hash --control-plane --certificate-key certificate-key3.4 配置负载均衡器# 安装HAProxy apt-get install -y haproxy # 配置HAProxy cat EOF /etc/haproxy/haproxy.cfg global log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners stats timeout 30s user haproxy group haproxy daemon defaults log global mode http option httplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 frontend kubernetes bind 192.168.1.200:6443 mode tcp option tcplog default_backend kubernetes-master-nodes backend kubernetes-master-nodes mode tcp option tcplog option tcp-check balance roundrobin default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100 server master1 192.168.1.101:6443 check server master2 192.168.1.102:6443 check server master3 192.168.1.103:6443 check EOF # 重启HAProxy systemctl restart haproxy systemctl enable haproxy4. 高可用网络配置4.1 配置Calico网络插件# 安装Calico kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml # 验证网络状态 kubectl get pods -n kube-system kubectl get nodes4.2 配置网络策略apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny namespace: default spec: podSelector: {} policyTypes: - Ingress - Egress5. 高可用存储配置5.1 配置StorageClassapiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: high-availability provisioner: kubernetes.io/aws-ebs parameters: type: gp2 iopsPerGB: 10 encrypted: true reclaimPolicy: Retain allowVolumeExpansion: true volumeBindingMode: WaitForFirstConsumer5.2 配置PersistentVolumeClaimapiVersion: v1 kind: PersistentVolumeClaim metadata: name: high-availability-pvc spec: storageClassName: high-availability accessModes: - ReadWriteOnce resources: requests: storage: 10Gi6. 高可用应用部署6.1 部署高可用应用示例apiVersion: apps/v1 kind: Deployment metadata: name: nginx-high-availability namespace: default spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80 resources: requests: cpu: 100m memory: 128Mi limits: cpu: 500m memory: 256Mi affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - nginx topologyKey: kubernetes.io/hostname --- apiVersion: v1 kind: Service metadata: name: nginx-service spec: selector: app: nginx ports: - port: 80 targetPort: 80 type: LoadBalancer7. 监控与告警7.1 部署Prometheus和Grafana# 使用Helm安装Prometheus和Grafana helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace # 查看监控组件 kubectl get pods -n monitoring7.2 配置高可用告警规则apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: kubernetes-high-availability-alerts namespace: monitoring spec: groups: - name: kubernetes-master rules: - alert: MasterDown expr: kube_node_status_condition{conditionReady,node~master.*} 0 for: 5m labels: severity: critical annotations: summary: Master node down description: Master node {{ $labels.node }} is down for more than 5 minutes - alert: EtcdDown expr: etcd_server_has_leader{jobetcd} 0 for: 5m labels: severity: critical annotations: summary: Etcd cluster down description: Etcd cluster has no leader for more than 5 minutes8. 故障转移与恢复8.1 模拟Master节点故障# 模拟Master节点故障 ssh master1 sudo systemctl stop kube-apiserver kube-controller-manager kube-scheduler # 查看集群状态 kubectl get nodes kubectl get pods -n kube-system # 恢复Master节点 ssh master1 sudo systemctl start kube-apiserver kube-controller-manager kube-scheduler # 验证集群状态 kubectl get nodes kubectl get pods -n kube-system8.2 模拟etcd节点故障# 模拟etcd节点故障 ssh etcd1 sudo systemctl stop etcd # 查看etcd集群状态 etcdctl endpoint health --endpointshttp://192.168.1.101:2379,http://192.168.1.102:2379,http://192.168.1.103:2379 # 恢复etcd节点 ssh etcd1 sudo systemctl start etcd # 验证etcd集群状态 etcdctl endpoint health --endpointshttp://192.168.1.101:2379,http://192.168.1.102:2379,http://192.168.1.103:2379️ 最佳实践集群规划Master节点数量应为奇数3或5个确保etcd集群的高可用每个Master节点配置至少2CPU/4GB内存Worker节点根据实际工作负载配置网络配置使用Calico或Cilium等成熟的网络插件配置网络策略限制Pod间通信确保网络带宽足够特别是etcd节点间的通信存储配置使用高可用的存储解决方案为关键应用配置PersistentVolumeClaim定期备份etcd数据监控与告警部署Prometheus和Grafana监控集群状态配置关键指标的告警建立监控Dashboard实时查看集群健康状态安全配置启用RBAC限制用户权限配置Pod安全策略定期更新Kubernetes版本和组件灾备与恢复定期备份etcd数据制定详细的故障转移和恢复流程定期进行故障演练确保团队熟悉恢复流程 总结Kubernetes集群的高可用性设计是生产环境的必备条件。通过本文的实践你应该已经掌握了多Master架构的设计与部署etcd集群的配置与管理负载均衡器的配置网络和存储的高可用配置应用的高可用部署监控与告警的设置故障转移与恢复的流程记住高可用性不是一蹴而就的需要持续的维护和优化。在实际生产环境中要根据业务需求和资源情况选择合适的高可用方案。susu碎碎念Master节点数量不要太多3个就足够了多了会增加复杂度etcd集群一定要使用奇数节点确保选举机制正常工作负载均衡器本身也要高可用避免成为单点故障定期备份etcd数据这是恢复集群的最后希望监控告警要设置合理避免告警风暴觉得有用点个赞再走咱们下期见