K8s集群部署Jenkins Master高可用部署实操技术栈Jenkins 2.440.x Kubernetes v1.32.13 Rocky Linux 8.6 Kubernetes Plugin Kaniko Helm 3.14.x操作环境 / 对接原理 / 详细步骤 / 完整命令 / 配置文件 / 验证流程 / 排错方案K8s集群部署Jenkins Master高可用部署实操操作环境Jenkins 版本 2.440.3部署于 K8s 集群 ci 命名空间已安装 Kubernetes、Kubernetes CLI、Docker Pipeline、Credentials Binding 插件K8s 集群版本 v1.32.13目标部署命名空间 default已配置 ServiceAccount jenkins-deployer 及 RBAC 权限镜像仓库 Harbor v2.10.x地址 harbor.example.com项目 library已配置机器人账号凭据代码仓库 GitLab/GitHub已配置 Webhook 触发 Jenkins 构建容器运行时 Containerd 1.7.xJenkins Agent 使用 Kaniko 镜像构建无需 Docker daemonHelm 3.14.x应用 Chart 存放于代码仓库 helm/ 目录Values 文件按环境区分对接原理Jenkins Pipeline 以 Groovy DSL 声明式描述 CI/CD 全流程通过 Kubernetes Plugin 在 K8s 集群中动态创建 Pod 作为构建 Agent任务完成后自动销毁实现资源弹性伸缩。Pipeline 流程包含代码检出、代码质量检查、单元测试、镜像构建Kaniko、镜像推送Harbor、部署到 K8skubectl/helm、健康验证、通知。Jenkins 通过 Kubernetes CLI Plugin 调用 kubeconfig 或 ServiceAccount Token 与 K8s APIServer 通信。凭据统一存储于 Jenkins Credentials通过 withCredentials 绑定到构建环境。K8s 侧通过 RBAC 为 jenkins-deployer ServiceAccount 授予最小权限。详细步骤1. K8s 部署 Jenkins Master# 创建命名空间和持久卷 kubectl create namespace ci # Helm 部署 Jenkins helm repo add jenkins https://charts.jenkins.io helm repo update cat jenkins-values.yaml EOF controller: serviceType: ClusterIP ingress: enabled: true hostName: jenkins.example.com resources: requests: cpu: 500m memory: 1Gi limits: cpu: 2 memory: 4Gi installPlugins: - kubernetes:3937.vd7b_82db_e347 - kubernetes-cli:4383.v2a_f9d2148296 - docker-workflow:572.v950f58993843 - credentials-binding:642.v8395a_b_b_c89e - git:5.2.1 - workflow-aggregator:596.v8c21c963d92d - sonar:2.17.2 - slack:684.v8330f8598869 serviceAccount: create: true name: jenkins agent: enabled: true image: jenkins/inbound-agent:latest persistence: enabled: true storageClass: rook-ceph-block size: 50Gi EOF helm install jenkins jenkins/jenkins -n ci -f jenkins-values.yaml # 等待启动 kubectl wait --forconditionready pod -l app.kubernetes.io/componentjenkins-controller -n ci --timeout300s # 获取初始密码 kubectl exec -n ci deployment/jenkins -- cat /var/jenkins_home/secrets/initialAdminPassword2. 配置 K8s 云与动态 Agent# Manage Jenkins - Manage Nodes and Clouds - Configure Clouds # 添加 Kubernetes 云 # Name: k8s # Kubernetes URL: https://kubernetes.default.svc.cluster.local # Kubernetes namespace: ci # Service account: jenkins # Jenkins URL: http://jenkins.ci.svc.cluster.local:8080 # Jenkins tunnel: jenkins-agent.ci.svc.cluster.local:50000 # Pod 模板配置 # Name: kaniko-builder # Labels: kaniko-builder # Containers: # - name: kaniko, image: gcr.io/kaniko-project/executor:v1.19.2-debug, command: sleep, args: 999999 # - name: kubectl, image: bitnami/kubectl:1.32, command: sleep, args: 999999 # - name: helm, image: alpine/helm:3.14.0, command: sleep, args: 999999 # 验证动态 Agent kubectl get pods -n ci -w # 触发构建后自动创建 jenkins-agent-xxx Pod验证流程# 1. 验证 Jenkins 运行 kubectl get pods -n ci -l app.kubernetes.io/componentjenkins-controller # Pod STATUS Running, READY 1/1 ​ # 2. 验证动态 Agent # 触发构建后查看 ci 命名空间 Pod kubectl get pods -n ci | grep jenkins-agent # 显示 jenkins-agent-xxx Pod 运行 ​ # 3. 验证镜像构建推送 # Harbor UI 查看 library/myapp 项目或 API: curl -u robot\$libraryci-bot:${HARBOR_TOKEN} \ https://harbor.example.com/api/v2.0/projects/library/repositories/myapp/artifacts # 显示新镜像标签 ​ # 4. 验证 K8s 部署 kubectl get deployment myapp -n default # 镜像已更新为新标签 kubectl rollout status deployment/myapp -n default # deployment successfully rolled out ​ # 5. 验证 RBAC 权限 kubectl auth can-i create deployments --assystem:serviceaccount:ci:jenkins-deployer -n default # 返回 yes kubectl auth can-i delete namespaces --assystem:serviceaccount:ci:jenkins-deployer # 返回 no ​ # 6. 验证 Webhook 触发 # 推送代码后 Jenkins 自动构建 curl http://jenkins.example.com/job/myapp/lastBuild/api/json | jq .building排错方案Jenkins Agent Pod 一直 Pending检查 ci 命名空间资源是否充足ServiceAccount 是否存在镜像是否可拉取kubectl describe pod 查看事件镜像构建失败Kaniko检查 Dockerfile 语法基础镜像是否可拉取Harbor 凭据是否正确/kaniko/.docker/config.json 是否生成镜像推送失败 401检查 Harbor 机器人账号权限Token 是否过期凭据 ID 是否正确withCredentials 变量绑定是否正确kubectl 部署失败 Forbidden检查 ServiceAccount RBAC 权限RoleBinding 是否绑定正确命名空间kubeconfig Token 是否有效kubectl 部署超时检查 Deployment 镜像是否可拉取Pod 是否启动成功健康检查是否通过kubectl describe pod 和 kubectl logs 排查Webhook 不触发检查 GitLab Webhook URLJenkins 构建触发器是否启用Secret Token 是否匹配网络是否可达查看 GitLab Webhook 最近交付记录凭据泄露风险确保不在 Jenkinsfile 硬编码密码使用 withCredentials 绑定开启 Credentials 掩码功能定期轮换凭据构建速度慢优化 Dockerfile 多阶段构建和缓存层启用 Kaniko 缓存使用本地 Maven/Gradle 缓存 PVC并行执行独立阶段增大 Agent 资源