SeaTunnel Kubernetes 混合集群模式部署指南Master 与 Worker 同进程的高可用集群搭建【免费下载链接】seatunnelSeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.项目地址: https://gitcode.com/GitHub_Trending/se/seatunnel本篇技术指南以 SeaTunnel Engine 的混合集群模式Hybrid Cluster Mode为主题完整讲解如何在 Kubernetes 中通过 StatefulSet、Headless Service 与 ConfigMap 搭建一个Master 与 Worker 运行于同一进程的多节点集群涵盖 Hazelcast 成员发现、RBAC 授权、健康检查与优雅停止、REST API 验证等全流程。读完本文你将能够独立在 Kubernetes 上部署一个 3 副本的 SeaTunnel 混合集群并理解其调度、执行与资源配比的核心原理。混合集群模式是什么在 SeaTunnel Engine 中集群由两类逻辑角色构成Master调度节点负责任务调度、作业状态管理、集群元数据维护与 REST API 服务Worker执行节点负责实际运行 SeaTunnel 任务Job占用 Slot 资源执行 Pipeline。混合集群模式Hybrid Cluster Mode下Master 与 Worker 运行在同一个 JVM 进程中。集群中的所有节点地位对等——每个节点都可以参与 Master 选举成为调度者同时也都承担任务执行职责。从启动脚本的实现可以印证这一点seatunnel-cluster.sh 中定义了三种节点角色其中master_and_worker是默认角色NODE_ROLEmaster_and_worker对应读取config/hazelcast.yaml、config/jvm_options与config/seatunnel.yaml启动org.apache.seatunnel.core.starter.seatunnel.SeaTunnelServer主类而master与worker角色则分别读取hazelcast-master.yaml与hazelcast-worker.yaml对应分离集群模式。适用场景混合模式部署简单、节点数少适合小规模集群或测试环境。由于调度与执行资源不隔离任务负载较高时可能影响 Master 的选举、调度与 REST API 稳定性生产环境更推荐使用分离集群模式将调度与执行资源隔离。部署原则在 Kubernetes 上部署混合集群需要遵循以下四条核心原则使用 StatefulSet 部署混合集群节点每个节点拥有稳定的网络标识Pod 名称seatunnel-0、seatunnel-1…配合顺序启动、顺序滚动更新等特性适合有状态分布式系统使用 Headless Service 提供 Hazelcast Kubernetes discoveryHazelcast 成员通过clusterIP: None的 Headless Service 发现彼此并组建成集群所有节点使用同一份hazelcast.yaml、hazelcast-client.yaml和seatunnel.yaml混合模式下所有节点角色相同配置天然一致只需通过 ConfigMap 统一挂载资源配置需同时考虑 Master 与 Worker 负载因为每个进程同时承担调度与执行JVM 内存、CPU 配额应比纯 Master 或纯 Worker 节点更高且要预留 Master 职责元数据、调度、REST所需的余量。创建 ConfigMap拆分配置职责生产环境建议按配置职责拆分 ConfigMap避免单个 YAML 过长也便于后续独立更新与审计。本文示例创建三个 ConfigMapHazelcast 服务端配置、Hazelcast 客户端配置、SeaTunnel Engine 配置。⚠️注意生产环境应将敏感信息如 HDFS 凭证、HTTP Basic Auth 密码放入 Secret以下示例只展示非敏感配置。Hazelcast 配置hazelcast.yaml是 SeaTunnel 集群的成员发现与网络基础。示例配置通过Hazelcast Kubernetes API 发现机制让节点自动互相发现apiVersion: v1 kind: ConfigMap metadata: name: seatunnel-hazelcast-config data: hazelcast.yaml: | hazelcast: cluster-name: seatunnel-cluster network: rest-api: enabled: true endpoint-groups: CLUSTER_WRITE: enabled: true DATA: enabled: true port: auto-increment: false port: 5801 join: kubernetes: enabled: true namespace: default service-name: seatunnel-cluster service-port: 5801 properties: hazelcast.invocation.max.retry.count: 20 hazelcast.tcp.join.port.try.count: 30 hazelcast.logging.type: log4j2 hazelcast.operation.generic.thread.count: 50 hazelcast.heartbeat.failuredetector.type: phi-accrual hazelcast.heartbeat.interval.seconds: 2 hazelcast.max.no.heartbeat.seconds: 180 hazelcast.heartbeat.phiaccrual.failuredetector.threshold: 10 hazelcast.heartbeat.phiaccrual.failuredetector.sample.size: 200 hazelcast.heartbeat.phiaccrual.failuredetector.min.std.dev.millis: 100关键字段说明配置项说明cluster-name集群名称所有节点必须一致否则无法加入同一集群同时需与 Hazelcast 客户端配置一致network.rest-api开启 Hazelcast REST API 与CLUSTER_WRITE、DATA端点组SeaTunnel Engine 依赖其进行集群管理与数据读写port.auto-increment: falseport: 5801固定使用 5801 端口避免端口自动递增导致与 Kubernetes Service 端口映射错位join.kubernetes开启 Kubernetes 成员发现通过service-name指定的 Headless Service 解析成员地址namespace指定运行命名空间hazelcast.operation.generic.thread.count通用操作线程数SeaTunnel 默认配置为 50可结合 CPU 核数调整hazelcast.heartbeat.*心跳与故障检测参数采用phi-accrual故障检测器心跳间隔 2 秒最大无心跳容忍 180 秒仓库自带的本地默认配置 config/hazelcast.yaml 采用tcp-ip发现member-list: localhost而 Kubernetes 部署必须将join段替换为kubernetes发现二者仅在成员发现策略上不同properties心跳与线程参数保持一致。备选使用 DNS 发现如果希望避免 Hazelcast 直接访问 Kubernetes API例如出于安全合规或 RBAC 简化考虑可以将join.kubernetes替换为基于 DNS 的发现join: kubernetes: enabled: true service-dns: seatunnel-cluster.default.svc.cluster.local service-dns-timeout: 10说明使用 DNS 发现时下文为 API 发现创建 RBAC章节不是成员发现所必需的。如果跳过 RBAC 清单也需要从 StatefulSet 中移除serviceAccountName: seatunnel或单独创建这个 ServiceAccount。Hazelcast Client 配置hazelcast-client.yaml供 SeaTunnel 引擎内部客户端连接集群使用例如seatunnel.sh提交任务时连接集群。关键点是cluster-members指向Headless Service 的 DNS 名称apiVersion: v1 kind: ConfigMap metadata: name: seatunnel-client-config data: hazelcast-client.yaml: | hazelcast-client: cluster-name: seatunnel-cluster properties: hazelcast.logging.type: log4j2 connection-strategy: connection-retry: cluster-connect-timeout-millis: 7000 network: cluster-members: # 如果 SeaTunnel 部署在其他命名空间需要将 default 替换为实际命名空间。 - seatunnel-cluster.default.svc.cluster.local:5801仓库默认配置 config/hazelcast-client.yaml 中cluster-connect-timeout-millis为 3000msKubernetes 场景由于 Pod 启动、服务发现需要时间建议调大到 7000ms 以增强连接韧性。若跨命名空间部署务必替换 DNS 中的default。SeaTunnel Engine 配置seatunnel.yaml是引擎级配置包含副本数、Slot、调度策略、Checkpoint 与 HTTP 服务。对照源码 ServerConfigOptions.java 可以确认各参数的默认值与语义apiVersion: v1 kind: ConfigMap metadata: name: seatunnel-engine-config data: seatunnel.yaml: | seatunnel: engine: backup-count: 1 history-job-expire-minutes: 1440 print-execution-info-interval: 300 classloader-cache-mode: true slot-service: dynamic-slot: false slot-num: 8 job-schedule-strategy: WAIT checkpoint: interval: 180000 timeout: 30000 storage: type: hdfs max-retained: 3 plugin-config: namespace: /seatunnel/checkpoint/ storage.type: hdfs fs.defaultFS: hdfs://namenode:8020 http: enable-http: true port: 8080各项配置与源码默认值对照配置项示例值源码默认值说明backup-count11Hazelcast 分区数据备份副本数backup-count个节点宕机不影响集群元数据可用性history-job-expire-minutes14401440历史作业状态保留时间分钟到期后自动清理print-execution-info-interval30060打印执行信息的间隔秒classloader-cache-modetruetrue类加载器缓存模式缓存开启时jar 相同的作业共享同一个 classloader显著减少类加载开销slot-service.dynamic-slotfalsetrue是否启用动态 Slot。false时使用固定 Slot 数即slot-numslot-service.slot-num8CPU 核数 × 2固定 Slot 数量仅当dynamic-slot: false时生效job-schedule-strategyWAITREJECT任务队列满时的策略REJECT拒绝新任务WAIT让任务排队等待checkpoint.interval180000300000两次 checkpoint 之间的间隔毫秒checkpoint.timeout3000030000单次 checkpoint 超时时间毫秒checkpoint.storage.typehdfslocalfilecheckpoint 存储类型生产集群建议使用 HDFS 等共享存储checkpoint.storage.max-retained320最多保留的 checkpoint 数量http.enable-httptruefalse是否开启引擎 HTTPREST服务http.port80808080HTTP 服务端口提示checkpoint.storage.plugin-config中的fs.defaultFS需指向实际可用的 HDFS NameNode 地址示例为hdfs://namenode:8020若无 HDFS也可像仓库默认配置 config/seatunnel.yaml 那样使用localfile或file:///本地文件系统仅适合单机测试多节点集群必须使用共享存储。此外仓库默认配置还包含telemetry.metric、telemetry.logs等可观测性项可在 ServerConfigOptions.java 中查阅。为 API 发现创建 RBAChazelcast.yaml中使用的namespace、service-name和service-port属于Hazelcast Kubernetes API 发现需要 Pod 具备读取 Pod/Service/Endpoints 的权限。在启用 RBAC 的集群中请先创建 ServiceAccount、Role 和 RoleBinding再启动 StatefulSetapiVersion: v1 kind: ServiceAccount metadata: name: seatunnel --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: seatunnel-hazelcast-discovery rules: - apiGroups: [] resources: [pods, services, endpoints] verbs: [get, list, watch] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: seatunnel-hazelcast-discovery subjects: - kind: ServiceAccount name: seatunnel roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: seatunnel-hazelcast-discoveryRBAC 最小权限说明Role 仅授予get、list、watch三个只读动词资源限定为pods、services、endpoints足够 Hazelcast 成员发现使用。若改用service-dns发现可以跳过本节同时需从 StatefulSet 移除serviceAccountName或自行创建该 ServiceAccount。创建 Service混合集群需要两个 ServiceHeadless Serviceseatunnel-clusterclusterIP: None供 Hazelcast 成员发现与解析 Pod IPpublishNotReadyAddresses: true确保未就绪的 Pod 也能被 DNS 解析到从而支持集群在节点全部启动前就开始组网普通 ClusterIP Serviceseatunnel暴露 REST API8080与 Hazelcast 端口5801供客户端与外部访问。apiVersion: v1 kind: Service metadata: name: seatunnel-cluster labels: app: seatunnel-cluster spec: clusterIP: None publishNotReadyAddresses: true ports: - name: hazelcast port: 5801 targetPort: 5801 selector: app: seatunnel component: hybrid --- apiVersion: v1 kind: Service metadata: name: seatunnel labels: app: seatunnel spec: type: ClusterIP ports: - name: rest-api port: 8080 targetPort: 8080 - name: hazelcast port: 5801 targetPort: 5801 selector: app: seatunnel component: hybrid注意两个 Service 的selector必须与 StatefulSet Pod 标签一致app: seatunnel、component: hybrid否则成员发现与流量转发都会失效。创建 StatefulSetStatefulSet 是混合集群的部署载体。核心要素serviceName: seatunnel-cluster绑定 Headless Service获得稳定网络标识命令直接使用/opt/seatunnel/bin/seatunnel-cluster.sh默认即master_and_worker角色通过SEATUNNEL_HOME与HAZELCAST_CLUSTER_NAME环境变量注入主目录与集群名三个 ConfigMap 分别以subPath方式挂载到/opt/seatunnel/config/下对应文件保证容器内读取到的就是 ConfigMap 中的三份配置资源配额需同时覆盖 Master 与 Worker 负载示例 requests 1 核 / 2Gilimits 2 核 / 4GiterminationGracePeriodSeconds: 120为优雅下线预留时间。apiVersion: apps/v1 kind: StatefulSet metadata: name: seatunnel labels: app: seatunnel component: hybrid spec: serviceName: seatunnel-cluster replicas: 3 selector: matchLabels: app: seatunnel component: hybrid template: metadata: labels: app: seatunnel component: hybrid spec: serviceAccountName: seatunnel containers: - name: app image: seatunnel:3.0.0 imagePullPolicy: IfNotPresent command: - /opt/seatunnel/bin/seatunnel-cluster.sh env: - name: SEATUNNEL_HOME value: /opt/seatunnel - name: HAZELCAST_CLUSTER_NAME value: seatunnel-cluster ports: - containerPort: 8080 name: rest-api - containerPort: 5801 name: hazelcast resources: requests: cpu: 1 memory: 2Gi limits: cpu: 2 memory: 4Gi volumeMounts: - name: hazelcast-config mountPath: /opt/seatunnel/config/hazelcast.yaml subPath: hazelcast.yaml - name: client-config mountPath: /opt/seatunnel/config/hazelcast-client.yaml subPath: hazelcast-client.yaml - name: engine-config mountPath: /opt/seatunnel/config/seatunnel.yaml subPath: seatunnel.yaml terminationGracePeriodSeconds: 120 volumes: - name: hazelcast-config configMap: name: seatunnel-hazelcast-config - name: client-config configMap: name: seatunnel-client-config - name: engine-config configMap: name: seatunnel-engine-config从启动脚本 seatunnel-cluster.sh 可以看到进程启动时会校验HAZELCAST_CONFIG指向的hazelcast.yaml存在-Dhazelcast.config参数因此 ConfigMap 挂载失败或路径错误会导致容器直接启动失败这是排障时首先检查的点。健康检查和优雅停止集群模式下建议为每个 SeaTunnel 容器添加startupProbe、就绪/存活探针和preStop钩子避免启动期误杀并减少滚动更新或节点驱逐对集群的影响。示例探针全部使用 TCP 探测 Hazelcast 端口 5801startupProbe: tcpSocket: port: 5801 periodSeconds: 10 failureThreshold: 30 readinessProbe: tcpSocket: port: 5801 initialDelaySeconds: 30 periodSeconds: 30 timeoutSeconds: 5 failureThreshold: 3 livenessProbe: tcpSocket: port: 5801 initialDelaySeconds: 30 periodSeconds: 30 timeoutSeconds: 5 failureThreshold: 3 lifecycle: preStop: exec: command: - /bin/sh - -c - | /opt/seatunnel/bin/stop-seatunnel-cluster.sh while kill -0 $(ps -ef | grep SeaTunnelServer | grep -v grep | awk {print $2}) 2/dev/null; do sleep 1 done各探针的设计意图startupProbefailureThreshold: 30×periodSeconds: 10给节点最长 300 秒的启动时间避免 JVM 与 Hazelcast 组网期间的误杀readinessProbe / livenessProbe就绪后以 30 秒周期探测failureThreshold: 3容忍 90 秒内的瞬时抖动preStop 钩子先执行 stop-seatunnel-cluster.sh 优雅关闭 SeaTunnelServer然后轮询等待进程真正退出确保节点在下线前完成 checkpoint 与状态迁移减少对集群和运行中作业的冲击。应用全部资源将上述清单分别保存为独立文件后按顺序应用。注意仅在使用 API 发现或 StatefulSet 保留serviceAccountName: seatunnel时应用seatunnel-rbac.yamlkubectl apply -f seatunnel-hazelcast-config.yaml kubectl apply -f seatunnel-client-config.yaml kubectl apply -f seatunnel-engine-config.yaml kubectl apply -f seatunnel-rbac.yaml kubectl apply -f seatunnel-services.yaml kubectl apply -f seatunnel-hybrid.yaml推荐顺序先应用三个 ConfigMap确保配置就绪其次 RBACAPI 发现模式下必须再创建 Service最后启动 StatefulSet。应用后可通过kubectl get pods -w观察 3 个 Pod 依次进入 Ready 状态。访问 REST API集群就绪后通过端口转发访问 REST API 并验证集群健康状态。SeaTunnel Engine 的 REST 端点定义在 RestConstant.java 中其中REST_URL_SYSTEM_MONITORING_INFORMATION /system-monitoring-informationkubectl port-forward svc/seatunnel 8080:8080 curl http://127.0.0.1:8080/system-monitoring-information返回的监控信息包含集群各节点的 CPU、内存、进程状态等数据可据此确认所有节点是否已成功组成一个集群3 个节点应同时出现在结果中。该端点也被引擎自带的 Web UIseatunnel-engine-ui 中getMonitors()调用用作数据源。除监控外引擎还提供/overview、/running-jobs、/job-info、/submit-job、/stop-job等 REST 端点见 RestConstant.java可用于作业管理与状态查询。使用建议与限制混合集群模式的本质是用一份资源同时支撑调度与执行因此节点数量建议控制在较小规模如 35 个避免所有节点同时参与调度与执行造成 Master 压力过大资源配额CPU/内存应同时考虑 Master 负载元数据、调度、REST与 Worker 负载任务执行、checkpoint并预留 Buffer当任务负载较高时执行负载可能影响 Master 选举、调度和 REST API 稳定性需要更稳定、可弹性扩缩的生产部署时请迁移到分离集群模式将调度与执行资源隔离Master 与 Worker 各自独立扩容、独立调优。参考资料本部署指南的英文原版docs/en/getting-started/kubernetes/hybrid-cluster-mode.md分离集群模式部署docs/zh/getting-started/kubernetes/separated-cluster-mode.mdKubernetes 日常运维指南docs/zh/getting-started/kubernetes/operations.md引擎配置选项源码ServerConfigOptions.java集群启动与停止脚本seatunnel-cluster.sh、stop-seatunnel-cluster.shHelm 部署模板参考deploy/kubernetes/seatunnel【免费下载链接】seatunnelSeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.项目地址: https://gitcode.com/GitHub_Trending/se/seatunnel创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考