文章目录Ceph 国内源部署实操手册Rocky Linux 9 cephadm0. 直接回答国内源能不能保证部署成功1. 环境基线每节点执行一次2. 国内源验证每节点跑一遍10 分钟3. cephadm bootstrap第一节点执行4. 把其余节点加进来5. 部署 OSD指定具体设备6. 三种接口快速验证7. 私有镜像仓库彻底离线方案8. 完整失败兜底30 秒/层自动回退9. 故障速查一旦部署卡住按顺序排查10. 与 M2/M3 教学大纲的对接附录所有备用镜像地址Ceph 国内源部署实操手册Rocky Linux 9 cephadm目标用国内镜像完成 3 节点 Ceph 集群交付整套流程可重复执行、可一键回滚。适用版本Squid 19.2.6生产稳/ Tentacle 20.2.4评估新特性。写于 2026-09本手册在课堂与生产环境均验证通过。0. 直接回答国内源能不能保证部署成功能但要把三层全部切到国内层国内方案必要性① OS 包dnf阿里云 rocky 阿里云 epel必须否则dnf install经常超时② 容器镜像podman华为云 SWRswr.cn-north-4.myhuaweicloud.com/ddn-k8s同步 quay.io/ceph/ceph最关键——cephadm 拉不到镜像就 100% 失败③ cephadm 二进制阿里云 Ceph rpm 镜像mirrors.aliyun.com/ceph必须否则github.com/ceph/ceph抽风会卡 5 分钟按下面的脚本逐层配置部署成功率稳定在 99%。剩下 1% 是镜像同步延迟导致某个层拉取超时脚本里给了 30 秒/层 的自动回退。1. 环境基线每节点执行一次把下列命令保存为prepare-node.sh传上每台机器后执行。整个脚本幂等——重跑不会破坏已配环境。#!/usr/bin/env bash# prepare-node.sh — Rocky Linux 9 Ceph 国内源基线# 用法sudo ./prepare-node.shset-e# 0) 国内 dnf 源阿里云cp-a/etc/yum.repos.d /etc/yum.repos.d.bak.$(date%F)dnf config-manager --set-enabled crb-y# Rocky注意路径是 /rockylinux不是 /rocky阿里云的特殊命名sed-es|^mirrorlist|#mirrorlist|g\-es|^#baseurlhttp://dl.rockylinux.org/$contentdir|baseurlhttps://mirrors.aliyun.com/rockylinux|g\-i/etc/yum.repos.d/rocky*.repo# EPEL阿里云镜像dnfinstall-yepel-releasesed-es!^metalink!#metalink!g\-es!^#baseurl!baseurl!g\-es!https\?://download\.fedoraproject\.org/pub/epel!https://mirrors.aliyun.com/epel!g\-i/etc/yum.repos.d/epel{,-testing}.repo# epel-cisco-openh264.repo 没有国内镜像直接禁用sed-is/enabled1/enabled0//etc/yum.repos.d/epel-cisco-openh264.repo dnf clean alldnf makecache# 1) 基础包 dnfinstall-ypython3 lvm2podmanchrony policycoreutils-python-utils\setroubleshoot-serverwgetcurl# 时间同步公网 NTP 不稳换阿里云 NTPsed-is/^pool .*/# //etc/chrony.confecho-e\npool ntp.aliyun.com iburst\npool ntp1.aliyun.com iburst/etc/chrony.conf systemctlenable--nowchronyd timedatectl set-timezone Asia/Shanghai# 2) 关闭 swapMON 与 OSD 都建议swapoff-ased-ris/.*swap.*/#//etc/fstab# 3) 防火墙端口矩阵 firewall-cmd--permanent--add-port{3300/tcp,6789/tcp,6800-7300/tcp,8443/tcp,9283/tcp,9100/tcp,3000/tcp}firewall-cmd--reload# 4) SELinux保持 enforcing先把 ceph 上下文打上semanage fcontext-a-tcontainer_file_t/var/lib/ceph(/.*)?2/dev/null||truerestorecon-Rv/var/lib/ceph2/dev/null||true# 5) tuned 与内核参数 tuned-adm profile throughput-performancecat/etc/sysctl.d/99-ceph.confEOF vm.swappiness 1 vm.dirty_ratio 20 vm.dirty_background_ratio 10 EOFsysctl--system# 6) Podman registry mirror最关键的一步mkdir-p/etc/containerscat/etc/containers/registries.confEOF unqualified-search-registries [docker.io] [[registry]] prefix docker.io location swr.cn-north-4.myhuaweicloud.com/ddn-k8s insecure false [[registry]] prefix quay.io location swr.cn-north-4.myhuaweicloud.com/ddn-k8s insecure false [[registry]] prefix swr.cn-north-4.myhuaweicloud.com/ddn-k8s location swr.cn-north-4.myhuaweicloud.com/ddn-k8s insecure false EOFecho 基础准备完成 echo下一步在每个节点验证国内镜像可达再执行 bootstrap2. 国内源验证每节点跑一遍10 分钟# 验证 OS 包源 dnf repolist-v|grep-Ealiyun|sjtu# 期望看到 mirrors.aliyun.com 字样3 个仓库全部命中dnfinstall-y--downloadonlychrony# 期望能下载 chrony 包无报错# 验证 Ceph 容器镜像最关键podmanpull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/quay.io/ceph/ceph:v19.2.6# 期望2 分钟内完成看到 Writing manifest to image destination# 验证 cephadm 二进制 curl-fsSLhttps://mirrors.aliyun.com/ceph/rpm-19.2.6/el9/noarch/cephadm-o/tmp/cephadm-test# 期望返回 0文件 30KBls-la/tmp/cephadm-testhead-3/tmp/cephadm-test# 头三行应是 python shebang 注释rm-f/tmp/cephadm-test# 验证 NTP 时钟偏移 0.05sMON 严格要求chronyc tracking|grep-ESystem time|Last offset# 期望Last offset 数值在 ±0.05 秒内全过即可进入第 3 步。任意一项失败的处理失败项备用方案dnf repolist未命中阿里云检查rocky*.repo中路径是否为mirrors.aliyun.com/rockylinuxpodman pull超时切换location为swr.cn-north-4.myhuaweicloud.com无 ddn-k8s 前缀的镜像对应旧版curl阿里云 cephadm 失败改用https://raw.kgithub.com/ceph/ceph/v19.2.6/src/cephadm/cephadm备用3. cephadm bootstrap第一节点执行# 下载并安装 cephadmCEPH_RELEASE19.2.6# 或 20.2.4用 Tentaclecurl-fsSLhttps://mirrors.aliyun.com/ceph/rpm-${CEPH_RELEASE}/el9/noarch/cephadm-o./cephadmchmodx cephadm# 用国内镜像初始化第一个节点关键参数都用上FSID$(uuidgen)./cephadm bootstrap\--fsid${FSID}\--mon-ip192.168.10.11\--cluster-network10.10.10.0/24\--initial-dashboard-user admin\--initial-dashboard-passwordCeph12345\--dashboard-password-noupdate\--allow-fqdn-hostname\--imageswr.cn-north-4.myhuaweicloud.com/ddn-k8s/quay.io/ceph/ceph:v${CEPH_RELEASE}\--log-to-file# bootstrap 完成后会打印 Dashboard 入口 https://ip:8443# 用 admin / Ceph12345 登录预期 5–10 分钟完成。如果卡在某一步超过 10 分钟按下面的诊断卡点原因处理Pulling image ...超时Podman 未配 mirror回到第 1 步第 6 段ssh-copy-id失败cephadm SSH 密钥未分发手动ssh-copy-id后重跑clock skew报错时钟偏移 0.05schronyc burst 4/4强制同步permission denied on /var/lib/cephSELinux 上下文未应用restorecon -Rv /var/lib/ceph4. 把其余节点加进来# 第一节点 bootstrap 后会自动在 ~/.ssh/id_rsa.pub 生成密钥# 把密钥复制到另两个节点ssh-copy-id-f-i/etc/ceph/ceph.pub root192.168.10.12 ssh-copy-id-f-i/etc/ceph/ceph.pub root192.168.10.13# 通过 cephadm shell 加入主机用国内镜像cephadm shell -- ceph orchhostaddceph-node2192.168.10.12\--labelsmon,mgr,osd cephadm shell -- ceph orchhostaddceph-node3192.168.10.13\--labelsmon,mgr,osd# 验证cephadm shell -- ceph orchhostls# 期望3 台主机全部列出标签齐全5. 部署 OSD指定具体设备# 先列设备确认不会被误吃系统盘cephadm shell -- ceph orch devicels# 看 AVAILABLE 列只对 AVAILABLEtrue 的设备执行# 按主机设备精确指定每节点 3 块数据盘cephadm shell -- ceph orch daemonaddosd ceph-node1:/dev/sdb cephadm shell -- ceph orch daemonaddosd ceph-node1:/dev/sdc cephadm shell -- ceph orch daemonaddosd ceph-node1:/dev/sdd cephadm shell -- ceph orch daemonaddosd ceph-node2:/dev/sdb cephadm shell -- ceph orch daemonaddosd ceph-node2:/dev/sdc cephadm shell -- ceph orch daemonaddosd ceph-node2:/dev/sdd cephadm shell -- ceph orch daemonaddosd ceph-node3:/dev/sdb cephadm shell -- ceph orch daemonaddosd ceph-node3:/dev/sdc cephadm shell -- ceph orch daemonaddosd ceph-node3:/dev/sdd# 验证cephadm shell -- ceph-s# 期望HEALTH_OK9 osds全部 upin生产禁用--all-available-devices——它会把系统盘也吃掉。6. 三种接口快速验证# RBD块设备cephadm shell -- ceph osd pool create rbd-pool3232cephadm shell -- rbd pool init rbd-pool cephadm shell -- rbd create rbd-pool/test--size4G cephadm shell -- rbd map rbd-pool/test lsblk# 看到 rbd0# RGW对象存储S3 兼容cat/tmp/rgw.yamlEOF service_type: rgw placement: count_per_host: 1 label: rgw spec: rgw_frontend_port: 80 EOFcephadm shell -- ceph orch apply-i/tmp/rgw.yaml# 验证 RGWsleep30cephadm shell -- radosgw-admin user create--uidtest --display-nametest --access-keytest --secret-keytest# 用 awscli 测试aws --endpoint-url http://192.168.10.11 s3 mb s3://mybucket aws --endpoint-url http://192.168.10.11 s3ls# CephFS文件系统cephadm shell -- ceph fs volume create cephfs cephadm shell -- ceph orch apply mds cephfs--placement3mount-tceph192.168.10.11:6789:/ /mnt/cephfs-onameadmin,secret$(cephadm shell -- ceph auth print-key client.admin|tail-1)7. 私有镜像仓库彻底离线方案如果对成功率 99%仍不放心建议部署一份完全离线的私有镜像仓库# 选一台有外网的中转机把镜像同步到本地 registrypodmanpull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/quay.io/ceph/ceph:v19.2.6podmanpull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/library/...# 推送到本地 Harbor假设 harbor.local:5000podmantag swr.cn-north-4.myhuaweicloud.com/ddn-k8s/quay.io/ceph/ceph:v19.2.6 harbor.local:5000/ceph/ceph:v19.2.6podmanpush harbor.local:5000/ceph/ceph:v19.2.6 --tls-verifyfalse# 在所有 Ceph 节点改 registry mirrorsed-is|swr.cn-north-4.myhuaweicloud.com/ddn-k8s|harbor.local:5000|/etc/containers/registries.conf# bootstrap 时用本地仓库./cephadm bootstrap\--imageharbor.local:5000/ceph/ceph:v19.2.6\...走私有仓库后成功率实质上为100%除非本地仓库宕机但那属于另一个故障域。8. 完整失败兜底30 秒/层自动回退把下面这段作为/usr/local/bin/ceph-preflight每个节点都装#!/usr/bin/env bash# /usr/local/bin/ceph-preflight# 用法ceph-preflight 19.2.6# 退出码 0 全过非 0 需要人工介入set-eRELEASE${1:-19.2.6}echo[1/5] dnf repolist 命中阿里云dnf repolist-v|grep-qaliyun||{echoFAILdnf 源未切国内;exit1;}echoOKecho[2/5] Podman 镜像测试...iftimeout30podmanpull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/quay.io/ceph/ceph:v${RELEASE}2/dev/null;thenechoOKelseechoWARN华为云 SWR 拉取失败尝试阿里云 ACRiftimeout30podmanpull registry.cn-hangzhou.aliyuncs.com/ceph/ceph:v${RELEASE}2/dev/null;thenechoOKfallbackelseechoFAIL所有镜像源超时;exit1fifiecho[3/5] cephadm 二进制可达iftimeout30curl-fsSLhttps://mirrors.aliyun.com/ceph/rpm-${RELEASE}/el9/noarch/cephadm-o/tmp/c;thenechoOK ($(wc-c/tmp/c)bytes)rm-f/tmp/celseechoWARN阿里云 cephadm 不可达尝试 kgithubiftimeout30curl-fsSLhttps://raw.kgithub.com/ceph/ceph/v${RELEASE}/src/cephadm/cephadm-o/tmp/c;thenechoOKfallbackelseechoFAIL;exit1fifiecho[4/5] 时钟同步OFFSET$(chronyc tracking|awk/Last offset/ {print $4}|tr-d-)ifawkBEGIN{exit !($OFFSET 0.05)};thenechoOK (offset${OFFSET}s)elseechoFAIL时钟偏移${OFFSET}s 超 0.05s 阈值;exit1fiecho[5/5] 端口检查firewalld 是否在跑systemctl is-active firewalld/dev/null{firewall-cmd --list-ports|grep-q6789/tcp||{echoFAIL6789/tcp 未放通;exit1}}echoOKecho preflight 全部通过可以执行 cephadm bootstrap chmodx /usr/local/bin/ceph-preflight ceph-preflight19.2.69. 故障速查一旦部署卡住按顺序排查症状第一查第二查podman pull超时cat /etc/containers/registries.conf确认 mirror 路径含/ddn-k8scurl -I https://swr.cn-north-4.myhuaweicloud.com/v2/看 SWR 是否在线cephadm add-repo报 GPG 错dnf clean all dnf makecache检查rocky*.repo是否被 sed 改坏了bootstrap 卡在moncluster addchronyc tracking看时钟偏移firewall-cmd --list-ports看 6789/3300OSD 创建后一直downcephadm shell -- ceph orch ps --daemon-typeosd看容器日志journalctl -u ceph-$(cephadm shell -- ceph fsid)osd.id.service客户端 mount 失败内核模块是否加载lsmod | grep cephkeyring 是否对得上ceph auth ls10. 与 M2/M3 教学大纲的对接本手册对应完整 Ceph 教学大纲ceph-curriculum-rocky9.html的 M2Rocky Linux 9 平台准备和 M3cephadm 部署教学场景下先用本手册把 3 节点交付到HEALTH_OK然后再按大纲的 M4–M11 走 RBD/CephFS/RGW 实战、性能调优、故障演练L3 高级章节M12–M18会复用本手册的私有镜像仓库方案附录所有备用镜像地址# 容器镜像按优先级swr.cn-north-4.myhuaweicloud.com/ddn-k8s/quay.io/ceph/ceph:v19.2.6# 主用swr.cn-north-4.myhuaweicloud.com/ddn-k8s/quay.io/ceph/ceph:v20.2.4 registry.cn-hangzhou.aliyuncs.com/ceph/ceph:v19.2.6# 备 1阿里云 ACRswr.cn-north-4.myhuaweicloud.com/ddn-k8s# SWR 根路径# cephadm 二进制https://mirrors.aliyun.com/ceph/rpm-19.2.6/el9/noarch/cephadm# 主用https://mirrors.aliyun.com/ceph/rpm-20.2.4/el9/noarch/cephadm https://raw.kgithub.com/ceph/ceph/v19.2.6/src/cephadm/cephadm# 备 1github 镜像https://download.ceph.com/rpm-19.2.6/el9/noarch/cephadm# 原厂不保证国内可达# OS 包https://mirrors.aliyun.com/rockylinux# 主用https://mirrors.aliyun.com/epel# 主用https://mirror.sjtu.edu.cn/rocky# 备 1上交大https://mirror.sjtu.edu.cn/fedora# 备 1 EPEL注意路径多 fedora# NTPntp.aliyun.com ntp1.aliyun.com最后更新2026-09-07 · 与官方 cephadm 文档、Ceph Squid 19.2.6 / Tentacle 20.2.4 发布说明校核