通过 Kind 快速生成集群并部署 Calico BGP FullMesh 模式#!/bin/bash set -v # 1. Prepare NoCNI environment cat EOF | HTTP_PROXY HTTPS_PROXY http_proxy https_proxy kind create cluster --namecalico-bgp-fullmesh --imagekindest/node:v1.27.3 --config- kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 networking: disableDefaultCNI: true nodes: - role: control-plane - role: worker - role: worker EOF # 2. Remove taints controller_node_ipkubectl get node -o wide --no-headers | grep -E control-plane|bpf1 | awk -F {print $6} kubectl taint nodes $(kubectl get nodes -o name | grep control-plane) node-role.kubernetes.io/control-plane:NoSchedule- kubectl get nodes -o wide # 3. Collect startup message controller_node_name$(kubectl get nodes -o jsonpath{range .items[*]}{.metadata.name}{\n}{end} | grep control-plane) if [ -n $controller_node_name ]; then timeout 1 docker exec -t $controller_node_name bash -c cat EOF /root/monitor_startup.sh #!/bin/bash ip -ts monitor all /root/startup_monitor.txt 21 EOF chmod x /root/monitor_startup.sh /root/monitor_startup.sh else echo No such controller_node! fi # 4. Install CNI[Calico v3.23.2] kubectl apply -f calico.yaml # 5. Wait all pods ready kubectl wait --timeout100s --forconditionReadytrue pods --all -A## calico.yaml ## https://gitee.com/rowan-wcni/wcni-kind/blob/master/LabasCode/calico/05-calico-fullmesh/calico.yaml创建测试 Pod实际就是 Nginx仅用于后续互访时抓包apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: wluo name: wluo spec: selector: matchLabels: app: wluo template: metadata: labels: app: wluo spec: containers: - image: burlyluo/nettool:latest name: nettoolbox env: - name: NETTOOL_NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName securityContext: privileged: true查看部署结果rootnetwork-demo:~# kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE default wluo-4x7jf 1/1 Running 0 24h default wluo-7fxsn 1/1 Running 0 24h default wluo-8n4nq 1/1 Running 0 24h kube-system calico-kube-controllers-7bdccfc7d8-qglg7 1/1 Running 0 24h kube-system calico-node-gck4p 1/1 Running 0 24h kube-system calico-node-rjrqq 1/1 Running 0 24h kube-system calico-node-twjm9 1/1 Running 0 24h kube-system coredns-5d78c9869d-2nzzv 1/1 Running 0 24h kube-system coredns-5d78c9869d-brtkv 1/1 Running 0 24h kube-system etcd-calico-bgp-fullmesh-control-plane 1/1 Running 0 24h kube-system kube-apiserver-calico-bgp-fullmesh-control-plane 1/1 Running 0 24h kube-system kube-controller-manager-calico-bgp-fullmesh-control-plane 1/1 Running 0 24h kube-system kube-proxy-6v24w 1/1 Running 0 24h kube-system kube-proxy-nmnsz 1/1 Running 0 24h kube-system kube-proxy-pcldm 1/1 Running 0 24h kube-system kube-scheduler-calico-bgp-fullmesh-control-plane 1/1 Running 0 24h验证效果查询 Pod 路由、ARP、网络设备信息## 查看 Pod 信息 rootnetwork-demo:~# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE wluo-4x7jf 1/1 Running 0 29h 10.244.73.64 calico-bgp-fullmesh-worker2 wluo-7fxsn 1/1 Running 0 29h 10.244.83.128 calico-bgp-fullmesh-worker wluo-8n4nq 1/1 Running 0 29h 10.244.140.132 calico-bgp-fullmesh-control-plane1.查看 Pod 网卡信息只要 Calico 用了 BGP就会创建 tunl0 设备但用不用它取决于 IPIP 模式的配置。因为目前使用的 BGP 模式所以状态是 DOWN。rootnetwork-demo:~# kubectl exec -it wluo-8n4nq -- ip -d link show tunl0 2: tunl0NONE: NOARP mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 promiscuity 0 minmtu 0 maxmtu 0 ipip any remote any local any ttl inherit nopmtudisc numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 rootnetwork-demo:~# kubectl exec -it wluo-8n4nq -- ip address show eth0 4: eth0if10: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc noqueue state UP group default link/ether 6a:77:39:aa:1e:ce brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.244.140.132/32 scope global eth0 valid_lft forever preferred_lft forever2.查看 Pod 路由信息与 Calico IPIP 模式一样容器默认网关的 IP 169.254.1.1 是什么其实无所谓。因为通过scope link配置后这条路由被标记为本地链路路由通信走的是二层转发依赖的是 MAC 地址而非 IP 地址。宿主机 calixxxx 网口开了 proxy_arpARP 广播寻找 169.254.1.1 时 calixxxx 网口就可以充当这个地址把 Pod 流量接过来了rootnetwork-demo:~# kubectl exec -it wluo-8n4nq -- ip route show default via 169.254.1.1 dev eth0 169.254.1.1 dev eth0 scope link rootnetwork-demo:~# kubectl exec -it wluo-8n4nq -- route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 169.254.1.1 0.0.0.0 UG 0 0 0 eth0 169.254.1.1 0.0.0.0 255.255.255.255 UH 0 0 0 eth03.查看 Pod ARP 信息rootnetwork-demo:~# kubectl exec -it wluo-8n4nq -- ip neighbor show 172.18.0.2 dev eth0 lladdr ee:ee:ee:ee:ee:ee STALE 169.254.1.1 dev eth0 lladdr ee:ee:ee:ee:ee:ee STALE rootnetwork-demo:~# kubectl exec -it wluo-8n4nq -- arp -n Address HWtype HWaddress Flags Mask Iface 172.18.0.2 ether ee:ee:ee:ee:ee:ee C eth0 169.254.1.1 ether ee:ee:ee:ee:ee:ee C eth0查询 Node 节点 BGP、路由表、网络设备信息## 查看节点信息 rootnetwork-demo:~# kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP calico-bgp-fullmesh-control-plane Ready control-plane 29h v1.27.3 172.18.0.2 calico-bgp-fullmesh-worker Ready none 29h v1.27.3 172.18.0.4 calico-bgp-fullmesh-worker2 Ready none 29h v1.27.3 172.18.0.31.查询节点 BGP 信息Calicoctl 工具安装Install calicoctl | Calico Documentationbirdc 工具安装apt-get install bird2rootnetwork-demo:~# docker exec -it calico-bgp-fullmesh-control-plane calicoctl node status Calico process is running. IPv4 BGP status ----------------------------------------------------------------- | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | ----------------------------------------------------------------- | 172.18.0.3 | node-to-node mesh | up | 2026-04-08 | Established | | 172.18.0.4 | node-to-node mesh | up | 2026-04-08 | Established | ----------------------------------------------------------------- ## 查询 calico bird 启动的协议实例及其状态 rootnetwork-demo:~# docker exec -it calico-bgp-fullmesh-control-plane birdc -s /run/calico/bird.ctl show protocol BIRD v0.3.3birdv1.6.8 ready. name proto table state since info static1 Static master up 2026-04-08 kernel1 Kernel master up 2026-04-08 device1 Device master up 2026-04-08 direct1 Direct master up 2026-04-08 Mesh_172_18_0_3 BGP master up 2026-04-08 Established # BGP 邻居已建立 Mesh_172_18_0_4 BGP master up 2026-04-08 Established # BGP 邻居已建立当前节点 IP 为 172.18.0.2与另外两个节点172.18.0.3、172.18.0.4各建立了一条 BGP 邻居关系。三条 TCP 连接 两个 unix control socket这正是 Calico BGP full-mesh 模式 的典型特征集群中每个节点都与其他所有节点直接建立 BGP peer用来交换 Pod 路由信息。rootnetwork-demo:~# docker exec -it calico-bgp-fullmesh-control-plane ss -anp | grep bird tcp LISTEN 0 8 0.0.0.0:179 0.0.0.0:* users:((bird,pid2467,fd7)) tcp ESTAB 0 0 172.18.0.2:179 172.18.0.4:46881 users:((bird,pid2467,fd9)) tcp ESTAB 0 0 172.18.0.2:179 172.18.0.3:56887 users:((bird,pid2467,fd8))查询 calico bird BGP 协议注入到 BIRD 路由表里的路由条目## show route protocol Mesh_172_18_0_3 ## show route protocol Mesh_172_18_0_4 rootnetwork-demo:~# docker exec -it calico-bgp-fullmesh-control-plane birdc -s /run/calico/bird.ctl show route BIRD v0.3.3birdv1.6.8 ready. 0.0.0.0/0 via 172.18.0.1 on eth0 [kernel1 2026-04-08] * (10) 10.244.140.132/32 dev cali9c3ed6d06f3 [kernel1 2026-04-08] * (10) 10.244.140.129/32 dev caliae286712af1 [kernel1 2026-04-08] * (10) 10.244.140.128/26 blackhole [static1 2026-04-08] * (200) 10.244.140.128/32 dev cali995a1e92786 [kernel1 2026-04-08] * (10) 10.244.140.131/32 dev calibc67ab0a787 [kernel1 2026-04-08] * (10) 10.244.140.130/32 dev cali558835d1dd0 [kernel1 2026-04-08] * (10) ## BIRD 路由表中的 BGP 协议信息(Mesh_xxx) 10.244.83.128/26 via 172.18.0.4 on eth0 [Mesh_172_18_0_4 2026-04-08] * (100/0) [i] 10.244.73.64/26 via 172.18.0.3 on eth0 [Mesh_172_18_0_3 2026-04-08] * (100/0) [i] 172.18.0.0/16 dev eth0 [direct1 2026-04-08] * (240)2.查询节点路由信息Pod1 请求到 Node 后查询内核路由表发现到达 Pod210.244.73.64需要通过节点 eth0 网卡转发给172.18.0.3这里或许有个疑问转发流程中好像也没有看到 BGP 做了什么因为 BGP 的工作在前面已经做完了在内核路由表中可以看到转发给 Pod2 的路由中包含proto bird这就是 bird 进程注入的 bgp 路由。## 这里或许有个疑问上面 birdc show route 看到了很多 calico bird 路由表条目为什么内核路由表只查询到 3 条 ## 因为上面很多路由都是从内核路由表学到 bird 的 ## kernel1 协议负责 BIRD 与内核之间的双向同步它的 export 过滤器只会把非内核来源的路由写回内核 ## 具体可以与下面 ip route show 输出对应 rootnetwork-demo:~# docker exec -it calico-bgp-fullmesh-control-plane ip route show proto bird 10.244.73.64/26 via 172.18.0.3 dev eth0 10.244.83.128/26 via 172.18.0.4 dev eth0 blackhole 10.244.140.128/26 rootnetwork-demo:~# docker exec -it calico-bgp-fullmesh-control-plane ip route show default via 172.18.0.1 dev eth0 10.244.73.64/26 via 172.18.0.3 dev eth0 proto bird 10.244.83.128/26 via 172.18.0.4 dev eth0 proto bird 10.244.140.128 dev cali995a1e92786 scope link blackhole 10.244.140.128/26 proto bird 10.244.140.129 dev caliae286712af1 scope link 10.244.140.130 dev cali558835d1dd0 scope link 10.244.140.131 dev calibc67ab0a787 scope link 10.244.140.132 dev cali9c3ed6d06f3 scope link 172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.23.查询 Pod Peer 对网卡信息通过 ifX对端接口索引判断 Pod Peer## 此处查询到 Pod 网卡信息对端索引号为 10if10 rootnetwork-demo:~# kubectl exec -it wluo-8n4nq -- ip address show eth0 4: eth0if10: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc noqueue state UP group default link/ether 6a:77:39:aa:1e:ce brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.244.140.132/32 scope global eth0 valid_lft forever preferred_lft forever ## 通过网卡前面的编号本地接口索引10可以看出是一对 Pod Peer rootnetwork-demo:~# docker exec -it calico-bgp-fullmesh-control-plane ip address show | grep ^10 -A3 10: cali9c3ed6d06f3if4: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-50d666fd-bd4e-b805-3761-be2974402985Pod 网卡处抓包rootnetwork-demo:~# kubectl exec -it wluo-8n4nq -- curl -s 10.244.73.64 PodName: wluo-4x7jf | PodIP: eth0 10.244.73.64/32抓包时发现在 HTTP 请求结束后才出现 ARP 广播这是因为先前我已经请求过 10.244.73.64Pod 发起 HTTP 请求时内核发现网关 169.254.1.1 的 ARP 条目是 STALE 过期状态但并不会丢弃数据包而是先用缓存的 MAC 把 HTTP 请求发出去同时将 ARP 条目从 STALE 更新为 DELAYHTTP 完成后确认缓存 MAC 可达ARP 条目从 DELAY 更新为 REACHABLE可以通过ip neighbor del 169.254.1.1 dev eth0以及删除 Node 中本节点 Pod IP ARP 信息的方式重新抓包测试。这时 ARP 广播则会出现在 HTTP 请求前。Node 网卡出抓包可以看出这就是一个正常的包没有封装啥的。在本文查询节点路由信息处有过说明BGP 的工作在前面已经做完了在 Node 内核路由表中可以看到转发给 Pod2 的路由中包含proto bird这就是 bird 进程注入的 bgp 路由。