被 cilium BPF 劫走的 WireGuard 流量
Sep 1, 2026 00:00 · 4556 words · 10 minute read
现象
一句话描述问题现象：在公有云上走 WireGuard 访问 foo 集群内的服务不通。
网络架构图如上，有两条链路，一条通一条不通：
- 阿里云 instance1 走 service vip1 wireguard 到 foo 的 harbor 节点（harbor 不在 Kubernetes 集群中）✅ 比下面的链路多一跳，先路由至 171，再走 Kubernetes 的跨节点 overlay 到 gpu-118 业务节点
- 阿里云 instance2 走 service vip2 wireguard 到 foo 的 foo-i-master03 节点 ❌
排查
kubectl get svc -n ns-t-te-c637fngp2sne3h2q | grep if-dc6igsfwaodgnkof-service
if-dc6igsfwaodgnkof-service LoadBalancer 172.28.3.29 10.197.92.5 80/TCP,9998/TCP 53d
if-dc6igsfwaodgnkof-service-2 LoadBalancer 172.28.220.200 10.197.88.1 80/TCP,9998/TCP 2d16h
kubectl exec -it -n kube-system cilium-5759s -- cilium-dbg service list | grep "10.197.92.5"
208 10.197.92.5:80/TCP LoadBalancer 1 => 172.27.5.34:80/TCP (active)
209 10.197.92.5:9998/TCP LoadBalancer 1 => 172.27.5.34:9998/TCP (active)
kubectl exec -it -n kube-system cilium-5759s -- cilium-dbg service list | grep "10.197.88.1"
2013 10.197.88.1:80/TCP LoadBalancer 1 => 172.27.5.34:80/TCP (active)
2014 10.197.88.1:9998/TCP LoadBalancer 1 => 172.27.5.34:9998/TCP (active)
两个 Service 都是 LoadBalancer 类型，由 metalLB 实现。
- vip 10.197.92.5 -> Pod IP 172.27.5.34
- vip 10.197.88.1 -> Pod IP 172.27.5.34
两条链路的后端实际上是同一个推理服务 Pod，服务本身没问题。
在集群内的 foo-i-master03 节点上测试：
| 上通过 VIP 访问 | 通过 Cluster IP 访问 |
|---|---|
| 10.197.92.5 ✅ | 172.28.3.29 ✅ |
| 10.197.88.1 ❌ | 172.28.220.200 ✅ |
通过 Cluster IP 访问均没有问题说明 cilium 的 DNAT 和集群的 overlay 网络都正常。
找出丢包点
查看 foo-i-master03 节点上 cilium 的丢包：
kubectl exec -it -n kube-system cilium-5759s -- cilium monitor --type drop
xx drop (Policy denied) flow 0x460d64c7 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:33440 -> 39.102.60.242:80 tcp SYN
xx drop (Policy denied) flow 0x14179f0a to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:36978 -> 60.205.177.172:80 tcp SYN
xx drop (Policy denied) flow 0xee6e27df to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:39480 -> 39.102.60.242:80 tcp SYN
xx drop (Policy denied) flow 0x1416cdb0 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:35890 -> 39.102.60.242:80 tcp SYN
xx drop (Policy denied) flow 0x72e502ed to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:57910 -> 60.205.177.172:80 tcp SYN
xx drop (Policy denied) flow 0x588595ed to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:59668 -> 39.105.217.223:80 tcp SYN
xx drop (Policy denied) flow 0xd9d9842f to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:48364 -> 60.205.177.172:80 tcp SYN
xx drop (Policy denied) flow 0x9fc2f57c to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:49276 -> 39.105.217.223:80 tcp SYN
xx drop (Policy denied) flow 0x9fd8487b to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:60786 -> 60.205.177.172:80 tcp SYN
xx drop (Policy denied) flow 0x4941af40 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:53956 -> 60.205.177.172:80 tcp SYN
xx drop (Is a ClusterIP) flow 0x0 to endpoint 0, ifindex 183, file bpf_host.c:962, , identity 16777219->unknown: 10.2.10.1:57556 -> 172.28.0.1:443 tcp SYN
xx drop (Policy denied) flow 0x3d3e26ff to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:46462 -> 39.105.217.223:80 tcp SYN
xx drop (Policy denied) flow 0x30e5dc91 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:48212 -> 60.205.177.172:80 tcp SYN
xx drop (Policy denied) flow 0x9d029f69 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:36978 -> 60.205.177.172:80 tcp SYN
xx drop (Policy denied) flow 0x5726f3b5 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:49668 -> 60.205.177.172:80 tcp SYN
xx drop (Policy denied) flow 0xa03be84 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:57910 -> 60.205.177.172:80 tcp SYN
xx drop (Policy denied) flow 0x93f69c16 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:49054 -> 39.105.217.223:80 tcp SYN
xx drop (Policy denied) flow 0x72b625b4 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:46476 -> 39.105.217.223:80 tcp SYN
xx drop (Policy denied) flow 0xac64620f to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:60786 -> 60.205.177.172:80 tcp SYN
xx drop (Policy denied) flow 0x20396268 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:53956 -> 60.205.177.172:80 tcp SYN
xx drop (Is a ClusterIP) flow 0x0 to endpoint 0, ifindex 183, file bpf_host.c:962, , identity 16777219->unknown: 10.2.10.1:57556 -> 172.28.0.1:443 tcp SYN
xx drop (Policy denied) flow 0x47503867 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:54914 -> 39.102.60.242:80 tcp SYN
xx drop (Policy denied) flow 0x6ebd1d52 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:39480 -> 39.102.60.242:80 tcp SYN
xx drop (Policy denied) flow 0xa99d01bc to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:49054 -> 39.105.217.223:80 tcp SYN
xx drop (Policy denied) flow 0x53ad1da3 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:46476 -> 39.105.217.223:80 tcp SYN
xx drop (Policy denied) flow 0xa0c25956 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:52098 -> 60.205.177.172:80 tcp SYN
xx drop (Policy denied) flow 0x58ba5785 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:48226 -> 60.205.177.172:80 tcp SYN
xx drop (Policy denied) flow 0xa93632af to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:44328 -> 39.105.217.223:80 tcp SYN
xx drop (Policy denied) flow 0xf76ddc10 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:39766 -> 39.102.60.242:80 tcp SYN
xx drop (Policy denied) flow 0xcd7c44f6 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:36978 -> 60.205.177.172:80 tcp SYN
xx drop (Policy denied) flow 0x35c1fe11 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:49660 -> 39.105.217.223:80 tcp SYN
xx drop (Policy denied) flow 0x85567bcf to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:48946 -> 39.102.60.242:80 tcp SYN
xx drop (Policy denied) flow 0x3bece1bc to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:57910 -> 60.205.177.172:80 tcp SYN
xx drop (Policy denied) flow 0xff0799ab to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:55254 -> 39.102.60.242:80 tcp SYN
xx drop (Policy denied) flow 0x846450d9 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:34638 -> 39.105.217.223:80 tcp SYN
xx drop (Policy denied) flow 0xc5544dd5 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:60786 -> 60.205.177.172:80 tcp SYN
xx drop (Policy denied) flow 0xd75c3a16 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:53956 -> 60.205.177.172:80 tcp SYN
xx drop (Policy denied) flow 0xb81879b1 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:59152 -> 39.102.60.242:80 tcp SYN
xx drop (Is a ClusterIP) flow 0x0 to endpoint 0, ifindex 183, file bpf_host.c:962, , identity 16777219->unknown: 10.2.10.1:57556 -> 172.28.0.1:443 tcp SYN
xx drop (Policy denied) flow 0x52f2ceb6 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:49292 -> 39.105.217.223:80 tcp SYN
xx drop (Policy denied) flow 0x520b65e5 to endpoint 0, ifindex 69, file bpf_lxc.c:1651, , identity 50131->world: 172.27.2.127:48226 -> 60.205.177.172:80 tcp SYN
没看到 10.197.88.1 和 172.27.5.34 相关记录，表示 foo-i-master03 节点的 cilium 并未发生相关丢包。
在 foo-i-master03 节点上对所有网卡抓包（172.27.5.17 是后端 Pod IP）：
tcpdump -nne -i any 'tcp port 80 and host 172.27.5.17'
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
10:25:56.081452 cilium_vxlan Out ifindex 9 00:00:00:00:00:00 ethertype IPv4 (0x0800), length 80: 172.27.2.240.42750 > 172.27.5.17.80: Flags [S], seq 2869817211, win 64860, options [mss 1380,sackOK,TS val 2524038291 ecr 0,nop,wscale 7], length 0
10:26:02.063660 cilium_vxlan Out ifindex 9 00:00:00:00:00:00 ethertype IPv4 (0x0800), length 80: 172.27.2.240.38576 > 172.27.5.17.80: Flags [S], seq 1366030254, win 64860, options [mss 1380,sackOK,TS val 2524044273 ecr 0,nop,wscale 7], length 0
10:26:03.085393 cilium_vxlan Out ifindex 9 00:00:00:00:00:00 ethertype IPv4 (0x0800), length 80: 172.27.2.240.38576 > 172.27.5.17.80: Flags [S], seq 1366030254, win 64860, options [mss 1380,sackOK,TS val 2524045295 ecr 0,nop,wscale 7], length 0
10:26:05.101382 cilium_vxlan Out ifindex 9 00:00:00:00:00:00 ethertype IPv4 (0x0800), length 80: 172.27.2.240.38576 > 172.27.5.17.80: Flags [S], seq 1366030254, win 64860, options [mss 1380,sackOK,TS val 2524047311 ecr 0,nop,wscale 7], length 0
10:26:11.071336 cilium_vxlan Out ifindex 9 00:00:00:00:00:00 ethertype IPv4 (0x0800), length 80: 172.27.2.240.56192 > 172.27.5.17.80: Flags [S], seq 2167750376, win 64860, options [mss 1380,sackOK,TS val 2524053280 ecr 0,nop,wscale 7], length 0
10:26:12.077480 cilium_vxlan Out ifindex 9 00:00:00:00:00:00 ethertype IPv4 (0x0800), length 80: 172.27.2.240.56192 > 172.27.5.17.80: Flags [S], seq 2167750376, win 64860, options [mss 1380,sackOK,TS val 2524054287 ecr 0,nop,wscale 7], length 0
10:26:14.093427 cilium_vxlan Out ifindex 9 00:00:00:00:00:00 ethertype IPv4 (0x0800), length 80: 172.27.2.240.56192 > 172.27.5.17.80: Flags [S], seq 2167750376, win 64860, options [mss 1380,sackOK,TS val 2524056303 ecr 0,nop,wscale 7], length 0
数据包进入节点后，cilium 把 VIP 直接 DNAT 成推理服务后端 Pod IP 172.27.5.17。
这里抓包记录实际上有一个关键信息，我在排查的时候同时也 prompt AI 一起排查，但 AI 没能够及时看出这里的问题。该关键信息是可以直接缩短排查路径的。
cilium 的操作：
- 将 vip 10.197.88.1 DNAT 成 172.27.5.17（Pod IP）
- 将源 IP SNAT 成 172.27.2.240（foo-i-master03 节点的 cilium_host 设备的 IP）
- 将数据包送入 cilium_vxlan VTEP 设备，二次封包，走 VXLAN 隧道去 gpu-119 节点
SNAT 会导致在推理服务 Pod 中无法看出流量真正的来源，因为另一条链路没问题，推理服务正在 serving，无法判断出是否有数据包来自 instance2。
所以部署一个 nginx 服务来帮助测试与排查：
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
nodeName: gpu-119 # 将 Pod 固定到 gpu-119 节点上
containers:
- name: nginx
image: nginx:alpine
ports:
- name: http
containerPort: 80
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 200m
memory: 1Gi
---
apiVersion: v1
kind: Service
metadata:
name: nginx-2
annotations:
metallb.universe.tf/ip-allocated-from-pool: mizar-ip-pool2
labels:
mizar.k8s.io/user-job_kind: infini2
spec:
allocateLoadBalancerNodePorts: false
externalTrafficPolicy: Cluster
internalTrafficPolicy: Cluster
loadBalancerClass: metallb
type: LoadBalancer
selector:
app: nginx
ports:
- name: http
port: 80
targetPort: http
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: nginx-1
annotations:
metallb.universe.tf/ip-allocated-from-pool: mizar-ip-pool
labels:
mizar.k8s.io/user-job_kind: infini
spec:
allocateLoadBalancerNodePorts: false
externalTrafficPolicy: Cluster
internalTrafficPolicy: Cluster
loadBalancerClass: metallb
type: LoadBalancer
selector:
app: nginx
ports:
- name: http
port: 80
targetPort: http
protocol: TCP
kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-588c9b685f-b7544 1/1 Running 0 19h 172.27.5.17 gpu-119 <none> <none>
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 172.28.0.1 <none> 443/TCP 60d
nginx-1 LoadBalancer 172.28.55.234 10.197.92.7 80/TCP 79m
nginx-2 LoadBalancer 172.28.40.210 10.197.88.2 80/TCP 79m
和推理服务保持一致，两个 Service 两个 VIP 两条链路：
- vip 10.197.92.7 -> Pod IP 172.27.5.17 ✅
- vip 10.197.88.2 -> Pod IP 172.27.5.17 ❌
nginx Pod
只在 instance2 发包，在 nginx Pod 网络命名空间内抓包：
tcpdump -nne -i eth0 tcp and port 80
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
没有任何数据包，说明 TCP 包根本没到 Pod。
gpu-119 cilium
查看 gpu-119 节点上的 cilium 丢包：
kubectl exec -it -n kube-system cilium-5759s -- cilium-dbg monitor -vv --type drop --type trace --type policy-verdict | grep 172.27.5.17
也没看到任何与 Pod IP 172.27.5.17 相关的记录，说明 cilium BPF 中并未发生丢包。
foo-i-master03 bond0
在 foo-i-master03 节点的 bond0 上抓包，过滤 VXLAN 内封包 IP 为 172.27.5.17
tcpdump -nne -i bond0 \
'src host 192.168.71.192 and dst host 192.168.71.119 and udp dst port 8472 and udp[46:4] = 0xac1b0511'
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on bond0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
20:27:38.038003 c6:a8:07:0b:92:49 > 46:35:36:03:de:61, ethertype IPv4 (0x0800), length 124: 192.168.71.192.59598 > 192.168.71.119.8472: OTV, flags [I] (0x08), overlay 0, instance 2
00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 74: 172.27.2.240.59330 > 172.27.5.17.80: Flags [S], seq 1310639769, win 64860, options [mss 1380,sackOK,TS val 2473740246 ecr 0,nop,wscale 7], length 0
20:27:39.054431 c6:a8:07:0b:92:49 > 46:35:36:03:de:61, ethertype IPv4 (0x0800), length 124: 192.168.71.192.59598 > 192.168.71.119.8472: OTV, flags [I] (0x08), overlay 0, instance 2
00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 74: 172.27.2.240.59330 > 172.27.5.17.80: Flags [S], seq 1310639769, win 64860, options [mss 1380,sackOK,TS val 2473741263 ecr 0,nop,wscale 7], length 0
20:27:41.070507 c6:a8:07:0b:92:49 > 46:35:36:03:de:61, ethertype IPv4 (0x0800), length 124: 192.168.71.192.59598 > 192.168.71.119.8472: OTV, flags [I] (0x08), overlay 0, instance 2
00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 74: 172.27.2.240.59330 > 172.27.5.17.80: Flags [S], seq 1310639769, win 64860, options [mss 1380,sackOK,TS val 2473743279 ecr 0,nop,wscale 7], length 0
20:27:47.045591 c6:a8:07:0b:92:49 > 46:35:36:03:de:61, ethertype IPv4 (0x0800), length 124: 192.168.71.192.56675 > 192.168.71.119.8472: OTV, flags [I] (0x08), overlay 0, instance 2
00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 74: 172.27.2.240.51256 > 172.27.5.17.80: Flags [S], seq 4244890411, win 64860, options [mss 1380,sackOK,TS val 2473749254 ecr 0,nop,wscale 7], length 0
20:27:48.046385 c6:a8:07:0b:92:49 > 46:35:36:03:de:61, ethertype IPv4 (0x0800), length 124: 192.168.71.192.56675 > 192.168.71.119.8472: OTV, flags [I] (0x08), overlay 0, instance 2
00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 74: 172.27.2.240.51256 > 172.27.5.17.80: Flags [S], seq 4244890411, win 64860, options [mss 1380,sackOK,TS val 2473750255 ecr 0,nop,wscale 7], length 0
说明包走 VXLAN 成功出 foo-i-master03 机器了。
这里的抓包中关键信息又出现了一次，但 AI 还是没能够看出来。
gpu-119 bond0
在 gpu-119 bond0 上抓入包，过滤 VXLAN 内封包 IP 为 172.27.5.17
tcpdump -nne -s 0 -Q in -i bond0 'src host 192.168.71.192 and dst host 192.168.71.119 and udp dst port 8472 and udp[46:4] = 0xac1b0511'
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on bond0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
20:33:59.402199 c6:a8:07:0b:92:49 > 46:35:36:03:de:61, ethertype IPv4 (0x0800), length 124: 192.168.71.192.49067 > 192.168.71.119.8472: OTV, flags [I] (0x08), overlay 0, instance 2
00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 74: 172.27.2.240.36008 > 172.27.5.17.80: Flags [S], seq 483517714, win 64860, options [mss 1380,sackOK,TS val 2474121615 ecr 0,nop,wscale 7], length 0
20:34:05.376029 c6:a8:07:0b:92:49 > 46:35:36:03:de:61, ethertype IPv4 (0x0800), length 124: 192.168.71.192.53348 > 192.168.71.119.8472: OTV, flags [I] (0x08), overlay 0, instance 2
00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 74: 172.27.2.240.53118 > 172.27.5.17.80: Flags [S], seq 2775139000, win 64860, options [mss 1380,sackOK,TS val 2474127588 ecr 0,nop,wscale 7], length 0
20:34:06.378182 c6:a8:07:0b:92:49 > 46:35:36:03:de:61, ethertype IPv4 (0x0800), length 124: 192.168.71.192.53348 > 192.168.71.119.8472: OTV, flags [I] (0x08), overlay 0, instance 2
00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 74: 172.27.2.240.53118 > 172.27.5.17.80: Flags [S], seq 2775139000, win 64860, options [mss 1380,sackOK,TS val 2474128591 ecr 0,nop,wscale 7], length 0
20:34:08.394097 c6:a8:07:0b:92:49 > 46:35:36:03:de:61, ethertype IPv4 (0x0800), length 124: 192.168.71.192.53348 > 192.168.71.119.8472: OTV, flags [I] (0x08), overlay 0, instance 2
00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 74: 172.27.2.240.53118 > 172.27.5.17.80: Flags [S], seq 2775139000, win 64860, options [mss 1380,sackOK,TS val 2474130607 ecr 0,nop,wscale 7], length 0
有入包，说明 VXLAN 包成功达到 gpu-119 节点。
gpu-119 cilium
查看 cilium BPF 侧的丢包：
kubectl exec -it -n kube-system cilium-wbv4d -- cilium-dbg monitor -vv --type drop --type trace --type policy-verdict | grep "172.27.5.17 "
没有任何输出，说明包不在这丢的。
通过 cilium 全家桶的 pwru 工具分析丢包点：
root@gpu-119:~# pwru \
--filter-track-skb \
--output-tuple \
--output-tcp-flags \
--output-caller \
--timestamp=relative \
'tcp and dst host 172.27.5.17 and dst port 80'
2026/08/17 22:47:59 INFO Attaching kprobes via=kprobe
1475 / 1475 [--------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 370 p/s
2026/08/17 22:48:03 INFO Attached ignored=3
2026/08/17 22:48:03 INFO Listening for events..
2026/08/17 22:48:03 WARN Failed to retrieve all ifaces from all network namespaces. Some iface names might be not shown. error="open /proc/2069721/ns/net: no such file or directory"
SKB CPU PROCESS TIMESTAMP NETNS MARK/x IFACE PROTO MTU LEN TUPLE FUNC CALLER
0xff391ae9e95e7200 183 <empty>:0 0 4026531840 0 bond0:6 0x0800 0 74 192.168.71.192:42582->192.168.71.119:8472(udp) eth_type_trans vxlan_set_mac[vxlan]
0xff391ae9e95e7200 183 <empty>:0 1754 4026531840 0 cilium_vxlan:20 0x0800 0 60 192.168.71.192:42582->192.168.71.119:8472(udp) skb_release_head_state kfree_skb_reason
0xff391ae9e95e7200 183 <empty>:0 2163 4026531840 0 cilium_vxlan:20 0x0800 1500 60 192.168.71.192:42582->192.168.71.119:8472(udp) skb_free_head skb_release_data
0xff391ae9e95e7200 183 ~bin/pwru:118611 1007368312 4026531840 0 bond0:6 0x0800 0 74 192.168.71.192:42582->192.168.71.119:8472(udp) eth_type_trans vxlan_set_mac[vxlan]
0xff391ae9e95e7200 183 ~bin/pwru:118611 1736 4026531840 0 cilium_vxlan:20 0x0800 0 60 192.168.71.192:42582->192.168.71.119:8472(udp) kfree_skb_reason(SKB_DROP_REASON_NOT_SPECIFIED) vxlan_rcv[vxlan]
0xff391ae9e95e7200 183 ~bin/pwru:118611 1315 4026531840 0 cilium_vxlan:20 0x0800 0 60 192.168.71.192:42582->192.168.71.119:8472(udp) skb_release_head_state kfree_skb_reason
0xff391ae9e95e7200 183 ~bin/pwru:118611 1827 4026531840 0 cilium_vxlan:20 0x0800 1500 60 192.168.71.192:42582->192.168.71.119:8472(udp) skb_free_head skb_release_data
0xff391ae9e95e7e00 183 <empty>:0 0 4026531840 0 bond0:6 0x0800 0 74 192.168.71.192:42582->192.168.71.119:8472(udp) eth_type_trans vxlan_set_mac[vxlan]
0xff391ae9e95e7e00 183 <empty>:0 977 4026531840 0 cilium_vxlan:20 0x0800 0 60 192.168.71.192:42582->192.168.71.119:8472(udp) kfree_skb_reason(SKB_DROP_REASON_NOT_SPECIFIED) vxlan_rcv[vxlan]
0xff391ae9e95e7e00 183 <empty>:0 508 4026531840 0 cilium_vxlan:20 0x0800 0 60 192.168.71.192:42582->192.168.71.119:8472(udp) skb_release_head_state kfree_skb_reason
0xff391ae9e95e7e00 183 <empty>:0 714 4026531840 0 cilium_vxlan:20 0x0800 1500 60 192.168.71.192:42582->192.168.71.119:8472(udp) skb_release_data kfree_skb_reason
0xff391ae9e95e7e00 183 <empty>:0 471 4026531840 0 cilium_vxlan:20 0x0800 1500 60 192.168.71.192:42582->192.168.71.119:8472(udp) skb_free_head skb_release_data
0xff391ae9e95e7e00 183 <empty>:0 520 4026531840 0 cilium_vxlan:20 0x0800 1500 60 192.168.71.192:42582->192.168.71.119:8472(udp) kfree_skbmem kfree_skb_reason
0xff391b063809e800 149 <empty>:0 0 4026531840 0 bond0:6 0x0800 0 74 192.168.71.192:37207->192.168.71.119:8472(udp) eth_type_trans vxlan_set_mac[vxlan]
0xff391b063809e800 149 <empty>:0 1206 4026531840 0 cilium_vxlan:20 0x0800 0 60 192.168.71.192:37207->192.168.71.119:8472(udp) kfree_skb_reason(SKB_DROP_REASON_NOT_SPECIFIED) vxlan_rcv[vxlan]
0xff391b063809e800 149 <empty>:0 775 4026531840 0 cilium_vxlan:20 0x0800 0 60 192.168.71.192:37207->192.168.71.119:8472(udp) skb_release_head_state kfree_skb_reason
0xff391b063809e800 149 <empty>:0 635 4026531840 0 cilium_vxlan:20 0x0800 1500 60 192.168.71.192:37207->192.168.71.119:8472(udp) skb_release_data kfree_skb_reason
0xff391b063809e800 149 <empty>:0 638 4026531840 0 cilium_vxlan:20 0x0800 1500 60 192.168.71.192:37207->192.168.71.119:8472(udp) skb_free_head skb_release_data
0xff391b063809e800 149 <empty>:0 893 4026531840 0 cilium_vxlan:20 0x0800 1500 60 192.168.71.192:37207->192.168.71.119:8472(udp) kfree_skbmem kfree_skb_reason
0xff391b063809fd00 149 <empty>:0 0 4026531840 0 bond0:6 0x0800 0 74 192.168.71.192:37207->192.168.71.119:8472(udp) eth_type_trans vxlan_set_mac[vxlan]
0xff391b063809fd00 149 <empty>:0 894 4026531840 0 cilium_vxlan:20 0x0800 0 60 192.168.71.192:37207->192.168.71.119:8472(udp) kfree_skb_reason(SKB_DROP_REASON_NOT_SPECIFIED) vxlan_rcv[vxlan]
0xff391b063809fd00 149 <empty>:0 603 4026531840 0 cilium_vxlan:20 0x0800 0 60 192.168.71.192:37207->192.168.71.119:8472(udp) skb_release_head_state kfree_skb_reason
0xff391b063809fd00 149 <empty>:0 640 4026531840 0 cilium_vxlan:20 0x0800 1500 60 192.168.71.192:37207->192.168.71.119:8472(udp) skb_release_data kfree_skb_reason
0xff391b063809fd00 149 <empty>:0 565 4026531840 0 cilium_vxlan:20 0x0800 1500 60 192.168.71.192:37207->192.168.71.119:8472(udp) skb_free_head skb_release_data
0xff391b063809fd00 149 <empty>:0 718 4026531840 0 cilium_vxlan:20 0x0800 1500 60 192.168.71.192:37207->192.168.71.119:8472(udp) kfree_skbmem kfree_skb_reason
0xff391b063809fb00 149 <empty>:0 0 4026531840 0 bond0:6 0x0800 0 74 192.168.71.192:37207->192.168.71.119:8472(udp) eth_type_trans vxlan_set_mac[vxlan]
0xff391b063809fb00 149 <empty>:0 1036 4026531840 0 cilium_vxlan:20 0x0800 0 60 192.168.71.192:37207->192.168.71.119:8472(udp) kfree_skb_reason(SKB_DROP_REASON_NOT_SPECIFIED) vxlan_rcv[vxlan]
0xff391b063809fb00 149 <empty>:0 594 4026531840 0 cilium_vxlan:20 0x0800 0 60 192.168.71.192:37207->192.168.71.119:8472(udp) skb_release_head_state kfree_skb_reason
0xff391b063809fb00 149 <empty>:0 612 4026531840 0 cilium_vxlan:20 0x0800 1500 60 192.168.71.192:37207->192.168.71.119:8472(udp) skb_release_data kfree_skb_reason
0xff391b063809fb00 149 <empty>:0 475 4026531840 0 cilium_vxlan:20 0x0800 1500 60 192.168.71.192:37207->192.168.71.119:8472(udp) skb_free_head skb_release_data
0xff391b063809fb00 149 <empty>:0 558 4026531840 0 cilium_vxlan:20 0x0800 1500 60 192.168.71.192:37207->192.168.71.119:8472(udp) kfree_skbmem kfree_skb_reason
^C2026/08/17 22:48:14 INFO Received signal, exiting program..
2026/08/17 22:48:14 INFO Detaching kprobes...
1472 / 1472 [---------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 26 p/s
丢包点在 kfree_skb_reason(SKB_DROP_REASON_NOT_SPECIFIED) vxlan_rcv[vxlan]，源 IP 也能对上（来自 foo-i-master03），确实在解 VXLAN 封包时丢了，NOT_SPECIFIED 是因为当前的内核版本没有为 VXLAN 分支提供更细粒度的 reason。
此时 AI 给出的可能性为 VXLAN 内封包内的源 MAC 恰好等于本机 cilium_vxlan VTEP 设备的 MAC 地址，导致丢包，这种可能性微乎其微。
实际上，通过在 gpu-119 bond0 抓取另一条链路正常的 VXLAN 包对比：
-
正常的包
tcpdump -nne -s 0 -Q in -i bond0 'src host 192.168.71.192 and dst host 192.168.71.119 and udp dst port 8472 and udp[46:4] = 0xac1b0511' tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on bond0, link-type EN10MB (Ethernet), snapshot length 262144 bytes 23:46:11.060589 c6:a8:07:0b:92:49 > 46:35:36:03:de:61, ethertype IPv4 (0x0800), length 124: 192.168.71.192.49046 > 192.168.71.119.8472: OTV, flags [I] (0x08), overlay 0, instance 6 06:62:b2:21:c5:be > 06:62:b2:21:c5:be, ethertype IPv4 (0x0800), length 74: 172.27.2.240.43570 > 172.27.5.17.80: Flags [S], seq 99631676, win 65170, options [mss 1330,sackOK,TS val 2921688077 ecr 0,nop,wscale 7], length 0 23:46:11.060986 c6:a8:07:0b:92:49 > 46:35:36:03:de:61, ethertype IPv4 (0x0800), length 116: 192.168.71.192.49046 > 192.168.71.119.8472: OTV, flags [I] (0x08), overlay 0, instance 6 06:62:b2:21:c5:be > 06:62:b2:21:c5:be, ethertype IPv4 (0x0800), length 66: 172.27.2.240.43570 > 172.27.5.17.80: Flags [.], ack 238303052, win 510, options [nop,nop,TS val 2921688078 ecr 2925319524], length 0 23:46:11.061210 c6:a8:07:0b:92:49 > 46:35:36:03:de:61, ethertype IPv4 (0x0800), length 191: 192.168.71.192.49046 > 192.168.71.119.8472: OTV, flags [I] (0x08), overlay 0, instance 6 06:62:b2:21:c5:be > 06:62:b2:21:c5:be, ethertype IPv4 (0x0800), length 141: 172.27.2.240.43570 > 172.27.5.17.80: Flags [P.], seq 0:75, ack 1, win 510, options [nop,nop,TS val 2921688078 ecr 2925319524], length 75: HTTP: GET / HTTP/1.1 -
不正常的包
tcpdump -nne -s 0 -Q in -i bond0 'src host 192.168.71.192 and dst host 192.168.71.119 and udp dst port 8472 and udp[46:4] = 0xac1b0511' tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on bond0, link-type EN10MB (Ethernet), snapshot length 262144 bytes 23:49:00.327167 c6:a8:07:0b:92:49 > 46:35:36:03:de:61, ethertype IPv4 (0x0800), length 124: 192.168.71.192.58592 > 192.168.71.119.8472: OTV, flags [I] (0x08), overlay 0, instance 2 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 74: 172.27.2.240.46396 > 172.27.5.17.80: Flags [S], seq 1721381836, win 64860, options [mss 1380,sackOK,TS val 2485822538 ecr 0,nop,wscale 7], length 0 23:49:01.360099 c6:a8:07:0b:92:49 > 46:35:36:03:de:61, ethertype IPv4 (0x0800), length 124: 192.168.71.192.58592 > 192.168.71.119.8472: OTV, flags [I] (0x08), overlay 0, instance 2 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 74: 172.27.2.240.46396 > 172.27.5.17.80: Flags [S], seq 1721381836, win 64860, options [mss 1380,sackOK,TS val 2485823571 ecr 0,nop,wscale 7], length 0 23:49:03.372006 c6:a8:07:0b:92:49 > 46:35:36:03:de:61, ethertype IPv4 (0x0800), length 124: 192.168.71.192.58592 > 192.168.71.119.8472: OTV, flags [I] (0x08), overlay 0, instance 2 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 74: 172.27.2.240.46396 > 172.27.5.17.80: Flags [S], seq 1721381836, win 64860, options [mss 1380,sackOK,TS val 2485825583 ecr 0,nop,wscale 7], length 0
正常的链路中，VXLAN 的内封包的 mac 地址有效；但有问题的链路中，VXLAN 的内封包的 mac 地址全为 0，这是导致丢包的原因。实际上在之前抓包时就已经携带了该关键信息。
原因
因为 foo-i-master03 节点上问题数据包的入口是 wg-maas-prod2 WireGuard 的虚拟网卡（来自阿里云），这是个 L3/ARPHRD_NONE 设备，没有正常的 mac 地址，节点上的 cilium 为其 attach 了 BPF（bond0 也有），导致包被其劫走，harbor 节点不在 k8s 集群中，上面没有 cilium：
tc filter show dev wg-maas-prod2 ingress
filter protocol all pref 1 bpf chain 0
filter protocol all pref 1 bpf chain 0 handle 0x1 cil_from_netdev-wg-maas-prod2 direct-action not_in_hw id 17781 tag e064410d5c957a28 jited
ip -d link show wg-maas-prod2
183: wg-maas-prod2: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/none promiscuity 0 minmtu 0 maxmtu 2147483552
wireguard addrgenmode none numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
cat /sys/class/net/wg-maas-prod2/type
65534
# type 65534 表示 ARPHRD_NONE，不会有标准的以太网头。
有问题的链路：aliyun client -> wg-maas-prod2(foo-i-master03) -> cilium DNAT & SNAT(foo-i-master03) -> 送入 cilium_vxlan VTEP 设备时添加全零 mac 地址(foo-i-master03)
而没有问题的链路：aliyun client -> wg-maas-prod1(harbor) -> 命中路由直接转发往 foo-i-master01 节点 -> bond0 入站(foo-i-master01)
harbor 节点路由表：
ip r
10.197.92.0/22 via 192.168.71.171 dev bond0 proto static
流量在执行 Linux 路由查找（在内核中完成）前，就已经被 wg-maas-prod2 网卡上 attach 的 cilium BPF NAT 了，所以 wg-maas-prod2 节点上转发至 gpu-172 的静态路由失效。
ip r
10.197.88.0/22 via 192.168.71.172 dev bond0
解决方案
-
临时
卸掉 foo-i-master03 节点上 wg-maas-prod2 虚拟网卡上的 cilium BPF
tc filter show dev wg-maas-prod2 ingress filter protocol all pref 1 bpf chain 0 filter protocol all pref 1 bpf chain 0 handle 0x1 cil_from_netdev-wg-maas-prod2 direct-action not_in_hw id 17781 tag e064410d5c957a28 jited tc filter del dev wg-maas-prod2 ingress protocol all pref 1 handle 0x1 bpf tc filter show dev wg-maas-prod2 ingress卸完后 wg-maas-prod2 链路直接通了，流量顺利被路由至 gpu172 节点，再经过 VXLAN 隧道到达 gpu-119 节点。但以上修复在 cilium node plugin 重启后会还原。
-
持久化
修改 cilium 的配置文件，排除 wg-maas-prod2 设备；重启 foo-i-master03 节点上的 cilium node plugin。
为 foo-i-master03 创建专属的 CiliumNodeConfig 对象：
apiVersion: cilium.io/v2 kind: CiliumNodeConfig metadata: name: master03-devices namespace: kube-system spec: nodeSelector: matchLabels: kubernetes.io/hostname: foo-i-master03 defaults: devices: "bond0" direct-routing-device: "bond0"应用后只需重启该节点的 cilium。