模型初始化失败排查
Jul 12, 2026 00:00 · 3148 words · 7 minute read
一句话描述问题现象：某 Deployment 的三个副本，一个初始化成功，两个阻塞在模型初始化，初步怀疑加载共享文件系统中的 checkpoint 有问题。
Pod 中的业务进程阻塞，查看其日志一直卡在初始化模型：
+ ./workers/worker_main.py --production --platform 4090 --version Engine-v3.0.0-20260101 --module_name model_inference --loglevel WARN
-------------- celery@172.27.108.185:model_inference v5.5.3 (immunity)
--- ***** -----
-- ******* ---- Linux-5.15.0-144-generic-x86_64-with-glibc2.29 2026-01-16 20:11:12
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: director:0x7fe9336c2dc0
- ** ---------- .> transport: sentinel://:**@10.19.4.118:26379/5
- ** ---------- .> results: sentinel://:**@10.19.4.118:26379/6;sentinel://:**@10.19.4.120:26379/6;sentinel://:**@10.19.4.142:26379/6
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> tripo:Engine-v3.0.0-20260101:model_inference:None exchange=tripo:Engine-v3.0.0-20260101:model_inference:None(direct) key=tripo:Engine-v3.0.0-20260101:model_inference:None
/usr/local/lib/python3.8/dist-packages/celery/platforms.py:841: SecurityWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!
Please specify a different user using the --uid option.
User information: uid=0 euid=0 gid=0 egid=0
warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(
2026-01-16 20:11:13,492 - ip:172.27.108.185 - env:release - module_name:model_inference - version:Engine-v3.0.0-20260101 - task_id:no task_id - all_tasks.py[line:85] - INFO: 设置 TORCHINDUCTOR_COMPILE_THREADS=1 以限制编译线程数
2026-01-16 20:12:03,683 - ip:172.27.108.185 - env:release - module_name:model_inference - version:Engine-v3.0.0-20260101 - task_id:no task_id - worker_util.py[line:274] - INFO: initialize begin, input: ('cuda', '4090', ModuleConfig(structure_model_pt_path='/mnt/share/xxx_offical/distilled_models/ss_dmd_img_cfg3.pt', slat_model_pt_path='/mnt/share/xxx_offical/distilled_models/ss_dmd_img_cfg1_5.pt')), {}
Pod 定义：
containers:
- command:
- /bin/bash
- -ic
- terminated=0; touch /tmp/pid_list.txt; trap 'if [ $terminated -eq 0 ]; then echo "Received SIGTERM"; terminated=1; for pid in $(cat /tmp/pid_list.txt); do kill -SIGTERM $pid; done; wait; fi' SIGTERM; . /root/.bashrc && rm -rf /workspace/module-server* && mkdir -p /workspace/module-server && cd /workspace/module-server && unzip /mnt/share/server/code/module-server/v5.2.10_1a1e90ea_migration.zip && bash -x install_deps.sh; bash -x /workspace/module-server/init.sh Engine-v2.0.1_a79c368 model_inference production 4090 Engine-v3.0.0-20260101 WARN & wait
imagePullPolicy: Always
name: pytorch
readinessProbe:
exec:
command:
- /bin/sh
- -c
- ls /workspace/module-server/*.begin;return_value=$?;exit $((! return_value))
failureThreshold: 3
initialDelaySeconds: 1
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "32"
memory: 50Gi
nvidia.com/gpu: "1"
requests:
cpu: "8"
memory: 16Gi
nvidia.com/gpu: "1"
securityContext:
capabilities:
add:
- IPC_LOCK
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /mnt/share
name: sharefs
- mountPath: /dev/shm
name: cache-volume
volumes:
- hostPath:
path: /gpfs/gpfsinner/te-foo/public/
type: ""
name: sharefs
- emptyDir:
medium: Memory
name: cache-volume
先厘清业务逻辑：
- 主容器起来后解压共享文件存储中的一个 .zip 压缩包，解压后是初始化脚本 init.sh 和业务代码 *.py
- 执行初始化脚本 init.sh
- 在上述脚本中拉起业务 worker_main.py
git_tag=$1
module_name=$2
env=$3
platform=$4
version=$5
loglevel=$6
if [ $# -ge 7 ]; then
subtype="--subtypes ${@:7}"
else
subtype=""
fi
server_path="/workspace/module-server"
touch $server_path/$module_name.begin
rm -rf $server_path/init_dep.begin
work_dir=$server_path-$module_name-$version
rm -rf $work_dir
cp -r $server_path $work_dir
cd $work_dir
if [ $git_tag == "EMPTY" ]; then
echo "git_tag is EMPTY, skip unzip"
else
unzip -o "/mnt/share/server/code/${module_name}/${git_tag}.zip"
fi
unset http_proxy https_proxy
export HF_HOME="/mnt/share/pretrained_model/.cache/huggingface"
cores=$(nproc)
export OMP_NUM_THREADS=$((cores < 8 ? cores : 8))
export WORLD_SIZE=1
export CUDA_VISIBLE_DEVICES=0
export TRANSFORMERS_OFFLINE=1
export DIFFUSERS_OFFLINE=1
export HF_HUB_OFFLINE=1
check_files() {
for file in "$@"; do
if [ -e "${file}" ]; then
./${file} -h > /dev/null
if [ $? -eq 0 ]; then
echo "$file" # 返回文件名
return 0
fi
fi
done
return 1 # 如果没有找到任何文件，返回1
}
export DIRECTOR_HOME=$work_dir/tripo_workflows
export DIRECTOR_CONFIG=$work_dir/.env
# 调用函数并传递2个文件名
result=$(check_files "workers/worker_main.py" "binary_worker_main.3.11.bin" "binary_worker_main.3.10.bin")
# 检查函数的返回值
if [ $? -ne 0 ]; then
echo "not found available executable"
exit -1
fi
./${result} --${env} --platform ${platform} --version ${version} --module_name ${module_name} --loglevel ${loglevel} ${subtype} &
echo $! >> /tmp/pid_list.txt
两个模型文件作为输入：
- /mnt/share/xxx_offical/distilled_models/ss_dmd_img_cfg3.pt
- /mnt/share/xxx_offical/distilled_models/ss_dmd_img_cfg1_5.pt
排查过程
首先在 Pod 所在节点上找出容器进程树：
$ crictl inspect d5817fd3ecfeb9b33a1cb22fec79e86ed246bef6b60787c9bca296db4905686d | jq .info.pid
1040543
$ pstree -T -p 1040543
bash(1040543)---bash(1043952)---python(1043967)-+-python(1044074)
`-python(1044075)
在节点上已安装好 sysstat 工具包的前提下，通过 pidstat 观察进程的 CPU 相关使用率：
$ pidstat -u -p 1043967 5
Linux 5.15.0-144-generic (gpu-099) 09/16/25 _x86_64_ (144 CPU)
23:11:45 UID PID %usr %system %guest %wait %CPU CPU Command
23:11:50 0 1043967 0.60 0.00 0.00 0.00 0.60 118 python
23:11:55 0 1043967 0.40 0.00 0.00 0.00 0.40 118 python
23:12:00 0 1043967 0.60 0.00 0.00 0.00 0.60 116 python
23:12:05 0 1043967 0.40 0.20 0.00 0.00 0.60 66 python
23:12:10 0 1043967 0.60 0.00 0.00 0.00 0.60 66 python
23:12:15 0 1043967 0.20 0.00 0.00 0.00 0.20 66 python
23:12:20 0 1043967 0.60 0.00 0.00 0.00 0.60 66 python
23:12:25 0 1043967 0.40 0.00 0.00 0.00 0.40 66 python
23:12:30 0 1043967 0.40 0.00 0.00 0.00 0.40 66 python
不存在明显的 CPU 相关性能问题，继续观察进程磁盘 IO 相关数据：
-
1043967
$ pidstat -d -p 1043967 5 Linux 5.15.0-144-generic (gpu-099) 09/16/25 _x86_64_ (144 CPU) 23:16:07 UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command 23:16:12 0 1043967 0.00 0.00 0.00 0 python 23:16:17 0 1043967 0.00 0.00 0.00 0 python 23:16:22 0 1043967 0.00 0.00 0.00 0 python 23:16:27 0 1043967 0.00 0.00 0.00 0 python 23:16:32 0 1043967 0.00 0.00 0.00 0 python 23:16:37 0 1043967 0.00 0.00 0.00 0 python 23:16:42 0 1043967 0.00 0.00 0.00 0 python 23:16:47 0 1043967 0.00 0.00 0.00 0 python ^C Average: 0 1043967 0.00 0.00 0.00 0 python -
1044074
$ pidstat -d -p 1044074 5 Linux 5.15.0-144-generic (gpu-099) 09/16/25 _x86_64_ (144 CPU) 23:17:05 UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command 23:17:10 0 1044074 0.00 0.00 0.00 0 python 23:17:15 0 1044074 0.00 0.00 0.00 0 python 23:17:20 0 1044074 0.00 0.00 0.00 0 python 23:17:25 0 1044074 0.00 0.00 0.00 0 python 23:17:30 0 1044074 0.00 0.00 0.00 0 python 23:17:35 0 1044074 0.00 0.00 0.00 0 python 23:17:40 0 1044074 0.00 0.00 0.00 0 python 23:17:45 0 1044074 0.00 0.00 0.00 0 python ^C Average: 0 1044074 0.00 0.00 0.00 0 python -
1044075
$ pidstat -d -p 1044075 5 Linux 5.15.0-144-generic (gpu-099) 09/16/25 _x86_64_ (144 CPU) 23:17:54 UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command 23:17:59 0 1044075 0.00 0.00 0.00 0 python 23:18:04 0 1044075 0.00 0.00 0.00 0 python 23:18:09 0 1044075 0.00 0.00 0.00 0 python 23:18:14 0 1044075 0.00 0.00 0.00 0 python 23:18:19 0 1044075 0.00 0.00 0.00 0 python 23:18:24 0 1044075 0.00 0.00 0.00 0 python 23:18:29 0 1044075 0.00 0.00 0.00 0 python 23:18:34 0 1044075 0.00 0.00 0.00 0 python 23:18:39 0 1044075 0.00 0.00 0.00 0 python 23:18:44 0 1044075 0.00 0.00 0.00 0 python ^C Average: 0 1044075 0.00 0.00 0.00 0 python
说明相关进程此时无明显的磁盘 IO（文件读写）。
通过 lsof 查看相关进程打开的文件：
-
1043967
$ lsof -p 1043967 | grep "pt" python 1043967 root mem REG 0,837 6577762383 /usr/local/lib/python3.8/dist-packages/psycopg2_binary.libs/libk5crypto-b1f99d5c.so.3.1 (stat: No such file or directory) python 1043967 root mem REG 0,837 6577762381 /usr/local/lib/python3.8/dist-packages/psycopg2_binary.libs/libcrypto-ea28cefb.so.1.1 (stat: No such file or directory) python 1043967 root mem REG 0,837 4386913202 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (stat: No such file or directory) python 1043967 root mem REG 0,837 4409720635 /usr/lib/x86_64-linux-gnu/libpthread-2.31.so (stat: No such file or directory) -
1044074
lsof -p 1044074 | grep ".pt"
python 1044074 root mem REG 0,837 4409720635 /usr/lib/x86_64-linux-gnu/libpthread-2.31.so (stat: No such file or directory)
- 1044075
$ lsof -p 1044075 | grep ".pt"
python 1044075 root mem REG 0,837 6571362811 /usr/local/lib/python3.8/dist-packages/scipy/optimize/_direct.cpython-38-x86_64-linux-gnu.so (stat: No such file or directory)
python 1044075 root mem REG 0,837 6571362891 /usr/local/lib/python3.8/dist-packages/scipy/optimize/_lsap.cpython-38-x86_64-linux-gnu.so (stat: No such file or directory)
python 1044075 root mem REG 0,837 6571362805 /usr/local/lib/python3.8/dist-packages/scipy/optimize/_bglu_dense.cpython-38-x86_64-linux-gnu.so (stat: No such file or directory)
python 1044075 root mem REG 0,837 2230280033 /usr/local/lib/python3.8/dist-packages/scipy/optimize/_highs/_highs_constants.cpython-38-x86_64-linux-gnu.so (stat: No such file or directory)
python 1044075 root mem REG 0,837 2230280034 /usr/local/lib/python3.8/dist-packages/scipy/optimize/_highs/_highs_wrapper.cpython-38-x86_64-linux-gnu.so (stat: No such file or directory)
python 1044075 root mem REG 0,837 6571362802 /usr/local/lib/python3.8/dist-packages/scipy/optimize/__nnls.cpython-38-x86_64-linux-gnu.so (stat: No such file or directory)
python 1044075 root mem REG 0,837 6571362934 /usr/local/lib/python3.8/dist-packages/scipy/optimize/_zeros.cpython-38-x86_64-linux-gnu.so (stat: No such file or directory)
python 1044075 root mem REG 0,837 2230280043 /usr/local/lib/python3.8/dist-packages/scipy/optimize/_lsq/givens_elimination.cpython-38-x86_64-linux-gnu.so (stat: No such file or directory)
python 1044075 root mem REG 0,837 6571362896 /usr/local/lib/python3.8/dist-packages/scipy/optimize/_minpack.cpython-38-x86_64-linux-gnu.so (stat: No such file or directory)
python 1044075 root mem REG 0,837 6571362914 /usr/local/lib/python3.8/dist-packages/scipy/optimize/_slsqp.cpython-38-x86_64-linux-gnu.so (stat: No such file or directory)
python 1044075 root mem REG 0,837 6571362806 /usr/local/lib/python3.8/dist-packages/scipy/optimize/_cobyla.cpython-38-x86_64-linux-gnu.so (stat: No such file or directory)
python 1044075 root mem REG 0,837 6571362899 /usr/local/lib/python3.8/dist-packages/scipy/optimize/_moduleTNC.cpython-38-x86_64-linux-gnu.so (stat: No such file or directory)
python 1044075 root mem REG 0,837 6571362881 /usr/local/lib/python3.8/dist-packages/scipy/optimize/_lbfgsb.cpython-38-x86_64-linux-gnu.so (stat: No such file or directory)
python 1044075 root mem REG 0,837 2230280051 /usr/local/lib/python3.8/dist-packages/scipy/optimize/_trlib/_trlib.cpython-38-x86_64-linux-gnu.so (stat: No such file or directory)
python 1044075 root mem REG 0,837 6571362814 /usr/local/lib/python3.8/dist-packages/scipy/optimize/_group_columns.cpython-38-x86_64-linux-gnu.so (stat: No such file or directory)
python 1044075 root mem REG 0,837 6571362897 /usr/local/lib/python3.8/dist-packages/scipy/optimize/_minpack2.cpython-38-x86_64-linux-gnu.so (stat: No such file or directory)
python 1044075 root mem REG 0,837 4414569077 /usr/local/lib/python3.8/dist-packages/opencv_python_headless.libs/libcrypto-8c1ab3ad.so.1.1 (stat: No such file or directory)
python 1044075 root mem REG 0,837 4407161332 /usr/local/lib/python3.8/dist-packages/nvidia/cuda_cupti/lib/libcupti.so.12 (stat: No such file or directory)
python 1044075 root mem REG 0,837 6577762383 /usr/local/lib/python3.8/dist-packages/psycopg2_binary.libs/libk5crypto-b1f99d5c.so.3.1 (stat: No such file or directory)
python 1044075 root mem REG 0,837 6577762381 /usr/local/lib/python3.8/dist-packages/psycopg2_binary.libs/libcrypto-ea28cefb.so.1.1 (stat: No such file or directory)
python 1044075 root mem REG 0,837 4386913202 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (stat: No such file or directory)
python 1044075 root mem REG 0,837 4409720635 /usr/lib/x86_64-linux-gnu/libpthread-2.31.so (stat: No such file or directory)
均不存在两个 .pt 后缀的模型文件，此时可以彻底排除怀疑方向，即共享文件系统可能存在问题，因为业务进程还没走到这一步。
进一步排查进程阻塞原因，使用 strace 抓取系统调用：
-
1043967
$ strace -v -p 1043967 strace: Process 1043967 attached epoll_wait(11, [{events=EPOLLIN, data={u32=44, u64=44}}], 1023, 1118) = 1 recvfrom(44, "*-1\r\n", 65536, 0, NULL, NULL) = 5 sendto(44, "*5\r\n$5\r\nBRPOP\r\n$45\r\ntripo:Engine-"..., 178, 0, NULL, 0) = 178 epoll_ctl(11, EPOLL_CTL_ADD, 44, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=44, u64=44}}) = -1 EEXIST (File exists) epoll_ctl(11, EPOLL_CTL_ADD, 48, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=48, u64=48}}) = -1 EEXIST (File exists) epoll_ctl(11, EPOLL_CTL_ADD, 36, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=36, u64=36}}) = -1 EEXIST (File exists) epoll_wait(11, [], 1023, 375) = 0 wait4(589, 0x7ffc6643451c, WNOHANG, NULL) = 0 epoll_ctl(11, EPOLL_CTL_ADD, 44, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=44, u64=44}}) = -1 EEXIST (File exists) epoll_ctl(11, EPOLL_CTL_ADD, 48, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=48, u64=48}}) = -1 EEXIST (File exists) epoll_ctl(11, EPOLL_CTL_ADD, 36, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=36, u64=36}}) = -1 EEXIST (File exists) epoll_wait(11, [{events=EPOLLIN, data={u32=36, u64=36}}], 1023, 428) = 1 recvfrom(36, "*4\r\n$8\r\npmessage\r\n$20\r\n/5.celery"..., 65536, 0, NULL, NULL) = 814 ioctl(36, FIONBIO, [1]) = 0 recvfrom(36, "*4\r\n$8\r\npmessage\r\n$20\r\n/5.celery"..., 65536, 0, NULL, NULL) = 810 ioctl(36, FIONBIO, [0]) = 0 ioctl(36, FIONBIO, [1]) = 0 recvfrom(36, 0x1717f510, 65536, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) ioctl(36, FIONBIO, [0]) = 0 epoll_ctl(11, EPOLL_CTL_ADD, 44, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=44, u64=44}}) = -1 EEXIST (File exists) epoll_ctl(11, EPOLL_CTL_ADD, 48, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=48, u64=48}}) = -1 EEXIST (File exists) epoll_ctl(11, EPOLL_CTL_ADD, 36, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=36, u64=36}}) = -1 EEXIST (File exists) epoll_wait(11, [{events=EPOLLIN, data={u32=36, u64=36}}], 1023, 4) = 1 recvfrom(36, "*4\r\n$8\r\npmessage\r\n$20\r\n/5.celery"..., 65536, 0, NULL, NULL) = 866 ioctl(36, FIONBIO, [1]) = 0 recvfrom(36, 0x1717f510, 65536, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) ioctl(36, FIONBIO, [0]) = 0 epoll_ctl(11, EPOLL_CTL_ADD, 44, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=44, u64=44}}) = -1 EEXIST (File exists) epoll_ctl(11, EPOLL_CTL_ADD, 48, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=48, u64=48}}) = -1 EEXIST (File exists) epoll_ctl(11, EPOLL_CTL_ADD, 36, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=36, u64=36}}) = -1 EEXIST (File exists) epoll_wait(11, [{events=EPOLLIN, data={u32=36, u64=36}}], 1023, 4) = 1 recvfrom(36, "*4\r\n$8\r\npmessage\r\n$20\r\n/5.celery"..., 65536, 0, NULL, NULL) = 5798 ioctl(36, FIONBIO, [1]) = 0 recvfrom(36, "*4\r\n$8\r\npmessage\r\n$20\r\n/5.celery"..., 65536, 0, NULL, NULL) = 13980 ioctl(36, FIONBIO, [0]) = 0 recvfrom(36, "\"delivery_tag\": \"5522509d-ba82-4"..., 65536, 0, NULL, NULL) = 10839 ioctl(36, FIONBIO, [1]) = 0 recvfrom(36, "*4\r\n$8\r\npmessage\r\n$20\r\n/5.celery"..., 65536, 0, NULL, NULL) = 1652 ioctl(36, FIONBIO, [0]) = 0 ioctl(36, FIONBIO, [1]) = 0 recvfrom(36, 0x1717f510, 65536, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) ioctl(36, FIONBIO, [0]) = 0 openat(AT_FDCWD, "/proc/loadavg", O_RDONLY) = 58 read(58, "1.38 1.36 1.54 11/6358 21342\n", 64) = 29 close(58) = 0 getrandom("\x15\xce\x4b\x67\x37\x54\x63\x29\x20\x97\x44\xaa\x00\x68\xee\xc2", 16, 0) = 16 getpid() = 570 ioctl(42, FIONBIO, [1]) = 0 recvfrom(42, 0x1717f510, 65536, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) ioctl(42, FIONBIO, [0]) = 0 sendto(42, "*3\r\n$7\r\nPUBLISH\r\n$28\r\n/5.celerye"..., 803, 0, NULL, 0) = 803 recvfrom(42, ":75\r\n", 65536, 0, NULL, NULL) = 5 getpid() = 570 epoll_ctl(11, EPOLL_CTL_ADD, 44, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=44, u64=44}}) = -1 EEXIST (File exists) epoll_ctl(11, EPOLL_CTL_ADD, 48, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=48, u64=48}}) = -1 EEXIST (File exists) epoll_ctl(11, EPOLL_CTL_ADD, 36, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=36, u64=36}}) = -1 EEXIST (File exists) epoll_wait(11, [{events=EPOLLIN, data={u32=36, u64=36}}], 1023, 2000) = 1 recvfrom(36, "*4\r\n$8\r\npmessage\r\n$20\r\n/5.celery"..., 65536, 0, NULL, NULL) = 23221 ioctl(36, FIONBIO, [1]) = 0 recvfrom(36, "*4\r\n$8\r\npmessage\r\n$20\r\n/5.celery"..., 65536, 0, NULL, NULL) = 3335 ioctl(36, FIONBIO, [0]) = 0 ioctl(36, FIONBIO, [1]) = 0 recvfrom(36, 0x1717f510, 65536, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) ioctl(36, FIONBIO, [0]) = 0 epoll_ctl(11, EPOLL_CTL_ADD, 44, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=44, u64=44}}) = -1 EEXIST (File exists) epoll_ctl(11, EPOLL_CTL_ADD, 48, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=48, u64=48}}) = -1 EEXIST (File exists) epoll_ctl(11, EPOLL_CTL_ADD, 36, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=36, u64=36}}) = -1 EEXIST (File exists) epoll_wait(11, [{events=EPOLLIN, data={u32=44, u64=44}}], 1023, 1997) = 1 recvfrom(44, "*-1\r\n", 65536, 0, NULL, NULL) = 5 sendto(44, "*5\r\n$5\r\nBRPOP\r\n$45\r\ntripo:Engine-"..., 178, 0, NULL, 0) = 178 epoll_ctl(11, EPOLL_CTL_ADD, 44, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=44, u64=44}}) = -1 EEXIST (File exists) epoll_ctl(11, EPOLL_CTL_ADD, 48, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=48, u64=48}}) = -1 EEXIST (File exists) epoll_ctl(11, EPOLL_CTL_ADD, 36, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=36, u64=36}}) = -1 EEXIST (File exists) epoll_wait(11, [{events=EPOLLIN, data={u32=44, u64=44}}], 1023, 1799) = 1 recvfrom(44, "*-1\r\n", 65536, 0, NULL, NULL) = 5 sendto(44, "*5\r\n$5\r\nBRPOP\r\n$45\r\ntripo:Engine-"..., 178, 0, NULL, 0) = 178 epoll_ctl(11, EPOLL_CTL_ADD, 44, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=44, u64=44}}) = -1 EEXIST (File exists) epoll_ctl(11, EPOLL_CTL_ADD, 48, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=48, u64=48}}) = -1 EEXIST (File exists) epoll_ctl(11, EPOLL_CTL_ADD, 36, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=36, u64=36}}) = -1 EEXIST (File exists) epoll_wait(11, [{events=EPOLLIN, data={u32=36, u64=36}}], 1023, 793) = 1 recvfrom(36, "*4\r\n$8\r\npmessage\r\n$20\r\n/5.celery"..., 65536, 0, NULL, NULL) = 818 ioctl(36, FIONBIO, [1]) = 0 recvfrom(36, 0x1717f510, 65536, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) ioctl(36, FIONBIO, [0]) = 0 epoll_ctl(11, EPOLL_CTL_ADD, 44, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=44, u64=44}}) = -1 EEXIST (File exists) epoll_ctl(11, EPOLL_CTL_ADD, 48, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=48, u64=48}}) = -1 EEXIST (File exists) epoll_ctl(11, EPOLL_CTL_ADD, 36, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=36, u64=36}}) = -1 EEXIST (File exists) epoll_wait(11, [{events=EPOLLIN, data={u32=36, u64=36}}], 1023, 5) = 1 recvfrom(36, "*4\r\n$8\r\npmessage\r\n$20\r\n/5.celery"..., 65536, 0, NULL, NULL) = 810 ioctl(36, FIONBIO, [1]) = 0 recvfrom(36, 0x1717f510, 65536, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) ioctl(36, FIONBIO, [0]) = 0 epoll_ctl(11, EPOLL_CTL_ADD, 44, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=44, u64=44}}) = -1 EEXIST (File exists) epoll_ctl(11, EPOLL_CTL_ADD, 48, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=48, u64=48}}) = -1 EEXIST (File exists) epoll_ctl(11, EPOLL_CTL_ADD, 36, {events=EPOLLIN|EPOLLERR|EPOLLHUP, data={u32=36, u64=36}}) = -1 EEXIST (File exists) epoll_wait(11, [{events=EPOLLIN, data={u32=36, u64=36}}], 1023, 5) = 1 recvfrom(36, "*4\r\n$8\r\npmessage\r\n$20\r\n/5.celery"..., 65536, 0, NULL, NULL) = 2513 ioctl(36, FIONBIO, [1]) = 0 recvfrom(36, 0x1717f510, 65536, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)暂未阻塞，但存在一些可疑的重复行为。
-
1044074
strace -v -p 1044074 strace: Process 1044074 attached read(32,阻塞在 read syscall！
-
1044075
strace -v -p 1044075 strace: Process 1044075 attached read(46,同样阻塞在 read syscall！
下一步通过 /proc 找出进程文件描述符相关的文件：
-
1044074
$ ll /proc/1044074/fd total 0 dr-x------ 2 root root 0 Jan 16 20:11 ./ dr-xr-xr-x 9 root root 0 Jan 16 20:11 ../ lr-x------ 1 root root 64 Jan 16 20:11 0 -> /dev/null l-wx------ 1 root root 64 Jan 16 20:11 1 -> 'pipe:[414881916]' l-wx------ 1 root root 64 Jan 16 20:11 2 -> 'pipe:[414881917]' lr-x------ 1 root root 64 Jan 16 20:11 32 -> 'pipe:[414689796]'文件描述符 32 对应一个 Linux 管道（pipe）
-
1044075
$ ll /proc/1044075/fd | grep 46 lr-x------ 1 root root 64 Jan 16 20:11 13 -> pipe:[414689794] l-wx------ 1 root root 64 Jan 16 20:11 23 -> pipe:[414689795] l-wx------ 1 root root 64 Jan 16 20:11 33 -> pipe:[414689796] l-wx------ 1 root root 64 Jan 16 20:11 34 -> pipe:[414689798] lrwx------ 1 root root 64 Jan 16 20:13 46 -> socket:[414808825]文件描述符 46 对应 Linux 套接字（socket），说明正在进行网络 IO，通过 lsof 再次确认：
python 1044075 root 32u CHR 195,4 0t0 2539 /dev/nvidia4 python 1044075 root 33w FIFO 0,13 0t0 414689796 pipe python 1044075 root 34w FIFO 0,13 0t0 414689798 pipe python 1044075 root 35u a_inode 0,14 0 14758 [eventfd] python 1044075 root 36u CHR 195,4 0t0 2539 /dev/nvidia4 python 1044075 root 37u CHR 195,4 0t0 2539 /dev/nvidia4 python 1044075 root 38u CHR 195,4 0t0 2539 /dev/nvidia4 python 1044075 root 39u CHR 195,4 0t0 2539 /dev/nvidia4 python 1044075 root 40u CHR 195,4 0t0 2539 /dev/nvidia4 python 1044075 root 41u CHR 195,4 0t0 2539 /dev/nvidia4 python 1044075 root 42u CHR 195,4 0t0 2539 /dev/nvidia4 python 1044075 root 43u CHR 195,4 0t0 2539 /dev/nvidia4 python 1044075 root 44r FIFO 0,13 0t0 414881500 pipe python 1044075 root 45w FIFO 0,13 0t0 414881500 pipe python 1044075 root 46u sock 0,8 0t0 414808825 protocol: TCP
此时可以大致猜测下阻塞的原因：
- 1044074 阻塞在从管道中读取内容，因为对端（由 1044075 打开）迟迟不写入。
- 1044075 未向管道写内容，因为阻塞在网络 IO。
socket 一直打开着说明网络连接一直建立着，查看这条连接尝试找出对端：
$ ss -t -p -n | grep 1044075
ESTAB 0 0 172.27.108.185:60440 20.205.243.166:443 users:(("python",pid=1044075,fd=46))
进程正在与 20.205.243.166:443 通信，而这是 Github 的 IP 地址。
原因总结
执行管道写的业务进程在访问 Github 时因网络原因阻塞（未做 TCP IO 超时设置的恶果），进而导致另一个进程阻塞在管道读；业务并未推进到读取两个模型文件（.pt），和共享文件存储无关。