live-0-6dcff4f549-rvm6h pod failed occasionally

[haodengeke]

HW: ICE,SKY OS: CentOS 8.3.2011,RHEL 8.3, ubuntu, Case:CIR_BMRA_RegionDC_CDN_Xeon Command：ansible-playbook -i cdn_inventory.ini playbooks/cir.yml --extra-vars "profile=basic"

Result: kubectl get pod -A* NAMESPACE NAME READY STATUS RESTARTS AGE default cdn-8668b855ff-7rqrt 1/1 Running 0 106s default kafka-5b4b8845bc-wsv5b 1/1 Running 1 106s default live-0-6dcff4f549-rvm6h 0/1 CrashLoopBackOff 4 105s default redis-54fbcb48dd-ttb98 1/1 Running 0 106s default vod-0-b7cddccdd-qlxrn 1/1 Running 0 106s default vod-0-b7cddccdd-zfrhc 1/1 Running 0 106s default zookeeper-674686784d-dz4xt 1/1 Running 0 106s kube-system coredns-7677f9bb54-h8wck 1/1 Running 2 29m kube-system coredns-7677f9bb54-qkjrw 1/1 Running 1 29m kube-system dns-autoscaler-5b7b5c9b6f-hv6hd 1/1 Running 1 29m kube-system docker-registry-docker-registry-545dd99446-5x8p4 2/2 Running 0 26m kube-system kube-apiserver-aq09-00-wc 1/1 Running 0 106s kube-system kube-controller-manager-aq09-00-wc 1/1 Running 4 31m kube-system kube-flannel-jm7hf 1/1 Running 1 30m kube-system kube-flannel-qhwmr 1/1 Running 1 30m kube-system kube-multus-ds-amd64-52pm6 1/1 Running 1 29m kube-system kube-multus-ds-amd64-hbmqf 1/1 Running 1 29m kube-system kube-proxy-qmfjp 1/1 Running 1 30m kube-system kube-proxy-sgqwt 1/1 Running 2 30m kube-system kube-scheduler-aq09-00-wc 1/1 Running 1 19m kube-system kubernetes-dashboard-86c6f9df5b-pvgxn 1/1 Running 2 29m kube-system kubernetes-metrics-scraper-678c97765c-9fm99 1/1 Running 2 29m kube-system nginx-proxy-ar09-07-cyp 1/1 Running 2 30m kube-system node-feature-discovery-controller-5bf5c49849-q4knn 1/1 Running 0 22m kube-system node-feature-discovery-worker-skh7d 1/1 Running 0 22m monitoring collectd-tn28r 3/3 Running 0 18m monitoring node-exporter-c4bw8 2/2 Running 0 21m monitoring node-exporter-fdgmx 2/2 Running 0 21m monitoring prometheus-k8s-0 5/5 Running 1 21m monitoring prometheus-operator-5bb8cb9d8f-4nkd9 2/2 Running 0 21m monitoring tas-telemetry-aware-scheduling-575ccbc9d4-p5dd4 2/2 Running 4 19m

Error log for this issue: Events: Type Reason Age From Message

---

Warning FailedScheduling 47s default-scheduler 0/2 nodes are available: 2 pod has unbound immediate PersistentVolumeClaims. Warning FailedScheduling 47s default-scheduler 0/2 nodes are available: 2 pod has unbound immediate PersistentVolumeClaims. Normal Scheduled 45s default-scheduler Successfully assigned default/live-0-6dcff4f549-rj7g2 to av09-07-wp Normal AddedInterface 44s multus Add eth0 [10.244.1.25/24] Normal Pulled 24s (x3 over 44s) kubelet Container image "tc_transcode_xeon:latest" already present on machine Normal Created 24s (x3 over 44s) kubelet Created container live-0 Normal Started 24s (x3 over 43s) kubelet Started container live-0 Warning BackOff 10s (x4 over 39s) kubelet Back-off restarting failed container

[root@av09-06-wp build]# kubectl logs live-0-6dcff4f549-rj7g2 ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04) configuration: --prefix=/usr/local --libdir=/usr/local/lib --enable-shared --disable-static --disable-doc --disable-htmlpages --disable-manpages --disable-podpages --disable-txtpages --extra-cflags=-w --enable-gpl --enable-nonfree --enable-libass --enable-libfreetype --disable-xlib --disable-sdl2 --disable-hwaccels --disable-vaapi --enable-libv4l2 --enable-indev=v4l2 --enable-libopus --enable-libvpx --enable-libvorbis --enable-libx264 --enable-libx265 --enable-libsvtav1 --enable-libsvthevc --enable-libsvtvp9 --enable-libdav1d libavutil 56. 31.100 / 56. 31.100 libavcodec 58. 54.100 / 58. 54.100 libavformat 58. 29.100 / 58. 29.100 libavdevice 58. 8.100 / 58. 8.100 libavfilter 7. 57.100 / 7. 57.100 libswscale 5. 5.100 / 5. 5.100 libswresample 3. 5.100 / 3. 5.100 libpostproc 55. 5.100 / 55. 5.100 Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/var/www/archive/3115738.mp4': Metadata: major_brand : mp42 minor_version : 0 compatible_brands: mp42mp41isomavc1 creation_time : 2019-10-22T12:48:08.000000Z Duration: 00:00:18.79, start: 0.000000, bitrate: 25358 kb/s Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 3840x2160, 25102 kb/s, 24 fps, 24 tbr, 24 tbn, 48 tbc (default) Metadata: creation_time : 2019-10-22T12:48:08.000000Z handler_name : L-SMASH Video Handler encoder : AVC Coding Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 253 kb/s (default) Metadata: creation_time : 2019-10-22T12:48:08.000000Z handler_name : L-SMASH Audio Handler rtmp://cdn-service/dash/media_0_1: Input/output error

[lvcarry]

make parameters： cmake -DSCENARIO=cdn -DPLATFORM=Xeon -DNVODS=2 -DNLIVES=1 -DREGISTRY= /usr/src/cdn-transcode

[lvcarry]

we meet this issue in high reproduce rate. Pls help debug this issue.

[mnoiy]

The cause of this bug has been found. When deploying a CDN project on multiple machines environment, command: ‘make volume’ will transfer the video to the corresponding folder on working nodes, the user on the working node must have the same user as the user who deploys the CDN on the master node. If CDN is deployed as the root user on the master node, the working node must also use root privileges to receive video files. Therefore, the permission of the video storage folder :home/username/sdp/cache/video/ pushed to the working node is belong to root user. However, docker does not have root permission, so it is impossible to read the video file from the folder, which causes an input/output error. Want to avoid permission issues: ① Use a non-root user with the same name on each machine or ②Use ‘chown -R -v [non-root user] [name of the video folder pushed on the working node]’ command to change the owner of the cache folder on the worker node to a non-root user.

But after solving this problem, We found that there is a certain probability that a new problem will occur in the deployment of CDN in a multi-machine environment ： Error writing trailer of rtmp://cdn-service/dash/media_0_1: Broken pipe

Result: root@skl-yu02:~# kubectl get pods NAME READY STATUS RESTARTS AGE cdn-8668b855ff-9chpm 1/1 Running 0 98m kafka-5b4b8845bc-bz4wf 1/1 Running 0 98m live-0-5c4885b76c-kr42h 0/1 CrashLoopBackOff 13 ( ago) 98m redis-54fbcb48dd-ssh8c 1/1 Running 0 98m xcode-0-585c7fbb-7mrvm 1/1 Running 0 98m xcode-0-585c7fbb-g8djn 1/1 Running 0 98m zookeeper-674686784d-kpnn4 1/1 Running 0 98m

error pod logs： root@skl-yu02:/home/bihuanxu/wxs/CDN-Transcode-Sample/build# kubectl logs live-0-5c4885b76c-4x7bg ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04) configuration: --prefix=/usr/local --libdir=/usr/local/lib --enable-shared --disable-static --disable-doc --disable-htmlpages --disable-manpages --disable-podpages --disable-txtpages --extra-cflags=-w --enable-gpl --enable-nonfree --enable-libass --enable-libfreetype --disable-xlib --disable-sdl2 --disable-hwaccels --disable-vaapi --enable-libv4l2 --enable-indev=v4l2 --enable-libopus --enable-libvpx --enable-libvorbis --enable-libx264 --enable-libx265 --enable-libsvtav1 --enable-libsvthevc --enable-libsvtvp9 --enable-libdav1d libavutil 56. 31.100 / 56. 31.100 libavcodec 58. 54.100 / 58. 54.100 libavformat 58. 29.100 / 58. 29.100 libavdevice 58. 8.100 / 58. 8.100 libavfilter 7. 57.100 / 7. 57.100 libswscale 5. 5.100 / 5. 5.100 libswresample 3. 5.100 / 3. 5.100 libpostproc 55. 5.100 / 55. 5.100 Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/var/www/archive/3115738.mp4': Metadata: major_brand : mp42 minor_version : 0 compatible_brands: mp42mp41isomavc1 creation_time : 2019-10-22T12:48:08.000000Z Duration: 00:00:18.79, start: 0.000000, bitrate: 25358 kb/s Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 3840x2160, 25102 kb/s, 24 fps, 24 tbr, 24 tbn, 48 tbc (default) Metadata: creation_time : 2019-10-22T12:48:08.000000Z handler_name : L-SMASH Video Handler encoder : AVC Coding Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 253 kb/s (default) Metadata: creation_time : 2019-10-22T12:48:08.000000Z handler_name : L-SMASH Audio Handler Stream mapping: Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264)) Stream #0:0 -> #1:0 (h264 (native) -> hevc (libsvt_hevc)) Press [q] to stop, [?] for help [libx264 @ 0x55869808b8c0] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 [libx264 @ 0x55869808b8c0] profile High, level 3.1, 4:2:0, 8-bit [libx264 @ 0x55869808b8c0] 264 - core 161 - H.264/MPEG-4 AVC codec - Copyleft 2003-2021 - http://www.videolan.org/x264.html - options: cabac=1 ref=2 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=2 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=12 lookahead_threads=3 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=2 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=1 keyint=100 keyint_min=10 scenecut=40 intra_refresh=0 rc_lookahead=10 rc=abr mbtree=1 bitrate=8000 ratetol=1.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00 Output #0, flv, to 'rtmp://cdn-service/hls/media_0_0': Metadata: major_brand : mp42 minor_version : 0 compatible_brands: mp42mp41isomavc1 encoder : Lavf58.29.100 Stream #0:0(und): Video: h264 (libx264) ([7][0][0][0] / 0x0007), yuv420p, 856x480, q=-1--1, 8000 kb/s, 30 fps, 1k tbn, 30 tbc (default) Metadata: creation_time : 2019-10-22T12:48:08.000000Z handler_name : L-SMASH Video Handler encoder : Lavc58.54.100 libx264 Side data: cpb: bitrate max/min/avg: 0/0/8000000 buffer size: 0 vbv_delay: -1 SVT [version]: SVT-HEVC Encoder Lib v1.5.0 SVT [build] : GCC 7.5.0 64 bit LIB Build date: Mar 23 2021 11:03:42 ------------------------------------------- Number of logical cores available: 8 Number of PPCS 165 ------------------------------------------- SVT [config]: Main Profile Tier (auto) Level (auto) SVT [config]: EncoderMode / Tune : 9 / 1 SVT [config]: EncoderBitDepth / CompressedTenBitFormat / EncoderColorFormat : 8 / 0 / 1 SVT [config]: SourceWidth / SourceHeight / InterlacedVideo : 856 / 480 / 0 SVT [config]: Fps_Numerator / Fps_Denominator / Gop Size / IntraRefreshType : 30 / 1 / 100 / 1 SVT [config]: HierarchicalLevels / BaseLayerSwitchMode / PredStructure : 3 / 0 / 2 SVT [config]: BRC Mode / QP / LookaheadDistance / SceneChange : CQP / 32 / 17 / 1 SVT [config]: BitRateReduction / ImproveSharpness : 0 / 0 SVT [config]: tileColumnCount / tileRowCount / tileSliceMode / Constraint MV : 1 / 1 / 0 / 0 SVT [config]: De-blocking Filter / SAO Filter : 1 / 1 SVT [config]: HME / UseDefaultHME : 1 / 1 SVT [config]: MV Search Area Width / Height : 16 / 7 SVT [config]: HRD / VBV MaxRate / BufSize / BufInit : 0 / 0 / 0 / 90 ------------------------------------------- Output #1, flv, to 'rtmp://cdn-service/dash/media_0_1': Metadata: major_brand : mp42 minor_version : 0 compatible_brands: mp42mp41isomavc1 encoder : Lavf58.29.100 Stream #1:0(und): Video: hevc (libsvt_hevc) ([12][0][0][0] / 0x000C), yuv420p, 856x480, q=10-48, 8000 kb/s, 30 fps, 1k tbn, 30 tbc (default) Metadata: creation_time : 2019-10-22T12:48:08.000000Z handler_name : L-SMASH Video Handler encoder : Lavc58.54.100 libsvt_hevc av_interleaved_write_frame(): Broken pipe 665kB time=00:00:00.43 bitrate=12555.5kbits/s speed=0.0764x [flv @ 0x55869808a640] Failed to update header with correct duration. [flv @ 0x55869808a640] Failed to update header with correct filesize. [flv @ 0x5586980daa00] Failed to update header with correct duration. [flv @ 0x5586980daa00] Failed to update header with correct filesize. Error writing trailer of rtmp://cdn-service/dash/media_0_1: Broken pipe frame= 40 fps=5.3 q=-1.0 Lq=-0.0 size= 1578kB time=00:00:01.53 bitrate=8426.2kbits/s speed=0.205x video:1586kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown [libx264 @ 0x55869808b8c0] frame I:1 Avg QP: 6.85 size:140374 [libx264 @ 0x55869808b8c0] frame P:14 Avg QP: 9.51 size: 65756 [libx264 @ 0x55869808b8c0] frame B:25 Avg QP:12.39 size: 22020 [libx264 @ 0x55869808b8c0] consecutive B-frames: 5.0% 5.0% 90.0% [libx264 @ 0x55869808b8c0] mb I I16..4: 7.0% 15.4% 77.6% [libx264 @ 0x55869808b8c0] mb P I16..4: 0.9% 1.6% 0.9% P16..4: 62.2% 12.1% 10.9% 0.0% 0.0% skip:11.3% [libx264 @ 0x55869808b8c0] mb B I16..4: 0.1% 0.0% 0.0% B16..8: 15.2% 5.1% 2.8% direct:39.9% skip:36.9% L0:25.8% L1:41.5% BI:32.7% [libx264 @ 0x55869808b8c0] final ratefactor: 9.39 [libx264 @ 0x55869808b8c0] 8x8 transform intra:26.0% inter:34.9% [libx264 @ 0x55869808b8c0] coded y,uvDC,uvAC intra: 92.7% 99.3% 99.0% inter: 52.1% 31.6% 22.0% [libx264 @ 0x55869808b8c0] i16 v,h,dc,p: 40% 40% 7% 13% [libx264 @ 0x55869808b8c0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 4% 60% 13% 3% 2% 1% 7% 2% 8% [libx264 @ 0x55869808b8c0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 8% 38% 13% 4% 6% 2% 17% 2% 10% [libx264 @ 0x55869808b8c0] i8c dc,h,v,p: 41% 41% 5% 13% [libx264 @ 0x55869808b8c0] Weighted P-Frames: Y:14.3% UV:0.0% [libx264 @ 0x55869808b8c0] ref P L0: 85.4% 14.6% [libx264 @ 0x55869808b8c0] ref B L0: 77.0% 23.0% [libx264 @ 0x55869808b8c0] kb/s:7735.04 Conversion failed!