criu
criu copied to clipboard
checkpoint-restore
IP tool failed on route restore
Description criu version:3.17
I dump the init process of a container on a host, the cmd is:
dump cmd:
./criu dump --ghost-limit=500M -D /export/sine/dumpdata -j -R -t $pid --tcp-close --skip-in-flight\
--skip-mnt /etc/hosts \
--skip-mnt /etc/hostname \
--skip-mnt /etc/resolv.conf \
--external mnt[/var/log/mydb]:mydbmount \
--external mnt[/opt/mydb/data]:mydbdata \
--skip-mnt /dev/termination-log \
--skip-mnt /etc/config_info \
--skip-mnt /etc/container_config_info \
--skip-mnt /sys/fs/cgroup/perf_event \
--skip-mnt /sys/fs/cgroup/blkio \
--skip-mnt /sys/fs/cgroup/pids \
--skip-mnt /sys/fs/cgroup/hugetlb \
--skip-mnt /sys/fs/cgroup/devices \
--skip-mnt /sys/fs/cgroup/cpuset \
--skip-mnt /sys/fs/cgroup/memory \
--skip-mnt /sys/fs/cgroup/net_cls,net_prio \
--skip-mnt /sys/fs/cgroup/rdma \
--skip-mnt /sys/fs/cgroup/freezer \
--skip-mnt /sys/fs/cgroup/cpu,cpuacct \
--skip-mnt /sys/fs/cgroup/systemd \
-v4 -o /export/sine/sinedump.log
And I restore the process in a container on another host, the restore cmd:
restore cmd:
./criu restore --manage-cgroup=ignore --ghost-limit=500M -D /export/sine/dumpdata -j -d --tcp-close \
--external mnt[mydbmount]:/var/log/mydb \
--external mnt[mydbdata]:/opt/mydb/data \
-v4 -o /export/sine/sinerestore.log
Describe the results you received: And I received a error, like this:
(00.032155) 1: Try to restore a link 10:1:lo (00.032156) 1: Restoring link lo type 1 (00.032862) 1: Running ip addr restore RTNETLINK answers: File exists RTNETLINK answers: File exists (00.034026) 1: Running ip route restore RTNETLINK answers: Network is unreachable (00.034938) 1: Error (criu/util.c:641): exited, status=255 (00.034943) 1: Error (criu/net.c:1962): IP tool failed on route restore (00.046414) 1: Error (criu/util.c:1411): Can't wait or bad status: errno=0, status=65280 (00.074408) Error (criu/cr-restore.c:2536): Restoring FAILED.
complete restore log:
(00.000047) Version: 3.17 (gitid 496bcdb)
(00.000065) Running on 2ec7552e-3e6c-413c-be2c-b6631158a67d Linux 4.18.0-193.el8.jd_017.x86_64 #1 SMP Tue Apr 20 20:12:31 PDT 2021 x86_64
(00.000070) File /run/criu.kdat does not exist
(00.000083) sockets: Probing sock diag modules
(00.000115) sockets: Done probing
(00.002669) Error (criu/util.c:641): exited, status=3
(00.002710) Pagemap is fully functional
(00.002738) Found anon-shmem device at 5
(00.002755) Hugetlb size 2 Mb is supported but cannot get dev's number
(00.002761) Hugetlb size 1024 Mb is supported but cannot get dev's number
(00.002764) Reset 274974's dirty tracking
(00.002805) ... done
(00.002824) Dirty track supported on kernel
(00.002863) Found task size of 7ffffffff000
(00.007991) Restoring netdev veth idx 10
(00.008295) Dumping netns links
(00.008337) LD: Got link 1, type 772
(00.008343) LD: Got link 10, type 1
(00.009065) vdso: Parsing at 7fffff149000 7fffff14b000
(00.009075) vdso: PT_LOAD p_vaddr: 0
(00.009077) vdso: DT_HASH: 120
(00.009078) vdso: DT_STRTAB: 298
(00.009079) vdso: DT_SYMTAB: 1a8
(00.009080) vdso: DT_STRSZ: 5e
(00.009081) vdso: DT_SYMENT: 18
(00.009083) vdso: nbucket 3 nchain a bucket 7fffff149128 chain 7fffff149134
(00.009093) vdso: rt [vdso] 7fffff149000-7fffff14b000 [vvar] 7fffff146000-7fffff149000
(00.009443) vdso: Parsing at 7ff9926a3000 7ff9926a5000
(00.009453) vdso: PT_LOAD p_vaddr: 0
(00.009454) vdso: DT_HASH: b4
(00.009455) vdso: DT_STRTAB: 1c0
(00.009457) vdso: DT_SYMTAB: 130
(00.009458) vdso: DT_STRSZ: 95
(00.009459) vdso: DT_SYMENT: 10
(00.009460) vdso: nbucket 3 nchain 9 bucket 7ff9926a30bc chain 7ff9926a30c8
(00.009463) vdso: compat [vdso] 7fffff14c000-7fffff14e000 [vvar] 7fffff149000-7fffff14c000
(00.009781) cpu: x86_family 6 x86_vendor_id GenuineIntel x86_model_id Intel(R) Xeon(R) Gold 6267C CPU @ 2.60GHz
(00.009786) cpu: fpu: xfeatures_mask 0x2f5 xsave_size 2696 xsave_size_max 2696 xsaves_size 2568
(00.009790) cpu: fpu: x87 floating point registers xstate_offsets 0 / 0 xstate_sizes 160 / 160
(00.009792) cpu: fpu: AVX registers xstate_offsets 576 / 576 xstate_sizes 256 / 256
(00.009794) cpu: fpu: MPX CSR xstate_offsets 1024 / 832 xstate_sizes 64 / 64
(00.009796) cpu: fpu: AVX-512 opmask xstate_offsets 1088 / 896 xstate_sizes 64 / 64
(00.009797) cpu: fpu: AVX-512 Hi256 xstate_offsets 1152 / 960 xstate_sizes 512 / 512
(00.009799) cpu: fpu: AVX-512 ZMM_Hi256 xstate_offsets 1664 / 1472 xstate_sizes 1024 / 1024
(00.009801) cpu: fpu: Protection Keys User registers xstate_offsets 2688 / 2496 xstate_sizes 8 / 8
(00.009895) The new mount API (fsopen, fsmount) isn't supported
(00.009901) Time namespaces are not supported.
(00.009922) Warn (criu/kerndat.c:1336): Can't get pidfd
(00.009944) Warn (criu/kerndat.c:1453): CRIU was built without libnftables support
(00.010132) No MOVE_MOUNT_SET_GROUP kernel feature
(00.019335) No openat2 syscall support
(00.019516) ptrace(PTRACE_GET_RSEQ_CONFIGURATION) is not supported
(00.019630) Adjust mmap_min_addr 0x1000 -> 0x10000
(00.019633) Found mmap_min_addr 0x10000
(00.019639) files stat: fs/nr_open 6558576
(00.019669) Warn (criu/kerndat.c:1117): Can't keep kdat cache on non-tempfs
(00.019685) Will drop all TCP connections on restore
(00.019687) mnt-v2: Mounts-v2 requires MOVE_MOUNT_SET_GROUP support
(00.019688) Mount engine fallback to --mntns-compat-mode mode
(00.019693) rlimit: RLIMIT_NOFILE unlimited for self
(00.019736) cpu: fpu:1 fxsr:1 xsave:1 xsaveopt:1 xsavec:1 xgetbv1:1 xsaves:1
(00.019750) kernel pid_max=655300
(00.019751) Reading image tree
(00.019773) Add mnt ns 13 pid 1
(00.019775) Add net ns 10 pid 1
(00.019777) Add pid ns 9 pid 1
(00.019785) pstree pid_max=14
(00.019791) Will restore in 6c020000 namespaces
(00.019792) NS mask to use 6c020000
(00.019805) Collecting 51/56 (flags 3)
(00.019808) No memfd.img image
(00.019810) `- ... done
(00.019812) Collecting 40/54 (flags 2)
(00.019824) Collected [opt/jimdb/redis-server] ID 0x1
(00.019834) Collected [usr/lib64/libfreebl3.so] ID 0x2
(00.019837) Collected [usr/lib64/liblzma.so.5.0.99] ID 0x3
(00.019839) Collected [usr/lib64/libpcre.so.1.2.0] ID 0x4
(00.019843) Collected [usr/lib64/libcrypt-2.17.so] ID 0x5
(00.019846) Collected [usr/lib64/libselinux.so.1] ID 0x6
(00.019848) Collected [usr/lib64/libsasl2.so.3.0.0] ID 0x7
(00.019850) Collected [usr/lib64/libresolv-2.17.so] ID 0x8
(00.019853) Collected [usr/lib64/libkeyutils.so.1.5] ID 0x9
(00.019855) Collected [usr/lib64/libkrb5support.so.0.1] ID 0xa
(00.019857) Collected [usr/lib64/librt-2.17.so] ID 0xb
(00.019860) Collected [usr/lib64/libcrypto.so.1.0.1e] ID 0xc
(00.019862) Collected [usr/lib64/libssl.so.1.0.1e] ID 0xd
(00.019868) Collected [usr/lib64/libz.so.1.2.7] ID 0xe
(00.019870) Collected [usr/lib64/libldap-2.4.so.2.10.2] ID 0xf
(00.019873) Collected [usr/lib64/liblber-2.4.so.2.10.2] ID 0x10
(00.019875) Collected [usr/lib64/libcom_err.so.2.1] ID 0x11
(00.019877) Collected [usr/lib64/libk5crypto.so.3.1] ID 0x12
(00.019880) Collected [usr/lib64/libkrb5.so.3.3] ID 0x13
(00.019882) Collected [usr/lib64/libgssapi_krb5.so.2.2] ID 0x14
(00.019884) Collected [usr/lib64/libnspr4.so] ID 0x15
(00.019888) Collected [usr/lib64/libplc4.so] ID 0x16
(00.019891) Collected [usr/lib64/libplds4.so] ID 0x17
(00.019893) Collected [usr/lib64/libnssutil3.so] ID 0x18
(00.019895) Collected [usr/lib64/libnss3.so] ID 0x19
(00.019897) Collected [usr/lib64/libsmime3.so] ID 0x1a
(00.019900) Collected [usr/lib64/libssl3.so] ID 0x1b
(00.019902) Collected [usr/lib64/libssh2.so.1.0.1] ID 0x1c
(00.019904) Collected [usr/lib64/libidn.so.11.6.11] ID 0x1d
(00.019907) Collected [usr/lib64/libc-2.17.so] ID 0x1e
(00.019913) Collected [usr/lib64/libpthread-2.17.so] ID 0x1f
(00.019915) Collected [usr/lib64/libdl-2.17.so] ID 0x20
(00.019918) Collected [usr/lib64/libcurl.so.4.3.0] ID 0x21
(00.019920) Collected [usr/lib64/libm-2.17.so] ID 0x22
(00.019922) Collected [usr/lib64/ld-2.17.so] ID 0x23
(00.019925) Collected pipe entry ID 0x24 PIPE ID 0x7e14f6f
(00.019931) Collected [var/log/jimdb/6379.log] ID 0x25
(00.019938) epoll: Collected eventpoll: id 0x000026 flags 0x02
(00.019945) epoll: Collected eventpoll: id 0x000027 flags 0x02
(00.019965) eventfd: Collected : id 0x00002c flags 0x02 counter 0000000000000000
(00.019990) Collected [opt/jimdb/data] ID 0x37
(00.019992) Collected [.] ID 0x38
(00.019995) Collected [usr/sbin/sshd] ID 0x39
(00.019997) Collected [usr/lib64/libnss_files-2.17.so] ID 0x3a
(00.020001) Collected [usr/lib64/libnsl-2.17.so] ID 0x3b
(00.020003) Collected [usr/lib64/libutil-2.17.so] ID 0x3c
(00.020006) Collected [usr/lib64/libpam.so.0.83.1] ID 0x3d
(00.020008) Collected [usr/lib64/libaudit.so.1.0.0] ID 0x3e
(00.020010) Collected [usr/lib64/libwrap.so.0.7.6] ID 0x3f
(00.020013) Collected [usr/lib64/libfipscheck.so.1.2.1] ID 0x40
(00.020015) Collected [dev/null] ID 0x41
(00.020023) Collected [.] ID 0x44
(00.020025) Collected [.] ID 0x45
(00.020028) `- ... done
(00.020029) Collecting 46/68 (flags 0)
(00.020032) No remap-fpath.img image
(00.020033) `- ... done
(00.020048) No apparmor.img image
(00.020126) Running pre-restore scripts
(00.020155) Saved netns fd for links restore
(00.020194) mnt: Reading mountpoint images (id 13 pid 1)
(00.020201) mnt: Will mount 2921 from /
(00.020208) mnt: Will mount 2921 @ /tmp/.criu.mntns.s01VrQ/13-0000000000/run/secrets/kubernetes.io/serviceaccount /run/secrets/kubernetes.io/serviceaccount
(00.020210) mnt: Read 2921 mp @ /run/secrets/kubernetes.io/serviceaccount
(00.020214) mnt: Will mount 2920 from /var/log/jimdb (E)
(00.020216) mnt: Will mount 2920 @ /tmp/.criu.mntns.s01VrQ/13-0000000000/var/log/jimdb /var/log/jimdb
(00.020218) mnt: Read 2920 mp @ /var/log/jimdb
(00.020220) mnt: Will mount 2919 from /opt/jimdb/data (E)
(00.020222) mnt: Will mount 2919 @ /tmp/.criu.mntns.s01VrQ/13-0000000000/opt/jimdb/data /opt/jimdb/data
(00.020224) mnt: Read 2919 mp @ /opt/jimdb/data
(00.020226) mnt: Will mount 2915 from /
(00.020228) mnt: Will mount 2915 @ /tmp/.criu.mntns.s01VrQ/13-0000000000/dev/shm /dev/shm
(00.020232) mnt: Read 2915 mp @ /dev/shm
(00.020235) mnt: Will mount 2911 from /
(00.020236) mnt: Will mount 2911 @ /tmp/.criu.mntns.s01VrQ/13-0000000000/dev/mqueue /dev/mqueue
(00.020238) mnt: Read 2911 mp @ /dev/mqueue
(00.020251) mnt: Will mount 2864 from /
(00.020255) mnt: Will mount 2864 @ /tmp/.criu.mntns.s01VrQ/13-0000000000/sys/fs/cgroup /sys/fs/cgroup
(00.020256) mnt: Read 2864 mp @ /sys/fs/cgroup
(00.020259) mnt: Will mount 2863 from /
(00.020260) mnt: Will mount 2863 @ /tmp/.criu.mntns.s01VrQ/13-0000000000/sys /sys
(00.020262) mnt: Read 2863 mp @ /sys
(00.020264) mnt: Will mount 2862 from /
(00.020266) mnt: Will mount 2862 @ /tmp/.criu.mntns.s01VrQ/13-0000000000/dev/pts /dev/pts
(00.020267) mnt: Read 2862 mp @ /dev/pts
(00.020270) mnt: Will mount 2861 from /
(00.020271) mnt: Will mount 2861 @ /tmp/.criu.mntns.s01VrQ/13-0000000000/dev /dev
(00.020273) mnt: Read 2861 mp @ /dev
(00.020276) mnt: Will mount 2860 from /
(00.020281) mnt: Will mount 2860 @ /tmp/.criu.mntns.s01VrQ/13-0000000000/proc /proc
(00.020283) mnt: Read 2860 mp @ /proc
(00.020285) mnt: Will mount 2859 from /rootfs
(00.020287) mnt: Will mount 2859 @ /tmp/.criu.mntns.s01VrQ/13-0000000000/ /
(00.020289) mnt: Read 2859 mp @ /
(00.020291) mnt: Building mountpoints tree
(00.020293) mnt: Building plain mount tree
(00.020294) mnt: Working on 2859->1360
(00.020295) mnt: Working on 2860->2859
(00.020296) mnt: Working on 2861->2859
(00.020297) mnt: Working on 2862->2861
(00.020298) mnt: Working on 2863->2859
(00.020300) mnt: Working on 2864->2863
(00.020301) mnt: Working on 2911->2861
(00.020302) mnt: Working on 2915->2861
(00.020303) mnt: Working on 2919->2859
(00.020304) mnt: Working on 2920->2859
(00.020305) mnt: Working on 2921->2859
(00.020306) mnt: Resorting children of 2859 in mount order
(00.020308) mnt: Resorting children of 2921 in mount order
(00.020309) mnt: Resorting children of 2919 in mount order
(00.020310) mnt: Resorting children of 2920 in mount order
(00.020311) mnt: Resorting children of 2860 in mount order
(00.020313) mnt: Resorting children of 2861 in mount order
(00.020314) mnt: Resorting children of 2862 in mount order
(00.020315) mnt: Resorting children of 2911 in mount order
(00.020316) mnt: Resorting children of 2915 in mount order
(00.020317) mnt: Resorting children of 2863 in mount order
(00.020318) mnt: Resorting children of 2864 in mount order
(00.020319) mnt: Done:
(00.020320) mnt: [/](2859->1360)
(00.020322) mnt: [/run/secrets/kubernetes.io/serviceaccount](2921->2859)
(00.020324) mnt: <--
(00.020325) mnt: [/opt/jimdb/data](2919->2859)
(00.020326) mnt: <--
(00.020327) mnt: [/var/log/jimdb](2920->2859)
(00.020329) mnt: <--
(00.020330) mnt: [/proc](2860->2859)
(00.020331) mnt: <--
(00.020332) mnt: [/dev](2861->2859)
(00.020333) mnt: [/dev/pts](2862->2861)
(00.020335) mnt: <--
(00.020336) mnt: [/dev/mqueue](2911->2861)
(00.020337) mnt: <--
(00.020338) mnt: [/dev/shm](2915->2861)
(00.020339) mnt: <--
(00.020340) mnt: <--
(00.020342) mnt: [/sys](2863->2859)
(00.020343) mnt: [/sys/fs/cgroup](2864->2863)
(00.020360) mnt: <--
(00.020361) mnt: <--
(00.020362) mnt: <--
(00.020363) mnt: Start with 2859:/
(00.020367) mnt: Mountpoint 2859 (@/) moved to the root yard
(00.020403) No pidns-9.img image
(00.020459) Forking task with 1 pid (flags 0x6c020000)
(00.020880) PID: real 274986 virt 1
(00.020923) Wait until namespaces are created
(00.021821) Running setup-namespaces scripts
(00.021843) 1: Calling restore_sid() for init
(00.021855) 1: Restoring 1 to 1 sid
(00.022164) 1: Mount procfs in /tmp/crtools-proc.nU6jPO
(00.029275) 1: Collecting 44/37 (flags 2)
(00.029289) 1: No tty-info.img image
(00.029293) 1: `- ... done
(00.029294) 1: Collecting 45/51 (flags 0)
(00.029297) 1: No tty-data.img image
(00.029301) 1: `- ... done
(00.029303) 1: Restoring namespaces 1 flags 0x6c020000
(00.029578) 1: kernel/hostname nr 36
(00.029624) 1: kernel/domainname nr 6
(00.029888) 1: Restoring IPC namespace
(00.029898) 1: Restoring IPC variables
(00.030446) 1: Restoring IPC shared memory
(00.030455) 1: No ipcns-shm-11.img image
(00.030457) 1: Restoring IPC message queues
(00.030460) 1: No ipcns-msg-11.img image
(00.030461) 1: Restoring IPC semaphores sets
(00.030464) 1: No ipcns-sem-11.img image
(00.031747) 1: Try to restore a link 10:3:eth0
(00.031756) 1: Restoring link eth0 type 2
(00.031767) 1: Restoring netdev eth0 idx 3
(00.031773) 1: Restore ll addr (12:../6) for device
(00.031775) 1: Error (criu/net.c:1462): Unknown peer net namespace
(00.032050) 1: Skip eth0/accept_local, coincides with default
(00.032053) 1: Skip eth0/accept_redirects, coincides with default
(00.032054) 1: Skip eth0/accept_source_route, coincides with default
(00.032056) 1: Skip eth0/arp_accept, coincides with default
(00.032057) 1: Skip eth0/arp_announce, coincides with default
(00.032058) 1: Skip eth0/arp_filter, coincides with default
(00.032059) 1: Skip eth0/arp_ignore, coincides with default
(00.032061) 1: Skip eth0/arp_notify, coincides with default
(00.032062) 1: Skip eth0/bootp_relay, coincides with default
(00.032063) 1: Skip eth0/disable_policy, coincides with default
(00.032064) 1: Skip eth0/disable_xfrm, coincides with default
(00.032065) 1: Skip eth0/force_igmp_version, coincides with default
(00.032067) 1: Skip eth0/forwarding, coincides with default
(00.032068) 1: Skip eth0/igmpv2_unsolicited_report_interval, coincides with default
(00.032069) 1: Skip eth0/igmpv3_unsolicited_report_interval, coincides with default
(00.032070) 1: Skip eth0/log_martians, coincides with default
(00.032072) 1: Skip eth0/medium_id, coincides with default
(00.032073) 1: Skip eth0/promote_secondaries, coincides with default
(00.032074) 1: Skip eth0/proxy_arp, coincides with default
(00.032075) 1: Skip eth0/proxy_arp_pvlan, coincides with default
(00.032076) 1: Skip eth0/route_localnet, coincides with default
(00.032078) 1: Skip eth0/rp_filter, coincides with default
(00.032079) 1: Skip eth0/secure_redirects, coincides with default
(00.032080) 1: Skip eth0/send_redirects, coincides with default
(00.032081) 1: Skip eth0/shared_media, coincides with default
(00.032082) 1: Skip eth0/src_valid_mark, coincides with default
(00.032084) 1: Skip eth0/tag, coincides with default
(00.032085) 1: Skip eth0/ignore_routes_with_linkdown, coincides with default
(00.032086) 1: Skip eth0/drop_gratuitous_arp, coincides with default
(00.032087) 1: Skip eth0/drop_unicast_in_l2_multicast, coincides with default
(00.032089) 1: Skip eth0/accept_dad, coincides with default
(00.032090) 1: Skip eth0/accept_ra, coincides with default
(00.032092) 1: Skip eth0/accept_ra_defrtr, coincides with default
(00.032093) 1: Skip eth0/accept_ra_from_local, coincides with default
(00.032094) 1: Skip eth0/accept_ra_min_hop_limit, coincides with default
(00.032095) 1: Skip eth0/accept_ra_mtu, coincides with default
(00.032096) 1: Skip eth0/accept_ra_pinfo, coincides with default
(00.032098) 1: Skip eth0/accept_ra_rt_info_max_plen, coincides with default
(00.032099) 1: Skip eth0/accept_ra_rtr_pref, coincides with default
(00.032100) 1: Skip eth0/accept_redirects, coincides with default
(00.032101) 1: Skip eth0/accept_source_route, coincides with default
(00.032103) 1: Skip eth0/autoconf, coincides with default
(00.032104) 1: Skip eth0/dad_transmits, coincides with default
(00.032105) 1: Skip eth0/disable_ipv6, coincides with default
(00.032106) 1: Skip eth0/drop_unicast_in_l2_multicast, coincides with default
(00.032107) 1: Skip eth0/drop_unsolicited_na, coincides with default
(00.032109) 1: Skip eth0/force_mld_version, coincides with default
(00.032110) 1: Skip eth0/force_tllao, coincides with default
(00.032111) 1: Skip eth0/forwarding, coincides with default
(00.032112) 1: Skip eth0/hop_limit, coincides with default
(00.032118) 1: Skip eth0/ignore_routes_with_linkdown, coincides with default
(00.032120) 1: Skip eth0/keep_addr_on_down, coincides with default
(00.032121) 1: Skip eth0/max_addresses, coincides with default
(00.032122) 1: Skip eth0/max_desync_factor, coincides with default
(00.032123) 1: Skip eth0/mldv1_unsolicited_report_interval, coincides with default
(00.032125) 1: Skip eth0/mldv2_unsolicited_report_interval, coincides with default
(00.032126) 1: Skip eth0/ndisc_notify, coincides with default
(00.032128) 1: Skip eth0/optimistic_dad, coincides with default
(00.032129) 1: Skip eth0/proxy_ndp, coincides with default
(00.032130) 1: Skip eth0/regen_max_retry, coincides with default
(00.032131) 1: Skip eth0/router_probe_interval, coincides with default
(00.032133) 1: Skip eth0/router_solicitation_delay, coincides with default
(00.032134) 1: Skip eth0/router_solicitation_interval, coincides with default
(00.032135) 1: Skip eth0/router_solicitations, coincides with default
(00.032137) 1: Skip eth0/suppress_frag_ndisc, coincides with default
(00.032138) 1: Skip eth0/temp_prefered_lft, coincides with default
(00.032139) 1: Skip eth0/temp_valid_lft, coincides with default
(00.032140) 1: Skip eth0/use_oif_addrs_only, coincides with default
(00.032142) 1: Skip eth0/use_optimistic, coincides with default
(00.032143) 1: Skip eth0/use_tempaddr, coincides with default
(00.032155) 1: Try to restore a link 10:1:lo
(00.032156) 1: Restoring link lo type 1
(00.032862) 1: Running ip addr restore
RTNETLINK answers: File exists
RTNETLINK answers: File exists
(00.034026) 1: Running ip route restore
RTNETLINK answers: Network is unreachable
(00.034938) 1: Error (criu/util.c:641): exited, status=255
(00.034943) 1: Error (criu/net.c:1962): IP tool failed on route restore
(00.046414) 1: Error (criu/util.c:1411): Can't wait or bad status: errno=0, status=65280
(00.074408) Error (criu/cr-restore.c:2536): Restoring FAILED.
the dump host info: kernel:4.18.0-80.11.2.el8_0.x86_64 os:CentOS Linux release 8.0.1905 (Core) the restore container info: kernel:4.18.0-193.el8.jd_017.x86_64 os:CentOS Linux release 7.1.1503 (Core)
I look forward to your help!
It seems the destination container has an incompatible network setup with the source container.
You write "and I restore the process in a container on another host". What does that mean? The container already exists and you restore into the container? How is the network of the destination container configured?
1.The container already exists and you restore into the container? Uh,yes. The process I want to dump is a container internal process with a PID of 1, and I restore it into the destination container. 2. How is the network of the destination container configured? What command should I execute to view the network configuration? In fact, the network configuration of the two containers is almost identical.
destination network config:
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: dummy_jcr: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether b6:54:42:62:0d:48 brd ff:ff:ff:ff:ff:ff
23: eth0@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether b8:ce:f6:32:f7:1e brd ff:ff:ff:ff:ff:ff
inet 11.117.75.160/32 brd 11.117.75.160 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::b8ce:f600:1332:f71e/64 scope link
valid_lft forever preferred_lft forever
# ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
# ip route
default via 11.13.118.1 dev eth0
11.13.118.1 dev eth0 scope link
source network config:
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if103: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 12:5e:f6:1b:36:4d brd ff:ff:ff:ff:ff:ff
inet 11.53.66.216/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::105e:f6ff:fe1b:364d/64 scope link
valid_lft forever preferred_lft forever
# ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
# ip route
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
Do you checkpoint the container inside of the container or outside of the container?
Do you checkpoint the container inside of the container or outside of the container?
Outside. I understand that I can't dump a process with a pid of 1 in the container.
And the source container also does not disable seccomp , so when I checkpoint, I got an error:suspending seccomp failed: Operation not permitted.
If you checkpoint the process outside of the container CRIU will track all the namespace information. For the network namespace, for example you need to tell CRIU to join the network namespace of the existing container. Please have a look at the man page how to tell CRIU to join existing namespaces.
If you checkpoint the process outside of the container CRIU will track all the namespace information. For the network namespace, for example you need to tell CRIU to join the network namespace of the existing container. Please have a look at the man page how to tell CRIU to join existing namespaces.
Thanks for your tip. I'm not familiar with man page. Could you tell me the specific URL of how to tell CRIU to join existing namespaces?
I add --join-ns to restore,like this: --join-ns net:1 And it occurred another error: Unknown peer net namespace
error:
(00.000000) Added net:1 join namespace
(00.000061) Version: 3.17 (gitid 496bcdb)
(00.000066) Running on 2ec7552e-3e6c-413c-be2c-b6631158a67d Linux 4.18.0-193.el8.jd_017.x86_64 #1 SMP Tue Apr 20 20:12:31 PDT 2021 x86_64
(00.000070) File /run/criu.kdat does not exist
(00.000087) sockets: Probing sock diag modules
(00.000123) sockets: Done probing
(00.002220) Error (criu/util.c:641): exited, status=3
(00.002249) Pagemap is fully functional
(00.002274) Found anon-shmem device at 5
(00.002292) Hugetlb size 2 Mb is supported but cannot get dev's number
(00.002299) Hugetlb size 1024 Mb is supported but cannot get dev's number
(00.002302) Reset 312854's dirty tracking
(00.002346) ... done
(00.002364) Dirty track supported on kernel
(00.002403) Found task size of 7ffffffff000
(00.007894) Restoring netdev veth idx 10
(00.008244) Dumping netns links
(00.008291) LD: Got link 1, type 772
(00.008297) LD: Got link 10, type 1
(00.009078) vdso: Parsing at 7fffc013e000 7fffc0140000
(00.009090) vdso: PT_LOAD p_vaddr: 0
(00.009092) vdso: DT_HASH: 120
(00.009093) vdso: DT_STRTAB: 298
(00.009094) vdso: DT_SYMTAB: 1a8
(00.009095) vdso: DT_STRSZ: 5e
(00.009096) vdso: DT_SYMENT: 18
(00.009098) vdso: nbucket 3 nchain a bucket 7fffc013e128 chain 7fffc013e134
(00.009103) vdso: rt [vdso] 7fffc013e000-7fffc0140000 [vvar] 7fffc013b000-7fffc013e000
(00.009463) vdso: Parsing at 7fa7164bb000 7fa7164bd000
(00.009469) vdso: PT_LOAD p_vaddr: 0
(00.009471) vdso: DT_HASH: b4
(00.009472) vdso: DT_STRTAB: 1c0
(00.009473) vdso: DT_SYMTAB: 130
(00.009474) vdso: DT_STRSZ: 95
(00.009475) vdso: DT_SYMENT: 10
(00.009476) vdso: nbucket 3 nchain 9 bucket 7fa7164bb0bc chain 7fa7164bb0c8
(00.009486) vdso: compat [vdso] 7fffc0141000-7fffc0143000 [vvar] 7fffc013e000-7fffc0141000
(00.009824) cpu: x86_family 6 x86_vendor_id GenuineIntel x86_model_id Intel(R) Xeon(R) Gold 6267C CPU @ 2.60GHz
(00.009829) cpu: fpu: xfeatures_mask 0x2f5 xsave_size 2696 xsave_size_max 2696 xsaves_size 2568
(00.009833) cpu: fpu: x87 floating point registers xstate_offsets 0 / 0 xstate_sizes 160 / 160
(00.009836) cpu: fpu: AVX registers xstate_offsets 576 / 576 xstate_sizes 256 / 256
(00.009837) cpu: fpu: MPX CSR xstate_offsets 1024 / 832 xstate_sizes 64 / 64
(00.009839) cpu: fpu: AVX-512 opmask xstate_offsets 1088 / 896 xstate_sizes 64 / 64
(00.009841) cpu: fpu: AVX-512 Hi256 xstate_offsets 1152 / 960 xstate_sizes 512 / 512
(00.009842) cpu: fpu: AVX-512 ZMM_Hi256 xstate_offsets 1664 / 1472 xstate_sizes 1024 / 1024
(00.009844) cpu: fpu: Protection Keys User registers xstate_offsets 2688 / 2496 xstate_sizes 8 / 8
(00.009939) The new mount API (fsopen, fsmount) isn't supported
(00.009945) Time namespaces are not supported.
(00.009967) Warn (criu/kerndat.c:1336): Can't get pidfd
(00.009989) Warn (criu/kerndat.c:1453): CRIU was built without libnftables support
(00.010220) No MOVE_MOUNT_SET_GROUP kernel feature
(00.022664) No openat2 syscall support
(00.022927) ptrace(PTRACE_GET_RSEQ_CONFIGURATION) is not supported
(00.023069) Adjust mmap_min_addr 0x1000 -> 0x10000
(00.023072) Found mmap_min_addr 0x10000
(00.023078) files stat: fs/nr_open 6558576
(00.023108) Warn (criu/kerndat.c:1117): Can't keep kdat cache on non-tempfs
(00.023126) Will drop all TCP connections on restore
(00.023128) mnt-v2: Mounts-v2 requires MOVE_MOUNT_SET_GROUP support
(00.023129) Mount engine fallback to --mntns-compat-mode mode
(00.023135) rlimit: RLIMIT_NOFILE unlimited for self
(00.023181) cpu: fpu:1 fxsr:1 xsave:1 xsaveopt:1 xsavec:1 xgetbv1:1 xsaves:1
(00.023195) kernel pid_max=655300
(00.023197) Reading image tree
(00.023228) Add mnt ns 13 pid 1
(00.023231) Add net ns 10 pid 1
(00.023232) Add pid ns 9 pid 1
(00.023241) pstree pid_max=14
(00.023246) Will restore in 6c020000 namespaces
(00.023248) NS mask to use 6c020000
(00.023262) Collecting 51/56 (flags 3)
(00.023265) No memfd.img image
(00.023267) `- ... done
(00.023268) Collecting 40/54 (flags 2)
(00.023288) Collected [opt/jimdb/redis-server] ID 0x1
(00.023292) Collected [usr/lib64/libfreebl3.so] ID 0x2
(00.023295) Collected [usr/lib64/liblzma.so.5.0.99] ID 0x3
(00.023299) Collected [usr/lib64/libpcre.so.1.2.0] ID 0x4
(00.023302) Collected [usr/lib64/libcrypt-2.17.so] ID 0x5
(00.023304) Collected [usr/lib64/libselinux.so.1] ID 0x6
(00.023306) Collected [usr/lib64/libsasl2.so.3.0.0] ID 0x7
(00.023309) Collected [usr/lib64/libresolv-2.17.so] ID 0x8
(00.023311) Collected [usr/lib64/libkeyutils.so.1.5] ID 0x9
(00.023313) Collected [usr/lib64/libkrb5support.so.0.1] ID 0xa
(00.023316) Collected [usr/lib64/librt-2.17.so] ID 0xb
(00.023318) Collected [usr/lib64/libcrypto.so.1.0.1e] ID 0xc
(00.023322) Collected [usr/lib64/libssl.so.1.0.1e] ID 0xd
(00.023324) Collected [usr/lib64/libz.so.1.2.7] ID 0xe
(00.023327) Collected [usr/lib64/libldap-2.4.so.2.10.2] ID 0xf
(00.023329) Collected [usr/lib64/liblber-2.4.so.2.10.2] ID 0x10
(00.023331) Collected [usr/lib64/libcom_err.so.2.1] ID 0x11
(00.023333) Collected [usr/lib64/libk5crypto.so.3.1] ID 0x12
(00.023336) Collected [usr/lib64/libkrb5.so.3.3] ID 0x13
(00.023338) Collected [usr/lib64/libgssapi_krb5.so.2.2] ID 0x14
(00.023340) Collected [usr/lib64/libnspr4.so] ID 0x15
(00.023345) Collected [usr/lib64/libplc4.so] ID 0x16
(00.023347) Collected [usr/lib64/libplds4.so] ID 0x17
(00.023349) Collected [usr/lib64/libnssutil3.so] ID 0x18
(00.023352) Collected [usr/lib64/libnss3.so] ID 0x19
(00.023354) Collected [usr/lib64/libsmime3.so] ID 0x1a
(00.023356) Collected [usr/lib64/libssl3.so] ID 0x1b
(00.023358) Collected [usr/lib64/libssh2.so.1.0.1] ID 0x1c
(00.023361) Collected [usr/lib64/libidn.so.11.6.11] ID 0x1d
(00.023365) Collected [usr/lib64/libc-2.17.so] ID 0x1e
(00.023370) Collected [usr/lib64/libpthread-2.17.so] ID 0x1f
(00.023373) Collected [usr/lib64/libdl-2.17.so] ID 0x20
(00.023375) Collected [usr/lib64/libcurl.so.4.3.0] ID 0x21
(00.023377) Collected [usr/lib64/libm-2.17.so] ID 0x22
(00.023380) Collected [usr/lib64/ld-2.17.so] ID 0x23
(00.023383) Collected pipe entry ID 0x24 PIPE ID 0x7e14f6f
(00.023389) Collected [var/log/jimdb/6379.log] ID 0x25
(00.023400) epoll: Collected eventpoll: id 0x000026 flags 0x02
(00.023405) epoll: Collected eventpoll: id 0x000027 flags 0x02
(00.023425) eventfd: Collected : id 0x00002c flags 0x02 counter 0000000000000000
(00.023452) Collected [opt/jimdb/data] ID 0x37
(00.023454) Collected [.] ID 0x38
(00.023456) Collected [usr/sbin/sshd] ID 0x39
(00.023459) Collected [usr/lib64/libnss_files-2.17.so] ID 0x3a
(00.023463) Collected [usr/lib64/libnsl-2.17.so] ID 0x3b
(00.023465) Collected [usr/lib64/libutil-2.17.so] ID 0x3c
(00.023468) Collected [usr/lib64/libpam.so.0.83.1] ID 0x3d
(00.023470) Collected [usr/lib64/libaudit.so.1.0.0] ID 0x3e
(00.023472) Collected [usr/lib64/libwrap.so.0.7.6] ID 0x3f
(00.023475) Collected [usr/lib64/libfipscheck.so.1.2.1] ID 0x40
(00.023477) Collected [dev/null] ID 0x41
(00.023502) Collected [.] ID 0x44
(00.023506) Collected [.] ID 0x45
(00.023509) `- ... done
(00.023510) Collecting 46/68 (flags 0)
(00.023513) No remap-fpath.img image
(00.023515) `- ... done
(00.023534) No apparmor.img image
(00.023616) Running pre-restore scripts
(00.023647) Saved netns fd for links restore
(00.023933) mnt: Reading mountpoint images (id 13 pid 1)
(00.023941) mnt: Will mount 2921 from /
(00.023947) mnt: Will mount 2921 @ /tmp/.criu.mntns.y3mxIz/13-0000000000/run/secrets/kubernetes.io/serviceaccount /run/secrets/kubernetes.io/serviceaccount
(00.023949) mnt: Read 2921 mp @ /run/secrets/kubernetes.io/serviceaccount
(00.023954) mnt: Will mount 2920 from /var/log/jimdb (E)
(00.023956) mnt: Will mount 2920 @ /tmp/.criu.mntns.y3mxIz/13-0000000000/var/log/jimdb /var/log/jimdb
(00.023957) mnt: Read 2920 mp @ /var/log/jimdb
(00.023960) mnt: Will mount 2919 from /opt/jimdb/data (E)
(00.023962) mnt: Will mount 2919 @ /tmp/.criu.mntns.y3mxIz/13-0000000000/opt/jimdb/data /opt/jimdb/data
(00.023963) mnt: Read 2919 mp @ /opt/jimdb/data
(00.023966) mnt: Will mount 2915 from /
(00.023968) mnt: Will mount 2915 @ /tmp/.criu.mntns.y3mxIz/13-0000000000/dev/shm /dev/shm
(00.023973) mnt: Read 2915 mp @ /dev/shm
(00.023975) mnt: Will mount 2911 from /
(00.023977) mnt: Will mount 2911 @ /tmp/.criu.mntns.y3mxIz/13-0000000000/dev/mqueue /dev/mqueue
(00.023978) mnt: Read 2911 mp @ /dev/mqueue
(00.023980) mnt: Will mount 2864 from /
(00.023982) mnt: Will mount 2864 @ /tmp/.criu.mntns.y3mxIz/13-0000000000/sys/fs/cgroup /sys/fs/cgroup
(00.023983) mnt: Read 2864 mp @ /sys/fs/cgroup
(00.023986) mnt: Will mount 2863 from /
(00.023988) mnt: Will mount 2863 @ /tmp/.criu.mntns.y3mxIz/13-0000000000/sys /sys
(00.023989) mnt: Read 2863 mp @ /sys
(00.023991) mnt: Will mount 2862 from /
(00.023993) mnt: Will mount 2862 @ /tmp/.criu.mntns.y3mxIz/13-0000000000/dev/pts /dev/pts
(00.023994) mnt: Read 2862 mp @ /dev/pts
(00.023998) mnt: Will mount 2861 from /
(00.024000) mnt: Will mount 2861 @ /tmp/.criu.mntns.y3mxIz/13-0000000000/dev /dev
(00.024001) mnt: Read 2861 mp @ /dev
(00.024003) mnt: Will mount 2860 from /
(00.024005) mnt: Will mount 2860 @ /tmp/.criu.mntns.y3mxIz/13-0000000000/proc /proc
(00.024006) mnt: Read 2860 mp @ /proc
(00.024009) mnt: Will mount 2859 from /rootfs
(00.024011) mnt: Will mount 2859 @ /tmp/.criu.mntns.y3mxIz/13-0000000000/ /
(00.024012) mnt: Read 2859 mp @ /
(00.024015) mnt: Building mountpoints tree
(00.024016) mnt: Building plain mount tree
(00.024017) mnt: Working on 2859->1360
(00.024018) mnt: Working on 2860->2859
(00.024020) mnt: Working on 2861->2859
(00.024021) mnt: Working on 2862->2861
(00.024022) mnt: Working on 2863->2859
(00.024023) mnt: Working on 2864->2863
(00.024024) mnt: Working on 2911->2861
(00.024025) mnt: Working on 2915->2861
(00.024026) mnt: Working on 2919->2859
(00.024027) mnt: Working on 2920->2859
(00.024029) mnt: Working on 2921->2859
(00.024030) mnt: Resorting children of 2859 in mount order
(00.024032) mnt: Resorting children of 2921 in mount order
(00.024033) mnt: Resorting children of 2919 in mount order
(00.024034) mnt: Resorting children of 2920 in mount order
(00.024035) mnt: Resorting children of 2860 in mount order
(00.024036) mnt: Resorting children of 2861 in mount order
(00.024037) mnt: Resorting children of 2862 in mount order
(00.024039) mnt: Resorting children of 2911 in mount order
(00.024040) mnt: Resorting children of 2915 in mount order
(00.024041) mnt: Resorting children of 2863 in mount order
(00.024042) mnt: Resorting children of 2864 in mount order
(00.024043) mnt: Done:
(00.024044) mnt: [/](2859->1360)
(00.024046) mnt: [/run/secrets/kubernetes.io/serviceaccount](2921->2859)
(00.024047) mnt: <--
(00.024048) mnt: [/opt/jimdb/data](2919->2859)
(00.024050) mnt: <--
(00.024051) mnt: [/var/log/jimdb](2920->2859)
(00.024052) mnt: <--
(00.024053) mnt: [/proc](2860->2859)
(00.024054) mnt: <--
(00.024055) mnt: [/dev](2861->2859)
(00.024057) mnt: [/dev/pts](2862->2861)
(00.024058) mnt: <--
(00.024059) mnt: [/dev/mqueue](2911->2861)
(00.024060) mnt: <--
(00.024062) mnt: [/dev/shm](2915->2861)
(00.024063) mnt: <--
(00.024064) mnt: <--
(00.024065) mnt: [/sys](2863->2859)
(00.024066) mnt: [/sys/fs/cgroup](2864->2863)
(00.024068) mnt: <--
(00.024069) mnt: <--
(00.024070) mnt: <--
(00.024071) mnt: Start with 2859:/
(00.024075) mnt: Mountpoint 2859 (@/) moved to the root yard
(00.024115) No pidns-9.img image
(00.024167) Forking task with 1 pid (flags 0x6c020000)
(00.024663) PID: real 312866 virt 1
(00.024714) Wait until namespaces are created
(00.025637) Running setup-namespaces scripts
(00.025716) 1: Calling restore_sid() for init
(00.025726) 1: Restoring 1 to 1 sid
(00.026025) 1: Mount procfs in /tmp/crtools-proc.5k93um
(00.034517) 1: Collecting 44/37 (flags 2)
(00.034530) 1: No tty-info.img image
(00.034535) 1: `- ... done
(00.034536) 1: Collecting 45/51 (flags 0)
(00.034539) 1: No tty-data.img image
(00.034543) 1: `- ... done
(00.034545) 1: Restoring namespaces 1 flags 0x6c020000
(00.034818) 1: kernel/hostname nr 36
(00.034890) 1: kernel/domainname nr 6
(00.035155) 1: Restoring IPC namespace
(00.035159) 1: Restoring IPC variables
(00.035750) 1: Restoring IPC shared memory
(00.035759) 1: No ipcns-shm-11.img image
(00.035761) 1: Restoring IPC message queues
(00.035763) 1: No ipcns-msg-11.img image
(00.035764) 1: Restoring IPC semaphores sets
(00.035767) 1: No ipcns-sem-11.img image
(00.037024) 1: Try to restore a link 10:3:eth0
(00.037033) 1: Restoring link eth0 type 2
(00.037041) 1: Restoring netdev eth0 idx 3
(00.037047) 1: Restore ll addr (12:../6) for device
(00.037050) 1: Error (criu/net.c:1462): Unknown peer net namespace
(00.047533) 1: Error (criu/libnetlink.c:54): -17 reported by netlink: File exists
(00.047539) 1: Error (criu/net.c:1816): Can't restore link: -17
(00.068679) 1: Error (criu/util.c:1411): Can't wait or bad status: errno=0, status=65280
(00.093766) Error (criu/cr-restore.c:2536): Restoring FAILED.
Try to mark the network namespace as an external resource during dump with --external and restore it with --inheritfd.
Try to mark the network namespace as an external resource during dump with
--externaland restore it with--inheritfd.
I kind of understand what you mean and try to do it, but I don't know exactly how to do it. So I wanted to refer to this URL( https://criu.org/External_net_namespaces ), but there was no example. Could you help me with an example?
I found the usage through this URL.(https://github.com/checkpoint-restore/criu/commit/c2b21fbf4baa5b134e90d9daf863b770b7518c73) I add --external to dump, like this: --external net[4026534545]:host And I add --inheritfd to restore, like this: --inherit-fd fd[1]:host But It doesn't work. I don't know what NSFD is.
source container: # ls -l /proc/1/ns/net lrwxrwxrwx 1 root root 0 May 19 23:34 /proc/1/ns/net -> net:[4026534545]
the new error: (00.025485) Wait until namespaces are created (00.025533) 1: Found id host (fd 6) in inherit fd list (00.025600) 1: Error (criu/namespaces.c:258): Can't setns 6/net: Invalid argument (00.025607) 1: Error (criu/cr-restore.c:1795): Can't unshare net-namespace: Invalid argument (00.054975) Error (criu/cr-restore.c:2536): Restoring FAILED.
Am I using it the wrong way?
Using -j when dumping a container, do you really need it? Normally container init has own session. It is worth mentioning which container runtime you use to create containers, if it's something popular why don't you use it's builtin c/r support?
Probably the easiest way here is if we have an instruction on how to reproduce (better minimal reproduce) the issue.
A friendly reminder that this issue had no activity for 30 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}