gramine icon indicating copy to clipboard operation
gramine copied to clipboard

Published 20 hours ago • verified publisher icon gramineproject

Error "std::system_error" when running parameter server in gramine

Open gagandeep987123 opened this issue 2 years ago • 9 comments

I am attempting to run PyTorch’s parameter server in gramine.

The program uses PyTorch's multiprocessing which doesn't work.

As another attempt, I tried another approach where instead of calling a separate thread call, I am directly calling the function run_parameter_server(0,world_size) at line 275-277 which again fails with output

terminate called after throwing an instance of 'std::system_error'
 what():  Unexpected poll revent on the control pipe's reading fd: 24: Software caused connection abort

Can you please help me reason why it fails? For running the parameter server in python I used: python rpc_parameter_server.py --world_size=2 --rank=0 Likewise in gamine I am using: gramine-direct ./pytorch ./rpc_parameter_server.py --world_size=2 --rank=0

gagandeep987123 avatar Aug 05 '22 12:08 gagandeep987123

Could you provide us with a trace log of Gramine execution? (add loader.log_level = "all" to the manifest)

The program uses PyTorch's multiprocessing which doesn't work.

Why it does not work?

boryspoplawski avatar Aug 05 '22 12:08 boryspoplawski

The output is like 4.2 MB file with 55k lines. I am pasting last few lines of it as Github comment doesn't allow pasting the full output.


[::] debug: memory entry [0x7fffe5d8eb00]: 0x382fde218000-0x382fded48000
[::] debug: memory entry [0x7fffe5d8e6a0]: 0x382ff54ab000-0x382ff5830000
[::] debug: memory entry [0x7fffe5d8e608]: 0x382ff5830000-0x382ff59cc000
[::] debug: memory entry [0x7fffe5d8e570]: 0x382ff59cc000-0x382ff5ad1000
[::] debug: memory entry [0x7fffe5d8e4d8]: 0x382ff5ad1000-0x382ff600d000
[::] debug: memory entry [0x7fffe5d8e440]: 0x382ff600d000-0x382ff64ca000
[::] debug: memory entry [0x7fffe5d8dfe8]: 0x382ff66dc000-0x382ff66dd000
[::] debug: memory entry [0x7fffe5d8df50]: 0x382ff66dd000-0x382ff66de000
[::] debug: memory entry [0x7fffe5d8daf8]: 0x382ff68e3000-0x382ff68e4000
[::] debug: memory entry [0x7fffe5d8da60]: 0x382ff68e4000-0x382ff68e5000
[::] debug: memory entry [0x7fffe5d8d610]: 0x382ff75f1000-0x382ff760a000
[::] debug: memory entry [0x7fffe5d8d578]: 0x382ff760a000-0x382ff7653000
[::] debug: memory entry [0x7fffe5d8d4e0]: 0x382ff7653000-0x382ff76ac000
[::] debug: memory entry [0x7fffe5d8d448]: 0x382ff76ac000-0x382ff7785000
[::] debug: memory entry [0x7fffe5d8cfe0]: 0x382ff7985000-0x382ff7986000
[::] debug: memory entry [0x7fffe5d8cf48]: 0x382ff7986000-0x382ff7987000
[::] debug: memory entry [0x7fffe5d8ceb0]: 0x382ff7987000-0x382ff7988000
[::] debug: memory entry [0x7fffe5d8c960]: 0x382ff7b59000-0x382ff7b64000
[::] debug: memory entry [0x7fffe5d8c8c8]: 0x382ff7b64000-0x382ff7b67000
[::] debug: memory entry [0x7fffe5d8c558]: 0x382ff7b67000-0x382ff7b6a000
[::] debug: memory entry [0x7fffe5d8c3f0]: 0x382ff7d8e000-0x382ff7d8f000
[::] debug: memory entry [0x7fffe5d8c358]: 0x382ff7d8f000-0x382ff7d90000
[::] debug: memory entry [0x7fffe5d8c2c0]: 0x382ff7d90000-0x382ff7d94000
[::] debug: memory entry [0x7fffe5d8be58]: 0x382ff7f9c000-0x382ff7f9d000
[::] debug: memory entry [0x7fffe5d8bdc0]: 0x382ff7f9d000-0x382ff7f9e000
[::] debug: memory entry [0x7fffe5d8b948]: 0x382ff819e000-0x382ff819f000
[::] debug: memory entry [0x7fffe5d8b8b0]: 0x382ff819f000-0x382ff81a0000
[::] debug: memory entry [0x7fffe5d8b818]: 0x382ff81a0000-0x382ff81a1000
[::] debug: memory entry [0x7fffe5d8b780]: 0x382ff81a1000-0x382ff81a2000
[::] debug: memory entry [0x7fffe5d8ad48]: 0x382ff81a2000-0x382ff81e2000
[::] debug: memory entry [0x7fffe5d8acb0]: 0x382ff81e2000-0x382ff8222000
[::] debug: memory entry [0x7fffe5d8ac18]: 0x382ff8222000-0x382ff8322000
[::] debug: memory entry [0x7fffe5d8ab80]: 0x382ff8322000-0x382ff8422000
[::] debug: memory entry [0x7fffe5d8aae8]: 0x382ff8422000-0x382ff8462000
[::] debug: memory entry [0x7fffe5d8aa50]: 0x382ff8462000-0x382ff84a2000
[::] debug: memory entry [0x7fffe5d8a9b8]: 0x382ff84a2000-0x382ff84e2000
[::] debug: memory entry [0x7fffe5d8a7e8]: 0x382ff84f8000-0x382ff84f9000
[::] debug: memory entry [0x7fffe5d8a750]: 0x382ff84f9000-0x382ff84fa000
[::] debug: memory entry [0x7fffe5d8a270]: 0x382ff84fd000-0x382ff84fe000
[::] debug: memory entry [0x7fffe5d8a1d8]: 0x382ff84fe000-0x382ff84ff000
[::] debug: memory entry [0x7fffe5d89cf8]: 0x382ff8518000-0x382ff8519000
[::] debug: memory entry [0x7fffe5d89c60]: 0x382ff8519000-0x382ff851a000
[::] debug: memory entry [0x7fffe5d89750]: 0x382ff8541000-0x382ff8542000
[::] debug: memory entry [0x7fffe5d896b8]: 0x382ff8542000-0x382ff8543000
[::] debug: memory entry [0x7fffe5d89218]: 0x382ff854d000-0x382ff854e000
[::] debug: memory entry [0x7fffe5d89180]: 0x382ff854e000-0x382ff854f000
[::] debug: memory entry [0x7fffe5d88ca0]: 0x382ff8560000-0x382ff8561000
[::] debug: memory entry [0x7fffe5d88c08]: 0x382ff8561000-0x382ff8562000
[::] debug: memory entry [0x7fffe5d88898]: 0x382ff8562000-0x382ff85a2000
[::] debug: memory entry [0x7fffe5d88800]: 0x382ff85a2000-0x382ff85e2000
[::] debug: memory entry [0x7fffe5d88768]: 0x382ff85e2000-0x382ff8622000
[::] debug: memory entry [0x7fffe5d88598]: 0x382ff8629000-0x382ff862a000
[::] debug: memory entry [0x7fffe5d88500]: 0x382ff862a000-0x382ff862b000
[::] debug: memory entry [0x7fffe5d88048]: 0x382ff8631000-0x382ff8632000
[::] debug: memory entry [0x7fffe5d87fb0]: 0x382ff8632000-0x382ff8633000
[::] debug: memory entry [0x7fffe5d87a70]: 0x382ff863d000-0x382ff863e000
[::] debug: memory entry [0x7fffe5d879d8]: 0x382ff863e000-0x382ff863f000
[::] debug: memory entry [0x7fffe5d874d0]: 0x382ff865d000-0x382ff865e000
[::] debug: memory entry [0x7fffe5d87438]: 0x382ff865e000-0x382ff8662000
[::] debug: memory entry [0x7fffe5d86e50]: 0x382ff8662000-0x382ff86a2000
[::] debug: memory entry [0x7fffe5d86db8]: 0x382ff86a2000-0x382ff86e2000
[::] debug: memory entry [0x7fffe5d86d20]: 0x382ff86e2000-0x382ff8722000
[::] debug: memory entry [0x7fffe5d86c88]: 0x382ff8722000-0x382ff8762000
[::] debug: memory entry [0x7fffe5d86bf0]: 0x382ff8762000-0x382ff8862000
[::] debug: memory entry [0x7fffe5d86b58]: 0x382ff8862000-0x382ff88a2000
[::] debug: memory entry [0x7fffe5d86ac0]: 0x382ff88a2000-0x382ff88e2000
[::] debug: memory entry [0x7fffe5d86a28]: 0x382ff88e2000-0x382ff8922000
[::] debug: memory entry [0x7fffe5d86990]: 0x382ff8922000-0x382ff8962000
[::] debug: memory entry [0x7fffe5d868f8]: 0x382ff8962000-0x382ff89a2000
[::] debug: memory entry [0x7fffe5d86860]: 0x382ff89a2000-0x382ff89e2000
[::] debug: memory entry [0x7fffe5d867c8]: 0x382ff89e2000-0x382ff8a22000
[::] debug: memory entry [0x7fffe5d86730]: 0x382ff8a22000-0x382ff8a62000
[::] debug: memory entry [0x7fffe5d86698]: 0x382ff8a62000-0x382ff8aa2000
[::] debug: memory entry [0x7fffe5d86600]: 0x382ff8aa2000-0x382ff8ba2000
[::] debug: memory entry [0x7fffe5d86568]: 0x382ff8ba2000-0x382ff8be2000
[::] debug: memory entry [0x7fffe5d864d0]: 0x382ff8be2000-0x382ff8c22000
[::] debug: memory entry [0x7fffe5d86438]: 0x382ff8c22000-0x382ff8c25000
[::] debug: memory entry [0x7fffe5d86200]: 0x382ff8c3f000-0x382ff8c40000
[::] debug: memory entry [0x7fffe5d86168]: 0x382ff8c40000-0x382ff8c41000
[::] debug: memory entry [0x7fffe5d85e00]: 0x382ff8c41000-0x382ff8c43000
[::] debug: memory entry [0x7fffe5d85bc8]: 0x382ff8c6e000-0x382ff8c70000
[::] debug: memory entry [0x7fffe5d85b30]: 0x382ff8c70000-0x382ff8c71000
[::] debug: memory entry [0x7fffe5d85620]: 0x382ff8d4a000-0x382ff8d4b000
[::] debug: memory entry [0x7fffe5d85588]: 0x382ff8d4b000-0x382ff8d4c000
[::] debug: memory entry [0x7fffe5d850d0]: 0x382ff8d4f000-0x382ff8d50000
[::] debug: memory entry [0x7fffe5d85038]: 0x382ff8d50000-0x382ff8d51000
[::] debug: memory entry [0x7fffe5d84b78]: 0x382ff8d54000-0x382ff8d55000
[::] debug: memory entry [0x7fffe5d84ae0]: 0x382ff8d55000-0x382ff8d56000
[::] debug: memory entry [0x7fffe5d84628]: 0x382ff8d59000-0x382ff8d5a000
[::] debug: memory entry [0x7fffe5d84590]: 0x382ff8d5a000-0x382ff8d5b000
[::] debug: memory entry [0x7fffe5d840d0]: 0x382ff8f41000-0x382ff8f44000
[::] debug: memory entry [0x7fffe5d84038]: 0x382ff8f44000-0x382ff8f47000
[::] debug: memory entry [0x7fffe5d83cb8]: 0x382ff8f47000-0x382ff8f4f000
[::] debug: memory entry [0x7fffe5d83c20]: 0x382ff8f4f000-0x382ff8f51000
[::] debug: memory entry [0x7fffe5d83b88]: 0x382ff8f51000-0x382ff8f52000
[::] debug: memory entry [0x7fffe5d839b8]: 0x382ff8f81000-0x382ff8f82000
[::] debug: memory entry [0x7fffe5d83920]: 0x382ff8f82000-0x382ff8f83000
[::] debug: memory entry [0x7fffe5d83888]: 0x382ff8f83000-0x382ff8f85000
[::] debug: memory entry [0x7fffe5d834f0]: 0x382ff8f85000-0x382ff8f86000
[::] debug: memory entry [0x7fffe5d83458]: 0x382ff8f86000-0x382ff8fc6000
[::] debug: restored memory from checkpoint
[::] debug: receiving 81 PAL handles
[::] debug: restoring checkpoint at 0x7fffe5d81000 rebased from 0x7fffe5d81000
[::] debug: Creating pipe: pipe.srv:7febfcec7733f5484fdab8bc0e17e9ff656fbe677dbfe53a3d73813e4da1bf9f
[P3:T8:python3.8] debug: successfully restored checkpoint at 0x7fffe5d81000 - 0x7fffe5dad068
[P3:T8:python3.8] debug: Creating pipe: pipe.srv:24d312bc7c7d60862d77d779fe9c24192ba8ba4ee5834d5d87fff716c50d2243
[P3:T8:python3.8] debug: Creating pipe: pipe.srv:3
[P3:T8:python3.8] debug: Creating pipe: pipe.srv:303780db54199161a4483d89e0a08961a095247341f49d7aed3b5f9560f0736f
[P3:T8:python3.8] debug: ipc_get_id_owner: sending a request: 0
[P3:T8:python3.8] debug: Sending ipc message to 1
[P3:T8:python3.8] debug: Waiting for a response to 1
[P1:shim] debug: IPC worker: received IPC message from 3: code=6 size=21 seq=1
[P1:shim] debug: ipc_get_id_owner_callback: find_id_owner(0): 0
[P1:shim] debug: Sending ipc message to 3
[P3:shim] debug: IPC worker started
[P3:shim] debug: IPC worker: received IPC message from 1: code=0 size=21 seq=1
[P3:shim] debug: Got an IPC response from 1, seq: 1
[P3:T8:python3.8] debug: Waiting finished: 0
[P3:T8:python3.8] debug: ipc_get_id_owner: got a response: 0
[P1:T1:python3.8] trace: ---- return from shim_clone(...) = 0x8
[P1:T1:python3.8] trace: ---- shim_close(5) = 0x0
[P1:T1:python3.8] trace: ---- shim_close(6) = 0x0
[P1:T1:python3.8] trace: ---- shim_rt_sigprocmask(UNBLOCK, [SIGINT,SIGTERM,], [SIGINT,SIGTERM,], 0x8) = 0x0
[P1:T1:python3.8] trace: ---- shim_close(3) = 0x0
[P1:T1:python3.8] trace: ---- shim_getcwd(0x382f904064d0, 0x400) = 0x2
[P1:T1:python3.8] debug: Creating pipe: pipe.srv:f490896aaa90ef9cfe11171f8ec5620ffd1f7fb3105bd7e136f5d7ac6f8cb3c4
[P1:T1:python3.8] trace: ---- shim_pipe2(0x382ff8fc3f18, 524288) = 0x0
[P1:T1:python3.8] debug: Creating pipe: pipe.srv:f3faa19fd90916d29ab0f237d0a07cefa926f3e6baa0b5055eaeffb6d44e2abe
[P1:T1:python3.8] trace: ---- shim_pipe2(0x382ff8fc3f18, 524288) = 0x0
[P1:T1:python3.8] debug: Creating pipe: pipe.srv:3458f65df9f3811c0b5b35644de952a93137bc28809bbbf43d5618eec967cc2e
[P1:T1:python3.8] trace: ---- shim_pipe2(0x382ff8fc3d88, 524288) = 0x0
[P1:T1:python3.8] trace: ---- shim_clone(CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|[SIGCHLD], 0, 0, 0x382ff8c22a10, 0) ...
[P1:T1:python3.8] debug: Creating pipe: pipe.srv:fe76cb12da646426e8416ca95be810119ae1fc92ce96c4bd99ebc94f580618f9
[P3:T8:python3.8] debug: Shim process initialized
[P3:T8:python3.8] trace: ---- shim_set_robust_list(0x382ff8c22a20, 0x18) = 0x0
[P3:T8:python3.8] trace: ---- shim_fcntl(2, F_GETFD, 0) = 0x0
[P3:T8:python3.8] trace: ---- shim_fcntl(3, F_GETFD, 0) = 0x1
[P3:T8:python3.8] trace: ---- shim_fcntl(3, F_SETFD, 0) = 0x0
[P3:T8:python3.8] trace: ---- shim_close(5) = 0x0
[P3:T8:python3.8] trace: ---- shim_openat(AT_FDCWD, "/proc/self/fd", O_RDONLY|0x80000, 0000) = 0x5
[P3:T8:python3.8] trace: ---- shim_getdents64(5, 0x382ff8fc3750, 0x118) = 0x100
[P3:T8:python3.8] trace: ---- shim_close(6) = 0x0
[P3:T8:python3.8] trace: ---- shim_close(4) = 0x0
[P3:T8:python3.8] trace: ---- shim_getdents64(5, 0x382ff8fc3750, 0x118) = 0x20
[P3:T8:python3.8] trace: ---- shim_getdents64(5, 0x382ff8fc3750, 0x118) = 0x0
[P3:T8:python3.8] trace: ---- shim_close(5) = 0x0
[P3:T8:python3.8] trace: ---- shim_execve("/pytorch", [/pytorch,-c,from multiprocessing.resource_tracker import main;main(3),], [SHELL=/bin/bash,SESSION_MANAGER=local/gagan-VirtualBox:@/tmp/.ICE-unix/1691,unix/gagan-VirtualBox:/tmp/.ICE-unix/1691,]) ...
[P1:T1:python3.8] debug: allocating checkpoint store (size = 67108864, reserve = 33554432)
[::] debug: Gramine was built from commit: 1.2
[::] debug: Host: Linux
[::] debug: LibOS xsave_enabled 1, xsave_size 0x340(832), xsave_features 0x7
[::] debug: Initial VMA region 0x7fffebf33000-0x7fffebfd2000 (LibOS) bookkeeped
[::] debug: Initial VMA region 0x7ffff3fd2000-0x7ffff7fd2000 (pal_init_pool) bookkeeped
[::] debug: Initial VMA region 0x7ffff7fd6000-0x7ffff7fd8000 (vdso) bookkeeped
[::] debug: Initial VMA region 0x7ffff7fd2000-0x7ffff7fd6000 (vvar) bookkeeped
[::] debug: Initial VMA region 0x7fffebfd2000-0x7ffff3fd2000 (pal_internal_me) bookkeeped
[::] debug: ASLR top address adjusted to 0x5f1a69b58000
[P1:T1:python3.8] debug: complete checkpointing data
[::] debug: Shim loaded at 0x7fffebf33000, ready to initialize
[P1:T1:python3.8] debug: checkpoint of 181888 bytes created
[::] debug: checkpoint mapped at 0x7fffe5d81000-0x7fffe5dad680
[P3:T8:python3.8] trace: ---- return from shim_execve(...) = -2
[P3:T8:python3.8] trace: ---- shim_write(6, 0x82d79f, 0x8) ...
[P3:T8:python3.8] trace: ---- return from shim_write(...) = -9
[P3:T8:python3.8] trace: ---- shim_write(6, 0x382ff8fc3988, 0x1) ...
[P3:T8:python3.8] trace: ---- return from shim_write(...) = -9
[P3:T8:python3.8] trace: ---- shim_write(6, 0x82d7c0, 0x1) ...
[P3:T8:python3.8] trace: ---- return from shim_write(...) = -9
[P3:T8:python3.8] debug: ---- shim_exit_group (returning 255)
[P3:T8:python3.8] debug: Sending ipc message to 1
[P3:T8:python3.8] debug: Waiting for a response to 2
[::] debug: read checkpoint of 181888 bytes from parent
[::] debug: memory entry [0x7fffe5da69d8]: 0x8f8000-0x8f9000
[::] debug: memory entry [0x7fffe5da6940]: 0x8f9000-0x940000
[P1:shim] debug: IPC worker: received IPC message from 3: code=17 size=21 seq=2
[P1:shim] debug: clearing POSIX locks for pid 8
[P1:shim] debug: Sending ipc message to 3
[P3:shim] debug: IPC worker: received IPC message from 1: code=0 size=21 seq=2
[P3:shim] debug: Got an IPC response from 1, seq: 2
[P3:T8:python3.8] debug: Waiting finished: 0
[P3:T8:python3.8] debug: Sending ipc message to 1
[P3:T8:python3.8] debug: sync client shutdown: closing handles
[P3:T8:python3.8] debug: sync client shutdown: waiting for confirmation
[P3:T8:python3.8] debug: sync client shutdown: finished
[::] debug: memory entry [0x7fffe5da64d0]: 0x940000-0x963000
[P3:T8:python3.8] debug: ipc_release_id_range: sending a request: [8..8]
[P3:T8:python3.8] debug: Sending ipc message to 1
[P3:T8:python3.8] debug: ipc_release_id_range: ipc_send_message: 0
[P1:shim] debug: IPC worker: received IPC message from 3: code=2 size=37 seq=0
[P1:shim] debug: IPC callback from 3: IPC_MSG_CHILDEXIT(1, 8, 255, 0)
[P1:shim] debug: Child process (pid: 8) died
[P1:shim] debug: IPC worker: received IPC message from 3: code=4 size=25 seq=0
[P1:shim] debug: ipc_release_id_range_callback: release_id_range(8..8)
[P3:shim] debug: IPC worker: exiting worker thread
[::] debug: memory entry [0x7fffe5da6438]: 0x126f000-0x1290000
[P3:T8:python3.8] debug: process 3 exited with status 255
[::] debug: memory entry [0x7fffe5da6338]: 0x382f8f74d000-0x382f8f84d000
[::] debug: memory entry [0x7fffe5da62a0]: 0x382f8f84d000-0x382f8f88d000
[::] debug: memory entry [0x7fffe5da6208]: 0x382f8f90d000-0x382f8f94d000
[::] debug: memory entry [0x7fffe5da6170]: 0x382f8ff0d000-0x382f8ff4d000
[::] debug: memory entry [0x7fffe5da60d8]: 0x382f8ff4d000-0x382f9004d000
[::] debug: memory entry [0x7fffe5da6040]: 0x382f9004d000-0x382f9008d000
[::] debug: memory entry [0x7fffe5da5fa8]: 0x382f9008d000-0x382f900cd000
[::] debug: memory entry [0x7fffe5da5f10]: 0x382f900cd000-0x382f9010d000
[::] debug: memory entry [0x7fffe5da5e78]: 0x382f9010d000-0x382f9014d000
[::] debug: memory entry [0x7fffe5da5de0]: 0x382f9014d000-0x382f9018d000
[::] debug: memory entry [0x7fffe5da5d48]: 0x382f9018d000-0x382f901cd000
[::] debug: memory entry [0x7fffe5da5cb0]: 0x382f901cd000-0x382f9020d000
[::] debug: memory entry [0x7fffe5da5c18]: 0x382f9020d000-0x382f9024d000
[::] debug: memory entry [0x7fffe5da5b80]: 0x382f9024d000-0x382f9028d000
[::] debug: memory entry [0x7fffe5da5ae8]: 0x382f9028d000-0x382f902cd000
[::] debug: memory entry [0x7fffe5da5a50]: 0x382f902ce000-0x382f9030e000
[::] debug: memory entry [0x7fffe5da59b8]: 0x382f9030e000-0x382f9034e000
[::] debug: memory entry [0x7fffe5da5920]: 0x382f9034e000-0x382f9044e000
[::] debug: memory entry [0x7fffe5da5888]: 0x382f9044e000-0x382f9048e000
[::] debug: memory entry [0x7fffe5da57f0]: 0x382f9048e000-0x382f904ce000
[::] debug: memory entry [0x7fffe5da5758]: 0x382f904ce000-0x382f9050e000
[::] debug: memory entry [0x7fffe5da56c0]: 0x382f9050e000-0x382f9054e000
[::] debug: memory entry [0x7fffe5da5628]: 0x382f9054e000-0x382f9058e000
[::] debug: memory entry [0x7fffe5da5590]: 0x382f9058e000-0x382f905ce000
[::] debug: memory entry [0x7fffe5da54f8]: 0x382f905ce000-0x382f9060e000
[::] debug: memory entry [0x7fffe5da5460]: 0x382f9060e000-0x382f9064e000
[::] debug: memory entry [0x7fffe5da53c8]: 0x382f9064e000-0x382f9068e000
[::] debug: memory entry [0x7fffe5da5330]: 0x382f9068e000-0x382f906ce000
[::] debug: memory entry [0x7fffe5da5298]: 0x382f906ce000-0x382f9070e000
[::] debug: memory entry [0x7fffe5da5130]: 0x382f90922000-0x382f90923000
[::] debug: memory entry [0x7fffe5da5098]: 0x382f90923000-0x382f90924000
[::] debug: memory entry [0x7fffe5da4c30]: 0x382f90b9c000-0x382f90ba0000
[::] debug: memory entry [0x7fffe5da4888]: 0x382f90ba0000-0x382f90ba2000
[::] debug: memory entry [0x7fffe5da4720]: 0x382f91118000-0x382f9111c000
[::] debug: memory entry [0x7fffe5da4378]: 0x382f9111c000-0x382f9111e000
[::] debug: memory entry [0x7fffe5da4210]: 0x382f91361000-0x382f91362000
[::] debug: memory entry [0x7fffe5da4178]: 0x382f91362000-0x382f91363000
[::] debug: memory entry [0x7fffe5da3dd8]: 0x382f91363000-0x382f91373000
[::] debug: memory entry [0x7fffe5da3c70]: 0x382f915a4000-0x382f915a5000
[::] debug: memory entry [0x7fffe5da3bd8]: 0x382f915a5000-0x382f915a6000
[::] debug: memory entry [0x7fffe5da3b40]: 0x382f915a6000-0x382f915a8000
[::] debug: memory entry [0x7fffe5da3aa8]: 0x382f915a8000-0x382f915aa000
[::] debug: memory entry [0x7fffe5da3510]: 0x382f917d4000-0x382f917d5000
[::] debug: memory entry [0x7fffe5da3478]: 0x382f917d5000-0x382f917d6000
[::] debug: memory entry [0x7fffe5da33e0]: 0x382f917d6000-0x382f917dc000
[::] debug: memory entry [0x7fffe5da3348]: 0x382f917dc000-0x382f917e2000
[::] debug: memory entry [0x7fffe5da32b0]: 0x382f917e2000-0x382f917e8000
[::] debug: memory entry [0x7fffe5da3218]: 0x382f917e8000-0x382f917ee000
[::] debug: memory entry [0x7fffe5da3180]: 0x382f917ee000-0x382f917f4000
[::] debug: memory entry [0x7fffe5da2d28]: 0x382f9209e000-0x382f920a1000
[::] debug: memory entry [0x7fffe5da2c90]: 0x382f920a1000-0x382f920a6000
[::] debug: memory entry [0x7fffe5da27f8]: 0x382f920a6000-0x382f920a7000
[::] debug: memory entry [0x7fffe5da25c0]: 0x382f9215e000-0x382f92165000
[::] debug: memory entry [0x7fffe5da2528]: 0x382f92165000-0x382f92166000
[::] debug: memory entry [0x7fffe5da20e0]: 0x382f92370000-0x382f92371000
[::] debug: memory entry [0x7fffe5da2048]: 0x382f92371000-0x382f92374000
[::] debug: memory entry [0x7fffe5da1ba8]: 0x382f9241b000-0x382f9241c000
[::] debug: memory entry [0x7fffe5da1b10]: 0x382f9241c000-0x382f9241d000
[::] debug: memory entry [0x7fffe5da17a8]: 0x382f9241d000-0x382f9251d000
[::] debug: memory entry [0x7fffe5da1710]: 0x382f9251d000-0x382f9255d000
[P1:shim] debug: Unknown process (vmid: 0x3) disconnected
[::] debug: memory entry [0x7fffe5da1678]: 0x382f9255d000-0x382f9259d000
[::] debug: memory entry [0x7fffe5da15e0]: 0x382f9259d000-0x382f925dd000
[::] debug: memory entry [0x7fffe5da1410]: 0x382f92644000-0x382f92645000
[::] debug: memory entry [0x7fffe5da1378]: 0x382f92645000-0x382f92646000
[::] debug: memory entry [0x7fffe5da0ed0]: 0x382f926ac000-0x382f926ad000
[::] debug: memory entry [0x7fffe5da0e38]: 0x382f926ad000-0x382f926ae000
[::] debug: memory entry [0x7fffe5da0ad0]: 0x382f926ae000-0x382f926b0000
[::] debug: memory entry [0x7fffe5da0898]: 0x382f9272c000-0x382f92730000
[::] debug: memory entry [0x7fffe5da0800]: 0x382f92730000-0x382f92731000
[::] debug: memory entry [0x7fffe5da02f8]: 0x382f927b4000-0x382f927b5000
[::] debug: memory entry [0x7fffe5da0260]: 0x382f927b5000-0x382f927b6000
[::] debug: memory entry [0x7fffe5d9fd58]: 0x382f92816000-0x382f92819000
[::] debug: memory entry [0x7fffe5d9fcc0]: 0x382f92819000-0x382f9281d000
[::] debug: memory entry [0x7fffe5d9f910]: 0x382f9281d000-0x382f9291d000
[::] debug: memory entry [0x7fffe5d9f878]: 0x382f9291d000-0x382f92a1d000
[::] debug: memory entry [0x7fffe5d9f7e0]: 0x382f92a1d000-0x382f92a5d000
[::] debug: memory entry [0x7fffe5d9f748]: 0x382f92a5d000-0x382f92a9d000
[::] debug: memory entry [0x7fffe5d9f6b0]: 0x382f92a9d000-0x382f92b9d000
[::] debug: memory entry [0x7fffe5d9f618]: 0x382f92b9d000-0x382f92bdd000
[::] debug: memory entry [0x7fffe5d9f580]: 0x382f92bdd000-0x382f92c1d000
[::] debug: memory entry [0x7fffe5d9f4e8]: 0x382f92c1d000-0x382f92c5d000
[::] debug: memory entry [0x7fffe5d9f450]: 0x382f92c5d000-0x382f92c9d000
[::] debug: memory entry [0x7fffe5d9f3b8]: 0x382f92c9d000-0x382f92cdd000
[::] debug: memory entry [0x7fffe5d9f320]: 0x382f92cdd000-0x382f92ddd000
[::] debug: memory entry [0x7fffe5d9f288]: 0x382f92ddd000-0x382f92e1d000
[::] debug: memory entry [0x7fffe5d9f1f0]: 0x382f92e1d000-0x382f92e5d000
[::] debug: memory entry [0x7fffe5d9f158]: 0x382f92e5d000-0x382f92e9d000
[::] debug: memory entry [0x7fffe5d9f0c0]: 0x382f92e9d000-0x382f92edd000
[::] debug: memory entry [0x7fffe5d9ee88]: 0x382f92f06000-0x382f92f07000
[::] debug: memory entry [0x7fffe5d9edf0]: 0x382f92f07000-0x382f92f0a000
[::] debug: memory entry [0x7fffe5d9ea48]: 0x382f92f0a000-0x382f92f4a000
[::] debug: memory entry [0x7fffe5d9e9b0]: 0x382f92f4a000-0x382f92f8a000
[::] debug: memory entry [0x7fffe5d9e778]: 0x382f93010000-0x382f93019000
[::] debug: memory entry [0x7fffe5d9e6e0]: 0x382f93019000-0x382f9301d000
[::] debug: memory entry [0x7fffe5d9e370]: 0x382f9301d000-0x382f9305d000
[::] debug: memory entry [0x7fffe5d9e2d8]: 0x382f9305d000-0x382f9309d000
[::] debug: memory entry [0x7fffe5d9e240]: 0x382f930af000-0x382f930ef000
[::] debug: memory entry [0x7fffe5d9e070]: 0x382f9312d000-0x382f9312e000
[::] debug: memory entry [0x7fffe5d9dfd8]: 0x382f9312e000-0x382f9312f000
[::] debug: memory entry [0x7fffe5d9db30]: 0x382f93155000-0x382f93156000
[::] debug: memory entry [0x7fffe5d9da98]: 0x382f93156000-0x382f9315d000
[::] debug: memory entry [0x7fffe5d9d6f8]: 0x382f9315d000-0x382f9329e000
[::] debug: memory entry [0x7fffe5d9d660]: 0x382f9329e000-0x382f932de000
[::] debug: memory entry [0x7fffe5d9d5c8]: 0x382f932de000-0x382f933de000
[::] debug: memory entry [0x7fffe5d9d530]: 0x382f933de000-0x382f934de000
[::] debug: memory entry [0x7fffe5d9d498]: 0x382f934de000-0x382f935de000
[::] debug: memory entry [0x7fffe5d9d400]: 0x382f935de000-0x382f936de000
[::] debug: memory entry [0x7fffe5d9d368]: 0x382f936de000-0x382f9371e000
[::] debug: memory entry [0x7fffe5d9d2d0]: 0x382f9371e000-0x382f9375e000
[::] debug: memory entry [0x7fffe5d9d238]: 0x382f9375e000-0x382f9379e000
[::] debug: memory entry [0x7fffe5d9d1a0]: 0x382f9379e000-0x382f937de000
[::] debug: memory entry [0x7fffe5d9d108]: 0x382f937de000-0x382f9381e000
[::] debug: memory entry [0x7fffe5d9d070]: 0x382f9381e000-0x382f9385e000
[::] debug: memory entry [0x7fffe5d9cfd8]: 0x382f9385e000-0x382f9389e000
[::] debug: memory entry [0x7fffe5d9cf40]: 0x382f9389e000-0x382f938de000
[::] debug: memory entry [0x7fffe5d9cea8]: 0x382f938de000-0x382f9391e000
[::] debug: memory entry [0x7fffe5d9ce10]: 0x382f9391e000-0x382f9395e000
[::] debug: memory entry [0x7fffe5d9cd78]: 0x382f9395e000-0x382f9399e000
[::] debug: memory entry [0x7fffe5d9cce0]: 0x382f9399e000-0x382f939de000
[::] debug: memory entry [0x7fffe5d9cc48]: 0x382f939de000-0x382f93a1e000
[::] debug: memory entry [0x7fffe5d9cbb0]: 0x382f93a1e000-0x382f93a5e000
[::] debug: memory entry [0x7fffe5d9cb18]: 0x382f93a5e000-0x382f93a9e000
[::] debug: memory entry [0x7fffe5d9ca80]: 0x382f93a9e000-0x382f93ade000
[::] debug: memory entry [0x7fffe5d9c9e8]: 0x382f93ade000-0x382f93b1e000
[::] debug: memory entry [0x7fffe5d9c950]: 0x382f93b1e000-0x382f93b5e000
[::] debug: memory entry [0x7fffe5d9c8b8]: 0x382f93b5e000-0x382f93b9e000
[::] debug: memory entry [0x7fffe5d9c820]: 0x382f93b9e000-0x382f93bde000
[::] debug: memory entry [0x7fffe5d9c788]: 0x382f93bde000-0x382f93c1e000
[::] debug: memory entry [0x7fffe5d9c6f0]: 0x382f93c1e000-0x382f93c5e000
[::] debug: memory entry [0x7fffe5d9c658]: 0x382f93c5e000-0x382f93c9e000
[::] debug: memory entry [0x7fffe5d9c5c0]: 0x382f93c9e000-0x382f93cde000
[::] debug: memory entry [0x7fffe5d9c528]: 0x382f93cde000-0x382f93d1e000
[::] debug: memory entry [0x7fffe5d9c490]: 0x382f93d1e000-0x382f93d5e000
[::] debug: memory entry [0x7fffe5d9c3f8]: 0x382f93d5e000-0x382f93d9e000
[::] debug: memory entry [0x7fffe5d9c360]: 0x382f93d9e000-0x382f93dde000
[::] debug: memory entry [0x7fffe5d9c2c8]: 0x382f93dde000-0x382f93e1e000
[::] debug: memory entry [0x7fffe5d9c230]: 0x382f93e1e000-0x382f93e5e000
[::] debug: memory entry [0x7fffe5d9c198]: 0x382f93e5e000-0x382f93f5e000
[::] debug: memory entry [0x7fffe5d9bfc8]: 0x382f94016000-0x382f94017000
[::] debug: memory entry [0x7fffe5d9bf30]: 0x382f94017000-0x382f9403e000
[::] debug: memory entry [0x7fffe5d9bb68]: 0x382f9403e000-0x382f9403f000
[::] debug: memory entry [0x7fffe5d9b998]: 0x382f942e3000-0x382f9430f000
[::] debug: memory entry [0x7fffe5d9b900]: 0x382f9430f000-0x382f94311000
[::] debug: memory entry [0x7fffe5d9b588]: 0x382f94311000-0x382f94315000
[::] debug: memory entry [0x7fffe5d9b4f0]: 0x382f94315000-0x382f94355000
[::] debug: memory entry [0x7fffe5d9b458]: 0x382f94355000-0x382f94395000
[::] debug: memory entry [0x7fffe5d9b288]: 0x382f943b1000-0x382f943b2000
[::] debug: memory entry [0x7fffe5d9b1f0]: 0x382f943b2000-0x382f943b3000
[::] debug: memory entry [0x7fffe5d9abd8]: 0x382f94411000-0x382f94412000
[::] debug: memory entry [0x7fffe5d9ab40]: 0x382f94412000-0x382f94414000
[::] debug: memory entry [0x7fffe5d9a768]: 0x382f94414000-0x382f94415000
[::] debug: memory entry [0x7fffe5d9a6d0]: 0x382f94415000-0x382f94455000
[::] debug: memory entry [0x7fffe5d9a500]: 0x382f94458000-0x382f94459000
[::] debug: memory entry [0x7fffe5d9a468]: 0x382f94459000-0x382f9445a000
[::] debug: memory entry [0x7fffe5d99f80]: 0x382f94462000-0x382f94463000
[::] debug: memory entry [0x7fffe5d99ee8]: 0x382f94463000-0x382f94464000
[::] debug: memory entry [0x7fffe5d998e0]: 0x382f94472000-0x382f94473000
[::] debug: memory entry [0x7fffe5d99848]: 0x382f94473000-0x382f94474000
[::] debug: memory entry [0x7fffe5d99330]: 0x382f94482000-0x382f94483000
[::] debug: memory entry [0x7fffe5d99298]: 0x382f94483000-0x382f94484000
[::] debug: memory entry [0x7fffe5d98da8]: 0x382f9449c000-0x382f9449d000
[::] debug: memory entry [0x7fffe5d98d10]: 0x382f9449d000-0x382f944a0000
[::] debug: memory entry [0x7fffe5d98820]: 0x382f9452c000-0x382f9452d000
[::] debug: memory entry [0x7fffe5d98788]: 0x382f9452d000-0x382f94554000
[::] debug: memory entry [0x7fffe5d983d0]: 0x382f94554000-0x382f94555000
[::] debug: memory entry [0x7fffe5d98338]: 0x382f94555000-0x382f94655000
[::] debug: memory entry [0x7fffe5d982a0]: 0x382f94655000-0x382f94755000
[::] debug: memory entry [0x7fffe5d98208]: 0x382f94755000-0x382f94795000
[::] debug: memory entry [0x7fffe5d98170]: 0x382f94795000-0x382f947d5000
[::] debug: memory entry [0x7fffe5d980d8]: 0x382f947d5000-0x382f94815000
[::] debug: memory entry [0x7fffe5d97ea0]: 0x382f9482a000-0x382f9482b000
[::] debug: memory entry [0x7fffe5d97e08]: 0x382f9482b000-0x382f9482d000
[::] debug: memory entry [0x7fffe5d97918]: 0x382f94845000-0x382f94846000
[::] debug: memory entry [0x7fffe5d97880]: 0x382f94846000-0x382f94848000
[::] debug: memory entry [0x7fffe5d97388]: 0x382f94889000-0x382f9488a000
[::] debug: memory entry [0x7fffe5d972f0]: 0x382f9488a000-0x382f9488c000
[::] debug: memory entry [0x7fffe5d96f38]: 0x382f9488c000-0x382f9488d000
[::] debug: memory entry [0x7fffe5d96d68]: 0x382f948b7000-0x382f948b8000
[::] debug: memory entry [0x7fffe5d96cd0]: 0x382f948b8000-0x382f948bd000
[::] debug: memory entry [0x7fffe5d966b8]: 0x382f948d3000-0x382f948d4000
[::] debug: memory entry [0x7fffe5d96620]: 0x382f948d4000-0x382f948d5000
[::] debug: memory entry [0x7fffe5d96130]: 0x382f948d5000-0x382f94915000
[::] debug: memory entry [0x7fffe5d96098]: 0x382f94915000-0x382f94955000
[::] debug: memory entry [0x7fffe5d96000]: 0x382f94955000-0x382f94995000
[::] debug: memory entry [0x7fffe5d95f68]: 0x382f94995000-0x382f96995000
[::] debug: memory entry [0x7fffe5d95ed0]: 0x382f96995000-0x382f96996000
[::] debug: memory entry [0x7fffe5d95e38]: 0x382f96996000-0x382f969d6000
[::] debug: memory entry [0x7fffe5d95da0]: 0x382f969d6000-0x382f989d6000
[::] debug: memory entry [0x7fffe5d95d08]: 0x382f989d6000-0x382f989d7000
[::] debug: memory entry [0x7fffe5d95c70]: 0x382f989d7000-0x382f98a17000
[::] debug: memory entry [0x7fffe5d95bd8]: 0x382f98a17000-0x382f9aa17000
[::] debug: memory entry [0x7fffe5d95b40]: 0x382f9aa17000-0x382f9aa18000
[::] debug: memory entry [0x7fffe5d95aa8]: 0x382f9aa18000-0x382f9aa58000
[::] debug: memory entry [0x7fffe5d95a10]: 0x382f9aa58000-0x382f9ca58000
[::] debug: memory entry [0x7fffe5d95978]: 0x382f9ca58000-0x382f9ea58000
[::] debug: memory entry [0x7fffe5d95810]: 0x382f9ec92000-0x382f9ec93000
[::] debug: memory entry [0x7fffe5d95778]: 0x382f9ec93000-0x382f9ec94000
[::] debug: memory entry [0x7fffe5d956e0]: 0x382f9ec94000-0x382f9ec95000
[::] debug: memory entry [0x7fffe5d95268]: 0x382f9f10a000-0x382f9f10b000
[::] debug: memory entry [0x7fffe5d951d0]: 0x382f9f10b000-0x382f9f10d000
[::] debug: memory entry [0x7fffe5d95138]: 0x382f9f10d000-0x382f9f119000
[::] debug: memory entry [0x7fffe5d950a0]: 0x382f9f119000-0x382f9f122000
[::] debug: memory entry [0x7fffe5d94bc0]: 0x382fa0ebe000-0x382fa0ec4000
[::] debug: memory entry [0x7fffe5d94b28]: 0x382fa0ec4000-0x382fa0edd000
[::] debug: memory entry [0x7fffe5d94a90]: 0x382fa0edd000-0x382fa0eea000
[::] debug: memory entry [0x7fffe5d949f8]: 0x382fa0eea000-0x382fa0f48000
[::] debug: memory entry [0x7fffe5d94960]: 0x382fa0f48000-0x382fa0f8c000
[::] debug: memory entry [0x7fffe5d94350]: 0x382fa1421000-0x382fa1423000
[::] debug: memory entry [0x7fffe5d942b8]: 0x382fa1423000-0x382fa1445000
[::] debug: memory entry [0x7fffe5d94220]: 0x382fa1445000-0x382fa1465000
[::] debug: memory entry [0x7fffe5d94188]: 0x382fa1465000-0x382fa1469000
[::] debug: memory entry [0x7fffe5d940f0]: 0x382fa1469000-0x382fa146c000
[::] debug: memory entry [0x7fffe5d93d20]: 0x382fa146c000-0x382fa14ac000
[::] debug: memory entry [0x7fffe5d93b50]: 0x382fa14b2000-0x382fa14b3000
[::] debug: memory entry [0x7fffe5d93ab8]: 0x382fa14b3000-0x382fa14b4000
[::] debug: memory entry [0x7fffe5d935d8]: 0x382fa14e4000-0x382fa14e5000
[::] debug: memory entry [0x7fffe5d93540]: 0x382fa14e5000-0x382fa14e7000
[::] debug: memory entry [0x7fffe5d934a8]: 0x382fa14e7000-0x382fa14e9000
[::] debug: memory entry [0x7fffe5d93410]: 0x382fa14e9000-0x382fa14eb000
[::] debug: memory entry [0x7fffe5d92f30]: 0x382fa14eb000-0x382fa14ec000
[::] debug: memory entry [0x7fffe5d92e98]: 0x382fa14ec000-0x382fa152c000
[::] debug: memory entry [0x7fffe5d92e00]: 0x382fa152c000-0x382fa156c000
[::] debug: memory entry [0x7fffe5d92d68]: 0x382fa156c000-0x382fa15ac000
[::] debug: memory entry [0x7fffe5d92cd0]: 0x382fa15ac000-0x382fa16ac000
[::] debug: memory entry [0x7fffe5d92c38]: 0x382fa16ac000-0x382fa16ec000
[::] debug: memory entry [0x7fffe5d92ba0]: 0x382fa16ec000-0x382fa17ec000
[::] debug: memory entry [0x7fffe5d92b08]: 0x382fa17ec000-0x382fa182c000
[::] debug: memory entry [0x7fffe5d92a70]: 0x382fa182c000-0x382fa186c000
[::] debug: memory entry [0x7fffe5d929d8]: 0x382fa186c000-0x382fa18ac000
[::] debug: memory entry [0x7fffe5d92940]: 0x382fa18ac000-0x382fa18ec000
[::] debug: memory entry [0x7fffe5d928a8]: 0x382fa18ec000-0x382fa192c000
[::] debug: memory entry [0x7fffe5d92810]: 0x382fa192c000-0x382fa196c000
[::] debug: memory entry [0x7fffe5d92640]: 0x382fa1975000-0x382fa1976000
[::] debug: memory entry [0x7fffe5d925a8]: 0x382fa1976000-0x382fa1977000
[::] debug: memory entry [0x7fffe5d920b8]: 0x382fa1980000-0x382fa1981000
[::] debug: memory entry [0x7fffe5d92020]: 0x382fa1981000-0x382fa1982000
[::] debug: memory entry [0x7fffe5d91ad8]: 0x382fa19a8000-0x382fa19a9000
[::] debug: memory entry [0x7fffe5d91a40]: 0x382fa19a9000-0x382fa19ab000
[::] debug: memory entry [0x7fffe5d91440]: 0x382fa19ab000-0x382fa19ac000
[::] debug: memory entry [0x7fffe5d913a8]: 0x382fa19ac000-0x382fa19ec000
[::] debug: memory entry [0x7fffe5d91310]: 0x382fa19ec000-0x382fa1a2c000
[::] debug: memory entry [0x7fffe5d91278]: 0x382fa1a2c000-0x382fa1a6c000
[::] debug: memory entry [0x7fffe5d911e0]: 0x382fa1a6c000-0x382fa1aac000
[::] debug: memory entry [0x7fffe5d91148]: 0x382fa1aac000-0x382fa1aec000
[::] debug: memory entry [0x7fffe5d910b0]: 0x382fa1aec000-0x382fa1bec000
[::] debug: memory entry [0x7fffe5d91018]: 0x382fa1bec000-0x382fa1cec000
[::] debug: memory entry [0x7fffe5d90f80]: 0x382fa1cec000-0x382fa1dec000
[::] debug: memory entry [0x7fffe5d90ee8]: 0x382fa1dec000-0x382fa1eec000
[::] debug: memory entry [0x7fffe5d90e50]: 0x382fa1eec000-0x382fa1fec000
[::] debug: memory entry [0x7fffe5d90db8]: 0x382fa1fec000-0x382fa20ec000
[::] debug: memory entry [0x7fffe5d90d20]: 0x382fa20ec000-0x382fa21ec000
[::] debug: memory entry [0x7fffe5d90c88]: 0x382fa21ec000-0x382fa22ec000
[::] debug: memory entry [0x7fffe5d90bf0]: 0x382fa22ec000-0x382fa23ec000
[::] debug: memory entry [0x7fffe5d90b58]: 0x382fa23ec000-0x382fa24ec000
[::] debug: memory entry [0x7fffe5d90ac0]: 0x382fa24ec000-0x382fa25ec000
[::] debug: memory entry [0x7fffe5d90a28]: 0x382fa25ec000-0x382fa26ec000
[::] debug: memory entry [0x7fffe5d90990]: 0x382fa26ec000-0x382fa27ec000
[::] debug: memory entry [0x7fffe5d908f8]: 0x382fa27ec000-0x382fa28ec000
[::] debug: memory entry [0x7fffe5d90860]: 0x382fa28ec000-0x382fa29ec000
[::] debug: memory entry [0x7fffe5d907c8]: 0x382fa29ec000-0x382fa2aec000
[::] debug: memory entry [0x7fffe5d90730]: 0x382fa2aec000-0x382fa2bec000
[::] debug: memory entry [0x7fffe5d90698]: 0x382fa2bec000-0x382fa2cec000
[::] debug: memory entry [0x7fffe5d90600]: 0x382fa2cec000-0x382fa2dec000
[::] debug: memory entry [0x7fffe5d90568]: 0x382fa2dec000-0x382fa2eec000
[::] debug: memory entry [0x7fffe5d904d0]: 0x382fa2eec000-0x382fa2fec000
[::] debug: memory entry [0x7fffe5d90438]: 0x382fa2fec000-0x382fa30ec000
[::] debug: memory entry [0x7fffe5d903a0]: 0x382fa30ec000-0x382fa31ec000
[::] debug: memory entry [0x7fffe5d90308]: 0x382fa31ec000-0x382fa32ec000
[::] debug: memory entry [0x7fffe5d90270]: 0x382fa32ec000-0x382fa33ec000
[::] debug: memory entry [0x7fffe5d901d8]: 0x382fa33ec000-0x382fa34ec000
[::] debug: memory entry [0x7fffe5d90140]: 0x382fa34ec000-0x382fa35ec000
[::] debug: memory entry [0x7fffe5d8ffd8]: 0x382fa3864000-0x382fa3868000
[::] debug: memory entry [0x7fffe5d8ff40]: 0x382fa3868000-0x382fa386a000
[::] debug: memory entry [0x7fffe5d8fea8]: 0x382fa386a000-0x382fa386d000
[::] debug: memory entry [0x7fffe5d8fa30]: 0x382fa3adb000-0x382fa3add000
[::] debug: memory entry [0x7fffe5d8f998]: 0x382fa3add000-0x382fa3ae3000
[::] debug: memory entry [0x7fffe5d8f900]: 0x382fa3ae3000-0x382fa3ae5000
[::] debug: memory entry [0x7fffe5d8f868]: 0x382fa3ae5000-0x382fa3af3000
[::] debug: memory entry [0x7fffe5d8f418]: 0x382fa3dab000-0x382fa3daf000
[::] debug: memory entry [0x7fffe5d8f380]: 0x382fa3daf000-0x382fa3db1000
[::] debug: memory entry [0x7fffe5d8f2e8]: 0x382fa3db1000-0x382fa3db4000
[::] debug: memory entry [0x7fffe5d8f250]: 0x382fa3db4000-0x382fa3dbc000
[::] debug: memory entry [0x7fffe5d8edf8]: 0x382fdc7fa000-0x382fdc911000
[::] debug: memory entry [0x7fffe5d8ed60]: 0x382fdc911000-0x382fdc94f000
[::] debug: memory entry [0x7fffe5d8ecc8]: 0x382fdc94f000-0x382fdcb4f000
[::] debug: memory entry [0x7fffe5d8ec30]: 0x382fdcb4f000-0x382fdd6e8000
[::] debug: memory entry [0x7fffe5d8eb98]: 0x382fdd6e8000-0x382fde218000
[::] debug: memory entry [0x7fffe5d8eb00]: 0x382fde218000-0x382fded48000
[::] debug: memory entry [0x7fffe5d8e6a0]: 0x382ff54ab000-0x382ff5830000
[::] debug: memory entry [0x7fffe5d8e608]: 0x382ff5830000-0x382ff59cc000
[::] debug: memory entry [0x7fffe5d8e570]: 0x382ff59cc000-0x382ff5ad1000
[::] debug: memory entry [0x7fffe5d8e4d8]: 0x382ff5ad1000-0x382ff600d000
[::] debug: memory entry [0x7fffe5d8e440]: 0x382ff600d000-0x382ff64ca000
[::] debug: memory entry [0x7fffe5d8dfe8]: 0x382ff66dc000-0x382ff66dd000
[::] debug: memory entry [0x7fffe5d8df50]: 0x382ff66dd000-0x382ff66de000
[::] debug: memory entry [0x7fffe5d8daf8]: 0x382ff68e3000-0x382ff68e4000
[::] debug: memory entry [0x7fffe5d8da60]: 0x382ff68e4000-0x382ff68e5000
[::] debug: memory entry [0x7fffe5d8d610]: 0x382ff75f1000-0x382ff760a000
[::] debug: memory entry [0x7fffe5d8d578]: 0x382ff760a000-0x382ff7653000
[::] debug: memory entry [0x7fffe5d8d4e0]: 0x382ff7653000-0x382ff76ac000
[::] debug: memory entry [0x7fffe5d8d448]: 0x382ff76ac000-0x382ff7785000
[::] debug: memory entry [0x7fffe5d8cfe0]: 0x382ff7985000-0x382ff7986000
[::] debug: memory entry [0x7fffe5d8cf48]: 0x382ff7986000-0x382ff7987000
[::] debug: memory entry [0x7fffe5d8ceb0]: 0x382ff7987000-0x382ff7988000
[::] debug: memory entry [0x7fffe5d8c960]: 0x382ff7b59000-0x382ff7b64000
[::] debug: memory entry [0x7fffe5d8c8c8]: 0x382ff7b64000-0x382ff7b67000
[::] debug: memory entry [0x7fffe5d8c558]: 0x382ff7b67000-0x382ff7b6a000
[::] debug: memory entry [0x7fffe5d8c3f0]: 0x382ff7d8e000-0x382ff7d8f000
[::] debug: memory entry [0x7fffe5d8c358]: 0x382ff7d8f000-0x382ff7d90000
[::] debug: memory entry [0x7fffe5d8c2c0]: 0x382ff7d90000-0x382ff7d94000
[::] debug: memory entry [0x7fffe5d8be58]: 0x382ff7f9c000-0x382ff7f9d000
[::] debug: memory entry [0x7fffe5d8bdc0]: 0x382ff7f9d000-0x382ff7f9e000
[::] debug: memory entry [0x7fffe5d8b948]: 0x382ff819e000-0x382ff819f000
[::] debug: memory entry [0x7fffe5d8b8b0]: 0x382ff819f000-0x382ff81a0000
[::] debug: memory entry [0x7fffe5d8b818]: 0x382ff81a0000-0x382ff81a1000
[::] debug: memory entry [0x7fffe5d8b780]: 0x382ff81a1000-0x382ff81a2000
[::] debug: memory entry [0x7fffe5d8ad48]: 0x382ff81a2000-0x382ff81e2000
[::] debug: memory entry [0x7fffe5d8acb0]: 0x382ff81e2000-0x382ff8222000
[::] debug: memory entry [0x7fffe5d8ac18]: 0x382ff8222000-0x382ff8322000
[::] debug: memory entry [0x7fffe5d8ab80]: 0x382ff8322000-0x382ff8422000
[::] debug: memory entry [0x7fffe5d8aae8]: 0x382ff8422000-0x382ff8462000
[::] debug: memory entry [0x7fffe5d8aa50]: 0x382ff8462000-0x382ff84a2000
[::] debug: memory entry [0x7fffe5d8a9b8]: 0x382ff84a2000-0x382ff84e2000
[::] debug: memory entry [0x7fffe5d8a7e8]: 0x382ff84f8000-0x382ff84f9000
[::] debug: memory entry [0x7fffe5d8a750]: 0x382ff84f9000-0x382ff84fa000
[::] debug: memory entry [0x7fffe5d8a270]: 0x382ff84fd000-0x382ff84fe000
[::] debug: memory entry [0x7fffe5d8a1d8]: 0x382ff84fe000-0x382ff84ff000
[::] debug: memory entry [0x7fffe5d89cf8]: 0x382ff8518000-0x382ff8519000
[::] debug: memory entry [0x7fffe5d89c60]: 0x382ff8519000-0x382ff851a000
[::] debug: memory entry [0x7fffe5d89750]: 0x382ff8541000-0x382ff8542000
[::] debug: memory entry [0x7fffe5d896b8]: 0x382ff8542000-0x382ff8543000
[::] debug: memory entry [0x7fffe5d89218]: 0x382ff854d000-0x382ff854e000
[::] debug: memory entry [0x7fffe5d89180]: 0x382ff854e000-0x382ff854f000
[::] debug: memory entry [0x7fffe5d88ca0]: 0x382ff8560000-0x382ff8561000
[::] debug: memory entry [0x7fffe5d88c08]: 0x382ff8561000-0x382ff8562000
[::] debug: memory entry [0x7fffe5d88898]: 0x382ff8562000-0x382ff85a2000
[::] debug: memory entry [0x7fffe5d88800]: 0x382ff85a2000-0x382ff85e2000
[::] debug: memory entry [0x7fffe5d88768]: 0x382ff85e2000-0x382ff8622000
[::] debug: memory entry [0x7fffe5d88598]: 0x382ff8629000-0x382ff862a000
[::] debug: memory entry [0x7fffe5d88500]: 0x382ff862a000-0x382ff862b000
[::] debug: memory entry [0x7fffe5d88048]: 0x382ff8631000-0x382ff8632000
[::] debug: memory entry [0x7fffe5d87fb0]: 0x382ff8632000-0x382ff8633000
[::] debug: memory entry [0x7fffe5d87a70]: 0x382ff863d000-0x382ff863e000
[::] debug: memory entry [0x7fffe5d879d8]: 0x382ff863e000-0x382ff863f000
[::] debug: memory entry [0x7fffe5d874d0]: 0x382ff865d000-0x382ff865e000
[::] debug: memory entry [0x7fffe5d87438]: 0x382ff865e000-0x382ff8662000
[::] debug: memory entry [0x7fffe5d86e50]: 0x382ff8662000-0x382ff86a2000
[::] debug: memory entry [0x7fffe5d86db8]: 0x382ff86a2000-0x382ff86e2000
[::] debug: memory entry [0x7fffe5d86d20]: 0x382ff86e2000-0x382ff8722000
[::] debug: memory entry [0x7fffe5d86c88]: 0x382ff8722000-0x382ff8762000
[::] debug: memory entry [0x7fffe5d86bf0]: 0x382ff8762000-0x382ff8862000
[::] debug: memory entry [0x7fffe5d86b58]: 0x382ff8862000-0x382ff88a2000
[::] debug: memory entry [0x7fffe5d86ac0]: 0x382ff88a2000-0x382ff88e2000
[::] debug: memory entry [0x7fffe5d86a28]: 0x382ff88e2000-0x382ff8922000
[::] debug: memory entry [0x7fffe5d86990]: 0x382ff8922000-0x382ff8962000
[::] debug: memory entry [0x7fffe5d868f8]: 0x382ff8962000-0x382ff89a2000
[::] debug: memory entry [0x7fffe5d86860]: 0x382ff89a2000-0x382ff89e2000
[::] debug: memory entry [0x7fffe5d867c8]: 0x382ff89e2000-0x382ff8a22000
[::] debug: memory entry [0x7fffe5d86730]: 0x382ff8a22000-0x382ff8a62000
[::] debug: memory entry [0x7fffe5d86698]: 0x382ff8a62000-0x382ff8aa2000
[::] debug: memory entry [0x7fffe5d86600]: 0x382ff8aa2000-0x382ff8ba2000
[::] debug: memory entry [0x7fffe5d86568]: 0x382ff8ba2000-0x382ff8be2000
[::] debug: memory entry [0x7fffe5d864d0]: 0x382ff8be2000-0x382ff8c22000
[::] debug: memory entry [0x7fffe5d86438]: 0x382ff8c22000-0x382ff8c25000
[::] debug: memory entry [0x7fffe5d86200]: 0x382ff8c3f000-0x382ff8c40000
[::] debug: memory entry [0x7fffe5d86168]: 0x382ff8c40000-0x382ff8c41000
[::] debug: memory entry [0x7fffe5d85e00]: 0x382ff8c41000-0x382ff8c43000
[::] debug: memory entry [0x7fffe5d85bc8]: 0x382ff8c6e000-0x382ff8c70000
[::] debug: memory entry [0x7fffe5d85b30]: 0x382ff8c70000-0x382ff8c71000
[::] debug: memory entry [0x7fffe5d85620]: 0x382ff8d4a000-0x382ff8d4b000
[::] debug: memory entry [0x7fffe5d85588]: 0x382ff8d4b000-0x382ff8d4c000
[::] debug: memory entry [0x7fffe5d850d0]: 0x382ff8d4f000-0x382ff8d50000
[::] debug: memory entry [0x7fffe5d85038]: 0x382ff8d50000-0x382ff8d51000
[::] debug: memory entry [0x7fffe5d84b78]: 0x382ff8d54000-0x382ff8d55000
[::] debug: memory entry [0x7fffe5d84ae0]: 0x382ff8d55000-0x382ff8d56000
[::] debug: memory entry [0x7fffe5d84628]: 0x382ff8d59000-0x382ff8d5a000
[::] debug: memory entry [0x7fffe5d84590]: 0x382ff8d5a000-0x382ff8d5b000
[::] debug: memory entry [0x7fffe5d840d0]: 0x382ff8f41000-0x382ff8f44000
[::] debug: memory entry [0x7fffe5d84038]: 0x382ff8f44000-0x382ff8f47000
[::] debug: memory entry [0x7fffe5d83cb8]: 0x382ff8f47000-0x382ff8f4f000
[::] debug: memory entry [0x7fffe5d83c20]: 0x382ff8f4f000-0x382ff8f51000
[::] debug: memory entry [0x7fffe5d83b88]: 0x382ff8f51000-0x382ff8f52000
[::] debug: memory entry [0x7fffe5d839b8]: 0x382ff8f81000-0x382ff8f82000
[::] debug: memory entry [0x7fffe5d83920]: 0x382ff8f82000-0x382ff8f83000
[::] debug: memory entry [0x7fffe5d83888]: 0x382ff8f83000-0x382ff8f85000
[::] debug: memory entry [0x7fffe5d834f0]: 0x382ff8f85000-0x382ff8f86000
[::] debug: memory entry [0x7fffe5d83458]: 0x382ff8f86000-0x382ff8fc6000
[::] debug: restored memory from checkpoint
[::] debug: receiving 84 PAL handles
[::] debug: restoring checkpoint at 0x7fffe5d81000 rebased from 0x7fffe5d81000
[::] debug: Creating pipe: pipe.srv:9d43b171633bbeadb240734e8ea17119eea5601c41175cbc8d932ee259147445
[P4:T9:python3.8] debug: successfully restored checkpoint at 0x7fffe5d81000 - 0x7fffe5dad680
[P4:T9:python3.8] debug: Creating pipe: pipe.srv:57e016d238018f65f0e4af01bc5a3f9535e747cd126c458cfc0dc441e61ccdea
[P4:T9:python3.8] debug: Creating pipe: pipe.srv:4
[P4:T9:python3.8] debug: Creating pipe: pipe.srv:f8d950c59dba0e2239df671298df237b8782352f2cfa81e0525d067140aff492
[P4:T9:python3.8] debug: ipc_get_id_owner: sending a request: 0
[P4:T9:python3.8] debug: Sending ipc message to 1
[P4:T9:python3.8] debug: Waiting for a response to 1
[P1:shim] debug: IPC worker: received IPC message from 4: code=6 size=21 seq=1
[P1:shim] debug: ipc_get_id_owner_callback: find_id_owner(0): 0
[P1:shim] debug: Sending ipc message to 4
[P4:shim] debug: IPC worker started
[P4:shim] debug: IPC worker: received IPC message from 1: code=0 size=21 seq=1
[P4:shim] debug: Got an IPC response from 1, seq: 1
[P4:T9:python3.8] debug: Waiting finished: 0
[P4:T9:python3.8] debug: ipc_get_id_owner: got a response: 0
[P1:T1:python3.8] trace: ---- return from shim_clone(...) = 0x9
[P1:T1:python3.8] trace: ---- shim_close(8) = 0x0
[P1:T1:python3.8] trace: ---- shim_close(9) = 0x0
[P1:T1:python3.8] trace: ---- shim_fstat(7, 0x382ff8fc3bd0) = 0x0
[P1:T1:python3.8] trace: ---- shim_ioctl(7, TCGETS, 0x382ff8fc3c30) ...
[P1:T1:python3.8] trace: ---- return from shim_ioctl(...) = -38
[P1:T1:python3.8] trace: ---- shim_lseek(7, 0x0, SEEK_CUR) = -29
[P1:T1:python3.8] trace: ---- shim_write(7, 0x382f8ff524e0, 0x353) ...
[P1:T1:python3.8] trace: ---- return from shim_write(...) = 0x353
[P1:T1:python3.8] trace: ---- shim_getpid() = 0x1
[P1:T1:python3.8] trace: ---- shim_close(6) = 0x0
[P1:T1:python3.8] trace: ---- shim_close(5) = 0x0
[P1:T1:python3.8] trace: ---- shim_getpid() = 0x1
[P1:T1:python3.8] trace: ---- shim_wait4(9, 0x382ff8fc427c, 0, 0) ...
[P4:T9:python3.8] debug: Shim process initialized
[P4:T9:python3.8] trace: ---- shim_set_robust_list(0x382ff8c22a20, 0x18) = 0x0
[P4:T9:python3.8] trace: ---- shim_fcntl(4, F_GETFD, 0) = 0x1
[P4:T9:python3.8] trace: ---- shim_fcntl(4, F_SETFD, 0) = 0x0
[P4:T9:python3.8] trace: ---- shim_fcntl(5, F_GETFD, 0) = 0x1
[P4:T9:python3.8] trace: ---- shim_fcntl(5, F_SETFD, 0) = 0x0
[P4:T9:python3.8] trace: ---- shim_fcntl(6, F_GETFD, 0) = 0x1
[P4:T9:python3.8] trace: ---- shim_fcntl(6, F_SETFD, 0) = 0x0
[P4:T9:python3.8] trace: ---- shim_close(8) = 0x0
[P4:T9:python3.8] trace: ---- shim_openat(AT_FDCWD, "/proc/self/fd", O_RDONLY|0x80000, 0000) = 0x8
[P4:T9:python3.8] trace: ---- shim_getdents64(8, 0x382ff8fc3ac0, 0x118) = 0x100
[P4:T9:python3.8] trace: ---- shim_close(9) = 0x0
[P4:T9:python3.8] trace: ---- shim_close(7) = 0x0
[P4:T9:python3.8] trace: ---- shim_getdents64(8, 0x382ff8fc3ac0, 0x118) = 0x80
[P4:T9:python3.8] trace: ---- shim_close(3) = 0x0
[P4:T9:python3.8] trace: ---- shim_getdents64(8, 0x382ff8fc3ac0, 0x118) = 0x0
[P4:T9:python3.8] trace: ---- shim_close(8) = 0x0
[P4:T9:python3.8] trace: ---- shim_execve("/pytorch", [/pytorch,-c,from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=4, pipe_handle=6),--multiprocessing-fork,], [SHELL=/bin/bash,SESSION_MANAGER=local/gagan-VirtualBox:@/tmp/.ICE-unix/1691,unix/gagan-VirtualBox:/tm
[P4:T9:python3.8] trace: p/.ICE-unix/1691,]) ...
[P4:T9:python3.8] trace: ---- return from shim_execve(...) = -2
[P4:T9:python3.8] trace: ---- shim_write(9, 0x82d79f, 0x8) ...
[P4:T9:python3.8] trace: ---- return from shim_write(...) = -9
[P4:T9:python3.8] trace: ---- shim_write(9, 0x382ff8fc3cf8, 0x1) ...
[P4:T9:python3.8] trace: ---- return from shim_write(...) = -9
[P4:T9:python3.8] trace: ---- shim_write(9, 0x82d7c0, 0x1) ...
[P4:T9:python3.8] trace: ---- return from shim_write(...) = -9
[P4:T9:python3.8] debug: ---- shim_exit_group (returning 255)
[P4:T9:python3.8] debug: Sending ipc message to 1
[P4:T9:python3.8] debug: Waiting for a response to 2
[P1:shim] debug: IPC worker: received IPC message from 4: code=17 size=21 seq=2
[P1:shim] debug: clearing POSIX locks for pid 9
[P1:shim] debug: Sending ipc message to 4
[P4:shim] debug: IPC worker: received IPC message from 1: code=0 size=21 seq=2
[P4:shim] debug: Got an IPC response from 1, seq: 2
[P4:T9:python3.8] debug: Waiting finished: 0
[P4:T9:python3.8] debug: Sending ipc message to 1
[P4:T9:python3.8] debug: sync client shutdown: closing handles
[P4:T9:python3.8] debug: sync client shutdown: waiting for confirmation
[P4:T9:python3.8] debug: sync client shutdown: finished
[P4:T9:python3.8] debug: ipc_release_id_range: sending a request: [9..9]
[P4:T9:python3.8] debug: Sending ipc message to 1
[P4:T9:python3.8] debug: ipc_release_id_range: ipc_send_message: 0
[P1:shim] debug: IPC worker: received IPC message from 4: code=2 size=37 seq=0
[P1:shim] debug: IPC callback from 4: IPC_MSG_CHILDEXIT(1, 9, 255, 0)
[P1:shim] debug: Child process (pid: 9) died
[P1:shim] debug: IPC worker: received IPC message from 4: code=4 size=25 seq=0
[P1:shim] debug: ipc_release_id_range_callback: release_id_range(9..9)
[P4:shim] debug: IPC worker: exiting worker thread
[P1:T1:python3.8] trace: ---- return from shim_wait4(...) = -512
[P1:T1:python3.8] trace: ---- shim_wait4(9, 0x382ff8fc427c, 0, 0) ...
[P1:T1:python3.8] trace: ---- return from shim_wait4(...) = 0x9
[P1:T1:python3.8] trace: ---- shim_rt_sigaction([SIGINT], 0x382ff8fc4b30, 0x382ff8fc4bd0, 0x8) = 0x0
[P4:T9:python3.8] debug: process 4 exited with status 255
[P1:shim] debug: Unknown process (vmid: 0x4) disconnected
[P1:T1:python3.8] trace: ---- shim_sigaltstack(0, 0x382ff8fc4dc0) = 0x0
[P1:T1:python3.8] trace: ---- shim_sigaltstack(0x95ed10, 0) = 0x0
[P1:T1:python3.8] trace: ---- shim_munmap(0x382f9ca58000, 0x2000000) ...
[P1:T1:python3.8] trace: ---- return from shim_munmap(...) = 0x0
[P1:T1:python3.8] trace: ---- shim_munmap(0x382f9aa58000, 0x2000000) ...
[P1:T1:python3.8] trace: ---- return from shim_munmap(...) = 0x0
[P1:T1:python3.8] trace: ---- shim_munmap(0x382f98a17000, 0x2000000) ...
[P1:T1:python3.8] trace: ---- return from shim_munmap(...) = 0x0
[P1:T1:python3.8] trace: ---- shim_munmap(0x382f969d6000, 0x2000000) ...
[P1:T1:python3.8] trace: ---- return from shim_munmap(...) = 0x0
[P1:T1:python3.8] trace: ---- shim_munmap(0x382f94995000, 0x2000000) ...
[P1:T1:python3.8] trace: ---- return from shim_munmap(...) = 0x0
[P1:T1:python3.8] debug: ---- shim_exit_group (returning 0)
[P1:T1:python3.8] debug: clearing POSIX locks for pid 1
[P1:T1:python3.8] debug: sync client shutdown: closing handles
[P1:T1:python3.8] debug: sync client shutdown: waiting for confirmation
[P1:T1:python3.8] debug: sync client shutdown: finished
[P1:shim] debug: Async worker thread terminated
[P1:shim] debug: IPC worker: exiting worker thread
[P1:T1:python3.8] debug: process 1 exited with status 0
]0;gagan@gagan-VirtualBox: ~/examples/pytorch[01;32mgagan@gagan-VirtualBox[00m:[01;34m~/examples/pytorch[00m$ exit
exit

Script done on 2022-08-05 15:04:28+02:00 [COMMAND_EXIT_CODE="0"]

gagandeep987123 avatar Aug 05 '22 13:08 gagandeep987123

The output is like 4.2 MB file with 55k lines. I am pasting last few lines of it as Github comment doesn't allow pasting the full output.

You can attach the file, you don't have to paste the log inline.

There is no error message in this log, not like the one you pasted before. But I've noticed that execve("/pytorch", ... fails. Are you sure your manifest is correct with everything mounted properly? You can read more here: https://gramine.readthedocs.io/en/stable/manifest-syntax.html

boryspoplawski avatar Aug 05 '22 13:08 boryspoplawski

The earlier output was from the original code. I thought you wanted to see that. Attaching following the one with the reported error. output_approach.zip

gagandeep987123 avatar Aug 05 '22 13:08 gagandeep987123

Please first check your manifest file. As for the error in poll - hard to say, there is not much happening there, although there are some socket flag warning - for this I would recommend building Gramine manually and trying the code from current master - networking code was rewritten recently.

.rtf? This was supposed to be a simple text file.

The earlier output was from the original code. I thought you wanted to see that.

Sure, but I would expect some error, otherwise what's the problem?

boryspoplawski avatar Aug 05 '22 13:08 boryspoplawski

Can you please share any resource for that "I would recommend building Gramine manually and trying the code from the current master" I installed gramine using "https://gramine.readthedocs.io/en/latest/quickstart.html#install-gramine"

gagandeep987123 avatar Aug 05 '22 14:08 gagandeep987123

There's a building section on the same site you've linked: https://gramine.readthedocs.io/en/latest/devel/building.html

boryspoplawski avatar Aug 05 '22 14:08 boryspoplawski

I am still getting the error after building it from source. Attaching the log file. output_approach.zip

gagandeep987123 avatar Aug 05 '22 20:08 gagandeep987123

The "std::system_error" error is caused by unsupported sendto flag. strace result:

[pid 3841816] sendto(10, "\1\0\0\0\0\0\0\0", 8, MSG_NOSIGNAL|MSG_MORE, NULL, 0 <unfinished ...>
[pid 3841816] <... sendto resumed>)     = 8

gramine-direct result:

[P3:T10:python3.8] trace: ---- sendto(9, 0x32819dc210c8, 0x8, 49152, 0, 0) = -95

After I comment out the ckecking https://github.com/gramineproject/gramine/blob/d8217c8ad08fc2be289384b90100e29c0506c097/libos/src/sys/libos_socket.c#L616-L618 It can work. However, python rpc_parameter_server.py --world_size=2 --rank=1 will cause a gramine internal error. Need more investigation.

llly avatar Aug 15 '22 09:08 llly

Met same Pytorch issue again in other cases. https://github.com/pytorch/pytorch/blob/master/torch/csrc/distributed/c10d/Utils.hpp#L552 Pytouch add MSG_MORE to sendto flag. Gramine should ignore this flag instead of reporting error if it won't be supported.

llly avatar Sep 20 '22 02:09 llly