sgx-lkl
sgx-lkl copied to clipboard
lsds
Increase default LKL memory to 64M?
cryptsetup allocates/uses LKL memory (and not enclave mmap memory). The default size for LKL visible memory is 32MB. Using dm-crypt+dm-integrity I stumbled over the following. Increasing the size to 64M fixed it. Should 64M become a new default? It may be nontrivial for users to diagnose this themselves.
[ SGX-LKL ] SGX-LKL (OE) Git version fc08365-dirty LKL version 5.3.0 [DEBUG build (-O0)] [HARDWARE DEBUG]
[ SGX-LKL ] nproc=4 ETHREADS=4 CMDLINE="mem=32M" GETTIME_VDSO=0
[ SGX-LKL ] HW TLS support: conf->fsgsbase=1
[ SGX-LKL ] Registering disk 0 (path='.../disk1.img', mnt='/', [RW ])
[ SGX-LKL ] Registering disk 1 (path='.../disk2.img', mnt='/data/input', [RW ])
[ SGX-LKL ] Registering disk 2 (path='.../disk3.img', mnt='/data/output', [RW ])
[ SGX-LKL ] get_signed_libsgxlkl_path... result=/opt/sgx-lkl/lib/libsgxlkl.so.signed
[ SGX-LKL ] oe_create_enclave...
[ SGX-LKL ] sgxlkl_enclave_init(ethread_id=0)
[[ SGX-LKL ]] sgxlkl_enclave_init(): enter
[[ SGX-LKL ]] sgxlkl_enclave_show_attribute(): enclave base=0x7fbc00000000 size=8.042 GB
[[ SGX-LKL ]] sgxlkl_enclave_show_attribute(): enclave heap base=0x7fbc00b3f000 size=8.000 GB end=0x7fbe00b3f000
[[ SGX-LKL ]] sgxlkl_enclave_init(): calling _dlstart_c()
[[ SGX-LKL ]] _register_enclave_signal_handlers(): Registering OE exception handler...
[[ SGX-LKL ]] lkl_start_init(): kernel command line: 'mem=32M console=hvc0'
[[ SGX-LKL ]] lkl_start_init(): lkl_start_kernel() called
[ 0.000000] Linux version 5.3.0+ (@1784b8ead1a7) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #1 Thu Apr 23 10:42:08 UTC 2020
[ 0.000000] memblock address range: 0x7fbdfe7bc000 - 0x7fbe007bb000
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 8079
[ 0.000000] Kernel command line: mem=32M console=hvc0 virtio_mmio.device=256@0x1000000:1 virtio_mmio.device=273@0x2000000:2
[ 0.000000] Dentry cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] Memory available: 32172k/32764k RAM
[ 0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] NR_IRQS: 4096
[ 0.000000] lkl: irqs initialized
[ 0.000000] clocksource: lkl: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[ 0.000015] lkl: time and timers initialized (irq3)
[ 0.000701] printk: console [hvc0] enabled
[ 0.000743] pid_max: default: 4096 minimum: 301
[ 0.000825] Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[ 0.000891] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[ 0.003983] devtmpfs: initialized
[ 0.007684] random: get_random_bytes called from 0x00007fbc0065ab07 with crng_init=0
[ 0.007809] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.007889] xor: automatically using best checksumming function 8regs
[ 0.008125] NET: Registered protocol family 16
[ 0.015031] clocksource: Switched to clocksource lkl
[ 0.015450] NET: Registered protocol family 2
[ 0.016292] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes, linear)
[ 0.016352] TCP established hash table entries: 512 (order: 0, 4096 bytes, linear)
[ 0.016408] TCP bind hash table entries: 512 (order: 0, 4096 bytes, linear)
[ 0.016455] TCP: Hash tables configured (established 512 bind 512)
[ 0.016553] UDP hash table entries: 128 (order: 0, 4096 bytes, linear)
[ 0.016603] UDP-Lite hash table entries: 128 (order: 0, 4096 bytes, linear)
[ 0.016694] NET: Registered protocol family 1
[ 0.016799] virtio-mmio: Registering device virtio-mmio.0 at 0x1000000-0x10000ff, IRQ 1.
[ 0.016875] virtio-mmio: Registering device virtio-mmio.1 at 0x2000000-0x2000110, IRQ 2.
[ 0.017166] workingset: timestamp_bits=62 max_order=14 bucket_order=0
[ 0.018449] NET: Registered protocol family 38
[ 0.018513] io scheduler mq-deadline registered
[ 0.018552] io scheduler kyber registered
[ 0.124788] software IO TLB: mapped [mem 0x7fc054565000-0x7fc058565000] (64MB)
[ 0.126937] random: fast init done
[ 0.137450] loop: module loaded
[ 0.137949] device-mapper: ioctl: 4.40.0-ioctl (2019-01-18) initialised: [email protected]
[ 0.146540] wireguard: WireGuard 0.0.20191219 loaded. See www.wireguard.com for information.
[ 0.146603] wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
[ 0.146921] NET: Registered protocol family 10
[ 0.147761] Segment Routing with IPv6
[ 0.147819] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[ 0.148451] This architecture does not have kernel memory protection.
[ 0.148499] Run /init as init process
[[ SGX-LKL ]] lkl_start_init(): lkl_start_kernel() finished
[[ SGX-LKL ]] lkl_start_init(): creating LKL termination thread
[[ SGX-LKL ]] create_lkl_termination_thread(): enter
[[ SGX-LKL ]] lkl_start_init(): calling lkl_mount_virtial()
[[ SGX-LKL ]] lkl_termination_thread(): enter
[[ SGX-LKL ]] lkl_start_init(): calling init_random()
[[ SGX-LKL ]] init_random(): Adding entropy to entropy pool
[[ SGX-LKL ]] lkl_termination_thread(): Performed LKL syscall to get host task allocated (pid=36)
[ 0.149748] random: crng init done
[[ SGX-LKL ]] wg0 has public key BPY74ddJHpWW42M5wWmlFIuOGl95auTNPZJUTXvXKXs=
[[ SGX-LKL ]] aas_release_resources(): aas_release_resources: deallocate all resources
[ 0.153640] virtio_blk virtio2: [vda] 2017648 512-byte logical blocks (1.03 GB/985 MiB)
[ 0.154317] virtio_blk virtio3: [vdb] 9026720 512-byte logical blocks (4.62 GB/4.30 GiB)
[ 0.154735] virtio_blk virtio4: [vdc] 6291456 512-byte logical blocks (3.22 GB/3.00 GiB)
[[ SGX-LKL ]] lkl_mount_disk(): lkl_mount_disk(dev="/dev/vda", mnt="/mnt/vda", ro=0)
[[ SGX-LKL ]] lkl_mount_disk(): Activating crypto disk
[ 5.835475] EXT4-fs (dm-1): mounted filesystem without journal. Opts:
[[ SGX-LKL ]] lkl_mount_disk(): lkl_mount_disk(dev="/dev/vdb", mnt="/data/input", ro=0)
[[ SGX-LKL ]] lkl_mount_disk(): Activating crypto disk
[ 11.471980] host3 invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
[ 11.472047] Call Trace:
[ 11.472070] 0000000045b62742: [<7fbc000d9123>] 0x00007fbc000d9123
[ 11.472115] 000000002184529f: [<7fbc00086876>] 0x00007fbc00086876
[ 11.472170] 000000008c1b9c1e: [<7fbc003ee471>] 0x00007fbc003ee471
[ 11.472221] 00000000a05b6691: [<7fbc000d989b>] 0x00007fbc000d989b
[ 11.472263] 00000000bc9f1796: [<7fbc000d9489>] 0x00007fbc000d9489
[ 11.472321] 00000000c9321b8e: [<7fbc000d9c58>] 0x00007fbc000d9c58
[ 11.472367] 00000000b535166a: [<7fbc000f2f6b>] 0x00007fbc000f2f6b
[ 11.472413] 0000000074feb5d2: [<7fbc000fabf1>] 0x00007fbc000fabf1
[ 11.472459] 00000000c66dbb31: [<7fbc002818be>] 0x00007fbc002818be
[ 11.472504] 00000000981a4c0b: [<7fbc002819b6>] 0x00007fbc002819b6
[ 11.472551] 00000000dc7b3603: [<7fbc0020df9e>] 0x00007fbc0020df9e
[ 11.472597] 0000000032178354: [<7fbc0027138f>] 0x00007fbc0027138f
[ 11.472644] 0000000054c3665b: [<7fbc0027f735>] 0x00007fbc0027f735
[ 11.472690]
[ 11.472712] Mem-Info:
[ 11.472736] active_anon:0 inactive_anon:0 isolated_anon:0
[ 11.472736] active_file:1 inactive_file:1 isolated_file:0
[ 11.472736] unevictable:1 dirty:0 writeback:0 unstable:0
[ 11.472736] slab_reclaimable:256 slab_unreclaimable:1410
[ 11.472736] mapped:0 shmem:0 pagetables:0 bounce:0
[ 11.472736] free:179 free_pcp:0 free_cma:0
[ 11.473930] Node 0 active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:4kB unevictable:4kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[ 11.474064] Normal free:716kB min:716kB low:892kB high:1068kB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:4kB unevictable:4kB writepending:0kB present:32764kB managed:32172kB mlocked:0kB kernel_stack:168kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 11.474375] lowmem_reserve[]: 0 0
[ 11.474405] Normal: 1*4kB (U) 1*8kB (M) 2*16kB (UM) 1*32kB (U) 0*64kB 1*128kB (M) 0*256kB 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 716kB
[ 11.474489] 3 total pagecache pages
[ 11.474519] 8191 pages RAM
[ 11.474540] 0 pages HighMem/MovableOnly
[ 11.474570] 148 pages reserved
[ 11.474600] Unreclaimable slab info:
[ 11.474787] Name Used Total
[ 11.474824] bio-4 34KB 108KB
[ 11.474861] bio-3 8KB 15KB
[ 11.474899] bio-2 11KB 26KB
[ 11.474937] RAWv6 15KB 15KB
[ 11.474984] UDPv6 29KB 47KB
[ 11.475024] bio-1 3KB 3KB
[ 11.475097] RAW 7KB 7KB
[ 11.475282] UDP 7KB 7KB
[ 11.475321] request_queue 31KB 31KB
[ 11.475360] biovec-max 672KB 672KB
[ 11.475398] biovec-64 8KB 32KB
[ 11.475436] bio_integrity_payload 31KB 43KB
[ 11.475474] skbuff_head_cache 3KB 3KB
[ 11.475512] file_lock_cache 3KB 3KB
[ 11.475549] file_lock_ctx 3KB 3KB
[ 11.475739] proc_dir_entry 1559KB 1559KB
[ 11.475777] pde_opener 3KB 3KB
[ 11.475811] seq_file 3KB 3KB
[ 11.475850] kernfs_node_cache 1087KB 1087KB
[ 11.475888] mnt_cache 7KB 7KB
[ 11.475926] names_cache 32KB 32KB
[ 11.475964] nsproxy 6KB 7KB
[ 11.476002] vm_region 3KB 3KB
[ 11.476176] vm_area_struct 3KB 3KB
[ 11.476212] mm_struct 7KB 7KB
[ 11.476251] fs_cache 6KB 8KB
[ 11.476289] files_cache 7KB 7KB
[ 11.476327] signal_cache 39KB 39KB
[ 11.476366] sighand_cache 61KB 61KB
[ 11.476405] task_struct 61KB 63KB
[ 11.476594] cred_jar 8KB 8KB
[ 11.476634] pid 7KB 7KB
[ 11.476676] pool_workqueue 138KB 160KB
[ 11.476714] kmalloc-8k 120KB 128KB
[ 11.476762] kmalloc-4k 128KB 160KB
[ 11.476801] kmalloc-2k 256KB 256KB
[ 11.476839] kmalloc-1k 230KB 232KB
[ 11.476877] kmalloc-512 368KB 368KB
[ 11.477065] kmalloc-256 180KB 180KB
[ 11.477104] kmalloc-192 31KB 31KB
[ 11.477143] kmalloc-128 18KB 36KB
[ 11.477180] kmalloc-96 19KB 19KB
[ 11.477220] kmalloc-64 15KB 16KB
[ 11.477258] kmalloc-32 9KB 16KB
[ 11.477296] kmalloc-16 4KB 4KB
[ 11.477334] kmalloc-8 20KB 20KB
[ 11.477529] kmem_cache_node 8KB 8KB
[ 11.477564] kmem_cache 23KB 23KB
[ 11.477606] Tasks state (memory values in pages):
[ 11.477642] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 11.477706] [ 36] 0 36 162 0 0 0 0 host0
[ 11.477760] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),task=host0,pid=36,uid=0
[ 11.477825] Out of memory: Killed process 36 (host0) total-vm:648kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[ 11.478049] oom killer 36 (host0) has mm pinned by 1 (init)
[ 11.478281] host3 invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
[ 11.478342] Call Trace:
[ 11.478511] 00000000bdf6c896: [<7fbc000d9123>] 0x00007fbc000d9123
[ 11.478559] 00000000bc9f1796: [<7fbc00086876>] 0x00007fbc00086876
[ 11.478606] 00000000c9321b8e: [<7fbc000d9f31>] 0x00007fbc000d9f31
[ 11.478652] 00000000b535166a: [<7fbc000f2f6b>] 0x00007fbc000f2f6b
[ 11.478699] 0000000074feb5d2: [<7fbc000fabf1>] 0x00007fbc000fabf1
[ 11.478745] 00000000c66dbb31: [<7fbc002818be>] 0x00007fbc002818be
[ 11.478793] 00000000981a4c0b: [<7fbc002819b6>] 0x00007fbc002819b6
[ 11.478989] 00000000dc7b3603: [<7fbc0020df9e>] 0x00007fbc0020df9e
[ 11.479035] 0000000032178354: [<7fbc0027138f>] 0x00007fbc0027138f
[ 11.479081] 0000000054c3665b: [<7fbc0027f735>] 0x00007fbc0027f735
[ 11.479128]
[ 11.479150] Mem-Info:
[ 11.479173] active_anon:0 inactive_anon:0 isolated_anon:0
[ 11.479173] active_file:1 inactive_file:1 isolated_file:0
[ 11.479173] unevictable:1 dirty:0 writeback:0 unstable:0
[ 11.479173] slab_reclaimable:256 slab_unreclaimable:1410
[ 11.479173] mapped:0 shmem:0 pagetables:0 bounce:0
[ 11.479173] free:179 free_pcp:0 free_cma:0
[ 11.479520] Node 0 active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:4kB unevictable:4kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[ 11.479655] Normal free:716kB min:716kB low:892kB high:1068kB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:4kB unevictable:4kB writepending:0kB present:32764kB managed:32172kB mlocked:0kB kernel_stack:168kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 11.479971] lowmem_reserve[]: 0 0
[ 11.480001] Normal: 1*4kB (U) 1*8kB (M) 2*16kB (UM) 1*32kB (U) 0*64kB 1*128kB (M) 0*256kB 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 716kB
[ 11.480079] 3 total pagecache pages
[ 11.480109] 8191 pages RAM
[ 11.480131] 0 pages HighMem/MovableOnly
[ 11.480161] 148 pages reserved
[ 11.480340] Unreclaimable slab info:
[ 11.480371] Name Used Total
[ 11.480410] bio-4 34KB 108KB
[ 11.480450] bio-3 8KB 15KB
[ 11.480487] bio-2 11KB 26KB
[ 11.480525] RAWv6 15KB 15KB
[ 11.480563] UDPv6 29KB 47KB
[ 11.480602] bio-1 3KB 3KB
[ 11.480639] RAW 7KB 7KB
[ 11.480841] UDP 7KB 7KB
[ 11.480881] request_queue 31KB 31KB
[ 11.480919] biovec-max 672KB 672KB
[ 11.480957] biovec-64 8KB 32KB
[ 11.480995] bio_integrity_payload 31KB 43KB
[ 11.481033] skbuff_head_cache 3KB 3KB
[ 11.481073] file_lock_cache 3KB 3KB
[ 11.481266] file_lock_ctx 3KB 3KB
[ 11.481303] proc_dir_entry 1559KB 1559KB
[ 11.481341] pde_opener 3KB 3KB
[ 11.481379] seq_file 3KB 3KB
[ 11.481417] kernfs_node_cache 1087KB 1087KB
[ 11.481455] mnt_cache 7KB 7KB
[ 11.481492] names_cache 32KB 32KB
[ 11.481530] nsproxy 6KB 7KB
[ 11.481568] vm_region 3KB 3KB
[ 11.481765] vm_area_struct 3KB 3KB
[ 11.481802] mm_struct 7KB 7KB
[ 11.481840] fs_cache 6KB 8KB
[ 11.481879] files_cache 7KB 7KB
[ 11.481917] signal_cache 39KB 39KB
[ 11.481954] sighand_cache 61KB 61KB
[ 11.481991] task_struct 61KB 63KB
[ 11.482029] cred_jar 8KB 8KB
[ 11.482223] pid 7KB 7KB
[ 11.482261] pool_workqueue 138KB 160KB
[ 11.482298] kmalloc-8k 120KB 128KB
[ 11.482336] kmalloc-4k 128KB 160KB
[ 11.482373] kmalloc-2k 256KB 256KB
[ 11.482411] kmalloc-1k 230KB 232KB
[ 11.482448] kmalloc-512 368KB 368KB
[ 11.482486] kmalloc-256 180KB 180KB
[ 11.482525] kmalloc-192 31KB 31KB
[ 11.482711] kmalloc-128 18KB 36KB
[ 11.482749] kmalloc-96 19KB 19KB
[ 11.482787] kmalloc-64 15KB 16KB
[ 11.482824] kmalloc-32 9KB 16KB
[ 11.482862] kmalloc-16 4KB 4KB
[ 11.482900] kmalloc-8 20KB 20KB
[ 11.482938] kmem_cache_node 8KB 8KB
[ 11.482976] kmem_cache 23KB 23KB
[ 11.483165] Tasks state (memory values in pages):
[ 11.483203] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 11.483266] [ 36] 0 36 162 0 0 0 0 host0
[ 11.483319] Out of memory and no killable processes...
[ 11.483358] Kernel panic - not syncing: System is deadlocked on memory
I think that we should decide this based on the requirements of the cryptsetup replacement. @mikbras?
Isn't this independent of cryptsetup? The kernel-level dm-* modules would still be there.
Isn't this independent of cryptsetup? The kernel-level dm-* modules would still be there.
That is correct. The kernel formats integrity volumes and depletes memory when doing so (even with vicsetup).
