sys/targets: picking DataOffset is hard

[ramosian-glider]

trafficstars

At some point the DataOffset address which is used for fixed mmap() mappings started to clash with the mmap() calls issued by libc to prepare the heap (see Aleksandr's investigation in #5674). We then changed DataOffset from 0x20000000 to 0x200000000000, but it started clashing with the userspace ASan mappings (ASan is used during executor testing).

Changing DataOffset to 0x400000000000 fixed the problem on all the existing syzbot instances, but it stopped working in my local config, where I had CONFIG_ARM64_VA_BITS_39 and CONFIG_PGTABLE_LEVELS=3 (this address didn't exist, so mmap() returned ENOMEM).

If at some point we need to bring different VA configs to syzbot, this will blow again, so we might need to come up with a more flexible way to determine DataOffset (or we need to somehow prevent mmaps from libc from clashing with our memory).

I think we don't strictly need to work around ASan mappings, but different kernel configurations alone can complicate this task already.

[ramosian-glider]

One idea that came to my mind previously (I haven't tried it) was to replace DataOffset with a big page-aligned static array. Dunno how much memory we need though (and whether the compiler will agree to allocate that array without linker magic).

[ramosian-glider]

Marco's suggestion was to implement a tiny malloc wrapper that would allocate from our own heap. It is unclear whether this will prevent libc from calling mmap/brk to initialize the heap (or for any other purpose).

[a-nogikh]

One more case where 0x400000000000 doesn't work: https://github.com/google/syzkaller/issues/5810

[a-nogikh]

Marco's suggestion was to implement a tiny malloc wrapper that would allocate from our own heap. It is unclear whether this will prevent libc from calling mmap/brk to initialize the heap (or for any other purpose).

There seem to be __*_hook variables that should let us override the malloc implementation. If that won't require any extra linker or compiler options, it could well be worth it given that syz-executor mallocs only a small amount of memory.

https://www.gnu.org/software/libc/manual/html_node/Hooks-for-Malloc.html

[dvyukov]

Are there still any action items here? If not, please close.

[a-nogikh]

The problem is still not solved completely - we have only picked some data offsets that work +- fine. In particular, I am curious to see others' opinions on https://www.gnu.org/software/libc/manual/html_node/Hooks-for-Malloc.html - it could be a reasonably simple and reliable way to address the problem.

Also, after having returned to the old value for arm64 (https://github.com/google/syzkaller/commit/034534df050681c9e95b14ebd841afaa9db31390), we still face the original problem - malloc conflicts with DataOffset. That leads to occasional TestFuzz failures while updating syzkaller on arm64 native syz-ci's.

[dvyukov]

Ack. But note __malloc_hook is glibc-specific and is not the way to go.