Document which Rust types can be placed into eBPF maps

[p-e-w]

It appears that the RedBPF toolchain imposes currently undocumented restrictions on map value types that go beyond those expressed by the generic constraints:

• All variants of an enum must have the same size in memory.
• Struct fields are (mostly?) required to be aligned to multiples of 32 bits, though sometimes 64-bit alignment is required, and some structs don't work at all despite their fields being aligned.
• Tweaking alignments with #[repr(align(n))] has no effect whatsoever. Whether or not #[repr(C)] is used doesn't seem to matter either.
• Double-width types like u128 cannot be placed in nested structs. They sometimes work in flat structs though.
• Fields with array types are hit-and-miss, and whether they work or not appears to depend on the phase of the moon.
• Getting any of this wrong results in an incomprehensible error message from libbpf, which gives no hint whatsoever about what is actually going on.

It would be great if there was documentation explaining precisely what the requirements for Rust types are so they can be placed into the eBPF map types provided by RedBPF.

It might also be a good idea to mark all methods exposed by the map types as unsafe. The reason is that normally, you can rely on Rust's type system to tell you whether your arguments have acceptable types. But here there are secondary requirements that cannot be expressed using trait bounds, but will nevertheless result in catastrophic program failure when disregarded. That's pretty much the definition of unsafety.

[rhdxmr]

I wrote some note about valid structs for BPF maps. I hope this note will be written to other forms of documentation later.

While developing BPF programs that deal with BPF maps, errors due to illegal data definition frequently occur. It is basically FFI and is naturally a part of unsafe Rust, so we have to be careful. If something goes wrong, infamous defined behaviors will happen.

Here I am focusing on defining structs. unions and enums are off the table here.

I couldn't come up with all possible ways to define valid structures. But users can avoid pesky problems if they follow the conservative rules I suggest below.

1. Use #[repr(C)] representation

   Since the default representation does not guarantee anything about the layout, we can not control anything when BPF verifier complaints about the structure. Hence, let's use repr(C) always and control the layout of the structure and understand the specific size of the structure and figure out the alignments of all fields in the structure. Handling BPF maps is conducted by calling BPF helpers and it is naturally FFI. So repr(C) deserves to be used in BPF programs.

2. Add padding fields explicitly

   You may see the error if you don't add padding fields correctly:

   invalid indirect read from stack R3 off -128+92 size 96

   You should add padding fields not only at the tail of structure, but also before the fields that need to be aligned properly using padding.

   Remember the principle of layout:

   • each field of structure needs to be aligned correctly
   • whole size of structure is multiple of alignment of the structure
   • Refer to the rust references for more information: 1 2

3. Limit the maximum alignment of structure to 8 bytes

   In BPF programs, to store values in BPF map bpf_map_update_elem is called. It is wrapped by redbpf_probes::maps::HashMap::set. Inside kernel, storing values into BPF maps involves copying values. But Linux kernel copies values to the 8 bytes aligned memory.

   So even if a user defines a structure whose alignment is 32 bytes, the alignment will not be retained during copying the value into a BPF map. Thus in BPF programs, the pointer returned by bpf_map_lookup_elem is not aligned to 32 bytes. But redbpf_probes::maps::HashMap::get immediately creates a reference from the pointer by &*. However creating a reference of it is counted as undefined behavior because rust compiler always expects reference to be properly aligned. If this assumption is imploded, all code that is built upon the assumption is invalid.

   • See these kernel source code that allocates memory aligned to 8 bytes: 1
     2
   • See this kernel source code that copy values to memory where address is aligned to 8 bytes. Note that l_new->key is aligned to 8 bytes and round_up(key_size, 8) rounds up to 8 bytes. 1

[rhdxmr]

I also mentioned the alignment issue the item 3 describes, at PR #215 ..