blazesym icon indicating copy to clipboard operation
blazesym copied to clipboard

[Bug] failed to symbolize addresses: entity not found

Open MilesChan opened this issue 7 months ago • 11 comments

maps.rs pub(crate) fn parse_path_name( ... let maps_file = PathBuf::from(format!("/proc/{pid}/map_files/{vma_start:x}-{vma_end:x}")); )

handle_entry_addr addr=0x7f3f263110, entry=MapsEntry { range: 0x7f3f263000..0x7f3f264000, perm: r-x, offset: 0x4000, path_name: Some(Path(EntryPath { maps_file: "/proc/32011/map_files/7f3f263000-7f3f264000", symbolic_path: "/apex/com.android.runtime/lib64/bionic/libdl.so", _non_exhaustive: () })) }

xxx:/ # ls -lh /proc/32011/map_files/7f3f263000-7f3f264000 ls: /proc/32011/map_files/7f3f263000-7f3f264000: No such file or directory

xxx:/ # ls -lh /proc/32011/map_files/7f3f263000-7f3f267000
lr-------- 1 u0_a327 u0_a327 64 2025-05-15 14:13 /proc/32011/map_files/7f3f263000-7f3f267000 -> /apex/com.android.runtime/lib64/bionic/libdl.so

haotian:/ # cat /proc/32011/maps |grep 7f3f26 7f3f25f000-7f3f260000 r--p 00000000 07:98 48 /apex/com.android.runtime/lib64/bionic/libdl.so 7f3f260000-7f3f263000 ---p 00000000 00:00 0 [page size compat] 7f3f263000-7f3f264000 r-xp 00004000 07:98 48 /apex/com.android.runtime/lib64/bionic/libdl.so 7f3f264000-7f3f267000 ---p 00000000 00:00 0 [page size compat]

MilesChan avatar May 15 '25 06:05 MilesChan

/proc/32011/map_files/7f3f263000-7f3f264000 doesn't exist

in cat /proc/32011/maps 7f3f263000-7f3f264000 r-xp 00004000 07:98 48 /apex/com.android.runtime/lib64/bionic/libdl.so 7f3f264000-7f3f267000 ---p 00000000 00:00 0 [page size compat]

these two areas are merged as /proc/32011/map_files/7f3f263000-7f3f267000

MilesChan avatar May 15 '25 06:05 MilesChan

Interesting, I've never seen anything like that and I can't find any references to this "page size compat" string. Perhaps it's some Android thing?

d-e-s-o avatar May 15 '25 13:05 d-e-s-o

I suppose it may be a compatibility feature for when different page sizes are available at runtime. I guess the kernel has to work with a minimum size of 4k in your case, but the app is running with 16k pages and so this entry gets inserted. Of course I have no idea how to handle this, short of, I guess, parsing this [page size compat] thing and manually merging the ranges (lol...)

danielocfb avatar May 15 '25 18:05 danielocfb

My take is that this is an Android kernel bug introduced as part of the conversion from 4k pages to 16k: https://android.googlesource.com/kernel/common/+/8c2a805a857914324b077708b45c31c2f20d02da

The patch claims to be non-breaking but that is clearly not the case: my understanding is that what we are doing is the intended way of going about inferring maps file paths. I'd suggest submitting a bug report to them.

danielocfb avatar May 15 '25 20:05 danielocfb

I reached out to Kalesh who implemented most if not all of this logic.

You can probably disable https://docs.rs/blazesym/latest/blazesym/symbolize/source/struct.Process.html#structfield.map_files to circumvent the problem.

danielocfb avatar May 15 '25 20:05 danielocfb

I reached out to Kalesh who implemented most if not all of this logic.

You can probably disable https://docs.rs/blazesym/latest/blazesym/symbolize/source/struct.Process.html#structfield.map_files to circumvent the problem.

This will indeed circumvent the problem。I've tried the following changes and it goes well. but as the webpage(struct.Process.html) says: "map_files usage is generally strongly encouraged", maybe it's not the best solution.

patch.diff.txt

MilesChan avatar May 16 '25 03:05 MilesChan

My take is that this is an Android kernel bug introduced as part of the conversion from 4k pages to 16k: https://android.googlesource.com/kernel/common/+/8c2a805a857914324b077708b45c31c2f20d02da

The patch claims to be non-breaking but that is clearly not the case: my understanding is that what we are doing is the intended way of going about inferring maps file paths. I'd suggest submitting a bug report to them.

What you said makes sense, and I think improving the compatibility of blazesym will make this project more useful.

MilesChan avatar May 16 '25 04:05 MilesChan

I suppose it may be a compatibility feature for when different page sizes are available at runtime. I guess the kernel has to work with a minimum size of 4k in your case, but the app is running with 16k pages and so this entry gets inserted. Of course I have no idea how to handle this, short of, I guess, parsing this [page size compat] thing and manually merging the ranges (lol...)

manually merging the ranges sounds a good advice, I've tried to use peekable iterator implements a look-ahead mechanism that allows it to seamlessly merge adjacent memory mapping entries, and it works fine. Thanks for the idea.

merge.diff.txt

MilesChan avatar May 16 '25 06:05 MilesChan

Interesting, I've never seen anything like that and I can't find any references to this "page size compat" string. Perhaps it's some Android thing?

Thanks for your attention. yes, it's some Android thing. eBPF has become more available since Android 13(with GKI and BTF support), and the amount of Android devices is huge and increasing, I suppose improving the compatibility of blazesym with Android is a good idea. Thanks again.

MilesChan avatar May 16 '25 06:05 MilesChan

It sounds as if Kalesh is working on a patch for Android.

d-e-s-o avatar May 19 '25 17:05 d-e-s-o

It sounds as if Kalesh is working on a patch for Android.

Here it is: https://android-review.googlesource.com/c/kernel/common/+/3632493

Edit: He also mentioned that this will be backported to all released kernels, meaning everything out there is expected to end up being patched, eventually. Given that, I don't think we will land a workaround on the blazesym side.

d-e-s-o avatar May 19 '25 20:05 d-e-s-o

It sounds as if Kalesh is working on a patch for Android.

Here it is: https://android-review.googlesource.com/c/kernel/common/+/3632493

Edit: He also mentioned that this will be backported to all released kernels, meaning everything out there is expected to end up being patched, eventually. Given that, I don't think we will land a workaround on the blazesym side.

Got it. Thank you guys!

MilesChan avatar May 23 '25 02:05 MilesChan