Infrequently: segfault in FastMmap.so

Open UPi-nuance opened this issue 4 years ago • 1 comments

Hello,

I love Cache::FastMmap, except...

Infrequently, my web application crashes and will not restart until the cache file is deleted. The crash sometimes happens once a week, sometimes multiple times on a single day with no apparent pattern.

Version: '1.56' Perl: 5.32.1 Environment: Linux 5.10.0-9-amd64 #1 SMP Debian 5.10.70-1 (2021-09-30) x86_64 GNU/Linux

The workers initialize their cache with the following piece of code:

	$self->{cache} = Cache::FastMmap->new(
		share_file      => '/run/shm/webapp',
		expire_time     => 0,
		unlink_on_exit  => 0,
	);

Here is what /var/log/messages looks like. I have been unable to get other sort of diagnostics.

Nov 27 01:06:53 frontendsvr kernel: [1009530.623177] webapp.pl[2477416]: segfault at 8 ip 00007f4df256564e sp 00007ffcf7fcc660 error 4 in FastMmap.so[7f4df2561000+6000]
Nov 27 01:06:53 frontendsvr kernel: [1009530.623503] Code: 89 44 24 20 4d 39 ee 0f 83 1e 01 00 00 45 31 e4 89 e8 44 89 e5 4d 89 f4 4d 89 fe 41 89 c7 66 0f 1f 44 00 00 49 8b 34 24 31 d2 <8b> 46 08 41 f7 f7 0f 1f 40 00 89 d0 49 8d 1c 86 8b 03 85 c0 74 13
Nov 27 01:06:53 frontendsvr kernel: [1009530.762896] webapp.pl[2470298]: segfault at 8 ip 00007f4df256564e sp 00007ffcf7fcc660 error 4 in FastMmap.so[7f4df2561000+6000]
Nov 27 01:06:53 frontendsvr kernel: [1009530.762912] Code: 89 44 24 20 4d 39 ee 0f 83 1e 01 00 00 45 31 e4 89 e8 44 89 e5 4d 89 f4 4d 89 fe 41 89 c7 66 0f 1f 44 00 00 49 8b 34 24 31 d2 <8b> 46 08 41 f7 f7 0f 1f 40 00 89 d0 49 8d 1c 86 8b 03 85 c0 74 13
Nov 27 01:06:53 frontendsvr kernel: [1009530.882630] webapp.pl[2480764]: segfault at 8 ip 00007f4df256564e sp 00007ffcf7fcc660 error 4 in FastMmap.so[7f4df2561000+6000]
Nov 27 01:06:53 frontendsvr kernel: [1009530.882649] Code: 89 44 24 20 4d 39 ee 0f 83 1e 01 00 00 45 31 e4 89 e8 44 89 e5 4d 89 f4 4d 89 fe 41 89 c7 66 0f 1f 44 00 00 49 8b 34 24 31 d2 <8b> 46 08 41 f7 f7 0f 1f 40 00 89 d0 49 8d 1c 86 8b 03 85 c0 74 13
Nov 27 01:06:53 frontendsvr kernel: [1009530.910137] webapp.pl[2457890]: segfault at 8 ip 00007f4df256564e sp 00007ffcf7fcc660 error 4 in FastMmap.so[7f4df2561000+6000]
Nov 27 01:06:53 frontendsvr kernel: [1009530.910154] Code: 89 44 24 20 4d 39 ee 0f 83 1e 01 00 00 45 31 e4 89 e8 44 89 e5 4d 89 f4 4d 89 fe 41 89 c7 66 0f 1f 44 00 00 49 8b 34 24 31 d2 <8b> 46 08 41 f7 f7 0f 1f 40 00 89 d0 49 8d 1c 86 8b 03 85 c0 74 13
Nov 27 01:06:54 frontendsvr kernel: [1009531.204498] webapp.pl[2482236]: segfault at 8 ip 00007f4df256564e sp 00007ffcf7fcc660 error 4 in FastMmap.so[7f4df2561000+6000]
Nov 27 01:06:54 frontendsvr kernel: [1009531.204515] Code: 89 44 24 20 4d 39 ee 0f 83 1e 01 00 00 45 31 e4 89 e8 44 89 e5 4d 89 f4 4d 89 fe 41 89 c7 66 0f 1f 44 00 00 49 8b 34 24 31 d2 <8b> 46 08 41 f7 f7 0f 1f 40 00 89 d0 49 8d 1c 86 8b 03 85 c0 74 13
Nov 27 01:06:54 frontendsvr kernel: [1009531.306465] webapp.pl[2468108]: segfault at 8 ip 00007f4df256564e sp 00007ffcf7fcc660 error 4 in FastMmap.so[7f4df2561000+6000]
Nov 27 01:06:54 frontendsvr kernel: [1009531.306508] Code: 89 44 24 20 4d 39 ee 0f 83 1e 01 00 00 45 31 e4 89 e8 44 89 e5 4d 89 f4 4d 89 fe 41 89 c7 66 0f 1f 44 00 00 49 8b 34 24 31 d2 <8b> 46 08 41 f7 f7 0f 1f 40 00 89 d0 49 8d 1c 86 8b 03 85 c0 74 13
Nov 27 01:06:54 frontendsvr kernel: [1009531.718009] webapp.pl[2465247]: segfault at 8 ip 00007f4df256564e sp 00007ffcf7fcc660 error 4 in FastMmap.so[7f4df2561000+6000]
Nov 27 01:06:54 frontendsvr kernel: [1009531.718027] Code: 89 44 24 20 4d 39 ee 0f 83 1e 01 00 00 45 31 e4 89 e8 44 89 e5 4d 89 f4 4d 89 fe 41 89 c7 66 0f 1f 44 00 00 49 8b 34 24 31 d2 <8b> 46 08 41 f7 f7 0f 1f 40 00 89 d0 49 8d 1c 86 8b 03 85 c0 74 13
Nov 27 01:06:54 frontendsvr kernel: [1009531.861339] webapp.pl[2468513]: segfault at 8 ip 00007f4df256564e sp 00007ffcf7fcc660 error 4 in FastMmap.so[7f4df2561000+6000]
Nov 27 01:06:54 frontendsvr kernel: [1009531.861385] Code: 89 44 24 20 4d 39 ee 0f 83 1e 01 00 00 45 31 e4 89 e8 44 89 e5 4d 89 f4 4d 89 fe 41 89 c7 66 0f 1f 44 00 00 49 8b 34 24 31 d2 <8b> 46 08 41 f7 f7 0f 1f 40 00 89 d0 49 8d 1c 86 8b 03 85 c0 74 13
Nov 27 01:06:55 frontendsvr kernel: [1009531.976008] webapp.pl[2477412]: segfault at 8 ip 00007f4df256564e sp 00007ffcf7fcc660 error 4 in FastMmap.so[7f4df2561000+6000]
Nov 27 01:06:55 frontendsvr kernel: [1009531.976141] Code: 89 44 24 20 4d 39 ee 0f 83 1e 01 00 00 45 31 e4 89 e8 44 89 e5 4d 89 f4 4d 89 fe 41 89 c7 66 0f 1f 44 00 00 49 8b 34 24 31 d2 <8b> 46 08 41 f7 f7 0f 1f 40 00 89 d0 49 8d 1c 86 8b 03 85 c0 74 13
Nov 27 01:06:55 frontendsvr kernel: [1009532.036817] webapp.pl[2477599]: segfault at 8 ip 00007f4df256564e sp 00007ffcf7fcc660 error 4 in FastMmap.so[7f4df2561000+6000]
Nov 27 01:06:55 frontendsvr kernel: [1009532.036831] Code: 89 44 24 20 4d 39 ee 0f 83 1e 01 00 00 45 31 e4 89 e8 44 89 e5 4d 89 f4 4d 89 fe 41 89 c7 66 0f 1f 44 00 00 49 8b 34 24 31 d2 <8b> 46 08 41 f7 f7 0f 1f 40 00 89 d0 49 8d 1c 86 8b 03 85 c0 74 13

Any help would be appreciated.

Best regards, Peter

Nov 27 '21 16:11 UPi-nuance

Hi.

We use Cache::FastMmap extensively in many different processes, and I haven't seen crashes like this. However Cache::FastMmap does work by mmaping the entire cache file into memory. If something accidentally corrupts random memory, that could cause the cache file to become corrupted. Having said that, I'd expect random corruption like that to cause random other crashes as well, not just Cache::FastMmap ones, but it sounds like you don't see that.

Do you end up with a core dump that you can get a strack trace out of using gdb?

Thanks

Dec 10 '21 03:12 robmueller