linux icon indicating copy to clipboard operation
linux copied to clipboard

linux-5.4.51 cfg80211 fails to load regulatory.db when compiled with gcc-10.1.0 and glibc-2.31

Open juanitotc opened this issue 5 years ago • 20 comments

When compiling 5.4.51 natively on an RPi4 using bcm2711_defconfig with gcc-10.1.0 and glibc-2.31, cfg80211 fails to load regulatory.db - there are no errors or relevant warnings during the compilation.

Using a cross-compiler "arm-bcm2708hf-linux-gnueabihf-gcc (crosstool-NG 1.24.0.6-afaf7b9) 7.4.1 20181213" and glibc-2.28 with the same config except "CONFIG_CC_HAS_ASM_INLINE=y" (not supported by gcc-7) results in no problems.

In both cases wifi works.

[    1.215627] Asymmetric key parser 'x509' registered
[    1.491309] Loading compiled-in X.509 certificates
...
[60597.353367] cfg80211: Problem loading in-kernel X.509 certificate (-22)
[60597.365221] ------------[ cut here ]------------
[60597.365240] WARNING: CPU: 2 PID: 12380 at crypto/rsa-pkcs1pad.c:539 pkcs1pad_verify+0x158/0x174
[60597.365248] Modules linked in: cfg80211 rfkill snd_soc_bcm2835_i2s snd_soc_core snd_pcm_dmaengine snd_compress snd_bcm2835(C) snd_pcm snd_timer snd i2c_dev evdev squashfs spidev rpivid_mem raspberrypi_hwmon spi_bcm2835 i2c_bcm2835 zram zsmalloc uio_pdrv_genirq uio
[60597.365313] CPU: 2 PID: 12380 Comm: modprobe Not tainted 5.4.51-piCore-v7 #1
[60597.365317] Hardware name: BCM2711
[60597.365328] Workqueue: events request_firmware_work_func
[60597.365333] Backtrace: 
[60597.365346] [<c020df24>] (dump_backtrace) from [<c020e26c>] (show_stack+0x20/0x24)
[60597.365353]  r7:ffffffff r6:00000000 r5:60000013 r4:c129c7a0
[60597.365363] [<c020e24c>] (show_stack) from [<c0a5e38c>] (dump_stack+0xd0/0x114)
[60597.365372] [<c0a5e2bc>] (dump_stack) from [<c0221e28>] (__warn+0xe0/0x108)
[60597.365380]  r9:cfd14c00 r8:00000009 r7:0000021b r6:c0639948 r5:00000009 r4:c0dedd50
[60597.365389] [<c0221d48>] (__warn) from [<c02221f4>] (warn_slowpath_fmt+0x74/0xcc)
[60597.365395]  r7:c0639948 r6:0000021b r5:c0dedd50 r4:00000000
[60597.365404] [<c0222184>] (warn_slowpath_fmt) from [<c0639948>] (pkcs1pad_verify+0x158/0x174)
[60597.365411]  r8:cfd27300 r7:cfdf4880 r6:cfdacdc0 r5:00000000 r4:cfd27300
[60597.365421] [<c06397f0>] (pkcs1pad_verify) from [<c06432c4>] (public_key_verify_signature+0x258/0x304)
[60597.365428]  r9:cfd14c00 r8:cfd27300 r7:cfdf4880 r6:cfdacdc0 r5:00000000 r4:d1467c0c
[60597.365437] [<c064306c>] (public_key_verify_signature) from [<c06448cc>] (x509_check_for_self_signed+0xcc/0x128)
[60597.365444]  r9:000000ef r8:000002a8 r7:cfdacd80 r6:00000000 r5:cfdacdc0 r4:cfdf4780
[60597.365453] [<c0644800>] (x509_check_for_self_signed) from [<c0643cc0>] (x509_cert_parse+0x178/0x1c8)
[60597.365459]  r7:f082103c r6:cfdf4800 r5:cfdacf00 r4:cfdf4780
[60597.365468] [<c0643b48>] (x509_cert_parse) from [<c0644f94>] (pkcs7_extract_cert+0x3c/0x70)
[60597.365475]  r9:000000ef r8:0000004f r7:00000000 r6:000002a4 r5:00000002 r4:cfdacc80
[60597.365484] [<c0644f58>] (pkcs7_extract_cert) from [<c06a4270>] (asn1_ber_decoder+0x648/0x920)
[60597.365489]  r5:00000002 r4:00000040
[60597.365497] [<c06a3c28>] (asn1_ber_decoder) from [<c0644b18>] (pkcs7_parse_message+0xdc/0x1ac)
[60597.365504]  r10:00000000 r9:cfdacd00 r8:cfdacd00 r7:0000049e r6:f0821000 r5:c0e4040c
[60597.365509]  r4:cfdacc80
[60597.365519] [<c0644a3c>] (pkcs7_parse_message) from [<c037aa04>] (verify_pkcs7_signature+0x2c/0x80)
[60597.365527]  r9:00000080 r8:00000000 r7:eff3e600 r6:00000eb4 r5:f081b000 r4:f081b000
[60597.365641] [<c037a9d8>] (verify_pkcs7_signature) from [<bf1ae104>] (valid_regdb+0xd8/0x228 [cfg80211])
[60597.365647]  r6:bf21f554 r5:00000eb4 r4:f081b000
[60597.365827] [<bf1ae02c>] (valid_regdb [cfg80211]) from [<bf1b20d4>] (regdb_fw_cb+0x34/0x114 [cfg80211])
[60597.365835]  r10:00000000 r9:00000080 r8:00000000 r7:eff3e600 r6:cfdac900 r5:bf21f554
[60597.365840]  r4:cfdac840
[60597.365933] [<bf1b20a0>] (regdb_fw_cb [cfg80211]) from [<c078b628>] (request_firmware_work_func+0x60/0xa4)
[60597.365940]  r7:eff3e600 r6:eff3b340 r5:eee87e00 r4:cfdac8c0
[60597.365951] [<c078b5c8>] (request_firmware_work_func) from [<c023f468>] (process_one_work+0x21c/0x544)
[60597.365956]  r4:cfdac8c0
[60597.365964] [<c023f24c>] (process_one_work) from [<c023f7fc>] (worker_thread+0x6c/0x5d8)
[60597.365972]  r10:eff3b340 r9:00000008 r8:c1203d00 r7:eff3b358 r6:eee87e14 r5:eff3b340
[60597.365976]  r4:eee87e00
[60597.365984] [<c023f790>] (worker_thread) from [<c02460a8>] (kthread+0x16c/0x170)
[60597.365992]  r10:d0f57e84 r9:c023f790 r8:eee87e00 r7:d1466000 r6:00000000 r5:eee86a40
[60597.365996]  r4:efbb6540
[60597.366004] [<c0245f3c>] (kthread) from [<c02010ac>] (ret_from_fork+0x14/0x28)
[60597.366009] Exception stack(0xd1467fb0 to 0xd1467ff8)
[60597.366015] 7fa0:                                     00000000 00000000 00000000 00000000
[60597.366022] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[60597.366028] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[60597.366036]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0245f3c
[60597.366040]  r4:eee86a40
[60597.366046] ---[ end trace 8ac9463ac2468c2b ]---
[60597.366072] cfg80211: loaded regulatory.db is malformed or signature is missing/invalid

juanitotc avatar Aug 02 '20 05:08 juanitotc

Thanks for the heads up. I'm interested to hear the outcome of any investigation, but this is all unmodified upstream code.

pelwell avatar Aug 03 '20 13:08 pelwell

I see the "cfg80211: Problem loading in-kernel X.509 certificate" error and its accompanying stack trace when running the official 5.4.65+ kernel, which appears to have been compiled by a laughably outdated GCC 4.9.3. (Why is the project using such an ancient compiler anyway?)

My /lib/firmware/regulatory.db is from wireless-regdb-2019.06.03.tar.xz.

Wi-Fi works fine, though I'm assuming it's just using the compiled-in default regulatory settings and did not apply the database from disk.

I actually have two stack traces, the latter of which is equivalent to the one @juanitotc posted above. The former is emitted just prior to the "Problem loading in-kernel X.509 certificate (-22)" error:

[   18.604222] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[   18.839702] ------------[ cut here ]------------
[   18.839766] WARNING: CPU: 0 PID: 769 at crypto/rsa-pkcs1pad.c:539 pkcs1pad_verify+0x110/0x138
[   18.839779] Modules linked in: cfg80211(+) rfkill raspberrypi_hwmon binfmt_misc zram zsmalloc uio_pdrv_genirq uio fixed
[   18.839856] CPU: 0 PID: 769 Comm: systemd-udevd Not tainted 5.4.65+ #1341
[   18.839868] Hardware name: BCM2835
[   18.839878] Backtrace: 
[   18.839915] [<c0014d28>] (dump_backtrace) from [<c0015018>] (show_stack+0x20/0x24)
[   18.839938]  r6:00000009 r5:00000000 r4:00000000 r3:9e5172d2
[   18.839967] [<c0014ff8>] (show_stack) from [<c079059c>] (dump_stack+0x20/0x28)
[   18.840002] [<c079057c>] (dump_stack) from [<c0023c3c>] (__warn+0xdc/0x100)
[   18.840027] [<c0023b60>] (__warn) from [<c0023d10>] (warn_slowpath_fmt+0xb0/0xc0)
[   18.840051]  r9:c0977608 r8:0000021b r7:c04092b4 r6:00000009 r5:c0aa1028 r4:00000000
[   18.840078] [<c0023c64>] (warn_slowpath_fmt) from [<c04092b4>] (pkcs1pad_verify+0x110/0x138)
[   18.840102]  r9:d9b07a00 r8:d9a269c0 r7:00000000 r6:d9b672a0 r5:d9b672a0 r4:d9a269c0
[   18.840138] [<c04091a4>] (pkcs1pad_verify) from [<c040d1b0>] (public_key_verify_signature+0x258/0x304)
[   18.840158]  r7:c0aa1028 r6:d9b672a0 r5:00000000 r4:d9a51b04
[   18.840190] [<c040cf58>] (public_key_verify_signature) from [<c040e924>] (x509_check_for_self_signed+0xe4/0x10c)
[   18.840215]  r10:c0b04ecc r9:d98a0980 r8:000002a8 r7:d8ebaf20 r6:d9b53340 r5:00000000
[   18.840228]  r4:d9b64300
[   18.840254] [<c040e840>] (x509_check_for_self_signed) from [<c040db50>] (x509_cert_parse+0x170/0x1dc)
[   18.840274]  r7:bf0bbd4c r6:d9b673c0 r5:d9b64300 r4:d8ebaf20
[   18.840299] [<c040d9e0>] (x509_cert_parse) from [<c040e59c>] (x509_key_preparse+0x24/0x1a0)
[   18.840325]  r8:bf0bbd4c r7:d9a51ca0 r6:c0b07df0 r5:d9a51ca0 r4:c0b07e18 r3:c040e578
[   18.840351] [<c040e578>] (x509_key_preparse) from [<c040c094>] (asymmetric_key_preparse+0x5c/0x90)
[   18.840375]  r9:d98a0980 r8:bf0bbd4c r7:d9a51ca0 r6:c0b07df0 r5:c0aa1028 r4:c0b07e18
[   18.840416] [<c040c038>] (asymmetric_key_preparse) from [<c03d269c>] (key_create_or_update+0x158/0x430)
[   18.840438]  r7:00000000 r6:da215701 r5:c0aa1028 r4:da215700
[   18.842008] [<c03d2544>] (key_create_or_update) from [<bf103240>] (regulatory_init+0x164/0x274 [cfg80211])
[   18.842044]  r10:d9b53764 r9:000002a8 r8:bf0bbff4 r7:bf0d14d4 r6:bf0c50e0 r5:c0aa1028
[   18.842059]  r4:bf0bbd4c
[   18.845075] [<bf1030dc>] (regulatory_init [cfg80211]) from [<bf103068>] (cfg80211_init+0x68/0xdc [cfg80211])
[   18.845112]  r10:d9b53764 r9:d9b53740 r8:00000000 r7:c0aa1028 r6:d9a50000 r5:bf0d14c0
[   18.845124]  r4:00000000
[   18.846659] [<bf103000>] (cfg80211_init [cfg80211]) from [<c000aea4>] (do_one_initcall+0x4c/0x1c4)
[   18.846830]  r5:bf103000 r4:bf0d12e0
[   18.846880] [<c000ae58>] (do_one_initcall) from [<c0098c24>] (do_init_module+0x6c/0x214)
[   18.846907]  r9:d9b53740 r8:00000001 r7:bf0d12e0 r6:c0aa1028 r5:d8ebae40 r4:bf0d12e0
[   18.846936] [<c0098bb8>] (do_init_module) from [<c0097a70>] (load_module+0x1bf4/0x22c0)
[   18.846954]  r6:c0aa1028 r5:00000001 r4:d9a51f28
[   18.846979] [<c0095e7c>] (load_module) from [<c0098360>] (sys_finit_module+0xb8/0xcc)
[   18.847004]  r10:00000000 r9:00000000 r8:7fffffff r7:b6f19748 r6:0000000c r5:00000000
[   18.847016]  r4:c0aa1028
[   18.847041] [<c00982a8>] (sys_finit_module) from [<c0009000>] (ret_fast_syscall+0x0/0x28)
[   18.847058] Exception stack(0xd9a51fa8 to 0xd9a51ff0)
[   18.847079] 1fa0:                   00000000 00b0c488 0000000c b6f19748 00000000 00000000
[   18.847101] 1fc0: 00000000 00b0c488 00af3858 0000017b 00b12200 00578c0c 00000000 00af3858
[   18.847118] 1fe0: bea4f2d0 bea4f2c0 b6f10aec b6dfc530
[   18.847141]  r9:d9a50000 r8:c00091a4 r7:0000017b r6:00af3858 r5:00b0c488 r4:00000000
[   18.847156] ---[ end trace 4fcfc72692105f89 ]---
[   18.847219] cfg80211: Problem loading in-kernel X.509 certificate (-22)

whitslack avatar Sep 20 '20 01:09 whitslack

recently I tried to upgrade the kernel of my rpi2 using gcc-9.3.0 and glibc-2.31 in the cross compile toolchain, and I believe to have hit a similar issue. Can't say for sure, since my usb keyboard wasn't working, and neither did network (wired) or ssh.

Is there any way to get the boot log file in such a scenario?

stefson avatar Sep 20 '20 04:09 stefson

Is there any way to get the boot log file in such a scenario?

You can wire up an RS-232 level shifter to the Pi's UART pins and specify console=ttyAMA0,115200 on the kernel command line to get the console log over a serial cable.

whitslack avatar Sep 20 '20 06:09 whitslack

I see the same errors with the latest default kernel:

Linux version 5.4.72+ (dom@buildbot) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #1356 Thu Oct 22 13:56:00 BST 2020

This is in admittedly unusual circumstances: I'm trying to boot RPIs with an NFS root over wifi, so I added an initramfs that contains a wpa_supplicant. It contains a copy of everything in /lib/firmware and /lib/crda, including sforshee's public key, from the stock image. I can't find any certificate files in there, however.

When booting the normal image, it does report

cfg80211: Loading compiled-in X.509 certificates for regulatory database
cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'

but when booted with the initramfs it says

cfg80211: Loading compiled-in X.509 certificates for regulatory database
....
cfg80211: Problem loading in-kernel X.509 certificate (-22)
...
cfg80211: loaded regulatory.db is malformed or signature is missing/invalid

(including the stack traces as reported above).

The source location in the first trace is crypto/rsa-pkcs1pad.c:539, which triggers a warning (and stack trace) when req->dst_len==0, i.e. when it is passed an empty request. It happens right after this:

...
Run /init as init process
Loading, please wait...
Starting version 241
cfg80211: Loading compiled-in X.509 certificates for regulatory database
...

I suspect that the root file system (or some parts thereof) are simply not available yet, although I did expect the kernel to have all the necessary bits compiled into it, so that it doesn't need a file system. I suppose it could also be a module load order problem or something of that sort.

wintersteiger avatar Nov 14 '20 20:11 wintersteiger

I resolved my "Problem loading in-kernel X.509 certificate" problem by supplying the modules libsha256 and sha256_generic beneath /lib/firmware. (Oops, didn't realize those would be needed for Wi-Fi functionality.) However, I don't know if this is OP's problem, as I'm using the precompiled kernel, not compiling it myself with GCC 10.

whitslack avatar Dec 11 '20 20:12 whitslack

Thanks @whitslack, I can confirm that this works for me too. I would have never suspected that there is a hidden dependency on the hash function/module; clearly the error reporting could be much improved here!

In my case, I just added

sha256_generic
libsha256

to /etc/initramfs-tools/modules and updated the initramfs, and now it's happy again.

wintersteiger avatar Dec 14 '20 12:12 wintersteiger

Just to be sure - are you saying that the kernel modules libsha256 and sha256_generic should be loaded on the host or copied under /lib/firmware on the host?

juanitotc avatar Dec 17 '20 13:12 juanitotc

I added them to my /etc/initramfs-tools/modules, which instructs upgrade-initramfs to copy them into the initramfs. I suspect that it may be enough to copy them to /lib/modules and the other modules load them when they can find them, but I didn't test that theory.

wintersteiger avatar Dec 17 '20 13:12 wintersteiger

cfg80211 automatically loads libsha256 (which loads sha256_generic) when it needs to check the signature of the regulatory database against the public key in the certificate. It doesn't need it afterward, and those two modules can be unloaded if you wish.

whitslack avatar Dec 17 '20 16:12 whitslack

cfg80211 automatically loads libsha256 (which loads sha256_generic) when it needs to check the signature of the regulatory database against the public key in the certificate. It doesn't need it afterward, and those two modules can be unloaded if you wish. Could you please tell me how to get libsha256 and sha256_generic modules? I am having the same problem. " cfg80211: loaded regulatory.db is malformed or signature is missing/invalid" and pi is using the default regulatory database. Thanks.

redngreen71 avatar Jul 02 '21 09:07 redngreen71

Could you please tell me how to get libsha256 and sha256_generic modules?

@redngreen71: They're in the firmware repository. Be sure that your modules version matches your kernel version. If you're using a Linux distribution, it may have packages for Raspberry Pi that ensure that your modules and kernel match.

whitslack avatar Jul 02 '21 14:07 whitslack

Thank you for your quick reply. I found sha256_generic module only, not libsha256. Could you please suggest how to to avoid the following problem? "cfg80211: loaded regulatory.db is malformed or signature is missing/invalid"

redngreen71 avatar Jul 02 '21 15:07 redngreen71

Mine's at /lib/modules/5.10.17+/kernel/lib/crypto/libsha256.ko.

wintersteiger avatar Jul 02 '21 15:07 wintersteiger

Mine's at /lib/modules/5.10.17+/kernel/lib/crypto/libsha256.ko. Thank you @wintersteiger . I don't have the module. So I need to load it.

redngreen71 avatar Jul 02 '21 15:07 redngreen71

@wintersteiger you mind to share the name of the module in the kernel .config? I compile the kernel myself

stefson avatar Jul 02 '21 15:07 stefson

I don't know why libsha256.ko is missing. https://github.com/raspberrypi/firmware/tree/master/modules/5.10.46-v7l%2B/kernel/lib/crypto

redngreen71 avatar Jul 02 '21 15:07 redngreen71

My kernel config says it must be one of these:

... $ zgrep SHA256 /proc/config.gz
# CONFIG_CRYPTO_SHA256_ARM is not set
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_LIB_SHA256=y

Strange that the lib is listed as y instead of m though.

wintersteiger avatar Jul 02 '21 16:07 wintersteiger

The version of wireless-regdb I use is wireless-regdb_2018.10.24, in order to modify the country code, txt needs to be modified, and after modification, run the make command to generate regulator. db, regulator. db.p7s, sforshee.key.pub.pem, regulator. bin, and other files. The four files generated by make and the modified db.txt are re-referenced when compiling do_install for wireless-regdb. The compiled wireless-regdb will display the following error "loaded regulation. db is malformed or signature is missing/invalid". Please help to find the cause of this problem [87.840392].(0)[7:kworker/u4:0]mac 0 MAC_MCR = 104f30a [87.840413].(0)[7:kworker/u4:0]mac 0 PHY link=0 state=8 [87.840442].(0)[7:kworker/u4:0]mtk_soc_eth 1510000. ethernet eth0: Link is Down [88.880418].(1)[7:kworker/u4:0]mac 0 MAC_MCR = 104f30a [88.880439].(1)[7:kworker/u4:0]mac 0 PHY link=0 state=8 [88.880469].(1)[7:kworker/u4:0]mtk_soc_eth 1510000. ethernet eth0: Link is Down [89.920468].(1)[22:kworker/u4:1]mac 0 MAC_MCR = 104f30a [89.920489].(1)[22:kworker/u4:1]mac 0 PHY link=0 state=8 [89.920520].(1)[22:kworker/u4:1]mtk_soc_eth 1510000. ethernet eth0: Link is Down [90.124517].(0)[2300:mount_mtklog.sh]MTK log dir not mount waiting! [90.602459].(1)[24:kauditd]audit: type=1400 audit(1711960163.280:89): avc: denied { search } for pid=7683 comm="modprobe" name="events" dev="tracefs" ino=1134 scontext=system_u:system_r:kmod_t:s0-s15:c0.c1023 tcontext=system_u:object_r:tracefs_t:s0 tclass=dir permissive=1 [90.613108].(0)[7683:modprobe]calling cfg80211_init+0x0/0xe4 [cfg80211] @7683 [90.613792].(0)[7683:modprobe]cfg80211: Loading compiled-in X.509 certificates for regulatory database [90.614682].(0)[7683:modprobe]cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'. [90.614693].(1)[24:kauditd]audit: type=1400 audit(1711960163.290:90): avc: denied { write } for pid=7683 comm="modprobe" scontext=system_u:system_r:kmod_t:s0-s15:c0.c1023 tcontext=system_u:system_r:kmod_t:s0-s15:c0.c1023 tclass=key permissive=1 [90.616510].(0)[6603:kworker/0:4]PKCS#7 signature not signed with a trusted key [90.617522].(0)[6603:kworker/0:4]cfg80211: loaded regulatory.db is malformed or signature is missing/invalid [90.628680].(0)[7683:modprobe]initcall cfg80211_init+0x0/0xe4 [cfg80211] returned 0 after 14951 usecs [90.648785].(1)[0:swapper/1][name:spm&][SPM] MCUSYSOFF Didn't enter low power scenario [90.960393].(1)[7:kworker/u4:0]mac 0 MAC_MCR = 104f30a [90.960415].(1)[7:kworker/u4:0]mac 0 PHY link=0 state=8 [90.960446].(1)[7:kworker/u4:0]mtk_soc_eth 1510000. ethernet eth0: Link is Down

anqiao-cell avatar Apr 09 '24 09:04 anqiao-cell

If I need to change the country code in db.txt, how should I modify it? I have tried to modify the db.txt file directly, it does not take effect

anqiao-cell avatar Apr 09 '24 11:04 anqiao-cell