crash
crash copied to clipboard
crash-utility
crash fails to load on ARM64 live kernel
crash 7.2.8 fails to load on ARM64 live kernel on 5.4.
# uname -a
Linux com7 5.4.42-xxxxxxxx-standard #1 SMP Wed Jun 3 05:17:19 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
# crash -s /proc/kcore vmlinux-stable-master-dev54
crash: /proc/kcore: cannot read vabits_actual
Is this a known issue? Currently when reading of vabits_actual symbol value from /proc/kallsyms isn't successful crash returns error (FATAL), instead should it continue to read tcr_el1_t1sz from VMCOREINFO to determine vabits_actual? Please share your thoughts.
Made a small source change to calculate the vabits_actual from vmcoreinfo's tcr_el1_t1sz, with the change I can load crash on live kernel. Tried few sample crash commands, they all seem to work.
# git diff arm64.c
diff --git a/arm64.c b/arm64.c
index 7662d71..d8515b2 100644
--- a/arm64.c
+++ b/arm64.c
@@ -3869,10 +3869,11 @@ arm64_calc_VA_BITS(void)
machdep->machspec->VA_BITS = value;
machdep->machspec->VA_START = _VA_START(machdep->machspec->VA_BITS_ACTUAL);
} else
- error(FATAL, "/proc/kcore: cannot read vabits_actual\n");
+ error(WARNING, "/proc/kcore: cannot read vabits_actual\n");
} else if (ACTIVE())
error(FATAL, "cannot determine VA_BITS_ACTUAL: please use /proc/kcore\n");
- else {
+
+ if (!machdep->machspec->VA_BITS_ACTUAL) {
if ((string = pc->read_vmcoreinfo("NUMBER(tcr_el1_t1sz)"))) {
/* See ARMv8 ARM for the description of
* TCR_EL1.T1SZ and how it can be used
com5:/tmp# /tmp/crash -s /proc/kcore /tmp/vmlinux
WARNING: /proc/kcore: cannot read vabits_actual
crash> sys
KERNEL: /tmp/vmlinux
DUMPFILE: /proc/kcore
CPUS: 8
DATE: Mon Jun 22 23:24:53 2020
UPTIME: 00:28:21
LOAD AVERAGE: 0.14, 0.05, 0.01
TASKS: 138
NODENAME: com5
RELEASE: 5.4.44-xxxxxxxx-standard
VERSION: #1 SMP Sat Jun 20 00:55:50 UTC 2020
MACHINE: aarch64 (unknown Mhz)
MEMORY: 7.8 GB
crash>
Hello Vijay,
On Wed, Jun 24, 2020 at 2:42 AM vijaybalakrishna [email protected] wrote:
crash 7.2.8 fails to load live system kernel on ARM64 on 5.4.
uname -a
Linux com7 5.4.42-xxxxxxxx-standard #1 SMP Wed Jun 3 05:17:19 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
crash -s /proc/kcore vmlinux-stable-master-dev54
crash: /proc/kcore: cannot read vabits_actual
Is this a known issue? Currently when reading of vabits_actual symbol value from /proc/kallsyms isn't successful crash returns error (FATAL), instead should it continue to read tcr_el1_t1sz from VMCOREINFO to determine vabits_actual? Please share your thoughts.
Made a small source change to calculate the vabits_actual from vmcoreinfo's tcr_el1_t1sz, with the change I can load crash on live kernel. Tried few sample crash commands, they all seem to work.
git diff arm64.c
diff --git a/arm64.c b/arm64.c index 7662d71..d8515b2 100644 --- a/arm64.c +++ b/arm64.c @@ -3869,10 +3869,11 @@ arm64_calc_VA_BITS(void) machdep->machspec->VA_BITS = value; machdep->machspec->VA_START = _VA_START(machdep->machspec->VA_BITS_ACTUAL); } else
error(FATAL, "/proc/kcore: cannot read vabits_actual\n");
error(WARNING, "/proc/kcore: cannot read vabits_actual\n"); } else if (ACTIVE()) error(FATAL, "cannot determine VA_BITS_ACTUAL: please use /proc/kcore\n");
else {
if (!machdep->machspec->VA_BITS_ACTUAL) { if ((string = pc->read_vmcoreinfo("NUMBER(tcr_el1_t1sz)"))) { /* See ARMv8 ARM for the description of * TCR_EL1.T1SZ and how it can be usedcom5:/tmp# /tmp/crash -s /proc/kcore /tmp/vmlinux WARNING: /proc/kcore: cannot read vabits_actual crash> sys KERNEL: /tmp/vmlinux DUMPFILE: /proc/kcore CPUS: 8 DATE: Mon Jun 22 23:24:53 2020 UPTIME: 00:28:21 LOAD AVERAGE: 0.14, 0.05, 0.01 TASKS: 138 NODENAME: com5 RELEASE: 5.4.44-xxxxxxxx-standard VERSION: #1 SMP Sat Jun 20 00:55:50 UTC 2020 MACHINE: aarch64 (unknown Mhz) MEMORY: 7.8 GB crash>
This is a known issue since the 52-bit VA changes were introduced in the arm64 architecture kernel code. Please see https://lkml.org/lkml/2020/6/16/1443 for details.
You can use my public github crash/makedumpfile code for your checks while we wait for the 52-bit VA changes to be accepted upstream.
Thanks, Bhupesh
Hi Bhupesh,
Thanks for taking a look. We already have your kernel patch https://lkml.org/lkml/2020/5/13/1348
Definition of vabits_actual
u64 __section(".mmuoff.data.write") vabits_actual;
EXPORT_SYMBOL(vabits_actual);
I wonder if type of symbols defined above are in /proc/kallsyms. From my testing on ARM64 5.4 live kernel, symbol_value_from_proc_kallsyms("vabits_actual") call fails (returns BADVAL), hence crash (live) fails with FATAL error. With your patch we can determine vabits_actual from VMCOREINFO's tcr_el1_t1sz, crash (live kernel) shouldn't error (FATAL), IIUC.
Thanks, VIjay
Did some more digging. I rebuild kernel with KALLSYMS_ALL config, which lets symbols from data section in /proc/kallsyms, now I can load crash on ARM64 live kernel.
Before moving to 5.4 kernel we ran with 4.19 and crash loaded fine on live kernel. We didn't have KALLSYMS_ALL config enabled in 4.19.
# zcat /proc/config.gz | egrep -e CONFIG_KALLSYMS_ALL -e CONFIG_DEBUG_KERNEL
CONFIG_KALLSYMS_ALL=y
CONFIG_DEBUG_KERNEL=y
# grep vabits_actual /proc/kallsyms
ffff800010ca3a08 r __ksymtab_vabits_actual
ffff800010cb9ce7 r __kstrtab_vabits_actual
ffff8000110ee010 D vabits_actual
overlakedevcom6:~# crash -s /proc/kcore /tmp/vmlinux
strings: standard output: Broken pipe
crash> sys
KERNEL: /tmp/vmlinux
DUMPFILE: /proc/kcore
CPUS: 8
DATE: Sat Jun 27 05:56:56 2020
UPTIME: 00:12:46
LOAD AVERAGE: 0.13, 0.04, 0.01
TASKS: 155
NODENAME: com6
RELEASE: 5.4.44-xxxxxxxx-standard
VERSION: #1 SMP Sat Jun 27 05:05:09 UTC 2020
MACHINE: aarch64 (unknown Mhz)
MEMORY: 7.8 GB
crash>
Hi Vijay,
On Tue, Jun 30, 2020 at 12:41 AM vijaybalakrishna [email protected] wrote:
Did some more digging. I rebuild kernel with KALLSYMS_ALL config, which lets symbols from data section in /proc/kallsyms, now I can load crash on ARM64 live kernel.
Before moving to 5.4 kernel we ran with 4.19 and crash loaded fine on live kernel. We didn't have KALLSYMS_ALL config enabled in 4.19.
zcat /proc/config.gz | egrep -e CONFIG_KALLSYMS_ALL -e CONFIG_DEBUG_KERNEL
CONFIG_KALLSYMS_ALL=y CONFIG_DEBUG_KERNEL=y
grep vabits_actual /proc/kallsyms
ffff800010ca3a08 r __ksymtab_vabits_actual ffff800010cb9ce7 r __kstrtab_vabits_actual ffff8000110ee010 D vabits_actual overlakedevcom6:~# crash -s /proc/kcore /tmp/vmlinux strings: standard output: Broken pipe crash> sys KERNEL: /tmp/vmlinux DUMPFILE: /proc/kcore CPUS: 8 DATE: Sat Jun 27 05:56:56 2020 UPTIME: 00:12:46 LOAD AVERAGE: 0.13, 0.04, 0.01 TASKS: 155 NODENAME: com6 RELEASE: 5.4.44-xxxxxxxx-standard VERSION: #1 SMP Sat Jun 27 05:05:09 UTC 2020 MACHINE: aarch64 (unknown Mhz) MEMORY: 7.8 GB crash>
Indeed, I was wondering why I was not able to reproduce this issue at my end. This explains the difference in the environment: I have KALLSYMS_ALL config enabled in my environment.
Thanks for sharing the same. Glad we were able to conclude why the issue was seen in the first place.
Regards, Bhupesh
First phase of support of the upcoming ARM64 kernel memory map patch introduced KALLSYMS_ALL dependency for running crash on ARM64 live kernel.
@vijaybalakrishna , could you pls tell me how to build crash binary running on arm64 host? I used make target=ARM64 but it still can only run in x86. Thanks.
Hi,
On Tue, Jul 14, 2020 at 12:31 PM tiger20081015 [email protected] wrote:
@vijaybalakrishna https://github.com/vijaybalakrishna , could you pls tell me how to build crash binary running on arm64 host? I used make target=ARM64 but it still can only run in x86. Thanks.
If you are compiling the crash binary directly on the arm64 host, use the following command line: $ make lzo
i.e. directly use the target gcc compiler (which is configured for arm64 host) and git clone the crash-utility code there and use the above command-line to compile the crash binary.
Hope this helps.
Thanks, Bhupesh
@bhupesh-sharma ,thanks for your reply! Actually I'm trying to compile the crash binary on the x86 host, while the binary aims to run on arm64 host. Is there any method to do this?
@bhupesh-sharma ,thanks for your reply! Actually I'm trying to compile the crash binary on the x86 host, while the binary aims to run on arm64 host. Is there any method to do this?
For that you need a cross compiler.
On 7/14/2020 11:27 PM, Santosh wrote:
@bhupesh-sharma <https://github.com/bhupesh-sharma> ,thanks for your reply! Actually I'm trying to compile the crash binary on the x86 host, while the binary aims to run on arm64 host. Is there any method to do this?For that you need a cross compiler.
I have cross compiled using instructions from
Cross Compile files on x86 Linux host for 96Boards ARM systems https://www.96boards.org/documentation/guides/crosscompile/commandline.html
- copy aarch64.cmake from above website
- install ARM 64bit toolchain sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu (and dependencies, like below, I didn't) sudo apt-get install build-essential autoconf libtool cmake pkg-config git python-dev swig3.0 libpcre3-dev nodejs-dev
- in the dir where Makefile present run cmake -DBUILDSWIG=NO -DBUILDSWIGNODE=NO -DBUILDSWIGPYTHON=NO -DCMAKE_TOOLCHAIN_FILE=../aarch64.cmake . make
- now check file type to see if it aarch64 binary
Vijay
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/crash-utility/crash/issues/54#issuecomment-658572191, or unsubscribe https://github.com/notifications/unsubscribe-auth/AP7XU7NB22MJ6ZZ32IQRCZLR3VD4DANCNFSM4OGBPHCA.
On 7/14/2020 12:01 AM, tiger20081015 wrote:
@vijaybalakrishna https://github.com/vijaybalakrishna , could you pls tell me how to build crash binary running on arm64 host? I used make target=ARM64 but it still can only run in x86. Thanks.
I haven't build crash myself (but yocto does this for us!) to run on arm64 host. I also run make target=ARM64, but generated crash tool runs on x86_64 to load ARM64 crash dumps.
Vijay
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/crash-utility/crash/issues/54#issuecomment-658007668, or unsubscribe https://github.com/notifications/unsubscribe-auth/AP7XU7M7VQCJDOBOH5PTGJ3R3P7DZANCNFSM4OGBPHCA.
Hi,
On Wed, Jul 15, 2020 at 7:27 AM tiger20081015 [email protected] wrote:
@bhupesh-sharma https://github.com/bhupesh-sharma ,thanks for your reply! Actually I'm trying to compile the crash binary on the x86 host, while the binary aims to run on arm64 host. Is there any method to do this?
Hmm.. it depends on your use-case:
- If you want to post-process/analyze only the vmcore dump file generated on an arm64 target on a x86_64 host, use the following steps:
- First clean existing compiled binaries using $ make clean
- Then run
On Thu, Jul 16, 2020 at 3:29 AM Bhupesh SHARMA [email protected] wrote:
Hi,
On Wed, Jul 15, 2020 at 7:27 AM tiger20081015 [email protected] wrote:
@bhupesh-sharma ,thanks for your reply! Actually I'm trying to compile the crash binary on the x86 host, while the binary aims to run on arm64 host. Is there any method to do this?
Sorry. It seemed I missed a few steps before sending out the email:
Hmm.. it depends on your use-case:
- If you want to post-process/analyze only the vmcore dump file generated on an arm64 target on a x86_64 host, use the following steps:
- First clean existing compiled binaries using $ make clean
- Then run: $ ./configure target=ARM64 $ make target=ARM64
- This will compile the crash binary for arm64 target but it still needs to be run on a x86_64 host: $ ./crash vmlinux(from arm64 system) vmcore(from arm64 system)
- If you want to perform live-analysis on an arm64 machine/target, use a aarch64 toolchain (you can get one from the Linaro website: https://releases.linaro.org/components/toolchain/binaries/) and use CROSS_COMPILATION techniques. Although this is a less frequently used use-case.
Please let me know which use-case you want to target, so that I can further share details.
Thanks, Bhupesh
@bhupesh-sharma ， I‘ve already set up crash for case 1. What I’m trying to do is live-analysis on arm64 host. I have some knowledge of cross compilation and already got arm64 cross compile tool chain. But I don't know how to add cross compile build for crash tool. Could you please give more detailed steps?