pkg audit crash

[ohauer]

I'm running 20+ FreeBSD systems all having the same OS revision and pkg from local PD build system

$> uname -a
FreeBSD foo.my.site 12.3-RELEASE-p5 FreeBSD 12.3-RELEASE-p5 GENERIC  amd64

$> freebsd-version -ku
12.3-RELEASE-p5
12.3-RELEASE-p5

On two system the command pkg audit results in pkg crashing but no core file is created.

Already tried the following:

• remove all packages
• install only pkg

#> pkg audit -F
vulnxml file up-to-date
0 problem(s) in 0 installed package(s) found.
Child process pid=9808 terminated abnormally: Segmentation fault

Any hint how I can help further to solve this issue?

[vstakhov]

You can run the whole thing in lldb: lldb pkg and then, in the lldb prompt, run the actual command run audit -F. After that, you should be able to catch the crash and print the trace using bt full command.

[ohauer]

Thanks @vstakhov Im getting the following output on both systems:

#> lldb /usr/local/sbin/pkg
(lldb) target create "/usr/local/sbin/pkg"
Current executable set to '/usr/local/sbin/pkg' (x86_64).
(lldb) run audit -F
Process 11505 launching
Process 11505 launched: '/usr/local/sbin/pkg' (x86_64)
vulnxml file up-to-date
0 problem(s) in 0 installed package(s) found.
Process 11505 stopped
* thread #1, name = 'pkg', stop reason = signal SIGSEGV: invalid address (fault address: 0x111)
    frame #0: 0x0000000000408fa0 pkg`ucl_hash_destroy + 128
pkg`ucl_hash_destroy:
->  0x408fa0 <+128>: movq   0x10(%rdi), %rbx
    0x408fa4 <+132>: callq  *%r12
    0x408fa7 <+135>: movq   %rbx, %rdi
    0x408faa <+138>: testq  %rbx, %rbx
(lldb) bt full
error: bt [<digit> | all]
(lldb) bt all
* thread #1, name = 'pkg', stop reason = signal SIGSEGV: invalid address (fault address: 0x111)
  * frame #0: 0x0000000000408fa0 pkg`ucl_hash_destroy + 128
    frame #1: 0x0000000000416612 pkg`ucl_object_unref + 306
    frame #2: 0x000000000048afc6 pkg`pkg_shutdown + 38
    frame #3: 0x0000000800ca8e75 libc.so.7`__cxa_finalize + 325
    frame #4: 0x0000000800c3b371 libc.so.7`exit + 33
    frame #5: 0x00000000002a4947 pkg`_start(ap=<unavailable>, cleanup=<unavailable>) at crt1.c:76:2
(lldb) quit
Quitting LLDB will kill one or more processes. Do you really want to proceed: [Y/n] y

and the following on one of the systems no having that issue:

# lldb /usr/local/sbin/pkg
(lldb) target create "/usr/local/sbin/pkg"
Current executable set to '/usr/local/sbin/pkg' (x86_64).
(lldb) run audit
Process 5280 launching
Process 5280 launched: '/usr/local/sbin/pkg' (x86_64)
0 problem(s) in 0 installed package(s) found.
Process 5280 exited with status = 0 (0x00000000)
(lldb) quit

all systems running the same pkg release from the same build

#> pkg -v
1.18.3

[bapt]

can you host somewhere you /var/db/local.sqlite and your /var/db/pkg/vuln.xml ?

[ohauer]

sure, pkg_crash_issue_2053.txz

The strange thing is pkg crashes on this systems even I removed all ports and also all files in /var/db/pkg, bootstrapped pkg fresh and do a pkg audit -F (only with pkg installed)

I tested now with pkg-static and get the same crash

# lldb /usr/local/sbin/pkg-static
(lldb) target create "/usr/local/sbin/pkg-static"
Current executable set to '/usr/local/sbin/pkg-static' (x86_64).
(lldb) run audit
Process 74003 launching
Process 74003 launched: '/usr/local/sbin/pkg-static' (x86_64)
0 problem(s) in 0 installed package(s) found.
Process 74003 stopped
* thread #1, name = 'pkg-static', stop reason = signal SIGSEGV: invalid address (fault address: 0x111)
    frame #0: 0x00000000003d5b00 pkg-static`ucl_hash_destroy + 128
pkg-static`ucl_hash_destroy:
->  0x3d5b00 <+128>: movq   0x10(%rdi), %rbx
    0x3d5b04 <+132>: callq  *%r12
    0x3d5b07 <+135>: movq   %rbx, %rdi
    0x3d5b0a <+138>: testq  %rbx, %rbx
(lldb) bt all
* thread #1, name = 'pkg-static', stop reason = signal SIGSEGV: invalid address (fault address: 0x111)
  * frame #0: 0x00000000003d5b00 pkg-static`ucl_hash_destroy + 128
    frame #1: 0x00000000003e1232 pkg-static`ucl_object_unref + 306
    frame #2: 0x0000000000559116 pkg-static`pkg_shutdown + 38
    frame #3: 0x00000000008ee93f pkg-static`__cxa_finalize(dso=0x0000000000000000) at atexit.c:240:5
    frame #4: 0x0000000000888dce pkg-static`exit(status=0) at exit.c:74:2
    frame #5: 0x00000000003be107 pkg-static`_start(ap=<unavailable>, cleanup=<unavailable>) at crt1.c:76:2

Additional test with the stripped down vuxml file located in the test_vuxml_also_crashes folder

pkg audit -f test_vuxml_also_crashes /vuln.xml pkg -> crash
pkg-static audit -f test_vuxml_also_crashes /vuln.xml pkg -> crash

[bapt]

I can't reproduce the issue here :(

1. Are you building from ports or are you using official packages?
2. if you are building from ports can you try: Can you try adding --with-asan to the CONFIGURE_ARGS (build with WITH_DEBUG on port make invocation) ? if 2 is possible, invoking pkg should provide you a big stacktrace

[ohauer]

Hope I have the required information covered in the lldb output: PKG_DEBUG.txt

Plz. note:

• not running in lldb, pkg audit did not crash and just hang until stopped with ^C.
• running in lldb the line "==13848==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x603000018c28 at pc 0x000000aeab04 bp 0x7fffffffd980 sp 0x7fffffffd978" was printed immediately, but lldb was not comming back to command promt until usage of ^C

[bapt]

when build with libasan, you are not supposed to run inside lldb, it is supposed to be able to print a stack trace on its own. that said, in the trace you do obtain via lldb the error seems kind of obvious, but I don't know how you can end up in that situation :(, which is painful because that mean I cannot test myself a fix.

[vstakhov]

https://github.com/freebsd/pkg/blob/master/libpkg/pkg_audit.c#L637 - n = 0 https://github.com/freebsd/pkg/blob/master/libpkg/pkg_audit.c#L641 - n = 0, allocate ret of 1 element https://github.com/freebsd/pkg/blob/master/libpkg/pkg_audit.c#L641 - n = 1, access ret[1] -> crash

Literally, it is off-by-one error for the case when the whole vulnxml is empty.

[bapt]

yes the error is obvious, but how n can be 0 in real life

[bapt]

please try https://people.freebsd.org/~bapt/patch-ohauer (add it to your file directory and rebuild)

[ocochard]

Same problem here with this command: pkg query \*\ %n,\ license:%L,\ %w

[bapt]

this is not the same problem

[ohauer]

Hi @bapt the patch does not solve the crashes. until now my solution was to remove all ports and use previous pkg-1.7.x, use `pkg lock pkg' and reinstall everything.

If there is anything I can try let me know. PS the pkg query from @ocochard also let pkg-1.8 crash the affected systems

[ohauer]

Done some test with only pkg from custom build (12.3-p0, 12.3-p5) and pkg from pkg.freebbsd,org installed and found perhaps the missing hint:

This crashes, note the line HANDLE_RC_SCRIPTS exist twice, removing the duplicate line fixed the crashes on the affected systems.

$> cat /usr/local/etc/pkg.conf
REPO_AUTOUPDATE = false;
HANDLE_RC_SCRIPTS = true;
HANDLE_RC_SCRIPTS = true;

$> pkg audit
0 problem(s) in 0 installed package(s) found.
Child process pid=25062 terminated abnormally: Bus error

no crash with with fixed pkg.conf:

$> cat /usr/local/etc/pkg.conf
REPO_AUTOUPDATE = false;
HANDLE_RC_SCRIPTS = true;

$> pkg audit
0 problem(s) in 0 installed package(s) found.

Even after installing all required packages on the system, pkg audit and the reported command from @ocochard runs without issues.

$> pkg query \*\ %n,\ license:%L,\ %w
* apr, license:APACHE20, http://apr.apache.org/
* bash, license:GPLv3+, https://www.gnu.org/software/bash/
* cmdwatch, license:GPLv2+, UNKNOWN
* db5, license:SLEEPYCAT, https://www.oracle.com/database/berkeley-db/db.html
* expat, license:MIT, https://github.com/libexpat/libexpat
* expect, license:PD, http://expect.sourceforge.net
* gdbm, license:GPLv3+, https://www.gnu.org.ua/software/gdbm/
* gettext-runtime, license:LGPL21+, https://www.gnu.org/software/gettext/
* gettext-runtime, license:GPLv3+, https://www.gnu.org/software/gettext/
* glib, license:LGPL20, https://www.gtk.org/
* gobgp, license:APACHE20, https://github.com/osrg/gobgp
* icu, license:ICU, http://site.icu-project.org/
...