[cpp] extractor crashed when creating database

[Godrichp]

I'm trying to create a codeql database for the openharmony project (https://github.com/openharmony/manifest). Although I can get a packed database, lots of files are not included in the database.

I checked the "database-create-xxx.log" and found lots of following exceptions as follows.

52832692_0.trap.zst, 1: java.io.IOException: Not enough input bytes
io.airlift.compress.zstd.ZstdInputStream.read(ZstdInputStream.java:86)
com.semmle.util.trap.CompressedFileInputStream$WrappedZstdInputStream.read(CompressedFileInputStream.java:54)
com.semmle.inmemory.trap.TrapInputStream.read(TrapInputStream.java:60)
com.semmle.inmemory.trap.TrapScanner.fill(TrapScanner.java:451)
com.semmle.inmemory.trap.TrapScanner.ensureNext(TrapScanner.java:428)
com.semmle.inmemory.trap.TrapScanner.nextToken(TrapScanner.java:61)
com.semmle.inmemory.trap.TRAPReader.scanTuplesAndLabels(TRAPReader.java:493)
com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:414)
com.semmle.inmemory.trap.ImportTasksProcessor.process(ImportTasksProcessor.java:234)
com.semmle.inmemory.trap.ImportTasksProcessor.lambda$importTrap$1(ImportTasksProcessor.java:154)
com.semmle.util.concurrent.FutureUtils.lambda$mapAsync_$8(FutureUtils.java:161)
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base/java.lang.Thread.run(Unknown Source)
at (start of line)

I then checked the "build-tracer.log".(build-tracer.log.zip) Lots of errors leads to the crash of the extractor. It seems that all code that uses unique_ptr_ in libc++ will raise exceptions. The following is a snippet of errors.

"../../third_party/skia/third_party/externals/icu/source/i18n/unicode/unumberrangeformatter.h", line 467: error: no instance of constructor "std::__h::unique_ptr<_Tp, _Dp>::unique_ptr [with _Tp=UFormattedNumberRange, _Dp=void (*)(UFormattedNumberRange *)]" matches the argument list
            argument types are: (UFormattedNumberRange *, void (UFormattedNumberRange *))
  U_DEFINE_LOCAL_OPEN_POINTER(LocalUFormattedNumberRangePointer, UFormattedNumberRange, unumrf_closeResult);
  ^
"/mnt/disk/source/openharmony/workingdir/prebuilts/clang/ohos/linux-x86_64/llvm/bin/../include/libcxx-ohos/include/c++/v1/__memory/unique_ptr.h", line 109: note: number of parameters of function "std::__h::unique_ptr<_Tp, _Dp>::unique_ptr(const std::__h::unique_ptr<UFormattedNumberRange, void (*)(UFormattedNumberRange *)> &) [with _Tp=UFormattedNumberRange, _Dp=void (*)(UFormattedNumberRange *)]" does not match the call
  class _LIBCPP_UNIQUE_PTR_TRIVIAL_ABI _LIBCPP_TEMPLATE_VIS unique_ptr {
                                                            ^
"/mnt/disk/source/openharmony/workingdir/prebuilts/clang/ohos/linux-x86_64/llvm/bin/../include/libcxx-ohos/include/c++/v1/__memory/unique_ptr.h", line 210: note: number of parameters of function template "std::__h::unique_ptr<_Tp, _Dp>::unique_ptr(std::__h::unique_ptr<_Up, _Ep> &&) noexcept [with _Tp=UFormattedNumberRange, _Dp=void (*)(UFormattedNumberRange *)]" does not match the call
    unique_ptr(unique_ptr<_Up, _Ep>&& __u) _NOEXCEPT
    ^
"/mnt/disk/source/openharmony/workingdir/prebuilts/clang/ohos/linux-x86_64/llvm/bin/../include/libcxx-ohos/include/c++/v1/__memory/unique_ptr.h", line 201: note: number of parameters of function "std::__h::unique_ptr<_Tp, _Dp>::unique_ptr(std::__h::unique_ptr<_Tp, _Dp> &&) noexcept [with _Tp=UFormattedNumberRange, _Dp=void (*)(UFormattedNumberRange *)]" does not match the call
    unique_ptr(unique_ptr&& __u) _NOEXCEPT
    ^
"/mnt/disk/source/openharmony/workingdir/prebuilts/clang/ohos/linux-x86_64/llvm/bin/../include/libcxx-ohos/include/c++/v1/__memory/unique_ptr.h", line 168: note: number of parameters of function template "std::__h::unique_ptr<_Tp, _Dp>::unique_ptr() noexcept [with _Tp=UFormattedNumberRange, _Dp=void (*)(UFormattedNumberRange *)]" does not match the call
    _LIBCPP_CONSTEXPR unique_ptr() _NOEXCEPT : __ptr_(__value_init_tag(), __value_init_tag()) {}
                      ^

Then I checked the unique_ptr_.h(unique_ptr.h.txt) But I cannot find anything wrong. I truly appreciate your help in solving this problem.

Additional Info:

1. Codeql version: the latest 2.17.2. Platform: Linux, Ubuntu 20.04

2. Reproduce procedure:

a.download the source code following the guide in this page (https://www.openharmony.cn/download/). Or just running the following commands.

curl https://gitee.com/oschina/repo/raw/fork_flow/repo-py3 -o /usr/local/bin/repo
chmod a+x /usr/local/bin/repo
repo init -u https://gitee.com/openharmony/manifest.git -b master --no-repo-verify
repo sync -c
repo forall -c 'git lfs pull'

b.generate gn files in the project folder.

./build.py --product-name=rk3568 --ccache=false --build-only-gn

c.run codeql commands to create database

codeql database create -l cpp -c "./prebuilts/build-tools/linux-x86/bin/ninja -w dupbuild=warn -C out/rk3568 icu" ../../codeql-database/icu-1 --overwrite -vv

[aibaars]

Thanks for the detailed report. I'll ask the team to look into the errors.

[jketema]

Hi @Godrichp,

Thanks for the report. The build-tracer.log file you've included seems to be from a database created with CodeQL 2.17.0. Would you be able to share the build-tracer.log for a database created with CodeQL 2.17.2? Some things have change related to template instantiation between 2.17.0 and 2.17.2 which might affect you, so I'd like to be sure the reported errors establish whether the same errors still occur or whether they have changed.

[jketema]

In addition could you share the complete database-create-xxx.log file? Preferably the one from the same database creation run that created the requested build-tracer.log file, as that will allow for matching up some file names mentioned in the log files.

[Godrichp]

Thanks for your reply. I run the building process again and the logs are attached.

build-tracer.log.1.zip database-create-20240509.095715.753.log

[jketema]

Hi @Godrichp,

Thanks for the logs, we'll investigate this further.

[jketema]

@Godrichp Apologies for not getting back earlier. A workaround for the crash was introduced in CodeQL 2.17.5. A proper fix will be part of CodeQL 2.19.2, which should be released in a few weeks.

[jketema]

A proper fix will be part of CodeQL 2.19.2, which should be released in a few weeks.

Unfortunately we had to back out the fix, because of some unforeseen consequences it had, and this fix will not be part of CodeQL 2.19.2. It is currently not clear when we will have a proper fix. I'm reopening this issue for now.

Note that the reported crash should in any case no longer occur.

[jketema]

Unfortunately we had to back out the fix, because of some unforeseen consequences it had, and this fix will not be part of CodeQL 2.19.2. It is currently not clear when we will have a proper fix. I'm reopening this issue for now.

This should be fixed in CodeQL 2.21.0, which will be released slightly over a month from now.