llvm-project
llvm-project copied to clipboard
Published
20 hours ago •
llvm
llvm
clang static analyzer produces invalid externalDefMap.txt file during CTU analysis
I am running clang static analyzer via CodeChecker with CTU analysis on Firefox (so it's a big project.) I am running into errors because it seems to generate an invalid externalDefMap and spits out the following error during the CodeChecker run:
error: error parsing index file: 'blah/ctu-analysis/reports/ctu-dir/x86_64/externalDefMap.txt' line: 2283 '<USR-Length>:<USR> <File-Path>' format expected
When I grab the externalDefMap file while it is running (because it disappears after completion) and I look for bad lines I have the following:
$ cat bad-externalDefMap.txt | egrep -e "^([0-9]+):[a-z]:([^ ]+) ([^ ]+)$" -v > bad-lines.txt
$ wc -l bad-lines.txt
314 bad-lines.txt
$ wc -l bad-externalDefMap.txt
204168 bad-externalDefMap.txt
The externalDefMap.txt is ~36MB so I'm not attaching the whole thing, but here is a sample of the bad lines:
59:c:@S@nsBoxFrame@F@operator ast/new#l#*$@N@mozilla@S@PresShell#S /home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsBoxFrame.cpp.ast
97:c:@N@mozilla@S@nsDisplayXULEventRedirector@F@operator ast/new#l#*$@N@mozilla@S@nsDisplayListBuilder#S /home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsBoxFrame.cpp.ast
153:c:@N@icu_71@S@UnicodeSet@F@applyPattern#&$@N@icu_71@S@RuleCharacterIterator#*1$@N@icu_71@S@SymbolTable#&$@N@icu_71@S@UnicodeString#i# ast/#I#&$@E@UErrorCode# /home/tom/Documents/moz/static-analysis/mozilla-unified/intl/icu/source/common/uniset_props.cpp.ast
80:c:@S@nsComputedDOMStyle@F@GetNonStaticPositionOffset#$@N@mozilla@E@Side#b# ast/#S1_# /home/tom/Documents/moz/static-analysis/mozilla-unified/layout/style/nsComputedDOMStyle.cpp.ast
47:c:@N@mozilla@S@SPSNALIterator@F@operator ast/bool#1 /home/tom/Documents/moz/static-analysis/mozilla-unified/dom/media/platforms/agnostic/bytestreams/H264.cpp.ast
69:c:@S@nsDocElementBoxFrame@F@operator ast/new#l#*$@N@mozilla@S@PresShell#S /home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsDocElementBoxFrame.cpp.ast
and some good lines:
20:c:@F@jpeg_fdct_ifast ast/home/tom/Documents/moz/static-analysis/mozilla-unified/media/libjpeg/jfdctfst.c.ast
19:c:@F@ArrayString#i# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/gfx/angle/checkout/src/common/angleutils.cpp.ast
67:c:@F@ArrayIndexString#&1$@N@std@S@vector>#i#$@N@std@S@allocator>#i# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/gfx/angle/checkout/src/common/angleutils.cpp.ast
95:c:@F@FormatStringIntoVector#*1C#*$@S@__va_list_tag#&$@N@std@S@vector>#C#$@N@std@S@allocator>#C# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/gfx/angle/checkout/src/common/angleutils.cpp.ast
23:c:@N@angle@DirtyPointer ast/home/tom/Documents/moz/static-analysis/mozilla-unified/gfx/angle/checkout/src/common/angleutils.cpp.ast
62:c:@N@mozilla@N@dom@S@PrefValue@F@operator=#&1$@S@nsTString>#C# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/objdir/ipc/ipdl/PrefsTypes.cpp.ast
Following are three examples of a bad line (prefaced with BAD> added by me of course) surrounded by good lines. This is pure speculation but maybe it's a threading issue with a buffer; as the strings seem like it might be something with interleving? Some of the offending strings occur shortly later in the file...
39:c:@N@js@N@wasm@S@WasmFrameIter@F@done#1 ast/home/tom/Documents/moz/static-analysis/mozilla-unified/js/src/wasm/WasmFrameIter.cpp.ast
104:c:@N@js@N@wasm@F@GenerateFunctionEpilogue#&$@N@js@N@jit@S@MacroAssembler#i#*$@N@js@N@wasm@S@FuncOffsets# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/js/src/wasm/WasmFrameIter.cpp.ast
45:c:@N@js@N@wasm@S@WasmFrameIter@F@displayURL#1 ast/home/tom/Documents/moz/static-analysis/mozilla-unified/js/src/wasm/WasmFrameIter.cpp.ast
89:c:@N@js@N@wasm@S@ProfilingFrameIterator@F@ProfilingFrameIterator#*1$@N@js@N@wasm@S@Frame# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/js/src/wasm/WasmFrameIter.cpp.ast
84:c:@N@js@N@wasm@F@ClearExitFP#&$@N@js@N@jit@S@MacroAssembler#$@N@js@N@jit@S@Register# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/js/src/wasm/WasmFrameIter.cpp.ast
131:c:@N@js@N@wasm@F@GenerateExitEpilogue#&$@N@js@N@jit@S@MacroAssembler#i#$@N@js@N@wasm@S@ExitReason#*$@N@js@N@wasm@S@CallableOffsets# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/js/src/wasm/WasmFrameIter.cpp.ast
156:c:@N@js@N@wasm@F@GenerateFunctionPrologue#&$@N@js@N@jit@S@MacroAssembler#&1$@N@js@N@wasm@S@TypeIdDesc#&1$@N@mozilla@S@Maybe>#i#*$@N@js@N@wasm@S@FuncOffsets# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/js/src/wasm/WasmFrameIter.cpp.ast
50:c:@S@nsDocElementBoxFrame@F@~nsDocElementBoxFrame# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsDocElementBoxFrame.cpp.ast
BAD> 69:c:@S@nsDocElementBoxFrame@F@operator ast/new#l#*$@N@mozilla@S@PresShell#S /home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsDocElementBoxFrame.cpp.ast
97:c:@S@nsDocElementBoxFrame@F@nsDocElementBoxFrame#*$@N@mozilla@S@ComputedStyle#*$@S@nsPresContext# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsDocElementBoxFrame.cpp.ast
68:c:@S@nsDocElementBoxFrame@F@QueryFrame#$@S@nsQueryFrame@E@FrameIID#1 ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsDocElementBoxFrame.cpp.ast
45:c:@S@nsDocElementBoxFrame@F@IsFrameOfType#i#1 ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsDocElementBoxFrame.cpp.ast
111:c:@S@nsDocElementBoxFrame@F@CreateAnonymousContent#&$@S@nsTArray>#$@S@nsIAnonymousContentCreator@S@ContentInfo# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsDocElementBoxFrame.cpp.ast
99:c:@S@nsDocElementBoxFrame@F@DestroyFrom#*$@S@nsIFrame#&$@N@mozilla@N@layout@S@PostFrameDestroyData# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsDocElementBoxFrame.cpp.ast
35:c:@S@nsDocElementBoxFrame@kFrameIID ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsDocElementBoxFrame.cpp.ast
34:c:@S@nsDocElementBoxFrame@kClassID ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsDocElementBoxFrame.cpp.ast
63:c:@S@nsDocElementBoxFrame@F@GetFrameName#&$@S@nsTSubstring>#q#1 ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsDocElementBoxFrame.cpp.ast
86:c:@S@nsDocElementBoxFrame@F@AppendAnonymousContentTo#&$@S@nsTArray>#*$@S@nsIContent#i# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsDocElementBoxFrame.cpp.ast
40:c:@S@nsDocElementBoxFrame@F@GetFrameId#1 ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsDocElementBoxFrame.cpp.ast
84:c:@F@NS_NewDocElementBoxFrame#*$@N@mozilla@S@PresShell#*$@N@mozilla@S@ComputedStyle# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsDocElementBoxFrame.cpp.ast
84:c:@S@nsBoxFrame@F@RemoveFrame#$@N@mozilla@N@layout@E@FrameChildListID#*$@S@nsIFrame# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsBoxFrame.cpp.ast
90:c:@S@nsBoxFrame@F@AppendDirectlyOwnedAnonBoxes#&$@S@nsTArray>#$@S@nsIFrame@S@OwnedAnonBox# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsBoxFrame.cpp.ast
55:c:@S@nsBoxFrame@F@GetXULPrefSize#&$@S@nsBoxLayoutState# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsBoxFrame.cpp.ast
51:c:@S@nsBoxFrame@F@AttributeChanged#I#*$@S@nsAtom#I# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsBoxFrame.cpp.ast
128:c:@S@nsBoxFrame@F@InsertFrames#$@N@mozilla@N@layout@E@FrameChildListID#*$@S@nsIFrame#*1$@S@nsLineList_iterator#&$@S@nsFrameList# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsBoxFrame.cpp.ast
115:c:@S@nsBoxFrame@F@BuildDisplayListForChildren#*$@N@mozilla@S@nsDisplayListBuilder#&1$@N@mozilla@S@nsDisplayListSet# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsBoxFrame.cpp.ast
BAD> 97:c:@N@mozilla@S@nsDisplayXULEventRedirector@F@operator ast/new#l#*$@N@mozilla@S@nsDisplayListBuilder#S /home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsBoxFrame.cpp.ast
58:c:@S@nsBoxFrame@F@QueryFrame#$@S@nsQueryFrame@E@FrameIID#1 ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsBoxFrame.cpp.ast
95:c:@S@nsBoxFrame@F@SetInitialChildList#$@N@mozilla@N@layout@E@FrameChildListID#&$@S@nsFrameList# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsBoxFrame.cpp.ast
175:c:@N@mozilla@S@nsDisplayXULEventRedirector@F@HitTest#*$@N@mozilla@S@nsDisplayListBuilder#&1$@S@nsRect#*$@N@mozilla@S@nsDisplayItem@S@HitTestState#*$@S@nsTArray>#*$@S@nsIFrame# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsBoxFrame.cpp.ast
109:c:@S@nsXULEventRedirectorWrapper@F@WrapItem#*$@N@mozilla@S@nsDisplayListBuilder#*$@N@mozilla@S@nsDisplayItem# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsBoxFrame.cpp.ast
74:c:@N@mozilla@S@nsDisplayXULEventRedirector@F@~nsDisplayXULEventRedirector# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsBoxFrame.cpp.ast
55:c:@N@mozilla@S@nsDisplayXULEventRedirector@F@ItemType#S ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsBoxFrame.cpp.ast
104:c:@S@nsBoxFrame@F@BuildDisplayList#*$@N@mozilla@S@nsDisplayListBuilder#&1$@N@mozilla@S@nsDisplayListSet# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/layout/xul/nsBoxFrame.cpp.ast
96:c:@N@icu_71@S@UnicodeSet@F@applyFilter#*Fr(#I#*v)#S2_#*1$@N@icu_71@S@UnicodeSet#&$@E@UErrorCode# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/intl/icu/source/common/uniset_props.cpp.ast
83:c:@N@icu_71@S@UnicodeSet@F@UnicodeSet#&1$@N@icu_71@S@UnicodeString#&$@E@UErrorCode# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/intl/icu/source/common/uniset_props.cpp.ast
76:c:@N@icu_71@S@UnicodeSet@F@resemblesPattern#&1$@N@icu_71@S@UnicodeString#I#S ast/home/tom/Documents/moz/static-analysis/mozilla-unified/intl/icu/source/common/uniset_props.cpp.ast
84:c:@N@icu_71@S@UnicodeSet@F@resemblesPropertyPattern#&1$@N@icu_71@S@UnicodeString#I#S ast/home/tom/Documents/moz/static-analysis/mozilla-unified/intl/icu/source/common/uniset_props.cpp.ast
91:c:@N@icu_71@S@UnicodeSet@F@resemblesPropertyPattern#&$@N@icu_71@S@RuleCharacterIterator#I#S ast/home/tom/Documents/moz/static-analysis/mozilla-unified/intl/icu/source/common/uniset_props.cpp.ast
95:c:@N@icu_71@S@UnicodeSet@F@applyPropertyAlias#&1$@N@icu_71@S@UnicodeString#S0_#&$@E@UErrorCode# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/intl/icu/source/common/uniset_props.cpp.ast
BAD> 153:c:@N@icu_71@S@UnicodeSet@F@applyPattern#&$@N@icu_71@S@RuleCharacterIterator#*1$@N@icu_71@S@SymbolTable#&$@N@icu_71@S@UnicodeString#i# ast/#I#&$@E@UErrorCode# /home/tom/Documents/moz/static-analysis/mozilla-unified/intl/icu/source/common/uniset_props.cpp.ast
128:c:@N@icu_71@S@UnicodeSet@F@applyPropertyPattern#&$@N@icu_71@S@RuleCharacterIterator#&$@N@icu_71@S@UnicodeString#&$@E@UErrorCode# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/intl/icu/source/common/uniset_props.cpp.ast
38:c:@aN@F@uniset_getUnicode32Instance_71 ast/home/tom/Documents/moz/static-analysis/mozilla-unified/intl/icu/source/common/uniset_props.cpp.ast
81:c:@N@icu_71@S@UnicodeSet@F@applyIntPropertyValue#$@E@UProperty#I#&$@E@UErrorCode# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/intl/icu/source/common/uniset_props.cpp.ast
151:c:@N@icu_71@S@UnicodeSet@F@applyPatternIgnoreSpace#&1$@N@icu_71@S@UnicodeString#&$@N@icu_71@S@ParsePosition#*1$@N@icu_71@S@SymbolTable#&$@E@UErrorCode# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/intl/icu/source/common/uniset_props.cpp.ast
85:c:@N@icu_71@S@UnicodeSet@F@applyPattern#&1$@N@icu_71@S@UnicodeString#&$@E@UErrorCode# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/intl/icu/source/common/uniset_props.cpp.ast
26:c:@S@SuperBlitter@F@flush# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/gfx/skia/skia/src/core/SkScan_AntiPath.cpp.ast
34:c:@S@SuperBlitter@F@~SuperBlitter# ast/home/tom/Documents/moz/static-analysis/mozilla-unified/gfx/skia/skia/src/core/SkScan_AntiPath.cpp.ast
I am running only clangsa (not clang-tidy) passing --timeout=60 -j 45 (among other options to certain checkers that are probably not relevant) to CodeChecker. I'm running clang trunk: clang version 16.0.0 (https://github.com/llvm/llvm-project.git 10a7ee0bac211810376f29a879a9f73ed2ab15fc) and I'm using a compile_commands.json file that firefox can generate quite easily with the build command ./mach build-backend -b CompileDB.
If anyone would like to recreate this situation I can provide instructions for how to set up the environment and run it, but warning - it's slow (on the order of multiple hours on a very fast machine) and takes 150GB+ of disk space at a minimum. I can also self-host the full externalDefMap file or related.
We had a commit related to parsing CTU extdefmaps in CodeChecker. https://github.com/Ericsson/codechecker/commit/14d901d41ee686089ab1d290d19c863e09d3dc1f Have you checked CodeChecker master? @tomrittervg
The version I was using was about a month before that commit; I will update and try again!