plasma-umass
dwarf::format_error
Hi,
Something is off, I am trying to run coz with CryptoMiniSat with ApproxMC but I am getting:
$ coz run --- ./approxmc blasted_case110.cnf.gz.no_w.cnf.gz -v2
[libcoz.cpp:100] bootstrapping coz
[libcoz.cpp:128] Including MAIN, which is /[...]/build/approxmc
terminate called after throwing an instance of 'dwarf::format_error'
what(): DW_FORM_sec_offset not expected for attribute (DW_AT)0x2137
I am using latest coz and latest libelfin -- both compiled from git. Do you know what could be the issue? Both ApproxMC and CryptoMiniSat are open-source, see https://github.com/msoos/cryptominisat and https://github.com/meelgroup/approxmc
But I am getting the error with plain CryptoMiniSat too, so it's probably in CryptoMiniSat. I am compiling with -g and mtune=auto, and I am linking in boost, zlib and m4ri. All are dynamically linked. Should I try without linking all these? I am using Arch Linux, so I think the debug symbols should be available for all libraries.
Do you know what that issue above could mean?
@ccurtsinger will chime in, but in the meanwhile, can you attach the input files (or at least one input file that reproduces the issue)? Thanks!
Ah, thanks for the reply! Below is a cleaner, reproducible setup with binaries attached.
I build cryptominisat as such:
git clone https://github.com/msoos/cryptominisat
cd cryptominisat
mkdir build
cd build
rm -rf cm* CM* lib* cryptomini* Testing* tests* pycryptosat include tests cusp* scalmc*
cmake -DENABLE_PYTHON_INTERFACE=OFF -DENABLE_TESTING=OFF -DNOZLIB=ON -DNOM4RI=ON ..
make -j6 VERBOSE=1
make test
This builds with no dependencies, effectively.
Example build line:
cd /home/soos/development/sat_solvers/cryptominisat/build/cmsat5-src && /usr/lib/ccache/bin/c++ -DBOOST_TEST_DYN_LINK -DCMS_TESTING_ENABLED -DYALSAT_FPU -Dcryptominisat5_EXPORTS -I/home/soos/development/sat_solvers/cryptominisat -I/home/soos/development/sat_solvers/cryptominisat/build/cmsat5-src -I/home/soos/development/sat_solvers/cryptominisat/utils/gtest/include -Wall -Wextra -Wunused -Wsign-compare -fno-omit-frame-pointer -Wtype-limits -Wuninitialized -Wno-deprecated -Wstrict-aliasing -Wpointer-arith -Wpointer-arith -Wformat-nonliteral -Winit-self -Wparentheses -Wunreachable-code -ggdb3 -Wlogical-op -Wrestrict -Wnull-dereference -Wdouble-promotion -Wshadow -Wformat=2 -Wextra-semi -pedantic -Wno-class-memaccess -fPIC -g -pthread -O2 -fPIC -std=gnu++11 -o CMakeFiles/cryptominisat5.dir/solver.cpp.o -c /home/soos/development/sat_solvers/cryptominisat/src/solver.cpp
Note the -g, so I am doing as prescribed, I think :)
Now I have a binary (attached), along with a library (attached):
$ ldd cryptominisat5_simple
linux-vdso.so.1 (0x00007ffd7238c000)
libcryptominisat5.so.5.6 => /home/soos/development/sat_solvers/cryptominisat/build/lib/libcryptominisat5.so.5.6 (0x00007f880374d000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f8803517000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f88033d1000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f88033b7000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f8803396000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f88031d3000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8803874000)
$ ldd /home/soos/development/sat_solvers/cryptominisat/build/lib/libcryptominisat5.so.5.6
linux-vdso.so.1 (0x00007ffe1f7f7000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007faa46fee000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007faa46ea8000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007faa46e8e000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007faa46e6d000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007faa46caa000)
/usr/lib64/ld-linux-x86-64.so.2 (0x00007faa4733d000)
Seems pretty clean to me. Then:
$ coz run --- ./cryptominisat5_simple UTI-20-10p0.cnf-unz
[libcoz.cpp:100] bootstrapping coz
[libcoz.cpp:128] Including MAIN, which is /home/soos/development/sat_solvers/cryptominisat/build/cryptominisat5_simple
terminate called after throwing an instance of 'dwarf::format_error'
what(): DW_FORM_sec_offset not expected for attribute (DW_AT)0x2137
In case the issue is that coz.h was not added along with the macro, that's actually not the issue. Here is a diff that adds them (I compiled the binaries attached with this):
diff --git a/src/searcher.cpp b/src/searcher.cpp
index e238cc5ae..2c275b1a5 100644
--- a/src/searcher.cpp
+++ b/src/searcher.cpp
@@ -28,16 +28,18 @@ THE SOFTWARE.
#include "varreplacer.h"
#include "clausecleaner.h"
#include "propbyforgraph.h"
#include <algorithm>
#include <sstream>
#include <cstddef>
#include <cmath>
#include <ratio>
+#include <coz.h>
+
#include "sqlstats.h"
#include "datasync.h"
#include "reducedb.h"
#include "sqlstats.h"
#include "watchalgos.h"
#include "hasher.h"
#include "solverconf.h"
#include "distillerlong.h"
@@ -1179,16 +1181,17 @@ lbool Searcher::search()
#ifdef USE_GAUSS
gqhead = qhead;
#endif
if (update_bogoprops) {
confl = propagate<update_bogoprops>();
} else {
confl = propagate_any_order_fast();
}
+ COZ_PROGRESS_NAMED("propagate finished")
if (!confl.isNULL()) {
//manipulate startup parameters
if (!update_bogoprops) {
if (VSIDS &&
((stats.conflStats.numConflicts & 0xfff) == 0xfff) &&
var_decay_vsids < conf.var_decay_vsids_max
) {
cryptominisat5_simple.gz libcryptominisat5.so.5.6.gz
The exception is coming from libelfin as it tries to figure out what type the attribute tagged as DW_AT_sec_offset is. I think the attribute type is in the user-defined range, which is a bit odd, so libelfin doesn't know what type it should be. This is probably something we can skip over instead of failing, either by catching the exception or some other change. I'll try to reproduce locally and see what I can do about it.
DW_AT_sec_offset... hmmm maybe some security hardening flag I set in GCC? I'm guessing because of the "sec" here. Let me check.
Crap, it has nothing to do with it :S It's section offset:
http://wiki.dwarfstd.org/index.php?title=DW_FORM_sec_offset
Seems to be a new thing? I am admittedly using Arch Linux which tends to be pretty fresh :S
I have the same issue, also running in Arch If it can helps, it seems the error is that DW_AT (DW_AT_GNU_locviews, 0x2137) is not possible in the switch case statement for DW_AT parsing in libelfin ( https://github.com/aclements/libelfin/blob/master/dwarf/abbrev.cc#L137 )
It does not seems to be referenced in https://github.com/aclements/libelfin/blob/master/dwarf/data.hh also
I'd like to chime in with the error i'm seeing; in my case, the error is what(): DW_FORM_sec_offset not expected for attribute (DW_AT)0x2119, no other mention of 0x2119 in this thread so far, and i'm not sure where to look up what 0x2119 means (or is that an offset into the dwarf data or so?)
i have had success with setting -g1 -gdwarf-2 getting rid of 0x2119, but i don't know how 0x2137 came up and i have no clue if these flags will get rid of that section.
edit: FWIW, running my program with coz run --- ... gives empty profile.coz files, so not sure if this is the right way forward
Yeah I also tried coz with one of my program and it generated and empty profile.coz.
So I tried the provided examples and the the profiles were also empty.
I noticed the dwarf error which brought me here. Not sure if related.
I used gcc 8.3.0 to compile everything.
cat histogram/profile.coz
startup time=1568661327518287201
runtime time=7729293440
[benchmarks/matrix_multiply] Running benchmark on large input
../../coz run --- ./matrix_multiply 1000 > /dev/null
[libcoz.cpp:100] bootstrapping coz
[libcoz.cpp:128] Including MAIN, which is /home/setepenre/work/coz/benchmarks/matrix_multiply/matrix_multiply
[inspect.cpp:325] /lib/x86_64-linux-gnu/ld-2.28.so is not in scope
[inspect.cpp:325] /usr/lib/x86_64-linux-gnu/libelf++.so.0 is not in scope
[inspect.cpp:325] /usr/lib/coz-profiler/libcoz.so is not in scope
[inspect.cpp:325] /lib/x86_64-linux-gnu/libpthread-2.28.so is not in scope
[inspect.cpp:325] /usr/lib/x86_64-linux-gnu/libdwarf++.so.0 is not in scope
[inspect.cpp:513] Ignoring DWARF format error when reading line table: DW_FORM_sec_offset not expected for attribute (DW_AT)0x2137
[inspect.cpp:317] Including lines from executable /home/setepenre/work/coz/benchmarks/matrix_multiply/matrix_multiply
[inspect.cpp:325] /lib/x86_64-linux-gnu/libgcc_s.so.1 is not in scope
[inspect.cpp:325] /lib/x86_64-linux-gnu/libc-2.28.so is not in scope
[inspect.cpp:325] /lib/x86_64-linux-gnu/libdl-2.28.so is not in scope
[inspect.cpp:325] /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25 is not in scope
[inspect.cpp:325] /lib/x86_64-linux-gnu/libm-2.28.so is not in scope
[inspect.cpp:325] /lib/x86_64-linux-gnu/librt-2.28.so is not in scope
[profiler.cpp:75] Starting profiler thread
***** file size is 4000000
cat matrix_multiply/profile.coz
startup time=1568661604721678119
runtime time=935720435
Yes, I did. I actually changed the compiler to clang-9 for my program and the profile.coz was populated. So it hints towards an issue linked to gcc 8.3.0.
I also tried to compile provided the examples with clang-9 to see if it would solve the problem as well, but I had a linking error.
$ CC=clang-9 CCX=clang++-9 make bench
make[1]: Entering directory '/home/setepenre/work/coz/benchmarks'
[benchmarks/histogram] Compiling histogram-pthread.c
[benchmarks/histogram] Downloading histogram inputs
--2019-09-16 15:41:14-- http://csl.stanford.edu/~christos/data/histogram.tar.gz
Resolving csl.stanford.edu (csl.stanford.edu)... 171.64.73.4
Connecting to csl.stanford.edu (csl.stanford.edu)|171.64.73.4|:80... connected.
HTTP request sent, awaiting response... [benchmarks/histogram] Linking histogram
/usr/bin/ld: obj/histogram-pthread.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: final link failed: nonrepresentable section on output
collect2: error: ld returned 1 exit status
make[2]: *** [../../common.mk:91: histogram] Error 1
make[2]: *** Waiting for unfinished jobs....
I can reproduce this problem (DW_FORM_sec_offset not expected for attribute (DW_AT)0x2137), with the following patch to ripgrep:
diff --git a/Cargo.lock b/Cargo.lock
index d4d5759..425ebc1 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -71,6 +71,15 @@ dependencies = [
"unicode-width 0.1.6 (registry+https://github.com/rust-lang/crates.io-index)",
]
+[[package]]
+name = "coz"
+version = "0.1.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "libc 0.2.62 (registry+https://github.com/rust-lang/crates.io-index)",
+ "once_cell 1.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
[[package]]
name = "crossbeam-channel"
version = "0.3.9"
@@ -299,6 +308,11 @@ dependencies = [
"libc 0.2.62 (registry+https://github.com/rust-lang/crates.io-index)",
]
+[[package]]
+name = "once_cell"
+version = "1.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+
[[package]]
name = "packed_simd"
version = "0.3.3"
@@ -379,6 +393,7 @@ version = "11.0.2"
dependencies = [
"bstr 0.2.7 (registry+https://github.com/rust-lang/crates.io-index)",
"clap 2.33.0 (registry+https://github.com/rust-lang/crates.io-index)",
+ "coz 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
"grep 0.2.4",
"ignore 0.4.10",
"jemallocator 0.3.2 (registry+https://github.com/rust-lang/crates.io-index)",
@@ -536,6 +551,7 @@ dependencies = [
"checksum cc 1.0.41 (registry+https://github.com/rust-lang/crates.io-index)" = "8dae9c4b8fedcae85592ba623c4fd08cfdab3e3b72d6df780c6ead964a69bfff"
"checksum cfg-if 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)" = "b486ce3ccf7ffd79fdeb678eac06a9e6c09fc88d33836340becb8fffe87c5e33"
"checksum clap 2.33.0 (registry+https://github.com/rust-lang/crates.io-index)" = "5067f5bb2d80ef5d68b4c87db81601f0b75bca627bc2ef76b141d7b846a3c6d9"
+"checksum coz 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "f8526901d1bef683a5c0edddd892f0e5f80bc52d9c4653da9cc2440a60a2e777"
"checksum crossbeam-channel 0.3.9 (registry+https://github.com/rust-lang/crates.io-index)" = "c8ec7fcd21571dc78f96cc96243cab8d8f035247c3efd16c687be154c3fa9efa"
"checksum crossbeam-utils 0.6.6 (registry+https://github.com/rust-lang/crates.io-index)" = "04973fa96e96579258a5091af6003abde64af786b860f18622b82e026cca60e6"
"checksum encoding_rs 0.8.17 (registry+https://github.com/rust-lang/crates.io-index)" = "4155785c79f2f6701f185eb2e6b4caf0555ec03477cb4c70db67b465311620ed"
@@ -552,6 +568,7 @@ dependencies = [
"checksum memchr 2.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "88579771288728879b57485cc7d6b07d648c9f0141eb955f8ab7f9d45394468e"
"checksum memmap 0.7.0 (registry+https://github.com/rust-lang/crates.io-index)" = "6585fd95e7bb50d6cc31e20d4cf9afb4e2ba16c5846fc76793f11218da9c475b"
"checksum num_cpus 1.10.1 (registry+https://github.com/rust-lang/crates.io-index)" = "bcef43580c035376c0705c42792c294b66974abbfd2789b511784023f71f3273"
+"checksum once_cell 1.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "891f486f630e5c5a4916c7e16c4b24a53e78c860b646e9f8e005e4f16847bfed"
"checksum packed_simd 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)" = "a85ea9fc0d4ac0deb6fe7911d38786b32fc11119afd9e9d38b84ff691ce64220"
"checksum pcre2 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "603da5e101220b9b306bf28e4f1f8703458ce2f64d2787b374e1a19494317180"
"checksum pcre2-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)" = "876c72d05059d23a84bd9fcdc3b1d31c50ea7fe00fe1522b4e68cd3608db8d5b"
diff --git a/Cargo.toml b/Cargo.toml
index d44c484..245a842 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -55,6 +55,7 @@ num_cpus = "1.8.0"
regex = "1.0.5"
serde_json = "1.0.23"
termcolor = "1.0.3"
+coz = "0.1"
[dependencies.clap]
version = "2.32.0"
diff --git a/src/main.rs b/src/main.rs
index 6c8826b..4948055 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -1,3 +1,5 @@
+extern crate coz;
+
use std::error;
use std::io::{self, Write};
use std::process;
@@ -55,6 +57,7 @@ fn main() {
fn try_main(args: Args) -> Result<()> {
use args::Command::*;
+ coz::begin!("main");
let matched =
match args.command()? {
Search => search(&args),
@@ -65,6 +68,7 @@ fn try_main(args: Args) -> Result<()> {
Types => types(&args),
PCRE2Version => pcre2_version(&args),
}?;
+ coz::end!("main");
if matched && (args.quiet() || !messages::errored()) {
process::exit(0)
} else if messages::errored() {
0x2119 and 0x2137 are GNU extensions to the DWARF format
https://github.com/gcc-mirror/gcc/blob/master/include/dwarf2.def#L441 https://github.com/gcc-mirror/gcc/blob/master/include/dwarf2.def#L454
Compiling with -gdwarf-3 moves the problem to
terminate called after throwing an instance of 'std::out_of_range'
what(): file name index 25 exceeds file table size of 22
Also a libelfin issue
Backing out a previous change (#96, removed by #128) appears to fix this issue. The code relied on exceptions while walking through compilation units in an effort to improve performance. Unfortunately, the change didn't actually save any work (libelfin was still walking through all line tables), but that prevented existing error handling from working.
Unfortunately, it doesn't seem to work, at least for me. Arch Linux here again.
I only tried the included benchmark examples, e.g., word_count. When I compile with gcc 9.2, profile.coz is empty, except for startup time and runtime. I get a dwarf format error:
[inspect.cpp:513] Ignoring DWARF format error when reading line table: DW_FORM_sec_offset not expected for attribute (DW_AT)0x2137
When I compile the example with clang 9.0, everything seems to be fine.
Which version of libelfin are you running? If coz did not crash and you just get an empty profile then this could be a different issue (maybe #109).
I use git master of libelfin.
Comparing the outputs of word_count compiled with gcc and clang, I guess, gcc's version does not work as it cannot walk the compilation units. Two exceptions get thrown. How can I attach a debugger to coz to set a catchpoint?
gcc 9.2
$ ../../coz run --- ./word_count.gcc ./word_count_datafiles/word_100MB.txt
[libcoz.cpp:100] bootstrapping coz
[libcoz.cpp:128] Including MAIN, which is /home/you/coz_git/benchmarks/word_count/word_count.gcc
[inspect.cpp:513] Ignoring DWARF format error when reading line table: DW_FORM_sec_offset not expected for attribute (DW_AT)0x2137
[inspect.cpp:513] Ignoring DWARF format error when reading line table: DW_FORM_sec_offset not expected for attribute (DW_AT)0x2137
[inspect.cpp:316] Including lines from executable /home/you/coz_git/benchmarks/word_count/word_count.gcc
[profiler.cpp:75] Starting profiler thread
clang 9.0
$ ../../coz run --- ./word_count.clang ./word_count_datafiles/word_100MB.txt
[libcoz.cpp:100] bootstrapping coz
[libcoz.cpp:128] Including MAIN, which is /home/you/coz_git/benchmarks/word_count/word_count.clang
[inspect.cpp:509] Included source file /home/you/coz_git/benchmarks/word_count/stddefines.h
[inspect.cpp:509] Included source file /home/you/coz_git/benchmarks/word_count/word_count-pthread.c
[inspect.cpp:509] Included source file /home/you/coz_git/include/coz.h
[inspect.cpp:509] Included source file /usr/include/stdlib.h
[inspect.cpp:509] Included source file /usr/include/sys/stat.h
[inspect.cpp:509] Included source file /home/you/coz_git/benchmarks/word_count/sort-pthread.c
[inspect.cpp:316] Including lines from executable /home/you/coz_git/benchmarks/word_count/word_count.clang
[profiler.cpp:75] Starting profiler thread
DWARF 4 is the default in gcc 9.2, but -gdwarf-3 gets around the issue. Thanks for the tip.
I created a tiny patch for libelfin which solves the issue for me. gcc 9 works now with the default -g.
https://github.com/tetzank/libelfin/commit/c4394ec2aca4f6d858385f8e80ba9440ade2269b
@tetzank great, I hope your patch gets accepted upstream. Fixes the problem for me. For whatever reason, putting -gdwarf-3 in $CLAGS didn't solve it for me. I'm on GCC 9.2.
This should have been at least partially addressed in libelfin with this commit: https://github.com/aclements/libelfin/commit/74f11163334653866c08c32430bbad33c1a7877f
@tetzank great, I hope your patch gets accepted upstream. Fixes the problem for me. For whatever reason, putting
-gdwarf-3in$CLAGSdidn't solve it for me. I'm on GCC 9.2.
I'm currently on ubuntu 20.04 with GCC 10.2 and was running into this issue. I went into the libcoz directory and added in the -gdwarf-3 to the Makefile for libcoz.so, and this fixed my issue and generated a profile. I also added it to the histogram benchmark, which i was testing with. At least one of those changes allowed coz to run correctly.
@ManuelMeraz Thanks Manuel, I applied the same changes as you said, the problem resolved!
I'm still getting this, except this time with what(): unknown compilation unit version 5. It seems like in my case the issue is with the the /build/gcc/src/gcc/libgcc/config/i386/crtfastmath.c compilation unit (which I found using this command).
Since that compilation unit is used to implement -ffast-math, I've removed that flag from my build & things seem to be working fine so far.
I'm getting this as well with what(): unknown compilation unit version 5 in coz-profiler 0.2.2, but only when I build my Rust app with:
[profile.release]
debug = 1
Commenting this block out and building in release mode causes the app to work again. However, I'm still getting empty profile.coz files after the program runs, despite annotating segments with coz::{begin!,end!} and coz::progress!. Not really sure how to move forward
