sentry-native icon indicating copy to clipboard operation
sentry-native copied to clipboard
verified publisher icon getsentry

HTTP error 400 when sending segfault report on linux aarch64

Open michirod opened this issue 4 years ago • 12 comments

Description

I'm deliberately crashing my application to test the segfault report. It works well on my workstation (linux, x86_64), but on the target device I get this error during the envelope upload.

I checked the envelope file and I couldn't fine anything weird in the first line.

When does the problem happen

  • [ ] During build
  • [x] During run-time
  • [ ] When capturing a hard crash

Environment

  • OS: Yocto linux 4.4.52 aarch64
  • Compiler: aarch64-linux-gcc (GCC) 7.3.0
  • CMake version and config: 3.18.2

Steps To Reproduce

Just make the code call this function and restart the application.

    inline void deliberate_segfault()
    {
        *((int*)nullptr) = 42;
    }

Log output Crash log:

[sentry] INFO entering breakpad minidump callback
[sentry] DEBUG merging scope into event
[sentry] DEBUG trying to read modules from /proc/self/maps
[sentry] DEBUG inspecting module "/usr/bin/application"
[sentry] DEBUG inspecting module "/lib64/libresolv-2.26.so"
[sentry] DEBUG inspecting module "/lib64/libnss_dns-2.26.so"
[sentry] DEBUG inspecting module "/lib64/libnss_mdns4_minimal.so.2"
[sentry] DEBUG inspecting module "/lib64/libnss_files-2.26.so"
[sentry] DEBUG inspecting module "/usr/lib64/libgmp.so.10.3.2"
[sentry] DEBUG inspecting module "/usr/lib64/libhogweed.so.4.3"
[sentry] DEBUG inspecting module "/usr/lib64/libunistring.so.2.0.0"
[sentry] DEBUG inspecting module "/usr/lib64/libidn.so.11.6.16"
[sentry] DEBUG inspecting module "/lib64/libz.so.1.2.11"
[sentry] DEBUG inspecting module "/usr/lib64/libgnutls.so.30.14.5"
[sentry] DEBUG inspecting module "/usr/lib64/libnettle.so.6.3"
[sentry] DEBUG inspecting module "/lib64/libc-2.26.so"
[sentry] DEBUG inspecting module "/lib64/libgcc_s.so.1"
[sentry] DEBUG inspecting module "/lib64/libm-2.26.so"
[sentry] DEBUG inspecting module "/usr/lib64/libcurl.so.4.5.0"
[sentry] DEBUG inspecting module "/lib64/libdl-2.26.so"
[sentry] DEBUG inspecting module "/lib64/libpthread-2.26.so"
[sentry] DEBUG inspecting module "/lib64/ld-2.26.so"
[sentry] DEBUG inspecting module "linux-gate.so"
[sentry] DEBUG inspecting module "/lib64/ld-2.26.so"
[sentry] DEBUG read 21 modules from /proc/self/maps
[sentry] DEBUG adding attachments to envelope
[sentry] DEBUG sending envelope
[sentry] DEBUG serializing envelope into buffer
[sentry] INFO crash has been captured

Application startup log:

[sentry] INFO using database path "/home/root/.sentry-native"
[sentry] DEBUG starting transport
[sentry] DEBUG starting background worker thread
[sentry] DEBUG starting backend
[sentry] DEBUG background worker thread started
[sentry] DEBUG sending envelope
[sentry] DEBUG submitting task to background worker thread
[sentry] DEBUG executing task on worker thread
*   Trying 35.188.42.15...
* TCP_NODELAY set
* Connected to sentry.io (35.188.42.15) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
*        server certificate verification OK
*        server certificate status verification SKIPPED
*        common name: sentry.io (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: C=US,ST=California,L=San Francisco,O=Functional Software\, Inc.,CN=sentry.io
*        start date: Tue, 02 Jun 2020 00:00:00 GMT
*        expire date: Tue, 07 Jun 2022 12:00:00 GMT
*        issuer: C=US,O=DigiCert Inc,CN=DigiCert SHA2 Secure Server CA
*        compression: NULL
* ALPN, server did not agree to a protocol
> POST /api/XXXXXX/envelope/ HTTP/1.1
Host: sentry.io
User-Agent: sentry.native/0.4.7
Accept: */*
x-sentry-auth:Sentry sentry_key=XXXXXXX, sentry_version=7, sentry_client=sentry.native/0.4.7
content-type:application/x-sentry-envelope
content-length:211429
* We are completely uploaded and fine
< HTTP/1.1 400 Bad Request
< Server: nginx
< Date: Wed, 17 Feb 2021 20:32:10 GMT
< Content-Type: application/json
< Content-Length: 117
< Connection: keep-alive
< access-control-expose-headers: x-sentry-rate-limits, retry-after, x-sentry-error
< vary: Origin
< x-envoy-upstream-service-time: 0
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< 
{"detail":"invalid event envelope","causes":["invalid item header","invalid unicode code point at line 1 column 88"]}* Connection #0 to host sentry.io left intact

michirod avatar Feb 17 '21 20:02 michirod

There are multiple items, and I’m not sure the header parser does offset its error messages. In any case, having the full envelope would be really valuable here.

Swatinem avatar Feb 18 '21 07:02 Swatinem

I see what you mean. I inspected further the envelop and I think I found the issue at line 4. {"type":"attachment","length":206840,"attachment_type":"event.minidump","filename":"1�$"}

The filename looks corrupted.

michirod avatar Feb 18 '21 08:02 michirod

Okay, this is super interesting.

This might be a bit far fetched, but…

We get the path to the minidump from breakpad here:

https://github.com/getsentry/sentry-native/blob/f75ae734b3e7c44ffc7934569b23f700669c7c8d/src/backends/sentry_backend_breakpad.cpp#L90

We attach it to the event (reading the file, which succeeds, otherwise it would not end up in the envelope)

https://github.com/getsentry/sentry-native/blob/f75ae734b3e7c44ffc7934569b23f700669c7c8d/src/backends/sentry_backend_breakpad.cpp#L107

The filename part comes from here:

https://github.com/getsentry/sentry-native/blob/f75ae734b3e7c44ffc7934569b23f700669c7c8d/src/backends/sentry_backend_breakpad.cpp#L118

Which is using strrchr internally:

https://github.com/getsentry/sentry-native/blob/f75ae734b3e7c44ffc7934569b23f700669c7c8d/src/path/sentry_path_unix.c#L181

A quick search brought me to here:

https://bugs.launchpad.net/qemu/+bug/1908551 and looking through some more code, I found this: https://sourceware.org/git/?p=glibc.git;a=commit;h=79160c06c7b74672c7f7368355ca0b59103b2d30

Long story short, I’m kind of questioning if we can even trust the libc functions.

Swatinem avatar Feb 18 '21 10:02 Swatinem

@michirod is this something you can reliably reproduce?

Can you maybe try to avoid the sentry__path_filename call, and see if that creates a valid envelope? Otherwise it might be worth looking at the json serializer again, but that does output otherwise correct things here.

Swatinem avatar Feb 18 '21 10:02 Swatinem

Wow thanks @Swatinem for this analysis. This is reproducible, I'll try to patch the code and avoid the sentry__path_filename call and see what happens. Can I just use the full path dump_path->path as filename value?

michirod avatar Feb 18 '21 10:02 michirod

Yes that should work.

Swatinem avatar Feb 18 '21 10:02 Swatinem

Ok, this is super weird. I patched the code, but the issue is still there. So I tried to debug the application with gdb and it appears that the path is already broken in the descriptor object. But I get different outcomes whether I use the debugger or not.

I added a few debug prints, as you can see in this patch:

diff --git a/src/backends/sentry_backend_breakpad.cpp b/src/backends/sentry_backend_breakpad.cpp
index 0172a1a..9903b03 100644
--- a/src/backends/sentry_backend_breakpad.cpp
+++ b/src/backends/sentry_backend_breakpad.cpp
@@ -87,8 +87,11 @@ sentry__breakpad_backend_callback(
     dump_path = sentry__path_append_str(tmp_path, ".dmp");
     sentry__path_free(tmp_path);
 #else
+    SENTRY_DEBUGF("succeded: %d", succeeded);
+    SENTRY_DEBUGF("descriptor.path(): %s", descriptor.path());
     dump_path = sentry__path_new(descriptor.path());
 #endif
+    SENTRY_DEBUGF("dump_path: %s", dump_path->path);
 
     SENTRY_WITH_OPTIONS (options) {
         sentry__write_crash_marker(options);
@@ -106,6 +109,7 @@ sentry__breakpad_backend_callback(
         sentry_envelope_item_t *item
             = sentry__envelope_add_from_path(envelope, dump_path, "attachment");
         if (item) {
+            SENTRY_DEBUGF("dump_path: 0x%x %s", item, dump_path->path);
             sentry__envelope_item_set_header(item, "attachment_type",
                 sentry_value_new_string("event.minidump"));
 
@@ -115,8 +119,9 @@ sentry__breakpad_backend_callback(
 #else
                 sentry_value_new_string(
 #endif
-                    sentry__path_filename(dump_path)));
+                    (dump_path->path)));
         }
+        SENTRY_DEBUGF("dump_path: %s", dump_path->path);
 
         // capture the envelope with the disk transport
         sentry_transport_t *disk_transport
diff --git a/src/path/sentry_path_unix.c b/src/path/sentry_path_unix.c
index 745fd0f..2b7d68e 100644
--- a/src/path/sentry_path_unix.c
+++ b/src/path/sentry_path_unix.c
@@ -398,6 +398,7 @@ sentry__path_touch(const sentry_path_t *path)
 char *
 sentry__path_read_to_buffer(const sentry_path_t *path, size_t *size_out)
 {
+    SENTRY_DEBUGF("Opening file: %s", path->path);
     int fd = open(path->path, O_RDONLY);
     if (fd < 0) {
         return NULL;

Now, if I run the application, when I trigger the crash I get this log:

[sentry] INFO entering breakpad minidump callback
[sentry] INFO succeded: 1
[sentry] INFO descriptor.path(): ��
[sentry] INFO dump_path: ��
[sentry] DEBUG merging scope into event
[sentry] DEBUG trying to read modules from /proc/self/maps
[sentry] DEBUG inspecting module "/home/root/application"
[sentry] DEBUG inspecting module "/lib64/libresolv-2.26.so"
[sentry] DEBUG inspecting module "/lib64/libnss_dns-2.26.so"
[sentry] DEBUG inspecting module "/lib64/libnss_mdns4_minimal.so.2"
[sentry] DEBUG inspecting module "/lib64/libnss_files-2.26.so"
[sentry] DEBUG inspecting module "/usr/lib64/libgmp.so.10.3.2"
[sentry] DEBUG inspecting module "/usr/lib64/libhogweed.so.4.3"
[sentry] DEBUG inspecting module "/usr/lib64/libunistring.so.2.0.0"
[sentry] DEBUG inspecting module "/usr/lib64/libidn.so.11.6.16"
[sentry] DEBUG inspecting module "/lib64/libz.so.1.2.11"
[sentry] DEBUG inspecting module "/usr/lib64/libgnutls.so.30.14.5"
[sentry] DEBUG inspecting module "/usr/lib64/libnettle.so.6.3"
[sentry] DEBUG inspecting module "/lib64/libgcc_s.so.1"
[sentry] DEBUG inspecting module "/lib64/libc-2.26.so"
[sentry] DEBUG inspecting module "/lib64/libm-2.26.so"
[sentry] DEBUG inspecting module "/usr/lib64/libcurl.so.4.5.0"
[sentry] DEBUG inspecting module "/lib64/libdl-2.26.so"
[sentry] DEBUG inspecting module "/lib64/libpthread-2.26.so"
[sentry] DEBUG inspecting module "/lib64/ld-2.26.so"
[sentry] DEBUG inspecting module "linux-gate.so"
[sentry] DEBUG inspecting module "/lib64/ld-2.26.so"
[sentry] DEBUG read 21 modules from /proc/self/maps
[sentry] DEBUG adding attachments to envelope
[sentry] INFO Opening file: ��
[sentry] INFO dump_path: 0x849eb230 ��
[sentry] INFO dump_path: ��
[sentry] DEBUG sending envelope
[sentry] DEBUG serializing envelope into buffer
[sentry] INFO crash has been captured
Segmentation fault (core dumped)

When I execute the application from within gdb I get this:

[sentry] INFO entering breakpad minidump callback
[sentry] INFO succeded: 0
[sentry] INFO descriptor.path(): ��
[sentry] INFO dump_path: ��
[sentry] DEBUG merging scope into event
[sentry] DEBUG trying to read modules from /proc/self/maps
[sentry] DEBUG inspecting module "/home/root/application"
[sentry] DEBUG inspecting module "/lib64/libresolv-2.26.so"
[sentry] DEBUG inspecting module "/lib64/libnss_dns-2.26.so"
[sentry] DEBUG inspecting module "/lib64/libnss_mdns4_minimal.so.2"
[sentry] DEBUG inspecting module "/lib64/libnss_files-2.26.so"
[sentry] DEBUG inspecting module "/usr/lib64/libgmp.so.10.3.2"
[sentry] DEBUG inspecting module "/usr/lib64/libhogweed.so.4.3"
[sentry] DEBUG inspecting module "/usr/lib64/libunistring.so.2.0.0"
[sentry] DEBUG inspecting module "/usr/lib64/libidn.so.11.6.16"
[sentry] DEBUG inspecting module "/lib64/libz.so.1.2.11"
[sentry] DEBUG inspecting module "/usr/lib64/libgnutls.so.30.14.5"
[sentry] DEBUG inspecting module "/usr/lib64/libnettle.so.6.3"
[sentry] DEBUG inspecting module "/lib64/libgcc_s.so.1"
[sentry] DEBUG inspecting module "/lib64/libc-2.26.so"
[sentry] DEBUG inspecting module "/lib64/libm-2.26.so"
[sentry] DEBUG inspecting module "/usr/lib64/libcurl.so.4.5.0"
[sentry] DEBUG inspecting module "/lib64/libdl-2.26.so"
[sentry] DEBUG inspecting module "/lib64/libpthread-2.26.so"
[sentry] DEBUG inspecting module "/lib64/ld-2.26.so"
[sentry] DEBUG inspecting module "linux-gate.so"
[sentry] DEBUG inspecting module "/lib64/ld-2.26.so"
[sentry] DEBUG read 21 modules from /proc/self/maps
[sentry] DEBUG adding attachments to envelope
[sentry] INFO Opening file: ��
[sentry] WARN failed to read envelope item from "��"
[sentry] INFO dump_path: ��
[sentry] DEBUG sending envelope
[sentry] DEBUG serializing envelope into buffer
[sentry] INFO crash has been captured

Thread 19 "application" received signal SIGSEGV, Segmentation fault.
0x0000000000a99de0 in ?? ()

The difference is that in the second case sentry can't read the file and the envelope doesn't contain any minidump, which makes sense. I can't understand how sentry doesn't complain when opening the minidump file in the first case. The path string is broken in both cases.

I've seen once the read WARNING without the debugger. In that case succeeded was 0, not sure if relevant.

michirod avatar Feb 18 '21 14:02 michirod

Thanks for investigating!

So this means that breakpad itself seems to be broken, unless sentry is broken when initializing, which I doubt since your first log prints the database path correctly.

Can you log this run directory here: https://github.com/getsentry/sentry-native/blob/f75ae734b3e7c44ffc7934569b23f700669c7c8d/src/backends/sentry_backend_breakpad.cpp#L197 If that is correct, then breakpad itself is broken somehow.

The other takeaway: Start using the succeeded flag.

Swatinem avatar Feb 18 '21 15:02 Swatinem

I guess you are right. This is the patch I've applied:

diff --git a/src/backends/sentry_backend_breakpad.cpp b/src/backends/sentry_backend_breakpad.cpp
index 0172a1a..16fe6b8 100644
--- a/src/backends/sentry_backend_breakpad.cpp
+++ b/src/backends/sentry_backend_breakpad.cpp
@@ -87,8 +87,11 @@ sentry__breakpad_backend_callback(
     dump_path = sentry__path_append_str(tmp_path, ".dmp");
     sentry__path_free(tmp_path);
 #else
+    SENTRY_DEBUGF("succeded: %d", succeeded);
+    SENTRY_DEBUGF("descriptor.path(): %s", descriptor.path());
     dump_path = sentry__path_new(descriptor.path());
 #endif
+    SENTRY_DEBUGF("dump_path: %s", dump_path->path);
 
     SENTRY_WITH_OPTIONS (options) {
         sentry__write_crash_marker(options);
@@ -106,6 +109,7 @@ sentry__breakpad_backend_callback(
         sentry_envelope_item_t *item
             = sentry__envelope_add_from_path(envelope, dump_path, "attachment");
         if (item) {
+            SENTRY_DEBUGF("dump_path: 0x%x %s", item, dump_path->path);
             sentry__envelope_item_set_header(item, "attachment_type",
                 sentry_value_new_string("event.minidump"));
 
@@ -115,8 +119,9 @@ sentry__breakpad_backend_callback(
 #else
                 sentry_value_new_string(
 #endif
-                    sentry__path_filename(dump_path)));
+                    (dump_path->path)));
         }
+        SENTRY_DEBUGF("dump_path: %s", dump_path->path);
 
         // capture the envelope with the disk transport
         sentry_transport_t *disk_transport
@@ -194,7 +199,10 @@ sentry__breakpad_backend_startup(
         = new google_breakpad::ExceptionHandler(current_run_folder->path, NULL,
             sentry__breakpad_backend_callback, NULL, !IsDebuggerActive(), NULL);
 #else
+    SENTRY_DEBUGF("current_run_folder->path: %s", current_run_folder->path);
     google_breakpad::MinidumpDescriptor descriptor(current_run_folder->path);
+    SENTRY_DEBUGF("descriptor.directory(): %s", descriptor.directory().c_str());
+    SENTRY_DEBUGF("descriptor.path(): %s", descriptor.path());
     backend->data = new google_breakpad::ExceptionHandler(
         descriptor, NULL, sentry__breakpad_backend_callback, NULL, true, -1);
 #endif
diff --git a/src/path/sentry_path_unix.c b/src/path/sentry_path_unix.c
index 745fd0f..2b7d68e 100644
--- a/src/path/sentry_path_unix.c
+++ b/src/path/sentry_path_unix.c
@@ -398,6 +398,7 @@ sentry__path_touch(const sentry_path_t *path)
 char *
 sentry__path_read_to_buffer(const sentry_path_t *path, size_t *size_out)
 {
+    SENTRY_DEBUGF("Opening file: %s", path->path);
     int fd = open(path->path, O_RDONLY);
     if (fd < 0) {
         return NULL;

And this is the log I get:

[sentry] INFO using database path "/home/root/.sentry-native"
[sentry] INFO Opening file: /home/root/.sentry-native/user-consent
[sentry] DEBUG starting transport
[sentry] DEBUG starting background worker thread
[sentry] DEBUG starting backend
[sentry] INFO current_run_folder->path: /home/root/.sentry-native/ebb1f213-b6d5-4018-a922-3fdbb0f9fd2c.run
[sentry] DEBUG background worker thread started
[sentry] INFO descriptor.directory(): �W9
[sentry] INFO descriptor.path(): (null)

[2021-02-18 17:40:07] [ 8768] [ipc    ] [D] [+3359] Force segmentation fault

[sentry] INFO entering breakpad minidump callback
[sentry] INFO succeded: 1
[sentry] INFO descriptor.path(): dY9
[sentry] INFO dump_path: dY9
[sentry] DEBUG merging scope into event
[sentry] DEBUG trying to read modules from /proc/self/maps
[sentry] DEBUG inspecting module "/home/root/application"
[sentry] DEBUG inspecting module "/lib64/libresolv-2.26.so"
[sentry] DEBUG inspecting module "/lib64/libnss_dns-2.26.so"
[sentry] DEBUG inspecting module "/lib64/libnss_mdns4_minimal.so.2"
[sentry] DEBUG inspecting module "/lib64/libnss_files-2.26.so"
[sentry] DEBUG inspecting module "/usr/lib64/libgmp.so.10.3.2"
[sentry] DEBUG inspecting module "/usr/lib64/libhogweed.so.4.3"
[sentry] DEBUG inspecting module "/usr/lib64/libunistring.so.2.0.0"
[sentry] DEBUG inspecting module "/usr/lib64/libidn.so.11.6.16"
[sentry] DEBUG inspecting module "/lib64/libz.so.1.2.11"
[sentry] DEBUG inspecting module "/usr/lib64/libgnutls.so.30.14.5"
[sentry] DEBUG inspecting module "/usr/lib64/libnettle.so.6.3"
[sentry] DEBUG inspecting module "/lib64/libgcc_s.so.1"
[sentry] DEBUG inspecting module "/lib64/libc-2.26.so"
[sentry] DEBUG inspecting module "/lib64/libm-2.26.so"
[sentry] DEBUG inspecting module "/usr/lib64/libcurl.so.4.5.0"
[sentry] DEBUG inspecting module "/lib64/libdl-2.26.so"
[sentry] DEBUG inspecting module "/lib64/libpthread-2.26.so"
[sentry] DEBUG inspecting module "/lib64/ld-2.26.so"
[sentry] DEBUG inspecting module "linux-gate.so"
[sentry] DEBUG inspecting module "/lib64/ld-2.26.so"
[sentry] DEBUG read 21 modules from /proc/self/maps
[sentry] DEBUG adding attachments to envelope
[sentry] INFO Opening file: dY9
[sentry] INFO dump_path: 0x90089230 dY9
[sentry] INFO dump_path: dY9
[sentry] DEBUG sending envelope
[sentry] DEBUG serializing envelope into buffer
[sentry] INFO crash has been captured
Segmentation fault (core dumped)

The descriptor.directory() seems broken. BUT if I go there with the debugger I actually see something normal, even if the log prints garbage:

Thread 1 "application" hit Breakpoint 4, 0x000000000214787c in sentry__breakpad_backend_startup (backend=0x29219e0, options=0x292f170) at src/backends/sentry_backend_breakpad.cpp:203
203	    google_breakpad::MinidumpDescriptor descriptor(current_run_folder->path);
(gdb) p current_run_folder->path
$33 = (sentry_pathchar_t *) 0x293a2b0 "/home/root/.sentry-native/650aac11-654a-424f-28b5-33b67cf98d41.run"
(gdb) n
204	    SENTRY_DEBUGF("descriptor.directory(): %s", descriptor.directory().c_str());
(gdb) 
205	    SENTRY_DEBUGF("descriptor.path(): %s", descriptor.path());
(gdb) 
207	        descriptor, NULL, sentry__breakpad_backend_callback, NULL, true, -1);
(gdb) p descriptor.directory().c_str()
$34 = (const std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::value_type *) 0x2923870 "/home/root/.sentry-native/650aac11-654a-424f-28b5-33b67cf98d41.run"
(gdb) p (const char*) descriptor.directory().c_str()
$35 = 0x2923330 "/home/root/.sentry-native/650aac11-654a-424f-28b5-33b67cf98d41.run"
(gdb) c
Continuing.

So I went on debugging and I think I found the broken line:

Thread 1 "application" hit Breakpoint 6, google_breakpad::ExceptionHandler::ExceptionHandler (this=0x29301a0, descriptor=..., filter=0x0, 
    callback=0x214757c <sentry__breakpad_backend_callback(google_breakpad::MinidumpDescriptor const&, void*, bool)>, callback_context=0x0, install_handler=true, server_fd=-1)
    at external/breakpad/src/client/linux/handler/exception_handler.cc:235
235	    minidump_descriptor_.UpdatePath();
(gdb) s
google_breakpad::MinidumpDescriptor::UpdatePath (this=0x29301c0) at external/breakpad/src/client/linux/handler/minidump_descriptor.cc:84
84	  assert(mode_ == kWriteMinidumpToFile && !directory_.empty());
(gdb) n
88	  if (!CreateGUID(&guid) || !GUIDToString(&guid, guid_str, sizeof(guid_str))) {
(gdb) 
92	  path_.clear();
(gdb) 
93	  path_ = directory_ + "/" + guid_str + ".dmp";
(gdb) p path_.c_str()
$37 = (const std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::value_type *) 0x29301e0 ""
(gdb) p directory_.c_str()
$38 = (const std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::value_type *) 0x2921c10 "/home/root/.sentry-native/650aac11-654a-424f-28b5-33b67cf98d41.run"
(gdb) p guid_str
$39 = "c5253f6d-f611-448d-68d58d8f-9c8c22b4"
(gdb) n
94	  c_path_ = path_.c_str();
(gdb) p path_.c_str()
$40 = (const std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::value_type *) 0x2921d90 "\034\222\002"
(gdb) n
95	}
(gdb) p c_path_
$41 = 0x29301e1 "\035\222\002"

As you can see when it gets to the line path_ = directory_ + "/" + guid_str + ".dmp";, directory_ still holds the correct path, but the assignment to path_ somehow fails and if I print it I get garbage.

I don't have any clue, besides a suspect on breakpad own <string> header, which might be different wrt my libc++ headers. But how would that compile?

michirod avatar Feb 18 '21 17:02 michirod

besides a suspect on breakpad own <string> header

I don’t think it brings its own string type.

It is indeed super weird. Also, why do you get "\034\222\002" for path_.c_str() but "\035\222\002" for c_path_, this does not make any sense.

If I understand the spec correctly, the + operator actually allocates intermediate objects. Not sure if the compiler can optimize this out. https://en.cppreference.com/w/cpp/string/basic_string/operator%2B This mentions to mutate the original string instead (to control allocator behavior).

I wonder if we using path_ += XXX for each of the elements would change anything here.

Swatinem avatar Feb 19 '21 10:02 Swatinem

@michirod have you continued your investigation? It is super weird that this ends up in a c++ std function. Maybe some kind of ABI mismatch problem?

We do see some random memory corruption especially on Android aarch64, and so far you were the only one able to reliably reproduce a similar problem.

Swatinem avatar Apr 20 '21 11:04 Swatinem

@Swatinem we have this issue in the backlog. I will work on this again for certain, I can't tell when though.

I do believe this can be some ABI mismatch problem. Since we are also going to change the libc++ version and compilation flavor shortly, I think I'll get back to this afterwards.

michirod avatar Apr 20 '21 16:04 michirod