getsentry
Uploading DIF from Windows fails with "Invalid checksum" eror
Environment
- Windows 10, x86_64
- sentry-cli v2.25.3
Steps to Reproduce
- Open Visual Studio 2019 Developer Command Prompt
- Build a C++ project using cmake with
CMAKE_BUILD_TYPEset toRelWithDebInfo - Run:
sentry-cli upload-dif -o %SENTRY_ORGANIZATION% -p %SENTRY_PROJECT% --log-level=debug .
Expected Result
All debug information files should be uploaded to Sentry.
Actual Result
The sentry-cli call fails with exit code 1. The following message is printed:
error: Invalid checksum
This runs on a Windows VM. Please note that I run similar jobs on macOS and Linux VMs on the same computing cluster, which also call sentry-cli upload-dif with similar inputs at some point. Neither of these ever fail because of Sentry, only the Windows job does.
Logs
Here are (sanitized) logs from a typical run: sentry_fail.log
Hi @petrmanek, thank you for reporting this issue. Have you ever been able to run this command successfully on Windows? If yes, what was the most recent version of the Sentry CLI where running the command succeeded?
Have you ever been able to run this command successfully on Windows?
Yes, in fact I was running this in my CI/CD pipeline as recently as 2 weeks ago. The only thing that changed was time and some features of my software. I recently switched my C++ standard from C++17 to C++20. Could that have anything to do with it?
If yes, what was the most recent version of the Sentry CLI where running the command succeeded?
Everything worked with v2.12.0 until about 2 weeks ago, when the error started appearing. Since then I tried updating to the latest version (v2.25.3 as if writing), which made no difference.
I recently switched my C++ standard from C++17 to C++20. Could that have anything to do with it?
Perhaps, if making that change could have changed the debug information file format. You can use the sentry-cli debug-files check command to check whether the debug information files are usable. More information about the debug information file formats we accept is available here.
Please try running the sentry-cli debug-files check command against the files which are causing the issue, and let me know what the output is.
Perhaps, if making that change could have changed the debug information file format
I will try that. A job is currently running so I should have results in approximately an hour.
In the meantime, looking at logs I do not see anything out of ordinary. Mostly DLLs (x86_64 library) and PDBs (x86_64 debug companion) and a single EXE (x86_64 executable). Correct me if I am wrong, but if sentry-cli identifies these files correctly and displays UPLOADED next to their name, it implies that they are considered useful, right?
I have a follow up: sentry dif check seems to recognize my files just fine even after switching to C++20. The following would be a typical output:
Correct me if I am wrong, but if sentry-cli identifies these files correctly and displays
UPLOADEDnext to their name, it implies that they are considered useful, right?
The code that the check command calls to determine whether the file is useful does not appear to get called from the upload code, so it could be possible that a file might not be considered usable even if it can get uploaded properly. Although I am not sure about that, since the code is pretty complex. But seeing that the files pass the dif check, there is probably a different problem.
Have you checked whether you are able to see the uploaded debug files in Sentry?
Apologies for a delayed response.
Have you checked whether you are able to see the uploaded debug files in Sentry?
Yes, they do and there appear to be no processing issues either.
The following log snippet of sentry-cli:
[...]
UPLOADED 3875da5b-901f-4639-8fa2-5d1883c454fe-1 (plugins\app\csvfile.dll; x86_64 library)
UPLOADED 7d509770-d1b8-4423-bf65-be6d87dc6162-1 (lib\scn.dll; x86_64 library)
UPLOADED f4f2beaa-2b80-4bed-95e9-cf7b2e5e91c4-1 (share\debug\app\dacscan.pdb; x86_64 debug companion)
UPLOADED 3ba9b8a0-3a72-4da4-9d78-080af7e842ee-1 (share\debug\app\mmfile.pdb; x86_64 debug companion)
UPLOADED 179b21bb-a7d4-44a2-a06d-3b9275daf552-1 (share\debug\app\fitpix.pdb; x86_64 debug companion)
UPLOADED 9bf1a6ce-5f6c-4040-bba8-bd66084e3f9d-1 (bin\crashpad_wer.pdb; x86_64 debug companion)
UPLOADED 2337ece1-3804-47cb-86cf-377af91bcd00-1 (plugins\app\livehist.dll; x86_64 library)
[...]
...nicely corresponds with what I see on sentry.io:
@petrmanek Just want to make sure I am understanding everything correctly: my understanding is that besides the error message showing up in the logs, everything appears to be working in Sentry as expected? Or have you noticed anything wrong in Sentry (such as an error message stating that debug files are missing when you view an error event)?
besides the error message showing up in the logs, everything appears to be working in Sentry as expected?
That is correct, as far as I can tell. DIFs appear to be identified as useful, uploaded and visible online. Only the process for some reason exits with a failing exit code and prints the error: Invalid checksum message before dying.
Or have you noticed anything wrong in Sentry (such as an error message stating that debug files are missing when you view an error event)?
I have been on a lookout for those. Over the last two weeks, I have not noticed any such incident (or anything similar for that matter).
@petrmanek That is good to hear, glad that everything appears to work as expected, besides the error message.
Since our team is currently busy with other projects, and since the only symptom appears to be the error message in the logs, it might take us some time before we are able to fully investigate this issue.
However, if you begin noticing any other problems as a result of this issue (such as DIFs not being uploaded to Sentry, or a missing debug file error), please let us know right away! We would, of course, prioritize investigating this issue if it begins to negatively impact your experience using Sentry.
Since our team is currently busy with other projects, and since the only symptom appears to be the error message in the logs, it might take us some time before we are able to fully investigate this issue.
As you wish. I will modify my CI's deployment script for Windows to allow for non-zero exit code to be returned by sentry-cli at this stage for as long as this issue remains open. Please let me know if you would like to run any tests.
However, if you begin noticing any other problems as a result of this issue (such as DIFs not being uploaded to Sentry, or a missing debug file error), please let us know right away! We would, of course, prioritize investigating this issue if it begins to negatively impact your experience using Sentry.
I will do that if/when I notice anything serious. Realistically speaking, however, my pipeline has 50+ jobs, so I am unlikely to spot anything unless I start seeing unsymbolicated crashes at sentry.io.
Ok thank you, and I appreciate your understanding! Also, if anyone else experiences this issue, please leave a comment, so we can understand how common this problem is.
We just started getting this error on our macOS GitHub Actions runner. We're not seeing releases & debug info in Sentry. The last release created from our CI was on May 24th, the last successful build we had.
We're using the Fastlane plugin, with the Sentry CLI installed via Homebrew. (version 2.31.2)
We started getting this error before we updated, as we were on a slightly older version of the CLI.
We haven't had any changes to our system before we started seeing this error.
We updated our Fastlane script to the following
sentry_debug_files_upload(
auth_token: SENTRY_AUTH_TOKEN,
org_slug: 'dev-peek',
project_slug: 'peekpro-inhouse',
wait: false,
log_level: 'trace'
)
Other than just changing our script to allow failures from this - is there another remedy?
CI debug logs
[10:48:49]: --------------------------------------- [10:48:49]: --- Step: sentry_debug_files_upload --- [10:48:49]: --------------------------------------- [10:48:49]: Using sentry-cli 2.31.2 [10:48:49]: Starting sentry-cli... [10:48:49]: DEBUG 2024-05-28 10:48:49.609613 -07:00 sentry-cli version: 2.31.2, platform: "darwin", architecture: "arm64" [10:48:49]: INFO 2024-05-28 10:48:49.610240 -07:00 sentry-cli was invoked with the following command line: "/Users/administrator/actions-runner/_work/nomad/nomad/vendor/bundle/ruby/2.6.0/gems/fastlane-plugin-sentry-1.22.1/bin/sentry-cli-Darwin-universal" "debug-files" "upload" "." "--wait" [10:48:49]: INFO 2024-05-28 10:48:49.610420 -07:00 Issuing a command for Organization: dev-peek Project: peekpro-inhouse [10:48:49]: DEBUG 2024-05-28 10:48:49.611011 -07:00 request GET https://sentry.io/api/0/organizations/dev-peek/chunk-upload/ [10:48:49]: DEBUG 2024-05-28 10:48:49.611029 -07:00 using token authentication [10:48:49]: DEBUG 2024-05-28 10:48:49.611045 -07:00 retry number 0, max retries: 0 [10:48:49]: DEBUG 2024-05-28 10:48:49.646666 -07:00 > GET /api/0/organizations/dev-peek/chunk-upload/ HTTP/1.1 [10:48:49]: DEBUG 2024-05-28 10:48:49.646678 -07:00 > Host: sentry.io [10:48:49]: DEBUG 2024-05-28 10:48:49.646680 -07:00 > Accept: */* [10:48:49]: DEBUG 2024-05-28 10:48:49.646682 -07:00 > Connection: TE [10:48:49]: DEBUG 2024-05-28 10:48:49.646684 -07:00 > TE: gzip [10:48:49]: DEBUG 2024-05-28 10:48:49.646686 -07:00 > User-Agent: sentry-cli/2.31.2 [10:48:49]: DEBUG 2024-05-28 10:48:49.647350 -07:00 > Authorization: *** [10:48:49]: DEBUG 2024-05-28 10:48:49.784833 -07:00
Just to update this: we are still seeing this on Windows, there has been no change since the issue was reported half a year ago.
@Mordil Please open a separate issue, since the problem you are describing seems to be different from the problem @petrmanek is experiencing.
@petrmanek only observed the problem on Windows and besides the error showing in the logs, everything in Sentry works properly.
On the other hand, from what you have written @Mordil, it sounds like you are noticing something wrong in Sentry, since you don't see releases and debug info in Sentry. You are also observing this problem on Mac, rather than Windows.
Please provide as much detail as possible when opening your issue; this will help me to reproduce your issue and to identify a fix/workaround.
@szokeasaurusrex Apologies, we were looking at the wrong organization when were verifying if release symbols were being uploaded or not.
We've confirmed that the correct release information is being sent to the correct organization in Sentry, and our issue is only the same as Petr reported, simply on macOS instead of just Windows.
We're going to update our CI script to allow this subprocess failure, though we'd ideally like to see a fix for this.
Okay @Mordil, glad to hear that everything is working as expected in Sentry, but sorry to hear that this problem is affecting you as well.
Would you be able to share a debug symbol file that reproduces this error (or the source code and steps to recreate such a debug file)? If your project is open source, you could just share the code/debug files that you noticed the problems with. Otherwise, if you have been able to create a minimal reproduction that you can share with me, it would greatly help me with debugging this problem.
Since recently I'm seeing this issue on my Linux builds, though I'm not sure if it's just the same error message or whether the entire behavior is the same. I have the impression the entire upload fails in my case.
An example build running into this issue can be seen at https://github.com/mapeditor/tiled/actions/runs/9592646514/job/26451582147 (see the "Upload symbols and sources to Sentry" step):
Sucessfully installed sentry-cli 2.32.1
Done!
INFO 2024-06-20 06:32:47.682623978 +00:00 sentry-cli was invoked with the following command line: "sentry-cli" "upload-dif" "--log-level=info" "--include-sources" "src" "."
INFO 2024-06-20 06:32:47.682699229 +00:00 Issuing a command for Organization: mapeditor Project: tiled
error: Invalid checksum
The last succesfull build is at https://github.com/mapeditor/tiled/actions/runs/9562961181/job/26360475510.
The problem actually started happening after updating from Sentry 0.6.7 to Sentry 0.7.6. In both cases sentry-cli 2.32.1 was used.
@bjorn, this might be the same problem.
The problem actually started happening after updating from Sentry 0.6.7 to Sentry 0.7.6
Which SDK version numbers are you referring to here?
Which SDK version numbers are you referring to here?
Whoops, that would be the version of the Sentry Native SDK at https://github.com/getsentry/sentry-native.
By downgrading the sentry-native version I've now confirmed that this uploading issue does not occur on 0.6.7 but does happen since the 0.7.0 version (this version made crashpad the the default, but I'm passing -DSENTRY_BACKEND=breakpad).
- Fails with 0.7.0: https://github.com/mapeditor/tiled/actions/runs/9643019019/job/26592093862 (never mind the green check, it's cause the step is marked
continue-on-error) - Works with 0.6.7: https://github.com/mapeditor/tiled/actions/runs/9643160290/job/26592537298
Okay, thank you for that information @bjorn, this is very helpful! One more question to ensure your issue is the same as what @petrmanek and @Mordil reported: despite the error message, are the debug files still showing up in Sentry?
For the record, with additional testing, debug symbols were NOT being sent to Sentry and we identified the issue with something from one of our dependencies: https://github.com/filestack/filestack-ios/issues/80
@szokeasaurusrex Yeah when this error occurs for me I don't see the debug files showing up in Sentry. I guess I'm not running into the same issue as @Mordil, since for me the only difference is whether I use Sentry Native 0.6.7 or Sentry Native 0.7.x.
For now I think I've been able to resolve this issue by only uploading the .debug files and not all .o files (see https://github.com/mapeditor/tiled/pull/3987, which changes from passing repository root to passing the AppDir directory which contains only the installed files). I'm not sure if the .o files add any information.
@bjorn Are you still seeing full stack traces in Sentry after https://github.com/mapeditor/tiled/pull/3987? If yes, then probably you don't need the .o files you were uploading before
Are you still seeing full stack traces in Sentry after mapeditor/tiled#3987?
I will only know for sure once somebody manages to crash the Tiled 1.11 AppImage, but that's only a matter of time unfortunately. :-)