go
go copied to clipboard
golang
proposal: runtime: use WER for GOTRACEBACK=crash on Windows
WER is a Windows built-in infrastructure provided by Microsoft that helps troubleshooting hardware and software problems. Many systems rely on the core dumps generated by WER for post-mortem analyses. These dumps are collected by a dedicated process running on the host and uploaded to the WER server and\or stored locally (see Collecting User-Mode Dumps).
#20498 added support for WER when setting GOTRACEBACK=crash, but was reverted and replaced by #49471. The reason to revert it was because WER uploads dumps to Microsoft by default, and this bit was not discussed in the original issue. See this comment https://github.com/golang/go/issues/49471#issuecomment-1362110361 to understand why I'm counterproposing #49471 after implementing it.
Here I'm proposing to revert the revert, enabling WER when GOTRACEBACK=crash is set. The arguments come next, but first the definition of the crash mode:
GOTRACEBACK=crash is like “system” but crashes in an operating system-specific manner instead of exiting. For example, on Unix systems, the crash raises SIGABRT to trigger a core dump.
I have two simple arguments in favor of WER:
-
WER is the Windows specific manner to crash applications. It is the default experience and many production systems rely on WER to do post-mortem analysis, not only from within Microsoft.
-
WER can be used for collecting user-mode dumps without adding any complexity to the Go runtime.
It's true that some users might not want their diagnosis data to be uploaded to Microsoft servers, but my point is that they already have enough knobs to turn it off or limit what's uploaded:
GOTRACEBACK=crashis not the default runtime behavior, one have to manually set via environment variable or programmatically via debug.SetTraceback. We can inform about the WER gotchas in the release notes and in theGOTRACEBACKdocs.- On Windows,
GOTRACEBACK=crashdoes nothing overGOTRACEBACK=system. If this proposal is approved, the only reason to setGOTRACEBACK=crashwould be to opt into WER. - Privacy-concerned users will most likely opt-out to uploading diagnosis data when setting up windows for the first time, which disables WER.
- WER can easily be disabled by running
Disable-WindowsErrorReportingon PowerShell. One can also disable just some applications using the system registry.
Side note: #20498 is a bug report, not a proposal, so I think it makes sense to create this as a formal proposal.
CC @zx2c4 @zhizheng052 @bhiggins @alexbrainman @aarzilli @aclements @golang/windows
Wasn't this already discussed and rejected in an earlier proposal? The privacy concerns seem real, especially given how hard opting out has become on Windows, and how little trust exists on the user side these days...
This is my personal summary: The earlier attempt was removed due to privacy concerns (which you raised on https://go.dev/cl/307372 -- thanks). Then we decided to adopt minidump in #49471. However, the actual implementation of minidump (https://go.dev/cl/458955) turns out to have some problems (see https://github.com/golang/go/issues/49471#issuecomment-1362110361). So this proposal suggests that we reconsider WER despite the privacy issues.
That is, I don't think we made a clear decision that we could not live with the privacy issues, because it seemed that minidump was a viable alternative. Now that we have learned more, it's possible that minidump is not a viable alternative. So let's consider whether we can live with the privacy issues.
I don't have an opinion myself.
IMO, it would be nice to have (and simple to implement) an opt-in only means for Go programs on Windows to work with WER. In addition to being opt-in only, there could be a warning in documentation that states that it's possible for WER to be configured to send crash dumps to Microsoft.
The privacy concerns seem real, especially given how hard opting out has become on Windows, and how little trust exists on the user side these days.
If this proposal is approved, WER would still be opt-in, so only users that need it would have it.
Opting out is just one PS command, Disable-WindowsErrorReporting, so I wouldn't say its hard. Additionally, the previous command only disables uploading dumps to the server, one can still use WER to generate local dumps.
@qmuntal Thank you for creating this proposal. I agree with your sentiment.
I always supported this functionality https://github.com/golang/go/issues/49471#issuecomment-1017412932 .
And it looks silly that Delve supports Go Windows minidumps, but Go tools do not provide tools to generate these dumps (see https://github.com/golang/go/issues/49471#issuecomment-1018200066 ).
Alex
My process address space can contain private keys. I can't put those in special memory regions because Go's GC. Ordinary users use my software who can't be fiddling around with powershell. Sometimes my software crashes. When this happens, uploading private keys to some error reporting server is a nonstarter. It must not happen. I can't imagine I'm the only one writing software with this situation, and I wouldn't want to choose the dangerous thing for other developers who might not even be aware that their private keys are sensitive - like casual users of TLS/HTTPS.
My opinion remains as in https://github.com/golang/go/issues/49471#issuecomment-963879095 (i.e. in favor of this).
Can we go with an explicit opt-in method to crash in a WER-compatible way (which really just means not swallowing the error)? This could involve a new GOTRACEBACK=crash-wer option and/or a runtime function. I’m concerned we’re just going to talk about blanket policies that don’t work for everyone.
On Sun, Dec 25, 2022 at 8:37 AM Alessandro Arzilli @.***> wrote:
My opinion remains as in #49471 (comment) https://github.com/golang/go/issues/49471#issuecomment-963879095
— Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/57441#issuecomment-1364707314, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAI4V3PCPEDFQIATT4OVCL3WPBZ5PANCNFSM6AAAAAATGONT4Y . You are receiving this because you were mentioned.Message ID: @.***>
Can we go with an explicit opt-in method to crash in a WER-compatible way (which really just means not swallowing the error)? This could involve a new GOTRACEBACK=crash-wer option and/or a runtime function.
We could, but I don't see how crash-wer would bring any benefit over crash, considering that the later is not currently implemented on Windows. It would add confusion to an already overloaded knob.
@zx2c4 ordinary users will not disable WER using a PS command, but neitheir will set GOTRACEBACK=crash unless someone guides them. I completly understand your privacy concerns, but I don't get why WER being opt-in is not enough of a mitigation.
Additionally, by default WER does not upload the full virtual memory, just the data segment, which can contain global variables, but not the heap nor the stack.
Thank you @qmuntal again for bring it forward again. I would actually argue that the proposal title is a bit misnomer and least as far as intent of the proposal is concerned. The proposal effectively intends to implement Go documented behavior “crashes in an operating system-specific manner”. No more nor less (never mind that this is on par with other platforms). On Windows it means either WER, custom crash handling, debugger activation, documented and used report entry in Windows Event log and usage of built-in and 3rd party tools. Another word, it is expected behavior for bazillion of applications running on Windows for many decades. Besides, it is a Go-runtime debugging and troubleshooting feature. If one is concerned with some side-effects for whatever reasons, do not activate it.
I got to this topic from https://github.com/golang/go/issues/20498 and I've spend the last hour or so trying to catch up with all the back and forths, so I'm sorry if I'm missing the point, but I just wanted to give my 2 cents.
As @qmuntal expressed in their opening statement, the documentation on GOTRACEBACK=crash is very opinionated:
GOTRACEBACK=crash is like “system” but crashes in an operating system-specific manner instead of exiting. For example, on Unix systems, the crash raises SIGABRT to trigger a core dump.
Based on that, IMHO, whether the SO sends or sends not the crash information to Microsoft is out of context from a Go program perspective. Microsoft says in their documentation:
Beginning with Windows Vista, Windows provides crash, non-response, and kernel fault error reporting by default without requiring changes to your application.
Then they further elaborate:
Because Windows automatically reports unhandled exceptions, application should not handle fatal exceptions.
First impression is that the decision to send or not send WER data to MS is in the scope of the system admin, not the developers. It is a bit old, but they document the controls of this feature in this article here.
Also, it is not clear to me if the way runtime is currently treating error is effectively "handling fatal exceptions" from a Windows perspective and that's why we are losing this functionality. If that's the case, GOTRACEBACK=crash should only disable any special treatment we do and let the application... er... crash. :) [By that I mean I wouldn't try to add any side effects like generating different types of dumps other than the default]
Another concern is that we are not generating Event Log ID 1000, as previously mentioned by @iglendd in https://github.com/golang/go/issues/49471#issuecomment-1266045020. This was the main reason why I got into this rabbit hole.
Finally, I'm more than happy to help with any coding if needed. I'm way too invested in this now :)
Wanted to chime in with some additional support for this proposal. I think it is worth considering the value of Windows local crash dump handling with WER, which is somewhat separate from uploading of reports to Microsoft (and the latter can be disabled, as @qmuntal points out).
- WER allows local crash dumps to be collected and stored in a directory. This is documented at Collecting User-Mode Dumps.
- WER allows the user to register their own application that is invoked when another application crashes. For instance, you could write a program that collects your own dumps, or hooks up a debugger to the crashing process. This is documented at Enabling Postmortem Debugging.
As mentioned above, the current approach of calling MiniDumpWriteDump from the crashing application can deadlock the process. It can also result in corrupted dumps due to the state of the process changing while the dump is being collected. So I think it is worth supporting a mechanism that uses the system facility for handling crashes.
This proposal has been added to the active column of the proposals project and will now be reviewed at the weekly proposal review meetings. — rsc for the proposal review group
GOTRACEBACK=crash does not upload things on any system today. It sounds like we should preserve that property, even if it's not the default privacy profile on Microsoft systems. It's also the case that code might contain debug.SetTraceback("crash") intending core dumps on Unix and then accidentally get reports to Microsoft on Windows.
Is there some way to force the WER code to write the local crash but not upload?
If not, then adding GOTRACEBACK=wer seems OK and very clear. Worth noting that debug.SetTraceback("wer") in a library would be interesting and maybe unexpected, but libraries are unlikely to do that.
Is there some way to force the WER code to write the local crash but not upload?
It is possible with admin permissions either running a PS script or modifying the registry, but AFAIK not possible within a process.
If not, then adding GOTRACEBACK=wer seems OK and very clear. Worth noting that debug.SetTraceback("wer") in a library would be interesting and maybe unexpected, but libraries are unlikely to do that.
I'm not particularly attached to crash, I just wanted to avoid having to set two knobs. GOTRACEBACK=wer looks clean and the intent is 100% clear, so I'm OK with this solution, which ideally should act like crash but enabling WER. Edit: I still think crash is a better fit, but I understand that it can't be used in this context if the not-upload-things-by-default property should be preserved.
Regarding GOTRACEBACK=crash vs GOTRACEBACK=wer I guess whatever works is fine but to me crash is better then wer to be honest. I guess I am with qmuntal on that point.
Regarding privacy and upload concerns I would argue that this is a completely orthogonal issue. Semantically and conceptually we are following exactly the same logic and behavior as on Unix - allowing system to decide what to do. And we would follow Go documentation literally (https://pkg.go.dev/runtime)
GOTRACEBACK=crash is like “system” but crashes in an operating system-specific manner instead of exiting. For example, on Unix systems, the crash raises SIGABRT to trigger a core dump.
In addition, it is not something a Go process would normally do. A user (or rather a developer) would set it up only for troubleshooting just to get their hands on a crash dump to find out a root cause. There are, by the way, more than a few ways to capture crash dump on Windows. In addition to WER one can also use external application like procdump or Dr.Watson, one can attach register post-mortem or debugger (and all of them are based on the existence of unhandled exception). And so, IMHO, the term wer is a bit ambiguous and misleading in this context. I think the term crash is more aptly describes intent,. behavior and semantics for both Go and system side.🙏
It is possible with admin permissions either running a PS script or modifying the registry, but AFAIK not possible within a process.
What about doing the opposite: is it possible to know if WER will upload anything from within the process?
is it possible to know if WER will upload anything from within the process?
I believe WER behaviour is controlled by these registry settings
https://learn.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps
(from @qmuntal message at the top https://github.com/golang/go/issues/57441#issue-1507449811 ).
So we can read the register settings.
Alex
is it possible to know if WER will upload anything from within the process?
I believe WER behaviour is controlled by these registry settings https://learn.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps
Yep. Specifically, the setting Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\Disabled controls if WER uploading is enabled or not, although there are other settings to control application-specific behavior.
@rsc @aarzilli just making sure we are on the same page: some of your comments go in the line of supporting local crash dumps trying to reuse WER infrastructure, which is a goal of this proposal, but not the only one. The main objective here is to support WER, as it provides other useful features already mentioned by myself and others in this issue, such as upload crash dumps to WER servers and post mortem debugging.
I found this line persuasive against GOTRACEBACK=crash always using WER:
It's also the case that code might contain debug.SetTraceback("crash") intending core dumps on Unix and then accidentally get reports to Microsoft on Windows.
Maybe it would be better if GOTRACEBACK=crash used WER only if WER was set for local dumps and a new value of GOTRACEBACK=wer used WER unconditionally (what would GOTRACEBACK=wer do on unix?).
Change https://go.dev/cl/474915 mentions this issue: runtime: crash with RaiseFailFastException on Windows
Maybe it would be better if GOTRACEBACK=crash used WER only if WER was set for local dumps and a new value of GOTRACEBACK=wer used WER unconditionally (what would GOTRACEBACK=wer do on unix?).
I like this idea. The concrete behavior would look like this:
- When
GOTRACEBACK=crashis set, do not disable WER if it is already disabled system-wide, i.e.Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\Disabledis set to 1. This will give WER a chance to generate a local dump without uploading the dump to Microsoft. There is no need to check if local dumps are configured, WER will do that for us. In fact, this will mostly cover #49471 without having to useMiniDumpWriteDump. - When
GOTRACEBACK=weris set, do not disable WER, let the OS manage how WER is executed, which means that local dumps can be generated if they are configured and dumps will be uploaded to Microsoft if WER is not disabled.
Clarification: It's confusing that WER can be disabled via registry and still make it generate local dumps, but that's how it works.
what would GOTRACEBACK=wer do on unix?
It could behave in the same way as when GOTRACEBACK was an unknown value: fallback to system.
@qmuntal I am not 100% sure I understand the difference you describe in the comment below
- When GOTRACEBACK=crash is set, do not disable WER if it is already disabled system-wide, i.e. Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\Disabled is set to 1. This will give WER a chance to generate a local dump without uploading the dump to Microsoft. There is no need to check if local dumps are configured, WER will do that for us. In fact, this will mostly cover https://github.com/golang/go/issues/49471 without having to use MiniDumpWriteDump.
- When GOTRACEBACK=wer is set, do not disable WER, let the OS manage how WER is executed, which means that local dumps can be generated if they are configured and dumps will be uploaded to Microsoft if WER is not disabled.
Are you suggesting to automatically disable WER for GOTRACEBACK=crash case if it is not disabled (yet)?
I do feel strange, on philosophical level, about an application attempting to control (interfere?) how system handles system behavior. As well as binding to a particular technology - besides WER as I have mention there are other non-WER related crash dump collection approaches (e.g. AeDebug -https://learn.microsoft.com/en-us/windows/win32/debug/configuring-automatic-debugging).
Another word I, personally, am not in favor to use GOTRACEBACK=wer. I feel that considering that it is a troubleshooting variable GOTRACEBACK=crash should be sufficient.
Are you suggesting to automatically disable WER for GOTRACEBACK=crash case if it is not disabled (yet)?
Nop. Go runtime is currently disabling WER for the running process using SetErrorMode (source). That call disables WER crash dump uploads and local dump generation. What I'm saying is to not call SetErrorMode if WER uploads are disabled system-wide so WER can generate local dumps (well, in fact this is @aarzilli's suggestion).
I do feel strange, on philosophical level, about an application attempting to control (interfere?) how system handles system behavior ...
Completely agree, but I don't want to die on this hill, if the Go community feels that GOTRACEBACK=crash should do its best to not upload anything, then so be it, GOTRACEBACK=wer is a good alternative.
I still think that if a sysadmin configures WER to send logs to Microsoft it is not the role of the Go runtime to bypass that.
For me GOTRACEBACK=crash should do whatever is configured at OS level, without trying to enable or disable anything. Of course, the documentation should make it clear that data might be sent to MS.
On Thu, 9 Mar 2023 at 19:44, Quim Muntal @.***> wrote:
Are you suggesting to automatically disable WER for GOTRACEBACK=crash case if it is not disabled (yet)?
Nop. Go runtime is currently disabling WER for the running process using SetErrorMode (source https://github.com/golang/go/blob/24017148ca8a669debbddba56d69769dc47d5c71/src/runtime/signal_windows.go#L13-L23). That call disables WER crash dump uploads and local dump generation. What I'm saying is to not call SetErrorMode if WER uploads are disabled system-wide and allow WER to chime in.
I do feel strange, on philosophical level, about an application attempting to control (interfere?) how system handles system behavior ...
Completely agree, but I don't want to die on this hill, if the Go community feels that GOTRACEBACK=crash should do its best to not upload anything, then so be it, GOTRACEBACK=wer is a good alternative.
— Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/57441#issuecomment-1462674256, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKGMJEQ4HWJQLVYIVAHKV3W3IXLXANCNFSM6AAAAAATGONT4Y . You are receiving this because you commented.Message ID: @.***>
-- Daniela Petruzalek Software Engineer github.com/danicat twitter.com/danicat83
@danicat Our concern is that configuring WER to send logs to Microsoft appears to be an opt-out setting in at least some installations. We want to be cautious about Go programs sending logs to Microsoft without a clear opt-in.
@ianlancetaylor IMHO I feel that we are mixing a bit of apples and oranges so to speak. Perhaps it is not intentional but the following phrase
We want to be cautious about Go programs sending logs to Microsoft without a clear opt-in
… is a perfect illustration of my point. On Windows, if an arbitrary application is crashed, the application will not be “sending logs” (or crash dump to be precise) to Microsoft. They generally cannot - they are dying or dead. But more importantly, it is the role and responsibility of Windows and not an application. By default, on a plain vanilla, unconfigured, Windows installation, indeed minidump of the crashed application will be sent to Microsoft (but it does not include heap or stacks). What I am, and @qmuntal and @danicat, are trying to say, it should not be the prerogative, responsibility or concern of an application developer (whether her or he is C++, JAVA or Go developer).
I firmly believe in this on a conceptual level. But practically speaking there are a number of important but at times subtle aspects as well.
- As I have mentioned before, the most important thing is that generating crash dump does not actually happen by default. It requires an administrator or installation an explicit act of setting
GOTRACEBACK=crash. This is, @ianlancetaylor, effectively the opt-in you have asked for. It is not opt-out, it is certainly opt-in as I see it at least - If we remove all abilities to generate crash dump, not only you block WER but you are effectively blocking the ability of any crash tracking utility to detect and report crashes of Go-based applications for both the authors of such applications and their user/customers.
- In enterprise settings this, as @qmuntal and @danicat already pointed out, will bypass hosts administrator configuration who want crashes to be reported in a certain way. By the way do know that an enterprise can set up on p[remises WER server where all crashes will be related.
- Even if one can block all possible crash generation from Go application code (e.g. with the noble goal of privacy) it will still not be 100% possible in cases of Go-runtime bugs and crashes which I doubt will ever be eradicated.
Another word, with effective and existing opt-in lack of this feature ALREADY reduces and will continue to reduce our ability to find and fix vulnerabilities and bugs. On the related note, similar concerns makes (to me of course) GOTRACEBACK=wer less palatable than already existing and adequate GOTRACEBACK=crash.
Setting GOTRACEBACK=crash is a standard debugging approach for Go programs on all operating systems. It is unfortunate that on some Windows systems that can lead to an opt-out sending information to Microsoft. As I said, we want to be cautious about that. You are basically suggesting that we should be less cautious, but I don't think you are engaging with the core of the argument. Consider the recent telemetry discussion (#58409) where there was significant opposition to an opt-out for sending significantly less sensitive data to Google.
Nobody is saying that we should remove all abilities to generate a crash dump. In this very issue people have suggested GOTRACEBACK=wer, and that doesn't pose any problems that I can see. In addition people have suggested having GOTRACEBACK=crash generate a crash dump if sending data to Microsoft has been disabled (if somebody has explicitly opted out). I don't know the technical feasibility of that but it also doesn't pose any problems that I can see.
Enterprises that are able to set up their own WER server or configure how crashes should be supported are entirely capable of setting GOTRACEBACK=wer by default on all their Windows systems, so I don't understand why you say that that is less palatable.
@ianlancetaylor Fair enough. I was significantly more concerned of having something to generate crash dump and if it is GOTRACEBACL=wer then it is sufficient to me (and to @qmuntal I don't want to die on this hill, if the Go community feels that GOTRACEBACK=crash should do its best to not upload anything).
But to answer on some of your questions, I do not think I have suggested that everyone should be less cautious. I was simply suggesting that these concerns, again only IMHO, simply are not the “business” of the Go Runtime or Go application developer. Perhaps a note on the documentation regarding GOTRACEBACL=crash privacy dangers would be sufficient (with a few links to disable known technologies like WER)?
I want also to add that technically GOTRACEBACK=crash is not safe for non-Windows platforms either. If a host has crash detection tools like RedHat ABRT, IBM Core Dump Handler and number of other tools deployed. If one enables GOTRACEBACL=crash on these hosts, then core dumps for Go applications may end up in other places specific for tools/technologies.
I did call GOTRACEBACK=wer less palatable for a few reasons:
- I see
GOTRACEBACK=crashas sufficient and adequate. - I do not want to have have multiple ways to track Go application crashes depending on environment and or platform.
- I do not like binding to WER technology because there are a few more ways on Windows (and other platforms) to collect and ship crash dumps automatically.
- I like the documentation for “
GOTRACEBACK=crash is like “system” but crashes in an operating system-specific manner instead of exiting” which is 100% correct IMHO. I do not want to have another document describingGOTRACEBACK=crashwhich is platform and technology specific, because it could be confusing and more importantly would give a false sense of safety due to existing or other popular tooling (already existing or future).
This is my personal opinion, I am not trying to get everyone or anyone to agree with my perspective. I am just trying to explain how I personally see and feel about it. I am certainly a Go newbie (and my apologies if I inadvertently said wrong things from Go community perspectives). I did spent though decades developing system software on Windows.