Endpoint is stopped when additional headers for error queue message result in a message too large for the transport

[adamralph]

See https://github.com/Particular/NServiceBus/issues/2956#issuecomment-147801149

An initial patch (5.2.7) was released which truncates the exception message to a max of 16,384 characters but it seems this is not enough in some cases.

From @burkhartt:

We actually noticed this on QA today :-/ (Thought it had fixed the issue b/c we had huge stack traces > and they weren't blowing up the queue lately.)

=ERROR REPORT==== 13-Oct-2015::16:08:25 === Error on AMQP connection <0.11316.60> (IP1 -> IP2, vhost: '/', user: 'tim', state: running), channel 2: {amqp_error,frame_error, "type 2, all octets = <<>>: {frame_too_large,189514,131064}",none}

This is running with the 5.2.7 version.

and

Retry 2 - HEADERS: [NServiceBus.MessageId] -- c0830ac6-959d-43d4-bfc3-a53100035a15 ........... [NServiceBus.CorrelationId] -- c0830ac6-959d-43d4-bfc3-a53100035a15 ........... [NServiceBus.MessageIntent] -- Send ........... [NServiceBus.Version] -- 5.2.7 ........... [NServiceBus.TimeSent] -- 2015-10-14 04:12:12:229934 Z ........... [NServiceBus.ContentType] -- text/xml ........... [NServiceBus.EnclosedMessageTypes] -- Engine.Messages.SalesForce.BookingPublishedForCrmCommand, Engine, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null ........... [NServiceBus.RelatedTo] -- 0cb50531-130c-46ba-93df-a531000358ed ........... [NServiceBus.ConversationId] -- cb0c7391-e683-4635-8fba-a531000358ed ........... [WinIdName] -- system ........... [Session.User.Identity.Name] -- system ........... [Country] -- ........... [NServiceBus.RabbitMQ.CallbackQueue] -- GAT.Cloud.AtlasAir.USAM-GATQAWEB1 ........... [NServiceBus.OriginatingMachine] -- USAM-GATQAWEB1 ........... [NServiceBus.OriginatingEndpoint] -- GAT.Cloud.AtlasAir ........... [$.diagnostics.originating.hostid] -- 814b04713d94785409c3f40fa4328670 ........... [NServiceBus.ReplyToAddress] -- GAT.Cloud.AtlasAir ........... [$.diagnostics.hostid] -- 65555f9599731e3d1131f12b2f643b83 ........... [$.diagnostics.hostdisplayname] -- USAM-GATQAWEB1 - BODY: System.Byte[] - MESSAGE: POST https://www.salesforce.com/services/apexrest/orders With data:

It seems the size of the message itself is leaving very little room for the exception message and stacktrace.

[burkhartt]

Thanks for looking into this. We ended up truncating the exception and referencing the payload in a different log entry to avoid running into this error. But if it gets fixed, that would be great also :-)

[danielmarbach]

FYI it is possible to use the header customization to remove or trim certain headers. The headers are mutable when passed to the customization. This acceptance test also shows it in action

[ramonsmits]

FYI: The following gist replaces the stacktrace with an MD5 and if its a new stacktrace writes the stack trace as a Debug log even. This reduces the header data significantly.

• https://gist.github.com/ramonsmits/5bed13a230c254b0694e9580e409af6f