Add ability to suppress Linkerd error response body and header

[chrismikehogan]

Issue Type:

• [ ] Bug report
• [X] Feature request

Problem

We are trying to use Linkerd's dtab feature to build very large, dynamic routing tables. The goal to use Linkerd (instead of, e.g. Consul) to control which versions of which applications get routed to when a a request for a particular service is received.

This requires us to dynamically build most of our dtab, and requires that the dtab have an explicit dentry for every service in our eco system. E.g.:

...
/svc/cool-thing => /tagged/ver-1.3/cool-thing;
/svc/lame-thing => 0.1 * /tagged/ver-2.0/lame-thing & 0.9 * /tagged/ver-1.8/lame-thing;
...

This means that our dtab will get very large! The example I'm working with at the moment is 1811 lines long, and 143kb.

When linkerd errors in some way (say when a requested service is not found), it currently 502s and sends back an error message that contains, among other things, the entire dtab in both the body and a response header.

Fortunately, Linkerd appears to truncate the response header field instead of blowing up. But I think it would be useful to be able to suppress this verbose response by default, exposing it only if the client provided a debug header.

Possible solution

Provide a configuration option to suppress verbose error responses by default. Add support for a debug header that results in the verbose error response.

[dadjeibaah]

👋 @chrismikehogan, Thanks for filing this. FWIW, there is a config parameter called clearContext that can be set at the router level of the config. It's a server side configuration so you won't have the ability to dynamically turn it on or off per request.

[chrismikehogan]

I saw that flag in the configuration docs, but it wasn't obvious to me that it would do what I wanted, in addition to what the docs said it does:

if true, all headers that set Linkerd contexts are removed from inbound requests. Useful for servers exposed on untrusted networks.

We definitely don't wan't the behavior described in the docs, even if it got us the suppressed error responses.

[dadjeibaah]

Ah, that's right, it also gets applied to inbound requests as well so that wouldn't work.

[dadjeibaah]

Thinking about this a little bit more, do you think it would be better to mimic what Linkerd does to the l5d-err header? So instead of sending back the entire dtab, Linkerd would truncate the dtab sent in the body if it is more than a specific number of lines?

The config parameter could definitely help but I feel that adding more implicit behavior through a header might increase the complexity of Linkerd's per-request handler. I am thinking that an error response with that much verbosity might not be as useful anyway and so it might be better to just truncate it. WDYT?

/cc @adleong

[adleong]

Truncating the response sounds great to me

[chrismikehogan]

Truncating would work perfectly for us as well, even more so if the level of truncation could be controlled.

We could take or leave the concept debug header adding additional info.

[andrii-minchekov]

It's a great idea to truncate a response body in case of a mesh error (request timeout or bad routing, etc) or just unify all errors to have a generic error body for all cases. It will much simplify client side because now it's a headache to handle all this different structures of a response body for each error. Looking forward to this important fix.