Tracking Issue for Dynamic Modules

[mathetake]

I will dedicate this is the tracking issue for the dynamic module feature. The description here will be updated as we make progress. This issue will serve as a central place for tracking the progress as well as the feature requests, etc.

General Status

It's available on the core Envoy image.

• The high-level documentation is here.
• The example repository: envoyproxy/envoy-dynamic-module-example
• Basic functionality such as header manipulation, body manipulation, local reply, and dynamic metadata are available.
• Dynamic Modules slack channel: #envoy-dynamic-modules in Envoy slack

Released in v1.34

• [x] (@mathetake) Add "remove header" callbacks. #38432
  • I think making set_xxx_header/trailer callbacks accept null pointer indicating removal would be simpler'
• [x] (@mathetake) Initial support for attributes https://github.com/envoyproxy/envoy/pull/38519
• [x] (@mathetake) Remove "WIP" from the config API. https://github.com/envoyproxy/envoy/pull/38956
  • This will be by the end of next March.

P0 (before v1.35)

• [x] (@mathetake) Add a minimal test target here to ensure the linking work on macos. My attempt: #38437
  • As indicated in #38363, macos linker doesn't support --export-dynamic so I suspect that the loading is not working right now. Since we are not running tests at all right now, I would like to add //test/extensions/dynamic_modules/http:filter_test there and fix the linker stuff accordingly.
• [x] (@mathetake) Build a foundational asynchronous callback mechanism for dynamic modules. https://github.com/envoyproxy/envoy/pull/39765
  • Proper life cycle management via something like a manual new/delete callback would be needed to provide the safe interface.
• [x] Callout related ABI. https://github.com/envoyproxy/envoy/pull/39151
• [ ] (help wanted) Add stats/metrics callbacks to allow dynamic modules to emit metrics.

Other outstanding items / Open questions

• Upstream HTTP filter support: https://github.com/envoyproxy/envoy/issues/38448 https://github.com/envoyproxy/envoy/pull/38686
• Network filter support
• Add more attributes implementation left as TODOs in https://github.com/envoyproxy/envoy/pull/38519
• filter state: https://github.com/envoyproxy/envoy/pull/38979
• Socket accessor: #38739
• Other language SDKs (Notably, C, C++ or Zig, maybe Go.)
  • This will be revisited after we have a good enough asynchronous callback mechanism.
  • Do we ever need/want to support other languages than Rust?
• After all the feature requests settle down, revisit the compatibility requirement and maybe relax it to work across multiple versions of Envoy.

[mathetake]

cc @bplotnick @thenewwazoo

[isker]

This might be unnecessary if we can provide a good enough asynchronous callback mechanism since dynamic modules can do whatever they want in the module including making outbound HTTP requests, etc.

Do you mean that in this situation, dynamic modules would be using arbitrary HTTP clients? We use proxy-wasm filters today to make outbound HTTP requests, and it's nice that those use Envoy clusters, because they get the same configuration, connection pooling, service discovery, statistics, etc that the rest of Envoy has.

[mathetake]

yeah i guess that makes sense - thank you for clarifying @isker. Let's add the envoy-managed callouts just like Lua and Wasm

[nfuden]

As part of P1 or P2 is there any desire to make a per route message for this filter?

I've been playing around with the filter and would love to be able to get its configuration a bit more granular. If it fits roadmap happy to spend some time to see if i can throw up something for it.

[mathetake]

yeah sure, indeed i expected that someone will request for it. happy to review!

[mathetake]

I just created #envoy-dynamic-modules for discussion around this feature

[mathetake]

https://github.com/envoyproxy/envoy/pull/38979

[hieusydo]

Hi @mathetake, is anyone working to support more attributes from envoy_dynamic_module_type_attribute_id ? If not, I'd like to add implementations for some attributes related to connection and upstream infos (e.g., connection.uri_san_peer_certificate & upstream.uri_san_peer_certificate)

[mathetake]

I think no one is working on that part, so sounds good and feel free to work on it! Happy to review

[luistilingue]

I'm working for conversational platform company and it uses (like many others) WebSockets and raw-TCP with a XMPP-based protocol for their apps due to mainly full-duplex communication, called LIME .

WebSockets uses persistent connections to allow the duplex flow and we're facing an issue for load balancing those protocols because the LB they are based on connection and not in its data. So a client could overload a single endpoint (e.g. k8s pod), because there isn't any client-lb in our apps (k8s endpoint aware).

I could overcome such limitations and create a DM to convert such stateful protocols to a HTTP/2 to talk to our backends, but acuatly they are limited to HTTP Filter only.

So, It'll very welcome see a Network Filter support to parsing paylods, etc, and make that changes in the proxy level, instead our entire SDKs and codebases.