Configurations on Ambassador Module not applied to the emissary ingress pods

[mohitreddy1996]

Describe the bug We have multiple EKS clusters with Emissary Ingress + Linkerd setup

We had defined Ambassador module in each to with configuration: "add_linkerd_headers": true

We recently to change envoy's log format, so tried modifying the Ambassador module. The changes are reflected on the module but they are not being applied. We also restarted the emissary ingress and apiext pods, but they don't seem to be using the latest configuration.

We also enabled debug logging for the emissary ingress pod by setting:

"AES_LOG_LEVEL": "debug",
"AMBASSADOR_DEBUG": "diagd",

We do not see (or at least seem to think) any errors on the emissary ingress pod. We do see this message:

DEBUG: V3Listener: Using log_format 'ACCESS [%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%"'

which seem to be coming from here: https://github.com/emissary-ingress/emissary/blob/b2b54f10ee38ff8068454b91b2676d762b6ed570/python/ambassador/envoy/v3/v3listener.py#L394

so is the ambassador_module not being loaded at all?

Ambassador Custom Resource for the module:

"apiVersion": "getambassador.io/v3alpha1",
"kind": "Module",
"metadata": {
    "name": "ambassador"
},
"spec": {
    "config": {
        "add_linkerd_headers": true,
        // https://www.getambassador.io/docs/edge-stack/latest/topics/using/circuit-breakers/ and
        //
        // https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/circuit_breaking.html
        //
        // NOTE: circuit breaker configured below is a GLOBAL circuit breaker. If there are more than one endpoint
        // configured in the future, consider making these limits at a `MAPPING` level.
        "circuit_breakers": {
            // Specifies the maximum number of connections that Ambassador Edge Stack will make to ALL hosts in the upstream cluster.
            "max_connections": 3072,
            // Specifies the maximum number of requests that will be queued while waiting for a connection.
            "max_pending_requests": 1024,
            // Specifies the maximum number of parallel outstanding requests to ALL hosts in a cluster at any given time.
            "max_requests": 3072,
            // default - "max_retries": 3,
        },
        "envoy_log_type": "text",
        "envoy_log_format": "%REQ(:METHOD)% %RESPONSE_CODE% %RESPONSE_FLAGS% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% %DURATION%"
    }
}

Also we don't see any debug logs for circuit breaker configuration as well. Not sure if we are missing something here..

We also looked at the ir.json, aconf.json files in /ambassador/snapshots/ directory of the pod. Could not find any of the above configured settings (looking at the documentation it seemed like module with name ambassador is special so it could be something which is handled as a special case in emissary, so didn't see it as a strong signal).

To Reproduce Steps to reproduce the behavior:

1. Create a kubernetes cluster
2. Define an Ambassador Module
3. Setup emissary with it
4. Modify the Ambassador Module to format envoy logging
5. restart the emissary ingress pods

Expected behavior The changes should be applied to the emissary ingress pods and further applied to envoy

Versions (please complete the following information):

• Ambassador: We use emissary ingress, image - docker.io/emissaryingress/emissary:2.2.1
• Kubernetes environment: EKS, running on v1.22.9
• Version - 2.2.1

[cindymullins-dw]

This has been tested recently and should work. It could be a config error such that it's just not picking up the module. Are you using ambassador_ids or single_namespace scope?

[mohitreddy1996]

Yes, we do set single_namespace scope. Our container has the following env vars:

we however do not set ambassador_ids

[cindymullins-dw]

With single namespace scope, Ambassador resources will only be in that one namespace ('emissary' by default), so the Module needs to be in that namespace as well and you probably have to set this manually. Can you try editing your Module resource to add the ‘emissary’ namespace like below? apiVersion": "getambassador.io/v3alpha1", "kind": "Module", "metadata": { "name": "ambassador" "namespace" : "emissary" }, "spec": {