Configuration not reloading when Mappings are updated

[joshbranham]

Describe the bug Our deployment(s) of emissary-ingress have been showing signs of not loading their config from Kubernetes state. For example, updating a mapping, then checking in the admin UI and the updated mapping is not loaded. Additionally the admin UI shows the config being many days old, even though mappings have changed. If you restart the deployment, the config will then be updated.

Configuration from 2023-09-27 13:49:44.491353 — 13 days, 1 hour, 37 minutes, 20 seconds ago

To Reproduce

1. Run emissary-ingress 3.7.2 on Kubernetes 1.25.8.
2. Apply a Mapping
3. Then, modify that mapping and re-apply it
4. Open the admin UI and you will see the old mapping config
5. Restart the emissary deployment and reload the admin ui, which will now show the right mapping data

Expected behavior I expected the mapping to update in emissary-ingress as it always has. Unclear if this broke with our upgrade to 3.7 sometime in July.

Versions (please complete the following information):

• Ambassador: 3.7.2
• Kubernetes environment: kOps on AWS
• Version: 1.25.8

Additional context https://datawire-oss.slack.com/archives/CAULN7S76/p1696951814408599