Clarification on "blessed" approach for running multiple different datadog agent setups within a single cluster

[dlmather]

Describe what happened: I need to run multiple (currently just two) different datadog-agent setups within a single cluster with very very slight variations. Ideally all I need that's different between the setups is different node selectors / affinity (so they don't overlap), slightly different labels, and one different env var. Everything else can be the same. It's not clear to me how to easily make this work with the datadog operator:

• it's unclear that creating multiple concurrent versions of the DatadogAgent object will work for this, for one, the operator seems to hit various issues and fail to converge. It seems that under this setup there would also be the creation of redundant cluster-agents which seems less than ideal.
• It's unclear to me that using something like extended daemonsets would support my use-case correctly
• the DAPs setup that's being worked on looks close to what I'd want as a simple system for overriding the base datadog agent configuration, but the set of settings that can be overwritten seems to be very limited still.

I'm seeking clarification on what the right path to configure this should be based on my requirements. For additional context, we deploy the datadog-operator currently through helm via the terraform helm add-on. If it will help for my use-case I can upgrade to any version, currently running a somewhat older one.