Deploying a new version of the Agent over an old one has unintended behaviors.

[miketheman]

Symptoms:

Traceback (most recent call last):
  File "/app/.apt/opt/datadog-agent/agent/dogstatsd.py", line 29, in <module>
    from aggregator import MetricsBucketAggregator, get_formatter
  File "/app/.apt/opt/datadog-agent/agent/aggregator.py", line 6, in <module>
    from checks.metric_types import MetricTypes
  File "/app/.apt/opt/datadog-agent/agent/checks/__init__.py", line 26, in <module>
    from checks import check_status
  File "/app/.apt/opt/datadog-agent/agent/checks/check_status.py", line 24, in <module>
    from utils.jmxfiles import JMXFiles
ImportError: No module named jmxfiles

After this, dogstatsd fails to start, and you get no metrics. :sob:

This occurs on slugs that have been pushed with versions of Datadog Agent prior to 5.4.x and then redeployed after.

This is due to a change it the internal file layout of the Agent, however sing the slug compiler will retrieve any existing .deb packages from the builder cache, and then install all of them sequentially, you may be left in a state where a file named utils.py exists in a place where the loader does not expect it to be.

Short-term mitigation

• install heroku plugin: heroku plugins:install heroku-repo
• clear the build cache: heroku repo:purge_cache
• redeploy

Longer-term

• Address removal of preexisting *.deb packages before running the install, as this can impact here, as well as overall slug size.

[miketheman]

I'm a little lost - @bjeanes would you have a good idea on how to handle this?

[miketheman]

One idea is to try and remove any files by most recent during compile in somewhere like here - but I'm not certain that's a great idea yet.

[bjeanes]

Hmm sorry I don't have a great understanding of our build system to have an intuitive sense of what is going on here ... :\

[brettlangdon]

@miketheman have you considered allowing people to pin the version of the agent they are using? that way they can make a more conscious effort to do an agent upgrade (and clear cache, etc), rather than being caught unawares with an upgrade?

[miketheman]

@brettlangdon That hadn't crossed my mind yet - do you have an idea on how that would be done? The part I want to ensure is that if a new Agent is released, nothing has to be done in this codebase - people either get the new one by default when they redeploy, or their pinning method (envvar?) is the only thing needing an update.

[brettlangdon]

Yeah, my thought was an optional environment variable to pin the version DATADOG_AGENT_VERSION="5.6.3"

Then just make sure to update the install version here: https://github.com/miketheman/heroku-buildpack-datadog/blob/a9cff3c98968e40ad2c5c4afe7e1f71785b635a3/bin/compile#L52

DATADOG_AGENT_PACKAGE="datadog-agent"
if [[ $DATADOG_AGENT_VERSION ]]; then
    DATADOG_AGENT_PACKAGE="datadog-agent=1:${DATADOG_AGENT_VERSION}-1"
fi
apt-get $APT_OPTIONS -y --force-yes -d install --reinstall --no-install-recommends $DATADOG_AGENT_PACKAGE | indent

Or something along those lines.

[jyee]

+1 to the optional pinning solution. I think if we combine that with defaulting to the most recent .deb for anything dynos without a pinned version, we'd cover the ease of staying up-to-date with the easy ability to maintain compatibility when needed.