NewRelic stops showing metrics after a while

[victorolinasc]

Describe the bug

Some of our production PODs are not logging events do NewRelic. They show CPU/RAM data but not, for example, transactions.

When looking into the application logs, it starts failing sending what was harvested and then gets a 502 response from NewRelic.

I've seen that the API client was supposed to retry on 500+ but it seems it does so only one time (here)

I am really not sure how to debug it further. If you can provide me with more instructions I might dig deeper here.

This application has a high throughput with an average of something like 80k/100k rpm. Perhaps not Google scale but still I think this is important to say.

Environment

• Elixir & Erlang version (elixir -v):

Erlang/OTP 24 [erts-12.0.4] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:no-native-stack]

Interactive Elixir (1.12.3)

• Agent version (mix deps | grep new_relic_agent):

"new_relic_agent": {:hex, :new_relic_agent, "1.27.6", "33de059c3f6585589ec652438eb4608b81075a271ca5e4e380ce4dc09626db24"