kamon-io
Connection error, retrying... (11 left) timeout
I have seen this error almost 6k times in a single project on the last 15 days:
at kamon.apm.KamonApmApiClient.postWithRetry(KamonApmApiClient.scala:101)
at kamon.apm.KamonApmApiClient.postSpans(KamonApmApiClient.scala:32)
at kamon.apm.KamonApm.$anonfun$reportSpans$2(KamonApm.scala:82)
at kamon.apm.KamonApm.$anonfun$reportSpans$2$adapted(KamonApm.scala:82)
at scala.Option.foreach(Option.scala:407)
at kamon.apm.KamonApm.reportSpans(KamonApm.scala:82)
at kamon.module.ModuleRegistry$$anon$2.$anonfun$run$4(ModuleRegistry.scala:214)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
Although I don't have the stack trace to confirm it's the same problem, I frequently get what appears to be the same timeouts...
2020-08-04 14:08:34,420 [warn] k.a.KamonApmApiClient - Connection error, retrying... (20 left) timeout 2020-08-04 14:08:58,510 [warn] k.a.KamonApmApiClient - Connection error, retrying... (20 left) timeout . . . 2020-08-04 14:38:37,315 [error] k.a.KamonApmApiClient - Ingestion error, no retries, dropping snapshot... timeout
And it appears at that point that it isn't reporting and from the Kamon dashboard side, I get no statistics and alerts start firing that I don't have enough instances (based on my alert levels of course).
[2021-06-04 14:02:33,644] [WARN] [kamon.apm.KamonApmApiClient] [] [main] - Connection error, retrying... (1 left) sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target [2021-06-04 14:02:38,772] [ERROR] [kamon.apm.KamonApmApiClient] [] [main] - Ingestion error, no retries, dropping snapshot... sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
