[BUG] Apm Agent 1.14.0 event sending error on composite span validation - validation rule 'min(2)' violated

[nugroho-exp]

APM Agent version

1.14.0

Environment

Operating system and version: Docker on Linux

.NET Framework/Core name and version : NET 6

Application Target Framework(s) : net6.0

Elastic/APM Server Version: 7.16.2

Describe the bug

We see a lot of event sending errors after upgrading to 1.14.0 with SpanCompressionEnabled feature enabled.

Expected behavior

The agent should send event with valid payload .

Actual behavior

Sometimes the agent send composite span with less than 2 so we get the following error:

validation error: span: composite: 'count': validation rule 'min(2)' violated

In the error log I can see composite object with count 0

"composite":{"count":0,"sum":0.0}

[Error] {"PayloadSenderV2"} Failed sending event. Events intake API absolute URL: https://xxxxxx.apm.eu-central-1.aws.cloud.es.io/intake/v2/events. APM Server response: status code: BadRequest, content: 
"{\"accepted\":7,\"errors\":[{\"message\":\"validation error: span: composite: 'count': validation rule 'min(2)' violated\",\"document\":\"{\\"span\\":{\\"action\\":\\"query\\",\\"context\\":{\\"db\\":{\\"instance\\":\\"x-test\\",\\"statement\\":\\"{ \\\\"find\\\\" : \\\\"Site__c\\\\", \\\\"filter\\\\" : { \\\\"ExternalSiteId__c\\\\" : \\\\"SIT-01558837\\\\" }, \\\\"limit\\\\" : 2, \\\\"$db\\\\" : \\\\"x-test\\\\", \\\\"lsid\\\\" : { \\\\"id\\\\" : CSUUID(\\\\"93a9632d-002e-4ce4-8383-ebe0017dbf51\\\\") }, \\\\"$clusterTime\\\\" : { \\\\"clusterTime\\\\" : Timestamp(1645627898, 7), \\\\"signature\\\\" : { \\\\"hash\\\\" : new BinData(0, \\\\"MBgrvDORMQHYS3DbIuoInQPWzNE=\\\\"), \\\\"keyId\\\\" : NumberLong(\\\\"7029680227286515718\\\\") } } }\\",\\"type\\":\\"mongo\\"},\\"destination\\":{\\"address\\":\\"pxxxx.mongodb.net\\",\\"port\\":1026,\\"service\\":{\\"name\\":\\"\\",\\"resource\\":\\"mongo\\",\\"type\\":\\"\\"}}},\\"duration\\":9.043,\\"id\\":\\"b10fe1610ca9a0b8\\",\\"name\\":\\"find\\",\\"outcome\\":\\"success\\",\\"parent_id\\":\\"adfd318b558d0c6c\\",\\"composite\\":{\\"count\\":0,\\"sum\\":0.0}............

[gregkalapos]

Thanks for reporting @nugroho-exp.

Could you please attach logs from the agent with trace level? That'd help a lot.

[gregkalapos]

Managed to figure this out without logs - fix is merged.

[nugroho-exp]

Thank you @gregkalapos . When will the fix be released?

[gregkalapos]

The 1.14.1 just went out, you should already see it - that contains the fix.

[nugroho-exp]

Great. Thank you @gregkalapos

[kruskall]

Hi @gregkalapos :wave: ,

Sorry for the ping, we bumped into the same issue. As part of the ops KPI review for APM Server, we're seeing the same error.message: validation error: span: composite: 'count': validation rule 'min(2)' violated, validation error: span: composite: 'count': validation rule 'min(2)' violated in the ecs logs

This seem to affect multiple versions including 1.22.0