Datadog re-appends `latest` tag on top of `image_tag` added through Tag Auto-discovery

[emalprokt]

Agent Environment Agent Version: 7.54.1 pulled through public.ecr.aws/datadog/agent:7.54.1

Describe what happened:

For example for a container with image docker.io/library/alpine:3.20.1 the Datadog agent, reports image_tag as 3.20.1. This is as expected.

For container with image docker.io/library/alpine@sha256:xxxx, the Datadog agent reports image_tag as latest. This must be a design choice? Further questions relating to this are asked below.

For a pod spec:

apiVersion: v1                                                                                    
kind: Pod                                                                                            
metadata:                                                                                            
   annotations:                                                                                       
     ad.datadoghq.com/alps.tags: '{"image_tag": "3.20.1_own"}'
spec:
  containers:
    - name: alps
       image: docker.io/library/alpine@sha256:xxxx

There are two image_tag tags and are reported as 3.20.1_own, latest

Describe what you expected: image_tag should've been 3.20.1_own and NOT another tag with latest

Question 1:

Why is latest the default tag is no tag is present? This does not make sense, shouldn't this be empty or null and not added as a tag? Code Reference 1, Code Reference 2

Question 2:

Why is the DD agent re-appending the latest tag to images with only digests, when the image_tag has already been added as annotation for Auto-Discovery?

Steps to reproduce the issue:

Just launch a pod with the spec with Datadog agent setup for the cluster to forward logs to Datadog.

apiVersion: v1                                                                                    
kind: Pod                                                                                            
metadata:                                                                                            
   annotations:                                                                                       
     ad.datadoghq.com/alps.tags: '{"image_tag": "3.20.1_own"}'
spec:
  containers:
    - name: alps
       image: docker.io/library/alpine@sha256:xxxx

We will see that all logs from this node in the K8s cluster will report two image_tag as 3.20.1_own and latest.

Additional environment details (Operating System, Cloud provider, etc):

The environment is aws-eks kubernetes v1.28 with the agent installed.