[Metricbeat] GCP cloudsql metadata

[gpop63]

What does this PR do?

While working on the GCP CloudSQL integration https://github.com/elastic/integrations/pull/4126 I discovered that there is no way to know which database type (PostgreSQL, MySQL or SQLServer) generated an event.

Adds cloudsql metadata.

An event would look like this with the added cloudsql labels.

 "gcp": {
    "labels": {
      "resource": {
        "region": "us-central",
        "database_id": "elastic-observability:postgres"
      },
      "cloudsql": {
        "version": "14",
        "name": "postgres"
      },
      "metrics": {
        "state": "SUSPENDED"
      }
    },
    "metrics": {
      "database": {
        "instance_state": {
          "value": false
        }
      }
    }
  },

@kaiyan-sheng suggested adding the cloudsql metadata under gcp not under labels but the stackdriver metadata only allows access to ECS and Labels.

Why is it important?

Checklist

• [ ] My code follows the style guidelines of this project
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

[mergify[bot]]

This pull request does not have a backport label. If this is a bug or security fix, could you label this PR @gpop63? 🙏. For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

:green_heart: Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-11-15T16:45:41.478+0000

• Duration: 54 min 41 sec

Test stats :test_tube:

Test	Results
Failed	0
Passed	1392
Skipped	87
Total	1479

:green_heart: Flaky test report

Tests succeeded.

:robot: GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[endorama]

@tommyers-elastic May you have a look at this comment:

@kaiyan-sheng suggested adding the cloudsql metadata under gcp not under labels but the stackdriver metadata only allows access to ECS and Labels.

and provide your opinion here? We can either try to fix this here, leave it like this for consistency or try to refactor (we would need an exploration phase to understand how this would play out)

[kaiyan-sheng]

Sorry I missed the comment here. I'm fine with leaving the fields under labels for consistency. I mentioned gcp because that's what the metadata are stored for AWS as an example, not a suggestion. But I'd rather have the consistency and no breaking change here.

[endorama]

@gpop63 I was reviewing this contribution, there is only one thing that we need to double check: this code uses the same caching logic as instance metadata. We discovered a bug in the caching logic there, and we decided to remove the cache as it wasn't been used properly.

Can you please remove the caching logic from this module? It will be reintroduced if the need arise in future updates.

[gpop63]

I have a question regarding cloud.id and cloud.name fields. CloudSQL seems to not have unique ints as instance ids, the id of an instance is a concatenation of the project name and the instance name {project}:{instance_name}. Is this ok?

"cloud": {
  "instance": {
    "id": "elastic-observability:postgres",
    "name": "postgres"
  },

[endorama]

@gpop63 If that is the GCP provided ID I would move forward with it. We need to make sure the mapping in the data stream is correct though, or there could be errors during ingestion.

[kaiyan-sheng]

I have a question regarding cloud.id and cloud.name fields. CloudSQL seems to not have unique ints as instance ids, the id of an instance is a concatenation of the project name and the instance name {project}:{instance_name}. Is this ok?

"cloud": {
  "instance": {
    "id": "elastic-observability:postgres",
    "name": "postgres"
  },

Actually looking at this comment, I don't think we should use cloud.instance.id and cloud.instance.name for these fields. In ECS, cloud.instance.id is defined as Instance ID of the host machine.. I would probably just use gcp.cloudsql.id and gcp.cloudsqp.name instead here.