[Azure Billing] Switch to Cost Management API for forecast data

[zmoog]

What does this PR do?

Switch the fetching of forecast data from the Azure Consumption API to the Cost Management API.

Why is it important?

The previous forecast endpoint from Consumption API is no longer maintained and does not support scope.

Additional information

The Consumption API version of the forecast endpoint was simple: it offered a simple filter with limited options. Instead, the Cost Management API version is a much richer endpoint with a dozen of options. The current implementation focuses on offering the same functionality using the new API.

The Azure Portal uses the Cost Management API (version 2021-10-01) to implement the Cost Management > Cost analysis page. We used the Azuire Portal as a reference to build the request for the Forecast data in this module.

This Azure Billing module is using an older version of the Cost Management API: 2019-10-01, it is the latest working [^1] version offered in the official Azure SDK. If we need to jump to a later version, we built a fallback solution: the github.com/zmoog/azure-sdk-clients repo contains an unofficial Azure SDK client generated from the OpenAPI spec using the same toolchain Microsoft uses.

[^1]: Version 2020-06-01 does not work for Forecast.

Checklist

• [x] My code follows the style guidelines of this project
• [x] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

On Azure:

• Create a new App registration.
• Take note of the "Application (client) ID"; it will be your CLIENT_ID.
• Visit App Registrations > App > Certificates & secrets > New client secret.
• Take note of the secret "Value"; it will be your CLIENT_SECRET.
• Visit Cost Management + Billing > Access control (IAM) > Add role assignment.
• Select role "Billing account reader" and the name of your app in "Users, groups, or app".

On your local environment:

• Enable the Azure Billing module and use the following configuration:

- module: azure
  metricsets:
    - billing
  enabled: true
  period: 24h
  client_id: '<CLIENT_ID>'
  client_secret: '<CLIENT_SECRET>'
  tenant_id: '<TENANT_ID>'
  subscription_id: '<SUBSCRIPTION_ID>'
  refresh_list_interval: 600s

• Build and run Metricbeat

$ cd x-pack/metricbeat
$ mage build 
>> build: Building metricbeat

$ ./metricbeat -e -E cloud.id=<CLOUD_ID> -E cloud.auth=<CLOUD_AUTH> -E output.elasticsearch.allow_older_versions=true

You may need to use output.elasticsearch.allow_older_versions=true if you built Metricbeat from sources and not using the latest version of ES.

Related issues

• Closes #32033

[elasticmachine]

:green_heart: Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-09-08T10:07:55.477+0000

• Duration: 53 min 3 sec

Test stats :test_tube:

Test	Results
Failed	0
Passed	1244
Skipped	87
Total	1331

:green_heart: Flaky test report

Tests succeeded.

:robot: GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify[bot]]

This pull request does not have a backport label. If this is a bug or security fix, could you label this PR @zmoog? 🙏. For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change) To fixup this pull request, you need to add the backport labels for the needed branches, such as:
• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[mergify[bot]]

This pull request is now in conflicts. Could you fix it? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b switch-azure-forecast-to-cost-management-api upstream/switch-azure-forecast-to-cost-management-api
git merge upstream/main
git push upstream switch-azure-forecast-to-cost-management-api