[Documentation] How-To Backfill Data Using Azure Cost Management (ACM) Managed Exports

[AErmie]

⚠️ Problem

The Configure Managed Exports documentation states: "To backfill historical data, you must run the config_RunBackfillJob pipeline for each month."

Backfilling multiple months (or years) of data can be tedious. If this is supported using the Azure Cost Management (ACM) Managed Export that is created by the FinOps Toolkit, it is not mentioned in the documentation.

🛠️ Solution

Update the documentation to provide guidance on backfilling multiple months of data at a time, versus requiring running the config_RunBackfillJob pipeline for each month.

ℹ️ Additional context

I recall it was mentioned/demonstrated that the Azure Cost Management (ACM) Managed Export could be used for this purpose. However, it is unclear how to do so. We manually ran the Export selected dates trigger for the monthly-costdetails export, but it is not clear that this will retrieve backfill data.

🙋‍♀️ Ask for the community

We could use your help:

1. Please vote this issue up (👍) to prioritize it.
2. Leave comments to help us solidify the vision.

[flanakin]

We will use this to track streamlining this pipeline. As a workaround, please use the Start-FinOpsCostExport PowerShell command to export data. You can specify -Backfill 13 to run it for 13 months (or whatever period you're looking for).

Please vote up (👍) this issue to push it up in the backlog.

[AErmie]

Thanks @flanakin, when attempting to run the Start-FinOpsCostExport command, I encounter the following error:

PS C:\Users\adine> Start-FinopsCostExport -Name 'CostExportBackfill12' -Backfill 12
Start-FinOpsCostExport: Export CostExportBackfill12 not found. Did you specify the correct scope?

According to the documentation, the -Name parameter is required, and should be "Name of the export." Does that mean it should be the actual Azure Cost Management managed export name? Since there are 2 created, my assumption is it should be the monthly-costdetails one, is that correct?

Also, the -Scope parameter (according to the documentation) is optional. The documentation states that this is the "Resource ID of the scope to export data for. If empty, defaults to current subscription context." Unfortunately, the documentation does not provide any examples of this (I can open a new GitHub Issue to request this be updated, if you prefer).

Since we are using the Enterprise Agreement Department as our scope, what would be the appropriate pattern expected? Would /providers/Microsoft.Billing/billingAccounts/1234567/departments/56789 be the correct pattern?

Update

Despite the documentation not providing a reference example, the following is the correct command to run (in our case):

  Start-FinopsCostExport -Name 'NAME_OF_ACM_EXPORT-monthly-costdetails' -Scope 'SCOPE_AS_ORIGINALLY_DEFINED_FOR_FTK' -Backfill 6

In my specific case, I do not have permissions at the Enterprise Agreement / Department level, so I encountered this error:

The user does not have valid claims on any enrollmentaccounts. (Request ID: ddb4a594-9885-4c5c-ae1b-0e2197782b54) (Code: BillingAccessDenied).
Start-FinOpsCostExport: Export NAME_OF_ACM_EXPORT-monthly-costdetails not found. Did you specify the correct scope?

[MSBrett]

Try this... Works around intermittend 429 messages. Update the export names and scopes as appropriate.

[CmdletBinding()]
param (
    [datetime]$startdate = '2024-04-01',
    [datetime]$enddate = '2025-03-31',
    $exportsToRun = @(
        'ftk-pricesheet',
        'ftk-focuscost'
    ),
    $exportScopes = @(
        '/providers/Microsoft.Billing/billingAccounts/123456'
    )
)

function GetData {
    param (
        $scopeToQuery
    )
    
    while ($startdate -lt $enddate)
    {
        [datetime]$nextdate = $startdate.AddMonths(1).AddDays(-1)
        if ($nextdate -gt $enddate) {
            $nextdate = $enddate
        }
        
        $nextdatestring = $nextdate.ToString('yyyy-MM-dd')
        $startdatestring = $startdate.ToString('yyyy-MM-dd')

        Write-Host "Exporting from $startdatestring to $nextdatestring" 
        Start-FinOpsCostExport -Scope $scopeToQuery -name $exportname -StartDate $startdatestring -EndDate $nextdatestring
        Start-Sleep -Seconds 5
        $startdate = $startdate.AddMonths(1)
    }
}

foreach ($exportname in $exportsToRun) {
    foreach ($exportScope in $exportScopes) {
        Write-Host "$exportname : $exportScope"
        GetData -scopeToQuery $exportScope
    }
}

[flanakin]

@MSBrett You don't need all that. Start-FinOpsCostExport already handles multi-month exports. You can just give it the start and end dates you want or use -Backfill for the last n months.

[flanakin]

@AErmie As you surmised, the only thing you're missing is the -Scope parameter. And yes, you'll use the Azure resource ID for the scope, so -Scope /providers/Microsoft.Billing/billingAccounts/1234567/departments/56789.

You can use either the daily or monthly. There's actually no difference between the two other than when they run by Cost Management.

We can use this issue to track adding -Scope examples. I've had that in my to do list for a while 😔

[AErmie]

Hey @flanakin, When we attempt to run the Start-FinopsCostExport PowerShell script, we experience the following error:

The user does not have valid claims on any enrollmentaccounts. (Request ID: fc0ff940-a00e-49d7-a92b-8747e55a2f1a) (Code: BillingAccessDenied).
Start-FinOpsCostExport: Export bcgov-mgd-lz-live-finops-hub-monthly-costdetails not found. Did you specify the correct scope?

Command Used:

PS C:\Users\adine> Get-Module -ListAvailable FinOpsToolkit

    Directory: C:\Users\adine\Documents\PowerShell\Modules

ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Script     0.9                   FinOpsToolkit                       Desk      {Add-FinOpsHubScope, Add-FinOpsServicePrincipal, Deploy-FinOpsHub, Get-FinOpsCostExport…}

Start-FinopsCostExport -Name 'bcgov-mgd-lz-live-finops-hub-monthly-costdetails' -Scope '/providers/Microsoft.Billing/billingAccounts/1234567/departments/123456' -Backfill 6

We know that the -Scope is correct, so our next question is: How does the PowerShell script actually work?

For example, my assumption would be that the PowerShell script effectively tells the FinOps Toolkit to "go do its thing" and export the backfill data. I would not assume that it somehow uses the individual user's credentials (that's triggering the Start-FinOpsCostExport command) to run the actual backfill export. Can you please confirm the script's expectations?

We also tried creating another Service Principal that has the same scope permissions as the System Assigned Managed Identity (though no permissions on the FinOps Toolkit resources deployed in Azure), and attempted to use that to execute the Start-FinOpsCostExport command, but still received the same error message.

Appreciate any guidance the team can provide (and also subsequent documentation updates).

[MSBrett]

The user does not have valid claims on any enrollmentaccounts. (Request ID: fc0ff940-a00e-49d7-a92b-8747e55a2f1a) (Code: BillingAccessDenied). Are you connected to the correct entra tenant and does the account have department reader permissions?

[AErmie]

The user does not have valid claims on any enrollmentaccounts. (Request ID: fc0ff940-a00e-49d7-a92b-8747e55a2f1a) (Code: BillingAccessDenied). Are you connected to the correct entra tenant and does the account have department reader permissions?

Hey @MSBrett, yes we are connected to the correct tenant, and we granted the account EA Department Reader permissions (using the same method that was followed for the FTK Managed Identity).

PS C:\Users\adine> az login --service-principal --username APP_ID --password PASSWORD --tenant TENANT_ID --allow-no-subscriptions
[
  {
    "cloudName": "AzureCloud",
    "id": "TENANT_ID",
    "isDefault": true,
    "name": "N/A(tenant level account)",
    "state": "Enabled",
    "tenantId": "TENANT_ID",
    "user": {
      "name": "APP_ID",
      "type": "servicePrincipal"
    }
  }
]
PS C:\Users\adine> Start-FinopsCostExport -Name 'bcgov-mgd-lz-live-finops-hub-monthly-costdetails' -Scope '/providers/Microsoft.Billing/billingAccounts/1234567/departments/123456' -Backfill 6
Write-Error: C:\Users\adine\Documents\PowerShell\Modules\FinOpsToolkit\0.9\Public\Get-FinOpsCostExport.ps1:133
Line |
 133 |  … $response = Invoke-Rest -Method GET -Uri $path -CommandName "Get-FinO …
     |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | The user does not have valid claims on any enrollmentaccounts. (Request ID: 21aff80b-f3e7-436b-836b-50d1bd5fa81e) (Code: BillingAccessDenied).
Start-FinOpsCostExport: Export bcgov-mgd-lz-live-finops-hub-monthly-costdetails not found. Did you specify the correct scope?

[MSBrett]

That cost management export - 'bcgov-mgd-lz-live-finops-hub-monthly-costdetails' - what scope was it created at in cost management. Was it perhaps created at the enrollment level rather than at the department level?

[AErmie]

Hey @MSBrett, the Cost Management exports were created as part of deploying the FinOps Toolkit. Our settings.json is as follows:

{
  "$schema": "https://aka.ms/finops/hubs/settings-schema",
  "type": "HubInstance",
  "version": "0.9",
  "learnMore": "https://aka.ms/finops/hubs",
  "scopes": [],
  "retention": {
    "msexports": {
      "days": 0
    },
    "ingestion": {
      "months": 13
    },
    "final": {
      "months": 25
    },
    "raw": {
      "days": 0
    }
  }
}

Interestingly, we did have the scopes set, but now it's empty! Q: Does this JSON file get reset with every upgrade?

Our settings.json was originally configured as:

{
  "$schema": "https://aka.ms/finops/hubs/settings-schema",
  "type": "HubInstance",
  "version": "0.9",
  "learnMore": "https://aka.ms/finops/hubs",
  "scopes": [
    {
      "scope": "/providers/Microsoft.Billing/billingAccounts/1234567/departments/56789"
    }
  ],
  "retention": {
    "msexports": {
      "days": 0
    },
    "ingestion": {
      "months": 13
    },
    "final": {
      "months": 25
    },
    "raw": {
      "days": 0
    }
  }
}

[MSBrett]

Hi @AErmie

Yes, I've seen the scope removed during upgrades too, but haven't had any other reports of it yet.

The cost management export you're trying to run - what scope was it created at? Is it at the same scope (department) as you're using in the scope variable?

What level of permissions were granted to the service principal?
Could you try using the PS module making sure to use the ObjectId of the app registration - https://github.com/microsoft/finops-toolkit/blob/dev/src/powershell/Public/Add-FinOpsServicePrincipal.ps1 It should throw an error saying the role assignment already exists.

[AErmie]

Hey @MSBrett, yes the Managed Export is configured at the same level (since it was created by the FTK deployment itself). This specifically is the Enterprise Agreement Department level (ie. /providers/Microsoft.Billing/billingAccounts/1234567/departments/56789).

The Service Principal was granted EA Department Reader, per the following: Assign the department reader role to the service principal

When I run the Add-FinOpsServicePrincipal command, I get the following error, but that's probably because I don't have access at the EA Department level myself.

PS C:\Users\adine> Add-FinOpsServicePrincipal -ObjectId xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -TenantId xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -BillingAccountId 1234567 -DepartmentId 56789
{"properties": { "PrincipalId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "PrincipalTenantId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "roleDefinitionId": "/providers/Microsoft.Billing/billingAccounts/1234567/departments/56789/billingRoleDefinitions/db609904-a47f-4794-9be8-9bd86fbffd8a" } }
OperationStopped: C:\Users\adine\Documents\PowerShell\Modules\FinOpsToolkit\0.9\Public\Add-FinOpsServicePrincipal.ps1:129
Line |
 129 |              throw $_.Exception
     |              ~~~~~~~~~~~~~~~~~~
     | Response status code does not indicate success: 403 (Forbidden).

[AErmie]

Quick update... We were able to run the Start-FinOpsCostExport command, but using the -Backfill # parameter threw the following error:

PS C:\Users\adine> Start-FinopsCostExport -Name 'bcgov-mgd-lz-live-finops-hub-monthly-costdetails' -Scope '/providers/Microsoft.Billing/billingAccounts/1234567/departments/56789' -Backfill 6
Write-Error: C:\Users\adine\Documents\PowerShell\Modules\FinOpsToolkit\0.9\Public\Start-FinOpsCostExport.ps1:183
Line |
 183 |  … $response = Invoke-Rest -Method POST -Uri $runpath -Body $body -Comma …
     |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Request properties validation failed: Invalid definition timePeriod; 'to' value cannot be in the future. (Code: BadRequest).
Write-Error: C:\Users\adine\Documents\PowerShell\Modules\FinOpsToolkit\0.9\Public\Start-FinOpsCostExport.ps1:183
Line |
 183 |  … $response = Invoke-Rest -Method POST -Uri $runpath -Body $body -Comma …
     |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Request properties validation failed: Invalid definition timePeriod; 'to' value cannot be in the future. (Code: BadRequest).
False

We re-ran the Start-FinOpsCostExport command, but used the -StartDate and -EndDate parameters instead. That seemed to work, despite a slight pause due to "too many requests."

PS C:\Users\adine> Start-FinopsCostExport -Scope '/providers/Microsoft.Billing/billingAccounts/1234567/departments/56789' -Name 'bcgov-mgd-lz-live-finops-hub-monthly-costdetails' -StartDate '2024-01-01' -EndDate '2024-06-30'
True
PS C:\Users\adine> Start-FinopsCostExport -Scope '/providers/Microsoft.Billing/billingAccounts/1234567/departments/56789' -Name 'bcgov-mgd-lz-live-finops-hub-monthly-costdetails' -StartDate '2024-07-01' -EndDate '2024-12-31'
Write-Error: C:\Users\adine\Documents\PowerShell\Modules\FinOpsToolkit\0.9\Public\Start-FinOpsCostExport.ps1:183
Line |
 183 |  … $response = Invoke-Rest -Method POST -Uri $runpath -Body $body -Comma …
     |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Too many requests. Please retry after 60 seconds. (Code: 429).
True

After successfully running those commands, the Data Factory shows multiple ETL pipelines running:

Q: Is there a reason why the Start-FinOpsCostExport PowerShell script needs to be run as a user with permissions at the target scope? Why wouldn't the script simply initiate a job to ACM (and use the FTK credentials)?

Most large organizations do not allow the Enterprise Agreement level permissions. This was our challenge (we did not have these permissions), which makes it very difficult to troubleshoot, etc. Additional, we cannot see the actual ACM Export in the portal (because of the scope it's created at).

[flanakin]

Someone mentioned that Start-FinOpsCostExport seems like it's asking for the wrong dates when it starts with the current month. I suspect that's a timezone issue, but I'm not 100% sure. I haven't looked at it yet.

[flanakin]

The Start-FinOpsCostExport command uses your Az credentials. That command doesn't know anything about FinOps hubs. We had plans to create a Start-FinOpsHubBackfill command, but the people who were going to work on that left before we got there 😔 There's a huge backlog of PowerShell ideas that we'd really like to deliver on, but don't have enough contributors to get there. If you know anyone with experience that would be open to helping, please send them our way! Every little bit helps!

[flanakin]

You really do need enrollment-level access. Without that, you won't be able to be able to get prices or reservation recommendations and you won't be able to calculate savings. That's a major limitation. I'm definitely familiar with the challenge. I'm not sure what we need to do to convey how important that is, but I'm happy to brainstorm ideas. I can work with the Cost Management team to document something, if that would help. This usually depends on the org.