finops-toolkit icon indicating copy to clipboard operation
finops-toolkit copied to clipboard

`config_ConfigureExports` Data Factory Pipeline Error with `Switch Export Type` Activity

Open AErmie opened this issue 2 months ago • 2 comments

🐛 Problem

After updating the settings.json scope configuration, the config_ConfigureExports is triggered but throws an error on the Switch Export Type step.

Switch Export Type Output

{
	"expression": "ea"
}

Digging into the internals of the Switch Export Type Activity, my theory is the root cause is because the EA monthly pricesheet export step is failing.

Image

👣 Repro steps

  1. Deploy v12 of the FTK
  2. Configure the settings.json scope for EA department (ie. "scope": "/providers/Microsoft.Billing/billingAccounts/1234567/departments/56789")
  3. Open Data Factory Studio
  4. Navigate to Monitor > Pipeline Runs
  5. Select the config_ConfigureExports pipeline run
  6. Review the inputs and outputs for each Activity
  7. See errors

🤔 Expected

Pipeline will run and complete exports fully.

📷 Screenshots

Image Image Image

ℹ️ Additional context

After upgrade the FTK to v12 on July 12, 2025, we (apparently) forgot to re-set the scope. Therefore, the data pipelines have been running (without errors) for 3 months. Only after correcting the scope do the data pipelines start to throw errors. No other changes have occurred in the environment.

Interestingly, the EA monthly pricesheet export Activities also throw errors, as follows:

{
	"error": {
		"code": "Unauthorized",
		"message": "Unauthorized. Authentication failed. RequestId: 6a9c4eb9-deda-4cea-90a5-e37499c0ec12"
	},
	"ADFHttpStatusCodeInResponse": "401",
	"ADFWebActivityResponseHeaders": {
		"x-ms-operation-identifier": "",
		"session-id": "5c72bea7-4dd1-468a-9f2a-2247681bc5e8",
		"x-ms-request-id": "6a9c4eb9-deda-4cea-90a5-e37499c0ec12",
		"x-ms-correlation-id": "230bcb6d-ae46-41ac-9e13-a1dd5501c6a0",
		"x-ms-correlation-request-id": "44919aba-5637-4db3-8347-a60fb7dc85ba",
		"Cache-Control": "private",
		"WWW-Authenticate": "Bearer",
		"X-Powered-By": "ASP.NET;ASP.NET",
		"x-ms-client-request-id": "75ef606b-c38b-427a-ad98-9828a1bbee00",
		"x-ms-ratelimit-remaining-tenant-reads": "249",
		"x-ms-routing-request-id": "CANADACENTRAL:20251015T200004Z:44919aba-5637-4db3-8347-a60fb7dc85ba",
		"Strict-Transport-Security": "max-age=31536000; includeSubDomains",
		"X-Content-Type-Options": "nosniff",
		"X-Cache": "CONFIG_NOCACHE",
		"X-MSEdge-Ref": "Ref A: A680DA5F9FC04E20ABD178C3FDD50853 Ref B: YTO221090813025 Ref C: 2025-10-15T20:00:00Z",
		"Date": "Wed, 15 Oct 2025 20:00:04 GMT",
		"Content-Length": "130",
		"Content-Type": "application/json"
	},
	"effectiveIntegrationRuntime": "AutoResolveIntegrationRuntime (Canada Central)",
	"executionDuration": 4,
	"durationInQueue": {
		"integrationRuntimeQueue": 0
	},
	"billingReference": {
		"activityType": "ExternalActivity",
		"billableDuration": [
			{
				"meterType": "AzureIR",
				"duration": 0.016666666666666666,
				"unit": "Hours"
			}
		]
	}
}

It's not clear why the EA monthly pricesheet export step is getting an "Unauthorized" error, when there have been no changes in the FTK configuration.

🙋‍♀️ Ask for the community

We could use your help:

  1. Please vote this issue up (👍) to prioritize it.
  2. Leave comments to help us solidify the vision.

AErmie avatar Oct 15 '25 20:10 AErmie

Re-checking the FTK this morning, and it seems that it is now collecting data for the current month (October).

Image Image

If it is working correctly, we now have to figure out why it's failing to perform the backfill. Will report additional details once obtained.

AErmie avatar Oct 16 '25 16:10 AErmie

When attempting to run the Backfill process (via PowerShell), I encounter the following error about permissions? But it's not clear why since the data processing (since fixing the scopes) seem to be working. So doesn't that mean the permissions are correct?

PS C:\Users\adine> Start-FinopsCostExport -Scope '/providers/Microsoft.Billing/billingAccounts/1234567/billingProfiles/123456' -Name 'bcgov-mgd-lz-live-finops-hub-monthly-costdetails' -StartDate '2025-07-01' -EndDate '2025-07-31'
Write-Error: C:\Users\adine\Documents\PowerShell\Modules\FinOpsToolkit\0.11\Public\Get-FinOpsCostExport.ps1:133
Line |
 133 |  … $response = Invoke-Rest -Method GET -Uri $path -CommandName "Get-FinO …
     |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | No claims present for the caller in the system (Code: 401).
Start-FinOpsCostExport: Export bcgov-mgd-lz-live-finops-hub-monthly-costdetails not found. Did you specify the correct scope?
PS C:\Users\adine> Get-Module -ListAvailable FinOpsToolkit

    Directory: C:\Users\adine\Documents\PowerShell\Modules

ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Script     0.11                  FinOpsToolkit                       Desk      {Add-FinOpsHubScope, Add-FinOpsServicePrincipal, Deploy-FinOpsHub, Get-FinOpsCostExport…}

AErmie avatar Oct 16 '25 17:10 AErmie