microsoft
Datafactory pipeline run error - BilledCost Column Missing
đ Problem
I'm seeing this pipeline error when any data that is dropped into the storage account from the billing export of my EA billing account.
Operation on target Convert CSV failed: ErrorCode=MappingColumnNameNotFoundInSourceFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Column 'BilledCost' specified in column mapping cannot be found in 'finops2_b8741481-faa2-4308-85cf-816fbebd1f76.csv' source file.,Source=Microsoft.DataTransfer.ClientLibrary,'
đŖ Repro steps
I deployed the latest finops hub today in my azure tenant, I followed the instructions for adding the export configuration to the letter. No matter how I configure the export I get the same error.
These are the fields in the raw CSV file from the export: DepartmentName,AccountName,AccountOwnerId,SubscriptionGuid,SubscriptionName,ResourceGroup,ResourceLocation,AvailabilityZone,UsageDateTime,ProductName,MeterCategory,MeterSubcategory,MeterId,MeterName,MeterRegion,UnitOfMeasure,UsageQuantity,ResourceRate,PreTaxCost,CostCenter,ConsumedService,ResourceType,InstanceId,Tags,OfferId,AdditionalInfo,ServiceInfo1,ServiceInfo2,Currency
đ¤ Expected
Normally this would simply process the data. I have an older version of the finops hub deployed and it works just fine. I noticed the new finops hub deployment configures datafactory a bit differently. We are looking to move to the latest finops hub version but ran into this issue.
đˇ Screenshots
N/A
âšī¸ Additional context
The components deployed thus far are from the below link, start at the "How to use this template" section. I have not made any customizations to the process thus far. https://learn.microsoft.com/en-us/samples/azure/azure-quickstart-templates/finops-hub/
đââī¸ Ask for the community
Please let me know if I'm doing something wrong, what settings to double check, and if more information is needed.
I found my issue. This other guide has info around a new preview export config with FOCUS. This is required with the new hub. The first set of instructions didn't mention this from what i saw.
https://microsoft.github.io/finops-toolkit/hubs
I'm leaving this open to remember to update the troubleshooting guide with this error before the 0.4 release.
@usvjosh Do you recall where you saw instructions that didn't mention this? We should find and update those and also see if we can add some more debugging into the report itself.
@flanakin I think the issue I had was that the instructions didn't say to use the FOCUS data export. There are other instructions out there for the v0.3 finops hub that do explain this requirement.
https://microsoft.github.io/finops-toolkit/hubs
I'm closing this as we have updated the docs to be more clear on this requirement: https://microsoft.github.io/finops-toolkit/hubs#%EF%B8%8F-why-finops-hubs