optscale icon indicating copy to clipboard operation
optscale copied to clipboard

Recommendations intermittent representation

Open SteveChurchillSWO opened this issue 2 months ago • 1 comments

Raising this issue on behalf of the SWO Customer Success team and the report below is taken from bug MPT-15371:

🧩 Problem Statement

It is being reported and observed, Possible Monthly Savings values are intermittently shown on FinOps for Cloud.

🔁 Steps Taken to Reproduce / Investigate the Issue

FinOps UI Check

  • Obtained User Role access
  • Navigate to Recommendations

Image

  • Navigated to Data Sources

    • Summary indicates values, I’d believe you’d expect to see emulated or represented in part or whole, in Recommendations

Image

  • By comparison the same validation was completed via local device (in region) and API responses were observed to be extended

**The original bug contains har files which appear unsupported so can not be attached.

FinOps Events

With Debug enabled, there are a notable number of events containing reference to;

  • ‘max retries exceeded’
  • Internal server error
Image

AppInsights

ContainerLogV2 | where LogMessage contains "9213a469-b73b-47ce-8442-c5631634eaf8" and LogLevel contains "error"

Using the above KQL there are a large number of Errors today along - with closer inspection many reference ‘recommendations’

ContainerLogV2 | where LogMessage contains "9213a469-b73b-47ce-8442-c5631634eaf8" and LogMessage contains "recommendations"

The majority of events are LogLevel Error with only intermittent ‘info’ events suggesting there is a platform level concern related to recommendations processes.

Expected Results

The Client and Marketplace CSM reported recommendations were present, then they have disappeared (no longer represented). With this in mind, it is expectation recommendations savings should be present

Actual Result

Recommendations Savings is suspiciously clean with $0 shown on all resources and in summary UI

📉 Issue Impact

Reportedly by MPT CSM, the current behaviour of FFC Recommendations is prohibiting further commercial engagement by the client.

Image

SteveChurchillSWO avatar Nov 13 '25 15:11 SteveChurchillSWO

I've seen similar behaviour in our automated tests where the possible savings card disappears from the Resources page, even though it is present on the recommendations page. Our automated tests fail due to the missing card, but will pass the next day when the card returns.

SteveChurchillSWO avatar Nov 13 '25 15:11 SteveChurchillSWO

We also find in bumiworker logs:

2025-11-19T11:16:58.609839162Z Error while processing security_optimization_notification (service) for organization 48509508-9338-4bc7-997e-44320d39ccfa - Empty module name
2025-11-19T11:16:58.609860436Z Traceback (most recent call last):
2025-11-19T11:16:58.609864380Z   File "/usr/src/app/bumiworker/bumiworker/tasks.py", line 507, in _execute
2025-11-19T11:16:58.609867306Z     data, options, error = call_module(
2025-11-19T11:16:58.609870911Z                            ^^^^^^^^^^^^
2025-11-19T11:16:58.609874635Z   File "/usr/src/app/bumiworker/bumiworker/modules/module.py", line 32, in call_module
2025-11-19T11:16:58.609878930Z     module = import_module(module_name, module_type)
2025-11-19T11:16:58.609881735Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-11-19T11:16:58.609884705Z   File "/usr/src/app/bumiworker/bumiworker/modules/module.py", line 26, in import_module
2025-11-19T11:16:58.609887862Z     return __import__('%s.%s.%s' % (import_base, module_type, module_name),
2025-11-19T11:16:58.609935599Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-11-19T11:16:58.609960738Z ValueError: Empty module name

ffaraone avatar Nov 19 '25 12:11 ffaraone

When no optimization is reported, the query:

db.checklists.aggregate(
	{
		'$match': {
		        $and': [
				{'organization_id': {'$eq': '9213a469-b73b-47ce-8442-c5631634eaf8'}},
				{'created_at': {'$eq': 1763614313}},
			]
		}
	}
)

doesn't return results (created_at should match last_completed of the checklist object, am I right).

ffaraone avatar Nov 20 '25 11:11 ffaraone

Hello @ffaraone @SteveChurchillSWO

We will investigate this issue. As soon as we have any conclusions, we will let you know immediately.

dsup-hystax avatar Nov 20 '25 15:11 dsup-hystax

Hi @SteveChurchillSWO , @ffaraone, fix merged into integration. Commit: c317e0e329760ceeda88c1599d185fc35004bbbe

stanfra avatar Nov 21 '25 11:11 stanfra