"Failed to retrieve regulatory compliance standards" when run against Azure

[djcater]

Describe the bug When I run ScoutSuite 5.10.0 against an Azure environment, I get the following errors in the output (added --debug for more info but the main ERROR lines are there without it):

2020-10-06 21:49:29 ubuntu scout[49736] INFO Launching Scout
2020-10-06 21:49:29 ubuntu scout[49736] INFO Authenticating to cloud provider
2020-10-06 21:49:29 ubuntu scout[49736] INFO To authenticate to the Resource Manager API, use a web browser to access https://microsoft.com/devicelogin and enter the XXXXXX code.
2020-10-06 21:54:02 ubuntu scout[49736] INFO To authenticate to the Azure AD Graph API, use a web browser to access https://microsoft.com/devicelogin and enter the XXXXXX code.
2020-10-06 21:54:15 ubuntu scout[49736] INFO Running against 2 subscriptions
2020-10-06 21:54:15 ubuntu scout[49736] INFO Gathering data from APIs
2020-10-06 21:54:15 ubuntu scout[49736] DEBUG Skipping the Azure Active Directory service
2020-10-06 21:54:15 ubuntu scout[49736] DEBUG Skipping the Azure RBAC service
2020-10-06 21:54:15 ubuntu scout[49736] DEBUG Skipping the SQL Database service
2020-10-06 21:54:15 ubuntu scout[49736] DEBUG Skipping the Storage Accounts service
2020-10-06 21:54:15 ubuntu scout[49736] DEBUG Skipping the Key Vault service
2020-10-06 21:54:15 ubuntu scout[49736] DEBUG Skipping the Network service
2020-10-06 21:54:15 ubuntu scout[49736] DEBUG Skipping the Virtual Machines service
2020-10-06 21:54:15 ubuntu scout[49736] DEBUG Skipping the App Services service
2020-10-06 21:54:15 ubuntu scout[49736] INFO Fetching resources for the Security Center service
2020-10-06 21:54:15 ubuntu scout[49736] ERROR securitycenter.py L98: Failed to retrieve regulatory compliance standards: 400 Client Error: Bad Request for url: https://management.azure.com/subscriptions/xxx-xxx-xxx/providers/Microsoft.Security/regulatoryComplianceStandards?api-version=2019-01-01-preview
Traceback (most recent call last):
  File "/home/xxx/ScoutSuite/ScoutSuite/providers/azure/facade/securitycenter.py", line 98, in get_regulatory_compliance_results
    compliance_standards = await run_concurrently(
  File "/home/xxx/ScoutSuite/ScoutSuite/providers/utils.py", line 25, in run_concurrently
    return await run_function_concurrently(function)
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/xxx/ScoutSuite/ScoutSuite/providers/azure/facade/securitycenter.py", line 99, in <lambda>
    lambda: list(client.regulatory_compliance_standards.list())
  File "/home/xxx/ScoutSuite/venv/lib/python3.8/site-packages/msrest/paging.py", line 143, in __next__
    self.advance_page()
  File "/home/xxx/ScoutSuite/venv/lib/python3.8/site-packages/msrest/paging.py", line 129, in advance_page
    self._response = self._get_next(self.next_link)
  File "/home/xxx/ScoutSuite/venv/lib/python3.8/site-packages/azure/mgmt/security/operations/_regulatory_compliance_standards_operations.py", line 99, in internal_paging
    raise exp
msrestazure.azure_exceptions.CloudError: 400 Client Error: Bad Request for url: https://management.azure.com/subscriptions/xxx-xxx-xxx/providers/Microsoft.Security/regulatoryComplianceStandards?api-version=2019-01-01-preview
2020-10-06 21:54:16 ubuntu scout[49736] ERROR securitycenter.py L98: Failed to retrieve regulatory compliance standards: 400 Client Error: Bad Request for url: https://management.azure.com/subscriptions/xxx-xxx-xxx/providers/Microsoft.Security/regulatoryComplianceStandards?api-version=2019-01-01-preview
Traceback (most recent call last):
  File "/home/xxx/ScoutSuite/ScoutSuite/providers/azure/facade/securitycenter.py", line 98, in get_regulatory_compliance_results
    compliance_standards = await run_concurrently(
  File "/home/xxx/ScoutSuite/ScoutSuite/providers/utils.py", line 25, in run_concurrently
    return await run_function_concurrently(function)
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/xxx/ScoutSuite/ScoutSuite/providers/azure/facade/securitycenter.py", line 99, in <lambda>
    lambda: list(client.regulatory_compliance_standards.list())
  File "/home/xxx/ScoutSuite/venv/lib/python3.8/site-packages/msrest/paging.py", line 143, in __next__
    self.advance_page()
  File "/home/xxx/ScoutSuite/venv/lib/python3.8/site-packages/msrest/paging.py", line 129, in advance_page
    self._response = self._get_next(self.next_link)
  File "/home/xxx/ScoutSuite/venv/lib/python3.8/site-packages/azure/mgmt/security/operations/_regulatory_compliance_standards_operations.py", line 99, in internal_paging
    raise exp
msrestazure.azure_exceptions.CloudError: 400 Client Error: Bad Request for url: https://management.azure.com/subscriptions/xxx-xxx-xxx/providers/Microsoft.Security/regulatoryComplianceStandards?api-version=2019-01-01-preview
2020-10-06 21:54:17 ubuntu scout[49736] INFO Running pre-processing engine
2020-10-06 21:54:17 ubuntu scout[49736] INFO Running rule engine

The process continues and appears to finish OK, so I can't immediately tell what would be different if this request hadn't failed, but I'm guessing there's a particular check that hasn't happened.

The exceptions JS file is empty:

exceptions =
{}

To Reproduce I ran each service in turn and narrowed it down the securitycenter service:

python scout.py azure --no-browser --report-dir results --timestamp --all-subscriptions --user-account-browser --tenant xxx-xxx-xxx --services securitycenter --debug

Additional context This is running ScoutSuite 5.10.0, set up as follows:

$ virtualenv -p python3 venv
$ source venv/bin/activate
$ pip install -r requirements.txt

Python version is 3.8.2 on Ubuntu 20.04.

Hope that helps. Thanks.

[x4v13r64]

Thanks for raising this and all the details, will review.

[x4v13r64]

I've been unable to reproduce. Could you please try branch https://github.com/nccgroup/ScoutSuite/tree/release/5.10.2 and update the requirements as per https://github.com/nccgroup/ScoutSuite/blob/release/5.10.2/requirements.txt? This will most likely resolve the issue.

Closing for now, please reopen if you still encounter the issue after doing the above.

[me0wday]

I'd like to re-open this as I am experiencing the same issue on the current 5.10.2 version. Same command as above. Fresh install.

Traceback (most recent call last):
  File "/root/scout/venv/lib/python3.8/site-packages/ScoutSuite/providers/azure/facade/securitycenter.py", line 98, in get_regulatory_compliance_results
    compliance_standards = await run_concurrently(
  File "/root/scout/venv/lib/python3.8/site-packages/ScoutSuite/providers/utils.py", line 25, in run_concurrently
    return await run_function_concurrently(function)
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/root/scout/venv/lib/python3.8/site-packages/ScoutSuite/providers/azure/facade/securitycenter.py", line 99, in <lambda>
    lambda: list(client.regulatory_compliance_standards.list())
  File "/root/scout/venv/lib/python3.8/site-packages/msrest/paging.py", line 143, in __next__
    self.advance_page()
  File "/root/scout/venv/lib/python3.8/site-packages/msrest/paging.py", line 129, in advance_page
    self._response = self._get_next(self.next_link)
  File "/root/scout/venv/lib/python3.8/site-packages/azure/mgmt/security/operations/_regulatory_compliance_standards_operations.py", line 99, in internal_paging
    raise exp
msrestazure.azure_exceptions.CloudError: 400 Client Error: Bad Request for url: https://management.azure.com/subscriptions/XXX-XXX-XXX/providers/Microsoft.Security/regulatoryComplianceStandards?api-version=2019-01-01-preview```