Bug report: flow list times out on big environments

[ved-leahcim]

Priority

(Medium) I'm annoyed but I'll live

Description

I want to create a report about all the flows in the company. When I use the following code it works for an environment with a few flows, but we also have environments with multiple thousands of flow, and there ich receive the following error:

$smallEnvFlows = m365 flow list --environmentName $smallEnv --asAdmin | ConvertFrom-Json
# This works (environment has less than 100 flows)

$bigEnvFlows = m365 flow list --environmentName $bigEnv --asAdmin | ConvertFrom-Json

# Returns the following error:
Error: The gateway did not receive a response from 'Microsoft.ProcessSimple' within the specified time period.

Is there a way around this error or can this be fixed? I'am using v7.1.0.

Steps to reproduce

m365 login

$bigEnvId = "c2548128-8c94-ffff-8c04-214b44309982"
$bigEnvFlows = m365 flow list --environmentName $bigEnvId --asAdmin | ConvertFrom-Json

Expected results

Expecting an enumeration of flows of the big environment.

Actual results

Error: The gateway did not receive a response from 'Microsoft.ProcessSimple' within the specified time period.

Diagnostics

No response

CLI for Microsoft 365 version

7.1.0

nodejs version

18.17.1

Operating system (environment)

macOS

Shell

PowerShell

cli doctor

{ "os": { "platform": "darwin", "version": "Darwin Kernel Version 23.0.0: Fri Sep 15 14:41:34 PDT 2023; root:xnu-10002.1.13~1/RELEASE_ARM64_T8103", "release": "23.0.0" }, "cliVersion": "7.1.0", "nodeVersion": "v18.17.1", "cliAadAppId": "31359c7f-bd7e-475c-86db-fdb8c937548e", "cliAadAppTenant": "common", "authMode": "DeviceCode", "cliEnvironment": "", "cliConfig": {}, "roles": [], "scopes": { "https://graph.microsoft.com": [ "AllSites.FullControl", "AppCatalog.ReadWrite.All", "AuditLog.Read.All", "ChannelMember.ReadWrite.All", "ChannelMessage.Read.All", "ChannelMessage.Send", "ChannelSettings.ReadWrite.All", "Chat.Read", "Chat.ReadWrite", "Directory.AccessAsUser.All", "Directory.ReadWrite.All", "Group.ReadWrite.All", "IdentityProvider.ReadWrite.All", "Mail.ReadWrite", "Mail.Send", "Place.Read.All", "Policy.Read.All", "Reports.Read.All", "SecurityEvents.Read.All", "ServiceHealth.Read.All", "ServiceMessage.Read.All", "ServiceMessageViewpoint.Write", "Tasks.ReadWrite", "Team.Create", "TeamMember.ReadWrite.All", "TeamsApp.ReadWrite.All", "TeamsAppInstallation.ReadWriteForUser", "TeamSettings.ReadWrite.All", "TeamsTab.ReadWrite.All", "TermStore.ReadWrite.All", "User.Invite.All", "User.ReadWrite.All", "profile", "openid", "email" ], "https://service.powerapps.com/": [ "User" ], "https://management.azure.com": [ "user_impersonation" ] } }

Additional Info

No response

[waldekmastykarz]

Thanks for reporting this @ved-leachim, and sorry you're running into it. Am I understanding this correctly that we can repro this issue on an environment with more than 100 flows?

[ved-leahcim]

Hey @waldekmastykarz thank for your reply.

I am unsure if 100 flows are enough, as mentioned we have an environment with multiple thousands of flows.

[waldekmastykarz]

Understood. Let us try and repro it, and we'll get back to you when we have a solution. We appreciate your help and taking the time to write it up 😊

[waldekmastykarz]

@ved-leachim would you be willing to run the following script so that we can test if setting the page size to 50 items would solve it or if we should decrease it further?

at=$(m365 util accesstoken get -r https://management.azure.com)
curl -H "authorization: Bearer $at" https://management.azure.com/providers/Microsoft.ProcessSimple/scopes/admin/environments/c2548128-8c94-ffff-8c04-214b44309982/flows\?api-version\=2016-11-01\&$top\=50

[ved-leahcim]

Hi @waldekmastykarz, thanks for your reply. Since I am not so strong with bash or zsh, I transformed your commands, to what I think should work, in powershell.

$at = $(m365 util accesstoken get -r https://management.azure.com)
$headers = @{ Authorization = "Bearer $at" }
$response = Invoke-RestMethod -ContentType "application/json" -URI "https://management.azure.com/providers/Microsoft.ProcessSimple/scopes/admin/environments/c2548128-8c94-ffff-8c04-214b44309982/flows/?api-version=2016-11-01/&`$top/=50" -Method "GET" -Headers $headers -UseBasicParsing

I receive what looks like a valid access token but when I use it in the last command I receive the following error:

{
  "error": {
    "code": "InvalidAuthenticationToken",
    "message": "The access token is invalid."
  }
}

According ms-docs this means the token is like not complete, which shouldn't be the case here. invalid-authentication-token

[waldekmastykarz]

That could be my bad. Could you please add -o text to the accesstoken get command? If your default output is set to JSON, then you likely get other characters which would be considered invalid.

[martinlingstuyl]

@ved-leachim would you be willing to run the following script so that we can test if setting the page size to 50 items would solve it or if we should decrease it further?

at=$(m365 util accesstoken get -r https://management.azure.com)

curl -H "authorization: Bearer $at" https://management.azure.com/providers/Microsoft.ProcessSimple/scopes/admin/environments/c2548128-8c94-ffff-8c04-214b44309982/flows\?api-version\=2016-11-01\&$top\=50

Using m365 request would be easy as well 😂

[martinlingstuyl]

So @ved-leachim, you can just run the following in powershell:

m365 request --url 'https://management.azure.com/providers/Microsoft.ProcessSimple/scopes/admin/environments/c2548128-8c94-ffff-8c04-214b44309982/flows?api-version=2016-11-01&$top=50'

it will get the token and execute the call.

[ved-leahcim]

Thank you for your reply and the command. I have been working with 'm365 login (Global Administrator Account)'

It apparently uses the PnP Management Shell Enterprise Application. Nevertheless I get the following error message:

'Error: Request failed with status code 403'

Does the app require additional API permission for this call?

[martinlingstuyl]

Hi @ved-leachim,

I forgot to say there's a guid in there that probably belongs to my tenant. You'll need to find out what guid your own tenant has. I suggest you run m365 flow list with the --debug flag to see the request url on your end so that you can correct it.

Does that make sense?

[ved-leahcim]

Hi @martinlingstuyl, upon reflection, I recognized that I had included the ID in my initial request. However, I overlooked the fact that I modified it for the public version of the request. Leveraging an effective ID is indeed beneficial.

Thank you!

[martinlingstuyl]

Are you sure though that this issue needs to be closed? We may need to look into the paging part still...

[waldekmastykarz]

@ved-leachim did you have a chance to look if the request that we proposed solves the issue you experienced originally? We'd appreciate your help.

[ved-leahcim]

@waldekmastykarz yeah, I tested the API and it works well. I can request 250 items per page, and after that I can use the nextLink link to get the next 250.

[waldekmastykarz]

Perfect! Thanks for confirming @ved-leachim