Calling deployed flows within another flow

[jonlwowski012]

From an architectural standpoint, I have an "orchestration" flow that is calling multiple other flows and then combining their results for external use. The orchestration flow is calling previous deployed flows (AWS Step Functions) that are their own ML models. The current way I am calling these flows is using boto3 directly.

            sfn = boto3.client("stepfunctions")
            sfn_response = sfn.start_execution(stateMachineArn=arn, name=f"{job_name}_{uuid.uuid4()}", input=json.dumps(inputs))

            exec_arn = sfn_response["executionArn"]
            status = "STARTING"

            while status not in FINAL_STATUSES:
                time.sleep(1.0)
                response = sfn.describe_execution(executionArn=exec_arn)
                status = response["status"]

However when doing this, I eventually hit a throttling error because AWS has a limit on how often you can get the status of step functions.

Is there a better way to do this?

[savingoyal]

Can you poll more infrequently?

[jonlwowski012]

Can you poll more infrequently?

The problem is that we could potentially have thousands of jobs running in parallel.

[savingoyal]

this use case is now well supported using our runner and deployer apis