Start Separate Container for Long Running, Sensitive Jobs

[rajbala]

I am using FastAPI and Python for an application with Copilot. One of the API routes looks like this:

@v1.post('/build')
async def build():

    # I want the following code to run in it's own container
    repo = clone_repo()
    build_npm(repo)

    return {"build_started" : "true"}

How do I start a separate container, starting with the clone_repo() function? I want there to be some isolation of this running process for performance and security reasons, but not quite sure which approach I should use with Copilot.

Perhaps there's an ideal usage pattern with Copilot that I am missing?

[efekarakus]

Hi @rajbala !

Ohh this is very interesting 🤔 I think there are few different options:

1. StateMachine Addon. Here is an existing user defining a StateMachine addon: https://github.com/aws/copilot-cli/issues/1507#issuecomment-710506833 that can be triggered programmatically when a request is received on POST /build. The general steps to follow here are:

   1. Move the current build implementation under a separate function that can be invoked based on a command override
   2. Define a StateMachine that will use the existing service TaskDefinition to run a task.
   3. On POST /build, execute the StateMachine.

2. Invoking a Copilot Scheduled Job. You can define a Scheduled Job for build with on.schedule: 'none' so that it's never triggered. When your service receives a request at POST /build, you can then invoke the scheduled job. The invocation can be done with the AWS SDK:

   import boto3
   client = boto3.client('stepfunctions')
   response = client.start_execution(
       stateMachineArn= "<the arn generated by the scheduled job>",
   )
   

3. Invoking RunTask directly. Alternatively, if you don't need any of the state machine's retry and timeout functionality, you can invoke ecs.RunTask directly. The ARN of the task definition should be deterministic based on the name of the service.

Hope these help!

[rajbala]

I thought that a Schedule Job might be the right approach, but I didn't see how to invoke the job programmatically.

Really appreciate the pointer to invoke it using start_execution(). :)

[rajbala]

@efekarakus One other question if possible: it appears that I can include an input parameter when calling start_execution() as follows:

response = client.start_execution(
    stateMachineArn='string',
    name='string',
    input='string',
    traceHeader='string'
)

How do I read that input parameter from within the Copilot Job?

[github-actions[bot]]

This issue is stale because it has been open 60 days with no response activity. Remove the stale label, add a comment, or this will be closed in 14 days.

[github-actions[bot]]

This issue is closed due to inactivity. Feel free to reopen the issue if you have any further questions!