Support initial state in Cognito

[scottsteen]

Hello.

We use the Dockerized moto server heavily in our testing strategy. An issue we're struggling to solve well is getting initial state into the mocked services. Currently, we have these test concerns mixed in with our production code. In other words, if a test flag is set, call the Create Pool & Client APIs during application start up & use the ids from the responses.

What I really want is to initialise the external services separately & feed the ids into the application. This will keep our application agnostic of what environment it is running in.

I had some thoughts on how this could be done (not an exhaustive list, I'm sure).

1. Extend the create APIs to accept an ID. This would allow us to provide a fixed id to both the set up script & the application. From reading other issues, I don't think this fits with your ethos of remaining as close to the AWS APIs as possible.
2. Build a moto specific API that allows us to create the data we need, with the values we need.
3. Provide the required data in file format that is slurped on server start up. Not sure how this would work for non-Docker environments, but I'd love to simply mount a volume containing my initial data into moto.

My preference from this list is option 3. It could even be extended to other services, for instance creating S3 buckets.

If there's an existing solution to this that I'm unaware, I am most defnitely all ears!

Thanks.

[bblommers]

Hi @scottsteen! Option 3 would definitely be the best approach to this. There is an outstanding feature request to implement an import/export functionality in Moto (#2882) - this sounds like a natural extension to that, to automatically import any data found in a specific directory.

A workaround until this is implemented (and you might have already considered this): If you have IoC, e.g. Terraform, you could create the infrastructure in Moto as an initial step in the testing pipeline. Terraform for sure has an option to point to a different endpoint URL that acts as an AWS backend (i.e., Moto) - other IOC-providers might have a similar solution.

[scottsteen]

Thanks for the quick response. Glad to hear it's not an outlandish request!

Sadly my python skills are slim to none, so not sure I'll be of much help in the PR department😅

[bblommers]

From Moto 3.1.14, it is possible to have a reproducible UserPoolID. This can be achieved by setting the environment variable MOTO_COGNITO_IDP_USER_POOL_ID_STRATEGY=HASH. See the documentation here: http://docs.getmoto.org/en/latest/docs/services/cognito-idp.html

Setting this variable will guarantee that, given a set of input parameters, the generated userpool ID will be always the same.

The scope of this feature request is much wider than just having a reproducible ID, but hopefully it can help separating the test concerns out of production code.

[scottsteen]

That's a great addition, I'll give it a bash.

In case anyone comes across this in the meantime, the implementation we've gone for (using docker-compose) is similar to your suggestion of using IaC. We didn't want the overhead of intialising Terraform, so we went with another service that simply makes aws cli calls to Moto. Here's a cut down example:

version: "3.7"
services:
  app:
    build: <our app>
    environment:
      - user_pool_config=file:/aws/configuration/cognito.properties
    volumes:
      - aws:/aws/configuration
    depends_on: 
      - user-pool

  user-pool: 
    image: amazon/aws-cli:${AWS_CLI_VERSION}
    environment:
      AWS_RETRY_MODE: standard
      AWS_MAX_ATTEMPTS: 10
      AWS_ACCESS_KEY_ID: ACCESS_KEY
      AWS_SECRET_ACCESS_KEY: SECRET_KEY
      AWS_DEFAULT_REGION: eu-west-2
      AWS_URL: http://aws:5000
      POOL_ID_PROPERTY: aws.cognito.user-pool-id
      CLIENT_ID_PROPERTY: aws.cognito.app.client-id
    entrypoint: /scripts/configure-user-pool.sh
    depends_on:
      - aws
    volumes:
      - ./cognito:/scripts
      - aws:/output
    
  aws:
    image: motoserver/moto:${MOTO_VERSION}
    expose:
      - "5000"

volumes:
  aws:

With the user-pool entry point script ./cognito/configure-user-pool.sh

#!/bin/sh

USER_POOL_ID=$(aws cognito-idp \
                   create-user-pool \
                   --pool-name 'my-cool-user-pool' \
                   --endpoint-url "$AWS_URL" \
                   --output text \
                   --query 'UserPool.Id')

CLIENT_ID=$(aws cognito-idp \
                create-user-pool-client \
                --user-pool-id "$USER_POOL_ID" \
                --client-name 'my-cool-client' \
                --endpoint-url "$AWS_URL" \
                --output text \
                --query 'UserPoolClient.ClientId')

cat <<EOF > /output/cognito.properties
$POOL_ID_PROPERTY=$USER_POOL_ID
$CLIENT_ID_PROPERTY=$CLIENT_ID
EOF

The user-pool service uses the aws cli to create the pool & client and writes the IDs from the responses to a properties file mounted as a named volume, sharing the created file between services. Our application, with this volume mounted, then understands how to read this properties file & configures itself with the values.

[bblommers]

@scottsteen

Moto 4.0.6 (just released) has the option to record requests, and replay them on another system/at another time, to ensure that the initial set of resources is the same. It is also possible to seed Moto now, which means that the identifiers are deterministic given a specific seed.

See the documentation for this here: http://docs.getmoto.org/en/latest/docs/configuration/recorder/index.html

We have a test case specifically to verify that a Cognito User Pool has a deterministic ID, that may be useful get an idea of how this works for your usecase: https://github.com/spulec/moto/blob/master/tests/test_cognitoidp/test_cognitoidp_replay.py

Note that the solution is slightly different from your suggested option 3, as it still requires you to make some requests after the server has started to load the data. I do believe it will solve your problem though.

I'll close this, but do let us know if there's any questions or feedback on this feature.