Unable to interact with the same Model using different Boto sessions

[neilferreira]

The current method of authenticating using PynamoDB is to specify an access key, secret access key and session token (optionally) within the metadata of a table which is converted into a boto session and attached to a "connection" as per https://github.com/pynamodb/PynamoDB/blob/master/pynamodb/models.py#L1049

I have the following use case:

• The IAM Role of my EC2 instance is permitted to write data to my DynamoDB table.
• The IAM Role of the Cognito user who is authenticating to my API is allowed to read specific records (using row-level permissions) from my DynamoDB table.

I decided to create this:

def get_model_class(boto_session):
    """ Returns the MyModel class authenticated with the supplied boto session
    """
    cls = MyModel
    credentials = boto_session.get_credentials().get_frozen_credentials()
    cls.Meta.aws_access_key_id = credentials.access_key
    cls.Meta.aws_secret_access_key = credentials.secret_key
    cls.Meta.aws_session_token = credentials.token
    return cls

Using this, I could simply do:

foo = get_model_class(boto3.session.Session())

and I would have MyModal authenticated with my EC2 instance

or:

foo = get_model_class(boto3.session.Session(..credentials of my cognito user))

which should then have MyModal authenticated with my Cognito user

What I then realized, is that the FIRST time that the credentials are set on a model, these are then applied to all further usages of that model (as per https://github.com/pynamodb/PynamoDB/blob/master/pynamodb/models.py#L1049)

The workaround was this:

def get_model_class(boto_session):
    """ Returns the MyModel class authenticated with the supplied boto session
    """
    cls = MyModel
    # If a connection has already been made to this table, clear it so that we know we're going to
    # authenticate as the correct user.
    if cls._connection:
        cls._connection = None

    # This is the first time we're using this Table, use our session's credentials to make this call.
    credentials = boto_session.get_credentials().get_frozen_credentials()
    cls.Meta.aws_access_key_id = credentials.access_key
    cls.Meta.aws_secret_access_key = credentials.secret_key
    cls.Meta.aws_session_token = credentials.token
    return cls

Recommendation: Allow a boto session to be suppliedd which specifies which credentials would be used within the connection when interacting with DynamoDB.

[ikonst]

You're changing a global variable on the model. If your application is something like a web worker which handles multiple requests, perhaps concurrently, you might end up having one request using another's credentials.

Did you check whether doing a copy.deepcopy on the model class would work to create a unique class for the duration of the operation? ( I didn't try this -- it's a shot on the dark...)

I do agree we need a better way of doing this. I'd also like to add a way of providing custom settings with an operation (such as query, scan etc.) to be able to control headers, timeouts etc. on a per-operation basis.

[beanaroo]

Being able to instantiate a Model with supplied Meta properties might be helpful for various use cases.

• Different table_name depending on stage/region/account
• Using the same Model across regions and accounts
• Using different Models across regions and accounts

https://github.com/pynamodb/PynamoDB/issues/204 would benefit from being able to supply a prepared boto3 Session too.

[beanaroo]

For reference, I can confirm deepcopy does not work for the above approach:

>>> id(ModelDev.Meta)
94844977756880
>>> id(ModelProd.Meta)
94844977756880

As per this SO answer:

instead of making a deep copy of the class-definition object, deepcopy() merely returns a reference to the same class-definition object.

To get the result I'm after, I've tweaked the above as:

def get_model_for_stage(model, stage, boto_session):

    cls = type(model.__name__, (Model,), vars(model).copy())

    meta = vars(cls.Meta).copy()

    meta['table_name'] = f'{model.Meta.table_name}-{stage}'
    meta['region'] = boto_session.region_name

    credentials = boto_session.get_credentials().get_frozen_credentials()
    meta['aws_access_key_id'] = credentials.access_key
    meta['aws_secret_access_key'] = credentials.secret_key
    meta['aws_session_token'] = credentials.token

    cls.Meta = type('Meta', (Protocol,), meta)

    return cls

[ikonst]

Basically yes, we need to unbundle models, tables and table connections.

I suppose that for backward compatibility we'd want to keep supporting the "bundled" API we have today. Perhaps an "unbundled" API would have something like:

1. Getting, querying and scanning through a ModelTableConnection instance
2. Models hydrated this way would keep an instance reference to the ModelTableConnection so they could perform saves and updates on the correct table with the correct credentials.
3. Make it possible to define Meta-less tables, so one could make it clearly an error to query not through a ModelTableConnection

[weegolo]

For reference, I can confirm deepcopy does not work for the above approach:

>>> id(ModelDev.Meta)
94844977756880
>>> id(ModelProd.Meta)
94844977756880

As per this SO answer:

instead of making a deep copy of the class-definition object, deepcopy() merely returns a reference to the same class-definition object.

To get the result I'm after, I've tweaked the above as:

def get_model_for_stage(model, stage, boto_session):

    cls = type(model.__name__, (Model,), vars(model).copy())

    meta = vars(cls.Meta).copy()

    meta['table_name'] = f'{model.Meta.table_name}-{stage}'
    meta['region'] = boto_session.region_name

    credentials = boto_session.get_credentials().get_frozen_credentials()
    meta['aws_access_key_id'] = credentials.access_key
    meta['aws_secret_access_key'] = credentials.secret_key
    meta['aws_session_token'] = credentials.token

    cls.Meta = type('Meta', (Protocol,), meta)

    return cls

@beanaroo can I ask where you're importing Protocol from?