AWS Client Parameters are stored as a dictionary and throw an error when assigned to an S3 bucket through the UI

[mthatt]

First check

• [X] I added a descriptive title to this issue.
• [X] I used the GitHub search to find a similar issue and didn't find it.
• [X] I searched the Prefect documentation for this issue.
• [X] I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

First I create an S3 block and an AWS Credentials block from the SDK. If I assign the credentials block to the S3 bucket in the UI, the Client Parameters are incorrectly stored as a dictionary. This leads to an error any time I try to use the S3 bucket I created.

I can see from the output of my loaded S3 bucket loaded_s3 = S3Bucket.load("s3-from-code") that aws_client_parameters is stored as a dictionary rather than an instance of AwsClientParameters as it should be:

AwsCredentials(region_name=None, profile_name=None, aws_access_key_id=None, aws_session_token=None, **aws_client_parameters={'config': None, 'verify': True, 'use_ssl': True, 'api_version': None, 'endpoint_url': None, 'verify_cert_path': None}**, aws_secret_access_key=None)

Which leads to TypeError: unhashable type: 'dict' whenever I try to work with the S3 bucket.

Reproduction

1. Create an S3 block from the SDK:

block = S3Bucket(bucket_name="se-demo-flow-code-store", bucket_path="se-demo-flow-code-store/weather/")
block.save(name="s3-from-code", overwrite=True)`

2. Create an AWS Credentials block from the SDK:

AwsCredentials(
    aws_access_key_id="access_key",
    aws_secret_access_key="secret_key",
    aws_session_token="session_token",
    region_name="us-east-2"
).save("aws-credentials-from-code", overwrite=True)`

3. Assign the credentials block to S3 in the UI

4. Load in S3 Bucket:

loaded_s3 = S3Bucket.load("s3-from-code")`

5. Try to copy an object to the S3 bucket:

loaded_s3.copy_object(
    "my_folder/notes.txt",
    "my_folder/notes_copy.txt"
)`

Error

Traceback (most recent call last):
  File "/Users/mihir/Documents/Code/create_aws_creds.py", line 18, in <module>
    loaded_s3.copy_object(
  File "/Users/mihir/anaconda3/envs/updated-prefect/lib/python3.11/site-packages/prefect/events/instrument.py", line 73, in inner
    raise exc
  File "/Users/mihir/anaconda3/envs/updated-prefect/lib/python3.11/site-packages/prefect/events/instrument.py", line 70, in inner
    return function(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mihir/anaconda3/envs/updated-prefect/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 259, in coroutine_wrapper
    return call()
           ^^^^^^
  File "/Users/mihir/anaconda3/envs/updated-prefect/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 432, in __call__
    return self.result()
           ^^^^^^^^^^^^^
  File "/Users/mihir/anaconda3/envs/updated-prefect/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 318, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mihir/anaconda3/envs/updated-prefect/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 179, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/mihir/anaconda3/envs/updated-prefect/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/Users/mihir/anaconda3/envs/updated-prefect/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 389, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "/Users/mihir/anaconda3/envs/updated-prefect/lib/python3.11/site-packages/prefect_aws/s3.py", line 1236, in copy_object
    s3_client = self.credentials.get_s3_client()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mihir/anaconda3/envs/updated-prefect/lib/python3.11/site-packages/prefect_aws/credentials.py", line 180, in get_s3_client
    return self.get_client(client_type=ClientType.S3)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mihir/anaconda3/envs/updated-prefect/lib/python3.11/site-packages/prefect_aws/credentials.py", line 171, in get_client
    return _get_client_cached(ctx=self, client_type=client_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mihir/anaconda3/envs/updated-prefect/lib/python3.11/site-packages/prefect_aws/credentials.py", line 122, in __hash__
    hash(self.aws_client_parameters),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: unhashable type: 'dict'

Versions

Version:             2.16.9
API version:         0.8.4
Python version:      3.11.8
Git commit:          083def52
Built:               Thu, Apr 4, 2024 3:11 PM
OS/Arch:             darwin/arm64
Profile:             default
Server type:         cloud

Additional context

No response

[paulusaptus]

I ran into the same problem (v 2.19.0). Downgrading to v2.16.8 solved the issue.

[extdponomarchuk]

I faced the same issue when changed the persistence result configuration from a S3 Block to a S3Bucket Block: prefect config set PREFECT_DEFAULT_RESULT_STORAGE_BLOCK=s3-bucket/my-bucket

  File "/home/root/ve/lib/python3.11/site-packages/prefect_aws/credentials.py", line 121, in __hash__
    hash(self.aws_client_parameters),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: unhashable type: 'dict'

[Rupin817]

Any update on this issues?

[bunchesofdonald]

Hi @Rupin817. This is fixed in our 0.5.0rc, we're currently working on porting that fix to <0.5.0, you can follow that work over here: https://github.com/PrefectHQ/prefect/issues/14147

[Rupin817]

@bunchesofdonald Hi, 0.5.0rc is only compatible with prefect 3 right? However it seems prefect 3 has compatible problem with your cloud, so it would be great to make this working for prefect 2.

[bunchesofdonald]

@Rupin817, right the RC is only compatible with Prefect 3. It shouldn't have compatibility issues with Cloud, but if you're running into problems, please open a GitHub issue.

As far as this issue, #14147 is to port the change we made to <0.5.0 so it's prefect 2 compatible, and that should be available in the next release.

However, the problem outlined in this issue is about the fact that the code creating the S3Bucket was doing overwrite=True which was wiping out the credentials saved via the UI. After this fix this will still be true, but the exception will be more helpful calling out that there are no credentials rather than this opaque unhashable type error.

Could you share the issue you're having? Maybe there's a workaround or another issue here that's unknown.

[Rupin817]

@bunchesofdonald I have reverted to 2.16.8 for our production.

I did read all details regarding the issue, I think the solution was sufficient. My case is more about we try to upgrade rather than downgrade at the beginning for the prefect aws package for this issue, and thus need upgrade to prefect 3.0, however apparently your prefect 3.0 docker image has compatibility issues with you cloud. Particularly, normally any upgrade of docker image or package will not impact any existing deployment, everything runs fine. The prefect 3.0 after update completely broke our all existing deployment - "This may be caused by attempting to run an image with a misspecified platform or architecture. " - I have discussed with Dylan on your slack, it seems we need redo all provisioning, which is not ideal and more work...

[bunchesofdonald]

I have discussed with Dylan on your slack, it seems we need redo all provisioning, which is not ideal and more work...

I'm sorry to hear that. I have this issue on my list for today, I'll see if I can move the release up a bit and get this out the door today/tomorrow so you're unblocked.