Possible issue running within docker compose?

[tonicebrian]

trafficstars

I'm using your docker image within a larger docker compose mainly to test Airflow pipelines. The fake s3 service definition is pretty simple:

   s3proxy: 
        image: lphoward/fake-s3
        ports:
            - "4569:4569"

The problem is that Airflow fails when trying to list contents of a bucket with Exception:

botocore.errorfactory.NoSuchKey:
 An error occurred (NoSuchKey) when calling the ListObjectsV2 operation: 
 The specified key does not exist

Digging a bit I saw that fake s3 was returning 404 on airflow calls, so I logged into one of the airflow workers and then issued:

$> curl 's3proxy:4569/mybucket?list-type=2&prefix=&delimiter='
<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>NoSuchKey</Code>
  <Message>The specified key does not exist</Message>
  <Key>coin-data</Key>
  <RequestId>1</RequestId>
  <HostId>2</HostId>
</Error>

it returns a 404, while if I do the same request this time from the host container using the exposed port it returns a 200 as expected

$> curl 'localhost:4569/mybucket?list-type=2&prefix=&delimiter='

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Name>mybucket</Name>
  <Prefix/>
  <Marker/>
  <MaxKeys>1000</MaxKeys>
  <IsTruncated>false</IsTruncated>
  <CommonPrefixes>
    <Prefix>h</Prefix>
  </CommonPrefixes>
</ListBucketResult>

Any idea why the different behaviour when querying from the outside and when running from same docker-compose.yml setup?

[tonicebrian]

After some digging I think it is related to https://github.com/jubos/fake-s3/issues/114 . Still looking for a solution in my docker-compose.yml

[waltermblair]

@tonicebrian did you ever find a docker-compose solution for this problem?

[tonicebrian]

No, I switched to https://github.com/findify/s3mock . Make sure you include some patches that aren't still merged https://github.com/findify/s3mock/pulls/tonicebrian into master.

[waltermblair]

I realized just now that naming service s3.localhost seems to work