S3 backend, AWS lib failing

[rs-garrick]

Playing with privatebin for my first time. I'd like to use the S3 backend. I'm trying both putting my access key in conf.php (not desired) and passing in my .aws directory. Ultimately, I'd want this to work with supplied IAM role.

docker run -p 8080:8080 -v $HOME/.aws:/.aws:ro -v $PWD/cfg:/srv/cfg:ro privatebin/s3

$ cat cfg/conf.php 
[main]
[model]
class = S3Storage                    
[model_options]  
region = "us-east-1"
version = "latest"  
bucket = "rs-privatebin"
accesskey = "AKIA....KGO"
secretkey = "fjTE....G4x/"

2022/12/15 19:22:56 [error] 21#21: *6 FastCGI sent in stderr: "PHP message: PHP Deprecated:  Return type of Aws\HandlerList::count() should either be compatible with Countable::count(): int, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /srv/vendor/aws/aws-sdk-php/src/HandlerList.php on line 284PHP message: PHP Deprecated:  Return type of Aws\Api\AbstractModel::offsetExists($offset) should either be compatible with ArrayAccess::offsetExists(mixed $offset): bool, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /srv/vendor/aws/aws-sdk-php/src/Api/AbstractModel.php on line 41PHP message: PHP Deprecated:  Return type of Aws\Api\AbstractModel::offsetGet($offset) should either be compatible with ArrayAccess::offsetGet(mixed $offset): mixed, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /srv/vendor/aws/aws-sdk-php/src/Api/AbstractModel.php on line 30PHP message: PHP Deprecated:  Return type of Aws\Api\AbstractModel::offsetSet($offset, $value) should either be compatible with ArrayAccess::offsetSet(mixed $offset, mixed $value): void, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /srv/vendor/aws/aws-sdk-php/src/Api/AbstractModel.php on line 36PHP message: PHP Deprecated:  Return type of Aws\Api\AbstractModel::offsetUnset($offset) should either be compatible with ArrayAccess::offsetUnset(mixed $offset): void, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /srv/vendor/aws/aws-sdk-php/src/Api/AbstractModel.php on line 46PHP message: PHP Deprecated:  Return type of Aws\Endpoint\Partition::offsetExists($offset) should either be compatible with ArrayAccess::offsetExists(mixed $offset): bool, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /srv/vendor/aws/aws-sdk-php/src/HasDataTrait.php on line 41PHP message: PHP Deprecated:  Return type of & 
2022/12/15 19:22:56 [error] 21#21: *6 FastCGI sent in stderr: "sed to temporarily suppress the notice in /srv/vendor/aws/aws-sdk-php/src/HasDataTrait.php on line 36PHP message: PHP Deprecated:  Return type of Aws\Command::offsetUnset($offset) should either be compatible with ArrayAccess::offsetUnset(mixed $offset): void, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /srv/vendor/aws/aws-sdk-php/src/HasDataTrait.php on line 46PHP message: PHP Deprecated:  Return type of Aws\Command::count() should either be compatible with Countable::count(): int, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /srv/vendor/aws/aws-sdk-php/src/HasDataTrait.php on line 56PHP message: PHP Deprecated:  Return type of Aws\Command::getIterator() should either be compatible with IteratorAggregate::getIterator(): Traversable, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /srv/vendor/aws/aws-sdk-php/src/HasDataTrait.php on line 13PHP message: PHP Fatal error:  Uncaught Error: Class "SimpleXMLElement" not found in /srv/vendor/aws/aws-sdk-php/src/Api/Parser/PayloadParserTrait.php:39
Stack trace:
#0 /srv/vendor/aws/aws-sdk-php/src/Api/ErrorParser/XmlErrorParser.php(28): Aws\Api\ErrorParser\XmlErrorParser->parseXml()
#1 [internal function]: Aws\Api\ErrorParser\XmlErrorParser->__invoke()
#2 /srv/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php(175): call_user_func()
#3 /srv/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php(98): Aws\WrappedHttpHandler->parseError()
#4 /srv/vendor/guzzlehttp/promises/src/Promise.php(204): Aws\WrappedHttpHandler->Aws\{closure}()
#5 /srv/vendor/guzzlehttp/promises/src/Promise.php(169): GuzzleHttp\Promise\Promise::callHandler()
#6 /srv/vendor/guzzlehttp/promises/src/RejectedPromise.php(42): GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}()
#7 /srv/vendor/guzzlehttp/promises/src/TaskQueue.php(48): GuzzleHttp\Promise\RejectedPromise::GuzzleHttp\Promise\{closure}()
#

[elrido]

I'm not sure the author of the S3 functionality did provide any other means of adding the keys than the conf.php and I suspect they mainly tested it with ceph, not AWS.

Reg. deprecation warnings, these would be either with the AWS SDK or Guzzle projects. We can of course update our dependency on the AWS SDK, when it gets fixed in these.

[elrido]

In the latest 1.5.1 image we are now using updated S3 libraries. When you switch to it, could you please check if that addresses some or all of the deprecation warnings?

[rugk]

@rs-garrick Did you find time to try that out yet?

[rs-garrick]

I'll give it a shot after New Year's.

[rs-garrick]

$ docker run -p 8080:8080 -v $HOME/.aws:/.aws:ro -v $PWD/cfg:/srv/cfg:ro privatebin/s3:1.5.1
Unable to find image 'privatebin/s3:1.5.1' locally
1.5.1: Pulling from privatebin/s3
c158987b0551: Already exists 
64b4b337b46b: Pull complete 
f9e3a5661a13: Pull complete 
4f4fb700ef54: Pull complete 
Digest: sha256:c31e448916bfb80dba0f15ec104b2b10158ff4450644270c3816846f5166ea8b
Status: Downloaded newer image for privatebin/s3:1.5.1
[03-Jan-2023 15:20:23] NOTICE: fpm is running, pid 10
[03-Jan-2023 15:20:23] NOTICE: ready to handle connections
172.17.0.1 - - [03/Jan/2023:15:20:32 +0000] "POST / HTTP/1.1" 500 5 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0" "-"
2023/01/03 15:20:32 [error] 18#18: *1 FastCGI sent in stderr: "PHP message: PHP Deprecated:  Return type of Aws\HandlerList::count() should either be compatible with Countable::count(): int, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /srv/vendor/aws/aws-sdk-php/src/HandlerList.php on line 284PHP message: PHP Deprecated:  Return type of Aws\Api\AbstractModel::offsetExists($offset) should either be compatible with ArrayAccess::offsetExists(mixed $offset): bool, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /srv/vendor/aws/aws-sdk-php/src/Api/AbstractModel.php on line 41PHP message: PHP Deprecated:  Return type of Aws\Api\AbstractModel::offsetGet($offset) should either be compatible with ArrayAccess::offsetGet(mixed $offset): mixed, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /srv/vendor/aws/aws-sdk-php/src/Api/AbstractModel.php on line 30PHP message: PHP Deprecated:  Return type of Aws\Api\AbstractModel::offsetSet($offset, $value) should either be compatible with ArrayAccess::offsetSet(mixed $offset, mixed $value): void, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /srv/vendor/aws/aws-sdk-php/src/Api/AbstractModel.php on line 36PHP message: PHP Deprecated:  Return type of Aws\Api\AbstractModel::offsetUnset($offset) should either be compatible with ArrayAccess::offsetUnset(mixed $offset): void, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /srv/vendor/aws/aws-sdk-php/src/Api/AbstractModel.php on line 46PHP message: PHP Deprecated:  Return type of Aws\Endpoint\Partition::offsetExists($offset) should either be compatible with ArrayAccess::offsetExists(mixed $offset): bool, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /srv/vendor/aws/aws-sdk-php/src/HasDataTrait.php on line 41PHP message: PHP Deprecated:  Return type of & 

[rs-garrick]

Also, I wasn't clear in my initial posting... this backend isn't working. The web UI loads, but the "Send" button results in " Could not create paste: server error or not responding". No objects are being created in S3.

[elrido]

Thank you for testing and reporting back. Sorry to hear that it isn't working - I'll raise a ticket in the main project and try to get in touch with the author of the backend.

[elrido]

running the compatibility-test.php from the AWS library in the image I found some PHP modules to be missing in the S3 image. These have now been added and we have also just tagged & released 1.5.1-alpine3.17.1 - I'm hoping that while this will still have the deprecation warnings, it should at least not error out on the missing simplexml module. I don't have a way to test it, tough, as we have not been given a unit test for the S3 backend, like we have for the filesystem, PDO and GCS one.

[elrido]

@rs-garrick, is this fixed or still a problem?

[jakheipcg]

I'm not entirely sure the issues are the same, but I played around with AWS AppRunner today. It works just fine, until I try to use the S3 backend. I've tried it with both, an instance role that allows accessing S3, as well as access/secret access key saved in the environment variables, as explained in the conf.php sample.

The frontend error message is identical to the message posted by @rs-garrick (Could not create paste: server error or not responding)

I do get the following error messages in the log when trying to save an issue:

05-22-2023 01:30:36 PM #4 /srv/lib/Model.php(85): PrivateBin\Data\S3Storage->__construct()
05-22-2023 01:30:36 PM #3 /srv/lib/Data/S3Storage.php(112): Aws\S3\S3Client->__construct()
05-22-2023 01:30:36 PM #2 /srv/vendor/aws/aws-sdk-php/src/S3/S3Client.php(358): Aws\AwsClient->__construct()
05-22-2023 01:30:36 PM #1 /srv/vendor/aws/aws-sdk-php/src/AwsClient.php(221): Aws\ClientResolver->resolve()
05-22-2023 01:30:36 PM #0 /srv/vendor/aws/aws-sdk-php/src/ClientResolver.php(370): Aws\ClientResolver::_apply_credentials()
05-22-2023 01:30:36 PM Stack trace:
05-22-2023 01:30:36 PM 2023/05/22 11:30:36 [error] 19#19: *12 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Uncaught InvalidArgumentException: Credentials must be an instance of Aws\Credentials\CredentialsInterface, an associative array that contains "key", "secret", and an optional "token" key-value pairs, a credentials provider function, or false. in /srv/vendor/aws/aws-sdk-php/src/ClientResolver.php:550

Any idea how to fix that issue?

[elrido]

The reportedly missing array keys are populated from the [model_options] configuration section, where these are called accesskey & secretkey: https://github.com/PrivateBin/PrivateBin/blob/5b458a17c2470a44ec241fc1791e28d1e2a81dc6/lib/Data/S3Storage.php#L20-L33

[jakheipcg]

I've used the section in the sample config file, that uses (should use) the default AWS SDK credential provider chain:

;[model]
; example of S3 configuration for AWS using its SDK default credential provider chain
; if relying on environment variables, the AWS SDK will look for the following:
; - AWS_ACCESS_KEY_ID
; - AWS_SECRET_ACCESS_KEY
; - AWS_SESSION_TOKEN (if needed)
; for more details, see https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/guide_credentials.html#default-credential-chain 
;class = S3Storage
;[model_options]
;region = "eu-central-1"
;version = "latest"
;bucket = "my-bucket"

Don't really like to hardcode credentials, but if this is the only option as of now, I'll go that way of course.

[elrido]

Those environment variables did get added to the php-fpm configuration, so they get passed through down to PHP if set: https://github.com/PrivateBin/docker-nginx-fpm-alpine/blob/6363ae3e58bbb242908280915435d020bf300a69/etc/php/php-fpm.d/zz-docker.conf#L15-L18

You may want to test the nginx unit image - it doesn't use php-fpm or the s6 services manager, but an application server that directly drives PHP, so there is no environment filtering.

Beyond this, we do have to rely on the SDK to pick these up automatically, so you may have to raise the issue in the AWS SDK repo, if it still doesn't work, after verifying that the variables do reach the PHP environment.