[Feeds API] Support stream in document->(up|down)load()

[quazardous]

Hi,

upload content are plain string. It's not memory friendly. You could allow using stream or file resource (since guzzle body is allowing it)

thx

[quazardous]

I've done a workaround by overriding theDocument class

I've added new function uploadStream() and downloadStream()

<?php

namespace XXX\Override\SellingPartnerApi;

use GuzzleHttp\Psr7\InflateStream;
use GuzzleHttp\Psr7\Utils;
use GuzzleHttp\RequestOptions;
use Psr\Http\Message\StreamInterface;
use ReflectionProperty;
use RuntimeException;
use SellingPartnerApi\Document as SellingPartnerApiDocument;
use SellingPartnerApi\ReportType;

class Document extends SellingPartnerApiDocument
{
    public function __construct(
        object $documentInfo,
        ?array $documentType = ReportType::__FEED_RESULT_REPORT,  // $documentType will be required in the next major version
        string $charset = null
    ) {
        parent::__construct($documentInfo, $documentType);
        if ($charset) {
            self::set_parent_property($this, 'contentType', $documentType['contentType'] . "; charset={$charset}");
        }
    }

    /**
     * Uploads data to the document specified in the constructor.
     *
     * @param resource|StreamInterface $feedDataStream The contents of the feed to be uploaded
     *
     * @return void
     */
    public function uploadStream($feedDataStream): void
    {

        /** @var \GuzzleHttp\Client $client */
        $client = self::get_parent_property($this, 'client');

        /** @var string $url */
        $url = self::get_parent_property($this, 'url');

        /** @var string $contentType */
        $contentType = self::get_parent_property($this, 'contentType');

        $response = $client->put($url, [
            RequestOptions::HEADERS => [
                "content-type" => $contentType,
                "host" => parse_url($url, PHP_URL_HOST),
            ],
            RequestOptions::BODY => $feedDataStream,
        ]);

        if ($response->getStatusCode() >= 300) {
            throw new RuntimeException("Upload failed ({$response->getStatusCode()}): {$response->getBody()}");
        }
    }

    /**
     * Downloads the document data.
     *
     * @param resource|string|StreamInterface|null $out
     * @param boolean $defalte
     * @return StreamInterface
     */
    public function downloadStream($out = null): StreamInterface
    {
        /** @var \GuzzleHttp\Client $client */
        $client = self::get_parent_property($this, 'client');

        /** @var string $url */
        $url = self::get_parent_property($this, 'url');

        /** @var string $compressionAlgo */
        $compressionAlgo = self::get_parent_property($this, 'compressionAlgo');

        try {
            $response = $client->request('GET', $url, ['stream' => true]);
        } catch (\GuzzleHttp\Exception\ClientException $e) {
            $response = $e->getResponse();
            if ($response->getStatusCode() == 404) {
                throw new RuntimeException("Document Report not Found ({$response->getStatusCode()}): {$response->getBody()}");
            } else {
                throw $e;
            }
        }

        $stream = $response->getBody();
        if (\strtolower($compressionAlgo) == "gzip") {
            $stream = new InflateStream($stream);
        }

        if ($out) {
            $out = Utils::streamFor($out);
            Utils::copyToStream($stream, $out);
            return $out;
        }

        return $stream;
    }

    protected static function get_parent_property(self $self, string $name)
    {
        $contentTypeProperty = new ReflectionProperty(SellingPartnerApiDocument::class, $name);
        $contentTypeProperty->setAccessible(true);
        return $contentTypeProperty->getValue($self);
    }

    protected static function set_parent_property(self $self, string $name, $value)
    {
        $contentTypeProperty = new ReflectionProperty(SellingPartnerApiDocument::class, $name);
        $contentTypeProperty->setAccessible(true);
        $contentTypeProperty->setValue($self, $value);
    }
}

[jlevers]

could you open a pr for this? ideally the stream would be integrated into the existing upload/download functions, so the user can just pass a stream or a string as they wish.

this needs to be PHP 7.3 compatible, so i would leave out the upload method return type hint -- the upload method i'm describing won't only return a StreamInterface, and i don't believe PHP 7.3 supports union types.

[quazardous]

Ok, I'll give it a try.

For the dowloading part, stream will not be compatible with any postprocess code, so maybe it should stay in a separated function.

What do you think ?

I could not poperly test the gzip part (I did not find a feed to trigger it).

[jlevers]

good point -- keeping the download function separate makes sense to me.

i think amazon has stopped gzipping reports, so i wouldn't sweat that part. i haven't seen a zipped report in a very long time.

[quazardous]

I can confirm that the gzip is still used (for 4400k+ SKUs GET_FLAT_FILE_OPEN_LISTINGS_DATA).

And the deflate stream is doing well.