[Bug]: SFTP not working - Could not copy from remote location

[HeicoSindelItShapes]

Expected behavior

A Json-File is stored on a SFTP-Server. When starting the import (manually) the file should be copied from the SFTP-Server in a tmp-folder to load it from there and import the data in pimcore

Actual behavior

After starting the job, an error ("Could not copy from remote location") appears

Steps to reproduce

1. setup a SFTP-Server with username and password login and a dedicated folder for the sftp (e.g. /var/sftp/uploads/)
2. fill in the login-data to the data-importer "Data Source" form, use type=SFTP and Format JSON
3. store a json-file (e.g. products.json) on the folder /var/sftp/uploads
4. set the remote path absolute (e.g. /var/sftp/uploads/products.json) or relative (/uploads/products.json)
5. start the job

[HeicoSindelItShapes]

Additional question: is the data-importer designed for importing bulk data (mostly 20.000 to 100.000 objects; sometimes up to 1.000.000 objects)? which type (sftp, push,...) is recommended when a daily update is necessary?

[ateles]

The reason pimcores SftpLoader doesn't work is because it uses the following copy-command that copies the files within the same sftp connection. It doesn't download the file to the local directory.

$filesystem = new Filesystem(new SftpAdapter($connectionProvider, '/'));
        try {
            $filesystem->copy($this->remotePath, $this->importFilePath);
        } catch (FilesystemException $e) {
            // Error stuff
        }

Since we're developing our own specialized SftpLoader (overriding the old one) we could instead create one SftpAdapter and one LocalFilesystemAdapter and transfer the stream from sftp to the local filesystem:

        $fileSystemSFTP = new Filesystem(new SftpAdapter($connectionProvider, '/'));
        $adapter = new LocalFilesystemAdapter('/');
        $filesystemLocal = new Filesystem($adapter);
        try {
                    $readStream = $fileSystemSFTP->readStream($file->path());
                    $filesystemLocal->writeStream($this->importFilePath, $readStream);
                    return $this->importFilePath;
        } catch (FilesystemException $e) {
                     // Error stuff
        }

[fashxp]

would you like to contribute that? would be great. thx!