gdal icon indicating copy to clipboard operation
gdal copied to clipboard

Add VSICopyFileRestartable() to allow restart of upload of large files

Open rouault opened this issue 1 year ago • 1 comments

Copy a source file into a target file in a way that can (potentially) be restarted.

This function provides the possibility of efficiently restarting upload of large files to cloud storage that implements upload in a chunked way, such as /vsis3/ and /vsigs/. For other destination file systems, this function may fallback to VSICopyFile() and not provide any smart restartable implementation.

Example of a potential workflow:

char* pszOutputPayload = NULL;
int ret = VSICopyFileRestartable(pszSource, pszTarget, NULL,
                                 &pszOutputPayload, NULL, NULL, NULL);
while( ret == 1 ) // add also a limiting counter to avoid potentiall endless looping
{
    // TODO: wait for some time

    char* pszOutputPayloadNew = NULL;
    const char* pszInputPayload = pszOutputPayload;
    ret = VSICopyFileRestartable(pszSource, pszTarget, pszInputPayload,
                                 &pszOutputPayloadNew, NULL, NULL, NULL);
    VSIFree(pszOutputPayload);
    pszOutputPayload = pszOutputPayloadNew;
}
VSIFree(pszOutputPayload);

rouault avatar May 03 '24 17:05 rouault

Coverage Status

coverage: 69.095% (+0.01%) from 69.081% when pulling a957dcc27d9669b76731fb2513a327944d71c062 on rouault:VSICopyFileRestartable into 4b1e7bf41fc62ef2d8e75e4bdcec3c37903f4093 on OSGeo:master.

coveralls avatar May 03 '24 19:05 coveralls