Concurrent chunking feature does not make good use of available connections when smaller files are present after an initial large file

[michaeljmx]

We use an S3 uploader with the default number of maximum connections (3) and concurrent chunking enabled:

chunking: { enabled: true, concurrent: { enabled: true } }

If there are several small (< 5MB) files in the queue, the uploader works correctly and uploads several files at the same time. However, if we first add a large file (e.g. more than 10MB, so it uses 3 chunks itself) and then multiple small files - after finishing uploading the first file the uploader starts uploading small files one by one, not three files at the same time as it should.

Could you please check what could cause the issue?

[rnicholus]

This is how the concurrent chunking feature is supposed to work. Please read the feature doc at http://docs.fineuploader.com/features/concurrent-chunking.html for more info.

[rnicholus]

Your question is a little hard to follow. If I have misinterpreted, please take some time to clarify.

[michaeljmx]

I understand that one file can be uploaded using multiple connections, and in this case there won't be connections left for other files. However, the problem is that after processing such large file, the uploader sometimes becomes stuck in the single-connection mode.

I'll explain below the test I performed. Sorry for the long post.

We use the following setting: maxConnections - 3 (default) chunking - enabled concurrent - enabled

First test I performed was to upload 5 small files (less than 5MB each). As I understand the size of 5MB is important here as it is the minimal size of chunk in S3, and we use the S3 uploader.

The upload process I saw was the following: Step 1: File 1 - uploading File 2 - uploading File 3 - uploading File 4 - queued File 5 - queued

Step 2: File 1 - uploaded File 2 - uploading File 3 - uploading File 4 - uploading File 5 - queued

Step 3: File 1 - uploaded File 2 - uploaded File 3 - uploading File 4 - uploading File 5 - uploading

Step 4: File 1 - uploaded File 2 - uploaded File 3 - uploaded File 4 - uploading File 5 - uploading

Step 2: File 1 - uploaded File 2 - uploaded File 3 - uploaded File 4 - uploaded File 5 - uploading

Step 5: File 1 - uploaded File 2 - uploaded File 3 - uploaded File 4 - uploaded File 5 - uploaded

So everything worked as expected.

[michaeljmx]

Next test was to upload one large file (about 15MB and 5 small files). And I saw the following ("File 0" is the large file):

Step 1: File 0 - uploading (using several threads) File 1 - queued File 2 - queued File 3 - queued File 4 - queued File 5 - queued

Step 2: File 0 - uploaded File 1 - uploading File 2 - queued File 3 - queued File 4 - queued File 5 - queued

Step 3: File 0 - uploaded File 1 - uploaded File 2 - uploading File 3 - queued File 4 - queued File 5 - queued

Step 4: File 0 - uploaded File 1 - uploaded File 2 - uploaded File 3 - uploading File 4 - queued File 5 - queued

Step 5: File 0 - uploaded File 1 - uploaded File 2 - uploaded File 3 - uploaded File 4 - uploading File 5 - queued

Step 6: File 0 - uploaded File 1 - uploaded File 2 - uploaded File 3 - uploaded File 4 - uploaded File 5 - uploading

Step 7: File 0 - uploaded File 1 - uploaded File 2 - uploaded File 3 - uploaded File 4 - uploaded File 5 - uploaded

[michaeljmx]

As you can see in the second test the uploader uploaded small files one by one. My interpretation is that because of some problem with connection management after processing the first file the uploader adds one connection at a time.

[rnicholus]

Thanks, I understand now. This is a known "limitation" of concurrent chunking. Implementing the feature was exceptionally challenging. In order to keep the complexity at bay, I opted to implement the feature such that it simply moves from one file to the next, uploading as many chunks as possible for one specific file. The target workflow is one that uploads a single very large file, and this also works fine for many large files (though there is no benefit of concurrent chunking in this context).

I'll look into enhancing the feature to accommodate other workflows as well in the future.