openneuro-cli repeated connection crashes on upload

[dmalt]

Describe the bug While trying to upload my dataset to openneuro.org I'm facing a repeated crashes. I have to restart the upload manually and add subjects one by one. Slowly but steadily the files are uploaded, but it takes several upload restarts for each subject.

For example:

base) ❯ openneuro upload --dataset ds003938 ~/asus_home/Data/metacognition/MEG1/rawdata_anon/ --verbose                                                         13:07:21
Adding files to "ds003938"
[email protected]

        Summary:                  Available Tasks:        Available Modalities:
        118 Files, 22.16GB        questions               MEG
        6 - Subjects              rest
        1 - Session


	If you have any questions, please post on https://neurostars.org/tags/bids.

=======================================================================
Files to be uploaded:
sub-05/meg/sub-05_task-questions_run-01_meg.fif - 945.1 MB
sub-05/meg/sub-05_task-questions_run-02_meg.fif - 968.0 MB
sub-05/meg/sub-05_task-questions_run-03_meg.fif - 882.7 MB
? Begin upload? Yes
=======================================================================
Starting a new upload (43598a91) to dataset: 'ds003938'
ds003938 [----------------------------------------] 0% | ETA: 0s | 0/3Retrying upload for sub-05/meg/sub-05_task-questions_run-01_meg.fif: Error: Request aborted
Retrying upload for sub-05/meg/sub-05_task-questions_run-03_meg.fif: Error: Request aborted
Retrying upload for sub-05/meg/sub-05_task-questions_run-02_meg.fif: Error: Request aborted
Retrying upload for sub-05/meg/sub-05_task-questions_run-01_meg.fif: Error [ERR_HTTP2_INVALID_SESSION]: The session has been destroyed
Retrying upload for sub-05/meg/sub-05_task-questions_run-02_meg.fif: Error [ERR_HTTP2_INVALID_SESSION]: The session has been destroyed
Retrying upload for sub-05/meg/sub-05_task-questions_run-01_meg.fif: Error [ERR_HTTP2_INVALID_SESSION]: The session has been destroyed
Error: read ECONNRESET
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:220:20) {
  errno: -104,
  code: 'ECONNRESET',
  syscall: 'read'
}
Error: read ECONNRESET
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:220:20) {
  errno: -104,
  code: 'ECONNRESET',
  syscall: 'read'
}
Error: read ECONNRESET
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:220:20) {
  errno: -104,
  code: 'ECONNRESET',
  syscall: 'read'
}
Retrying upload for sub-05/meg/sub-05_task-questions_run-03_meg.fif: Error [ERR_HTTP2_INVALID_SESSION]: The session has been destroyed
Retrying upload for sub-05/meg/sub-05_task-questions_run-02_meg.fif: Error [ERR_HTTP2_INVALID_SESSION]: The session has been destroyed
Retrying upload for sub-05/meg/sub-05_task-questions_run-01_meg.fif: Error [ERR_HTTP2_INVALID_SESSION]: The session has been destroyed
Retrying upload for sub-05/meg/sub-05_task-questions_run-03_meg.fif: Error [ERR_HTTP2_INVALID_SESSION]: The session has been destroyed
Retrying upload for sub-05/meg/sub-05_task-questions_run-02_meg.fif: Error [ERR_HTTP2_INVALID_SESSION]: The session has been destroyed
Retrying upload for sub-05/meg/sub-05_task-questions_run-01_meg.fif: Error [ERR_HTTP2_INVALID_SESSION]: The session has been destroyed
Retrying upload for sub-05/meg/sub-05_task-questions_run-03_meg.fif: Error [ERR_HTTP2_INVALID_SESSION]: The session has been destroyed
Retrying upload for sub-05/meg/sub-05_task-questions_run-02_meg.fif: Error [ERR_HTTP2_INVALID_SESSION]: The session has been destroyed
Retrying upload for sub-05/meg/sub-05_task-questions_run-01_meg.fif: Error [ERR_HTTP2_INVALID_SESSION]: The session has been destroyed

It usually takes 2-3 restarts to upload one subject but sometimes I have to restart up to 10-20 times before the subject gets uploaded.

Desktop

• OS: Arch linux, kernel: 5.15.5-arch-1
• nodejs v.17.0.1
• npm v.8.1.3
• openneuro v4.2.0

[dmalt]

Related suggestion: it would be helpful to have -y option to openneuro upload which would suppress the "Upload the data?" question to at least make possible the automation of the upload restarts.

[nellh]

Hi, @dmalt thanks for the bug report. I believe we've narrowed this down to a service issue that is now resolved, can you test this again and let us know if you are still seeing frequent errors?

[dmalt]

Hi, @nellh. Unfortunately the issue persists. Please let me know if you need any further information.

[nellh]

Hi, @nellh. Unfortunately the issue persists. Please let me know if you need any further information.

Do you know if you are uploading via an HTTP proxy? The uploader requires HTTP/2 and proxies can sometimes interfere with this.

[dmalt]

Not on my side. This might have something to do with my internet provider. I'll try another network and get back to you.

[dmalt]

I've tried uploading via mobile data on a small batch and it seems to be working fine. The batch was not big enough to be 100% sure. Any suggestions why exactly this might be happening so I can I further debug it?

[rosskempner]

Hi @dmalt @nellh , I am having the same issue.