Fetching from EMDB broken

[olibclarke]

Using latest commit of 1.06-pre. Tested with EMDB ID 16730. It seems to be downloading something, but nothing appears in display manager, and this is the only relevant info in the terminal:

PDB Accession Code: 16730
frame: 0x151350560
DEBUG:: extracted accession code handle mode n 114
DEBUG:: in coot_get_url_and_activate_curl_hook https://ftp.ebi.ac.uk/pub/databases/emdb/structures/EMD-16730/map/emd_16730.map.gz coot-download/emd_16730.map.gz

[olibclarke]

Progress in latest commit (15deae0) - I now see a progress bar, and then gets part of the way through, but then aborts:

** (coot-bin:41981): WARNING **: 21:25:12.227: Download failed. Status=28

This is for a map that is 97.5MB, which I just downloaded using ChimeraX with no problems, so the server isn't the issue. Maybe the timeout is too short for slowish connections?

[pemsley]

@hgonomeg, comment please? Is this the time-out issue?

[hgonomeg]

Yes, that's the timeout issue. See error code: https://curl.se/libcurl/c/libcurl-errors.html

This problem is not limited to EMDB fetches. All coot download operations that use coot_get_url() have a hard-coded 31 second timeout.

I suggest we get rid of the timeout (or increase it to a much much higher value) for connections where we display a progress bar.

In the future, all downloads without a progress bar should probably be moved to a background thread with a progress bar displayed.

[olibclarke]

I would definitely get rid of (or drastically increase) the timeout - EMDB maps can be up to 2GB, are often in the 300-500MB range, and that number is likely to increase as finer samplings for higher resolution structures become more routine.

[hgonomeg]

@pemsley Is it allright to ditch the timeout altogether? I'm not sure if it's practical for anything. Foreground downloads currently freeze the UI anyway. Better to have a freeze and a complete download than a freeze and incomplete download in my opinion.

[pemsley]

I bumped it up to 5 minutes in a7956aa695f8c19e52622447ca292434812d4e62.

That's 10 times longer.

[olibclarke]

Still maybe not long enough for large maps on slow connections...?

E.g. for a 300MB map, which is pretty typical, that means if the connection is slower than 1MB/s it will get most of the way through and then fail - frustrating.

I would prefer no time out (or absurdly long timeout), with a cancel/abort button so that if it is taking too long the user can choose