Error after updating to curl 7.84

[lbilli]

For a long time I've been running a task that retrieves data from the internet by sending a few thousands requests asynchronously using multi_add() and multi_run(). All has been very successful till now.

After updating to curl 7.84.0 the same task immediately aborts with the error:

Error in multi_run(timeout = 10, pool = pool) : 
  Unrecoverable error in select/poll

Few observations:

• reverting back to curl 7.83.1 fixes the issue
• the issue seems related to the number of requests in the pool: no problem with ~1000 requests whereas it fails when the number of requests exceeds 1500 (not sure where the exact cutoff is).

Any changes in curl 7.84.0 that might be causing this?

[jeroen]

You would have to ask @bagder or @jay if anyting relevant changed...

[jeroen]

@lbilli which OS do you use? Can you provide more info about your libcurl configuration: curl::curl_version() ?

Can you try to create a reproducible example? I realize it may be difficult with 1000 requests, but there is no other way for me to help finding the issue.

[lbilli]

OS is openSUSE tumbleweed with R 4.2.1 As far as I can tell, the error happens right after the first batch of 6 connections are established.

Here's a repro:

library(curl)
repro <- function(n) {
  urls <- paste0("https://httpbingo.org/get?q=", 1:n)
  make_handle <- function(url) new_handle(url=url)
  pool <- new_pool()
  fail <- function(msg) cat("failed connection:", msg, "\n")
  done <- function(data) cat("status:", data$status_code, "\n")
  for(u in urls)
    multi_add(make_handle(u), done=done, fail=fail, pool=pool)
  stat <- multi_run(timeout=10, pool=pool)
  cat("remaining:",  stat$pending, "\n")
}

> repro(10)    # OK
status: 200 
[...]
remaining: 0

> repro(1500)  # Fail
Error in multi_run(timeout = 10, pool = pool) : 
  Unrecoverable error in select/poll

> curl_version()
$version
[1] "7.84.0"

$ssl_version
[1] "OpenSSL/1.1.1q-fips"

$libz_version
[1] "1.2.11"

$libssh_version
[1] "libssh/0.9.6/openssl/zlib"

$libidn_version
[1] "2.3.3"

$host
[1] "x86_64-suse-linux-gnu"

$protocols
 [1] "dict"    "file"    "ftp"     "ftps"    "gopher"  "gophers" "http"   
 [8] "https"   "imap"    "imaps"   "ldap"    "ldaps"   "mqtt"    "pop3"   
[15] "pop3s"   "rtsp"    "scp"     "sftp"    "smb"     "smbs"    "smtp"   
[22] "smtps"   "telnet"  "tftp"   

$ipv6
[1] TRUE

$http2
[1] TRUE

$idn
[1] TRUE

[bagder]

@jeroen it looks like this is the commit that introduced this error that seems to trigger: https://github.com/curl/curl/commit/5912da253b64d

The bigger question is of course why select/poll returns error, but that requires deeper research.

[jeroen]

@lbilli I run your example on Debian with libcurl 7.84.0 but it does not error for me.

Update: I have also tested this in a docker container with suse tumbleweed on my machine, with exactly your configuration, but it does not error either.

So the problem may also be related to your local hardware, which makes it more difficult to find the problem. Perhaps you can look at /var/log/kern.log if there is any hint of your machine running out of memory or file descriptors or something?

@bagder is there any way to make libcurl print strerror(errno) upon the poll() failure?

[lbilli]

The issue got recently resolved somehow.