http sever occasionally return response status “Resource temporarily unavailable ”

[changweige]

After scanning nydus api server code, it does not return status “Resource temporarily unavailable ”, is this returned by http server crate?

        dist.put_multiple_files(20, Size(8, Unit.KB))
    
        image.set_backend(Backend.BACKEND_PROXY).create_image()
>       nc.pseudo_fs_mount(image.bootstrap_path, f"/pseudo{suffix}", conf.path(), None)

functional-test/test_nydus.py:505: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (<nydusd_client.NydusAPIClient object at 0x7f03d1558520>, '/home/runner/work/image-service/image-service/contrib/nydus-test/bs2', '/pseudo2', '/tmp/tmpixjok_airafs.config', None)
resp = <Response [500]>

    def wrapped(*args):
        resp = func(*args)
>       assert resp.status_code < 400 or resp.status_code == 501, resp.content.decode(
            "utf-8"
        )
E       AssertionError: Resource temporarily unavailable (os error 11)

nydusd_client.py:53: AssertionError

[changweige]

Seems poor nydusd api server never has a chance to process this failure api request

[imeoer]

Maybe the python HTTP client has opened more connections than the MAX_CONNECTIONS defined here for keeping its connection pool.

[jiangliu]

The dbs-uhttp limits the maximum concurrent connections to MAX_CONNECTIONS(10). Maybe we should enlarge the limit. The failure should be caused by the code below:

    fn handle_new_connection(&mut self) -> Result<()> {
        if self.connections.len() == MAX_CONNECTIONS {
            // If we want a replacement policy for connections
            // this is where we will have it.
            return Err(ServerError::ServerFull);
        }
    }

[sctb512]

The dbs-uhttp limits the maximum concurrent connections to MAX_CONNECTIONS(10). Maybe we should enlarge the limit. The failure should be caused by the code below:

    fn handle_new_connection(&mut self) -> Result<()> {
        if self.connections.len() == MAX_CONNECTIONS {
            // If we want a replacement policy for connections
            // this is where we will have it.
            return Err(ServerError::ServerFull);
        }
    }

It may not matter with MAX_CONNECTIONS, after I change it to 1000, I still get Resource temporarily unavailable (os error 11) error.

[imeoer]

@sctb512 Are you using the go or python HTTP client? May be we can use netstat to inspect the TCP connections of the client side first.

[sctb512]

@sctb512 Are you using the go or python HTTP client? May be we can use netstat to inspect the TCP connections of the client side first.

OK, I will try it.

[changweige]

I delve into the uHttp code a bit. It does not return an error. I think the root cause is Unix socket listener file is set to nonblocking. The solver could be making the client retry until it is connected or replace mio

[jiangliu]

The rust UnixListener has configured the listener queue depth as 128

The go unix dialer uses non-blocking to connect to server, which will cause EAGAIN when the listener queue is full in the linux kernel. So we have no chance to change the kernel behavior, and it's not easy to enlarge the queue depth.

Seems the way to solve it is "client retry"

[changweige]

will be solved by https://github.com/openanolis/dbs-uhttp/pull/18

[jiangliu]

new version of dbs-uhttp has been publish, need to update the cargo.lock file

[changweige]

Already updated dependency on dbs-uhttp. This problem is addressed.