C-API fdb_setup_network get "1510: Disk i/o operation failed" without trace log

[ZhangHuiGui]

Background

I want to record the c-api's client trace file when there is an error happened, but fdb-client didn't record the error trace log when error happend.

Environment

fdb-verion : 7.1.9 os : ubuntu 20.04

How to reproduce

1. keep environment clean and only start a fdbserver process.
2. adjust os config aio-max-nr to reproduce fdb-client's io-error

# cat /proc/sys/fs/aio-max-nr
65536
# echo 256 > /proc/sys/fs/aio-max-nr

3. startup three fdbcli and keep connect, and you will see aio-nr have arrived limited 256.
4. startup last fdbcli with trace-log : fdbcli --log --knob-min_trace_severity=10. You will see io-error which throwed by fdb_setup_network api's AsyncFileKAIO::init function. After that, there is no trace-log at current directory.

So @sfc-gh-ajbeamon , please take a simple look if this is a problem ?

[ZhangHuiGui]

Besides, i have seen that there is a TraceEvent in this code:

[sfc-gh-abeamon]

Is the concern here that the process is encountering an error in fdb_setup_network but not generating any trace logs? Unfortunately, this is the current behavior of the client because it does not open the trace file until the first fdb_create_database call, which happens sometime after fdb_setup_network.

If this is the problem, I was just talking with @sfc-gh-vgasiunas about this type of issue where an error happens in the client process before the trace file is opened. We discussed the idea of opening the trace file earlier and also of having the option to dump logs to stdout/stderr if it fails earlier than that.