td icon indicating copy to clipboard operation
td copied to clipboard

Too many open files

Open vincentneo opened this issue 2 years ago • 7 comments

I left my simulator running on my Mac for a while, and came back to a SIGABRT crash as well as this final console message: [1;31m[ 0][t 4][1689493045.501645088][EventFdBsd.cpp:44][#1][!ConnectionCreator][&err == -1] [PosixError : Too many open files : 24 : socketpair failed][0m

Judging from it, I guess it has something to do with file descriptor limits imposed by the OS, which for macOS is 256. (Unsure what it is for watchOS)

No, I could not reliably replicate this, and there could be a high chance that it's my fault, but thought I would open this nonetheless, in case its a real concern.

Usage observed on watchOS, latest master + #2501 + #2517

vincentneo avatar Jul 16 '23 07:07 vincentneo

This is expected after the option waitsForConnectivity is enabled. I could only cite, what I wrote to you, while we discussed addition of the option: "It should crash now if there is no connectivity for a very long time, but the user is online. If the user is offline, then I expect OS to kill the app earlier".

I also wrote, how to prevent the crash: "It would be better to implement task canceling when they aren't needed anymore. RawConnectionHttp has close() method, but it does nothing now, because the code doesn't keep track of active tasks".

levlam avatar Jul 16 '23 08:07 levlam

Ah got it! thought open files != tasks and assumed it was a separate issue.

That said, when this occured, user is both online and with stable connectivity.

vincentneo avatar Jul 16 '23 10:07 vincentneo

This is a real issue for watchOS devices, so it can be kept opened.

levlam avatar Jul 16 '23 11:07 levlam

By the way, I realised it always spams tons (like 1000s in span of 20 minutes) of [1;31m[ 1][t 3][1689529388.331466197][ConnectionCreator.cpp:900][#1][!ConnectionCreator] [Error : 0 : Socket is empty][0m, right before the too many files crash.

In one test example I observed the Socket is empty error message, 1267 times. A look at memory graph shows 1269 instances of NSURLSessionConfiguration, which is observed to climb at roughly similar rates with the error message.

Yet, it seems if I manage to keep tdlib busy, such as if I'm requesting a bunch of stuff, the rate of which the error "socket is empty"'s appearance, appears to be lessen.

Any relation to the mentioned issue as well?

vincentneo avatar Jul 16 '23 17:07 vincentneo

This is related to the crash. The line in the log is printed when a new "connection" is created. A new connection is created when the previous connection is dead. i.e. requests to the connection received no response. In this case the connection is closed and a new connection is created. But when waitsForConnectivity is set to true, the requests sent to the old connection are stuck in the waiting state instead of failing. This eventually leads to the crash because system resource are exhausted.

levlam avatar Jul 16 '23 18:07 levlam

Right, I retested it with the master branch build without any patches, and while getting 332 'socket is empty' errors in a short span of 4 minutes, there were only 34 instances of NSURLSessionConfiguration.

vincentneo avatar Jul 17 '23 04:07 vincentneo

The commit https://github.com/tdlib/td/pull/2501/commits/b73061fdc918a89e90f8de754f41a5f612e10ff9 should make things slightly better. Basically each task can at max last for 90s before timeout.

vincentneo avatar Jul 17 '23 09:07 vincentneo