too many opened files

[ederenn]

Name: Gert-Jan OS+lang+version (if applicable): MAC-js

GolemFactory.yagna.zip

[maaktweluit]

There are 2 sides to this issue:

• There is a simple config to add more open files, but do we want to ask our users to set this?
• We can ( and should ) investigate if we can reduce the open files for yagna and ya*pi

Other things to consider:

• Mac and linux default configs are very low ( 25x and 10xx respectively ).
• Can we recover / clean up when we detect open file limit is reached?

[nieznanysprawiciel]

Investigate what is opening so many files. Decide if we should try to optimize it.

[nieznanysprawiciel]

If someone encounters this problem again, please run lsof -p {process_id} for both yagna and running api.

[maaktweluit]

will be partially resolved by https://github.com/golemfactory/yagna/pull/1501

[mfranciszkiewicz]

Will be fully resolved when https://github.com/golemfactory/yapapi/pull/543 and js API equivalent are merged

[tworec]

This issue is still there. I was using yagna 0.10.0 (220c6399 2022-04-04 build #215) on MacOS 12.2.1 with default settings

$ ulimit -n
256

after playing with few yapapi examples I've got

[2022-04-06T21:23:14.666+0200 ERROR actix_server::accept] Error accepting connection: Too many open files (os error 24)

after running few examples. It have repeated few tens of times, like this

[2022-04-06T21:25:08.783+0200 INFO  actix_server::accept] Resume accepting connections on 127.0.0.1:7465
[2022-04-06T21:25:08.783+0200 ERROR actix_server::accept] Error accepting connection: Too many open files (os error 24)

and then yagna had even more serious issue

[2022-04-06T21:25:30.374+0200 ERROR r2d2] Unable to open the database file
[2022-04-06T21:25:30.374+0200 ERROR r2d2] Unable to open the database file
[2022-04-06T21:25:30.500+0200 INFO  ya_market::negotiation::requestor] Requestor 'golem-cli' [0x4c94f8883553ed92f7cffb01eab9968152291027] countered Proposal [R-df218c78f72d3057b25db24db9fc36cc91855e9e547e9ec164f539877b1dc66d] with [R-5e0c0f01fe544dadabdf91288c6f36cc72bd7d9cb794bb16f2a2c86d6bf99b5a]
[2022-04-06T21:25:30.779+0200 ERROR ya_persistence::executor] error: DatabaseError(__Unknown, "unable to open database file"), on: /Users/tworec/Library/Application Support/GolemFactory.yagna/erc20-driver.db, [lock: 19]
[2022-04-06T21:25:30.779+0200 ERROR r2d2] unable to open database file
[2022-04-06T21:25:30.796+0200 INFO  ya_market::negotiation::common] Received counter Proposal [R-af11aea9fc5ee444e4f2092b73a9efcf5f32427091c09661005f2efa308a1352] for Proposal [R-5e0c0f01fe544dadabdf91288c6f36cc72bd7d9cb794bb16f2a2c86d6bf99b5a] from [0x49ff6043a65766a832355e2899d13ba78b8819b8].
[2022-04-06T21:25:31.473+0200 INFO  ya_market::negotiation::common] New Proposal [R-43dc3dbce672613c3e91700d14b79b2ade3b5efe5192e1dd528be11465a57b2c] (Offer [1e56e1473c12454f9dd02d4dc17b92d6-bd0add096c852b8b3e2e350655c21481acab7bc6970ed35938bb296add0ba7e4], Demand [292105c4711f42f3b9fa8b7a3244ff58-0dfdfecea5f15c1885be775e5af468e3072fc721414d89d52864bc1ad48cf2f4])
[2022-04-06T21:25:31.584+0200 ERROR ya_persistence::executor] error: DatabaseError(__Unknown, "unable to open database file"), on: /Users/tworec/Library/Application Support/GolemFactory.yagna/erc20-driver.db, [lock: 20]
[2022-04-06T21:25:31.584+0200 ERROR r2d2] unable to open database file
[2022-04-06T21:25:31.625+0200 INFO  ya_market::negotiation::requestor] Requestor 'golem-cli' [0x4c94f8883553ed92f7cffb01eab9968152291027] countered Proposal [R-43dc3dbce672613c3e91700d14b79b2ade3b5efe5192e1dd528be11465a57b2c] with [R-b89f320c4f840278b56597e1e1958c627b0c982d569be81d43fedfd7d0a8e2d6]
[2022-04-06T21:25:31.731+0200 INFO  ya_market::negotiation::common] Received counter Proposal [R-07f118c0d3220f3349e68eb2c4770cdc50fa43d29afaa740f7ee5ebc04b4cd2a] for Proposal [R-b89f320c4f840278b56597e1e1958c627b0c982d569be81d43fedfd7d0a8e2d6] from [0x05ec10f2bd94c675be455e3d66eff74bdb804fd3].
[2022-04-06T21:25:31.958+0200 ERROR actix_server::accept] Error accepting connection: Too many open files (os error 24)
[2022-04-06T21:25:32.470+0200 INFO  actix_server::accept] Resume accepting connections on "/tmp/yagna.sock" (pathname)
[2022-04-06T21:25:33.186+0200 ERROR ya_persistence::executor] error: DatabaseError(__Unknown, "unable to open database file"), on: /Users/tworec/Library/Application Support/GolemFactory.yagna/erc20-driver.db, [lock: 21]
[2022-04-06T21:25:33.186+0200 ERROR r2d2] unable to open database file

Here is my datadir with logs yagna.datadir.zip

[nieznanysprawiciel]

@golemfactory/devops team will monitor open file descriptors after doing task: https://github.com/golemfactory/golem-internal/issues/393

[smitty-codes]

Seeing the same open files issue on MacOS 12.4 as well.

[nieznanysprawiciel]

Closing in favor of: https://github.com/golemfactory/yagna-triage/issues/232