maxreaders limit reached

[Gingeropolous]

May 25 19:56:23 phx-4.snipanet.com clearnet_exp.sh[10062]: "/home/monerosupport/.bitmonero/lmdb"
May 25 19:56:23 phx-4.snipanet.com clearnet_exp.sh[10062]: 2019-05-25 19:56:23.162        W Failed to create a read transaction for the db: MDB_READERS_FULL: Environment maxreaders limit reached
May 25 19:56:23 phx-4.snipanet.com clearnet_exp.sh[10062]: terminate called after throwing an instance of 'cryptonote::DB_ERROR_TXN_START'
May 25 19:56:23 phx-4.snipanet.com clearnet_exp.sh[10062]:   what():  Failed to create a read transaction for the db: MDB_READERS_FULL: Environment maxreaders limit reached
May 25 19:56:23 phx-4.snipanet.com clearnet_exp.sh[10062]: /home/monerosupport/conf_files/clearnet_exp.sh: line 17: 10064 Aborted                 (core dumped)

[moneroexamples]

Same issue as here with a solution: https://github.com/moneroexamples/onion-monero-blockchain-explorer/issues/169

Can start monero daemon with smaller concurrency using --max-concurrency 60 option. This should help.

[Gingeropolous]

well, thats a solution in monero.... @hyc mentioned there should be something fixable in the explorer code. stale lookers or stale something or others.

Anyway, its also possible to clean this up using:

mdb_stat -rr ~/.bitmonero/lmdb/

which can be found in

external/db_drivers/liblmdb/

of the monero directory

[moneroexamples]

Thanks. Will look into that. I had this issue in OpenMonero, but it was because OM uses a lot of threads. It was fixed by changing OM architecture.

Explorer does not create many threads. So this error here is more interesting. Maybe its due to stale readers then.

[moneroexamples]

Does the issue occur each time you start xmrblocks, or from time to time?

[moneroexamples]

concurrency,c option was added into devel branch. With this you can limit how many threads are used to server http requrets. By default it is based on your hardware, and this number can be large based on your server's cpu.

For example, to limit http requests to two threads:

/xmrblocks -c 2

[Gingeropolous]

yeah, stale readers. Thats what hyc said. And when I ran that mdbstat command, it found 104 stale readers or something.

[moneroexamples]

For now I don't know whey those stale readers originate from. As a temporary solution you could maybe use cron to schedule periodic cleanup of these readers using mdb_stat.

[Gingeropolous]

yeah thats what I've done. hyc wasn't happy. :)

[Gingeropolous]

FYI, this is still a problem, and I had to increase the frequency of the mdb_stat thing to every 5 minutes or something

[hyc]

you really should figure out why your readers aren't exiting cleanly...

[moneroexamples]

@Gingeropolous Could you provide the argument list how you execute the explorer?

I will try to replicate the issue by putting some load on my localhost using jmeter.

[Gingeropolous]

this is my launch file

user@phx-4:~/conf_files$ cat clearnet_exp.sh 
#!/bin/bash

cd /home/monerosupport/clearnet_explorer_v15_2/build/
#cd /home/monerosupport/onion-monero-blockchain-explorer/build/
./xmrblocks -p 8888 \
--enable-pusher 1 \
--enable-key-image-checker 1 \
--enable-output-key-checker 1 \
--enable-autorefresh-option 1 \
--enable-tx-cache 1 \
--enable-block-cache 1 \
--enable-json-api 1 \
--deamon-url http:://127.0.0.1:12345 \
--testnet-url https://testnet.xmrchain.net/ \
--stagenet-url https://stagenet.xmrchain.net/ \
--mainnet-url https://xmrchain.net/ \
--enable-mixin-details 1 

[moneroexamples]

T.L.D.R: Switch off caching, i.e., use --enable-tx-cache 0 --enable-block-cache 0 , and check if it helps.

I tested few scenarios using jmeter, and I found an issue leading to max readers error (assuming its the same for you).

Turns out that under some load (100-200 jmeter threads), explorer was crashing and readers were not de-allocated. Restarting the explorer, allocated new readers, but then due to cache issues it crashed again without de-allocating readers one more time. So after few such crashes, all readers were used up, resulting in MDB_READERS_FULL error.

Disabling caches solved the issue in my tests. Caches will be remove in future from the explorer, so its better not to use them anyway.

p.s. For future reference, useful command to monitor number of lmdb readers

watch -d -n 1 "./mdb_stat -r ~/.bitmonero/lmdb/ | wc -l"

[normoes]

Where can I find mdb_stat?

I have the same problem and would like to check the stale readers.

Thanks a lot.

[Gingeropolous]

this is the crontab i have on xmrchain.net

*/5 * * * * /home/monerosupport/monero_master/external/db_drivers/liblmdb/mdb_stat -rr ~/.bitmonero/lmdb/

[moneroexamples]

@Gingeropolous

The issue persists? I though that removing cashes from the explorer helped, as this is what I identified as a cause of the problem in my previews tests.

[normoes]

Well, to be honest it was the daemon (stagenet) actually who threw this error and did not start.

But since there only is the daemon and the explorer accessing the db I suspected the explorer and jumped onto this train.

[moneroexamples]

The explorer was causing this issue, as explained here https://github.com/moneroexamples/onion-monero-blockchain-explorer/issues/178#issuecomment-536166666

But I thought the issue was resolved. But maybe there is something new in the explorer that is causing the maxreader error. Will have to confirm with @Gingeropolous if this is the case.

[Gingeropolous]

@moneroexamples , I don't know. I apparently forgot about this crontab when I updated.

I am currently setting up a new server for xmrchain.net, and will attempt to run it clean to see if its all fixed.

[who-biz]

Was there ever investigation into what specifically caused this error? I see the mention of cache disabling above... but seems a rough fix.

@Gingeropolous thanks for the mdb_stat command above -- that worked.

But would like to get to the bottom of why this occurred, in the first place. I am using a fork of onion-monero-blockchain-explorer, which does appear to be the culprit. Had same operations happening on a few other nodes (sans explorer). Only the daemon/explorer node experienced the error.

[hyc]

If you've got a lot of users, you may legitimately be using up all the available reader slots. The default is 126, which must be shared with monerod itself. If you need more, you can use this little program to set whatever number you want. It must be run before any other program that uses the blockchain DB, and it needs to stay alive (as a background job, whatever) until any other programs start. lmdbrr.c.txt

gcc -o lmdbrr lmdbrr.c -pthread -llmdb
./lmdbrr ~/.bitmonero/lmdb 252 &
monerod 
xmrblocks &