influxdb icon indicating copy to clipboard operation
influxdb copied to clipboard

Too many open files -> Write failed

Open yareblo opened this issue 3 years ago • 1 comments

Running 2.06 on Ubuntu 18.04 (virtual environment, 6 CPUs, 16GB RAM)

ulimit -n set to 65535 via /etc/security/limits.conf

Cardinality does not seem to be too big (I have no reference):

Tag Value bucket1 Value Bucket 2 Value Bucket 3

A 0 4 4 B 6 37 37 C 0 4 4 _field 5 199 199 _measurement 1 3 3 _start 0 0 0 _stop 0 0 0

plus a telegraf bucket (normal configuration).

Writing data via python-influx-client synchronously.

-> Log shows tons of Reindexing-Messages -> After saving approx 20.000 values, write fails with "too many open files"-errors:

May 13 12:09:21 h2934423 influxd[4399]: ts=2021-05-13T10:09:21.670017Z lvl=info msg="index opened with 8 partitions" log_id=0U5IBfll000 service=storage-engine index=tsi May 13 12:09:21 h2934423 influxd[4399]: ts=2021-05-13T10:09:21.680668Z lvl=info msg="Reindexing TSM data" log_id=0U5IBfll000 service=storage-engine engine=tsm1 db_shard_id=45774 May 13 12:09:21 h2934423 influxd[4399]: ts=2021-05-13T10:09:21.681093Z lvl=info msg="Reindexing WAL data" log_id=0U5IBfll000 service=storage-engine engine=tsm1 db_shard_id=45774 May 13 12:09:21 h2934423 influxd[4399]: ts=2021-05-13T10:09:21.706835Z lvl=info msg="index opened with 8 partitions" log_id=0U5IBfll000 service=storage-engine index=tsi May 13 12:09:21 h2934423 influxd[4399]: ts=2021-05-13T10:09:21.707965Z lvl=info msg="Reindexing TSM data" log_id=0U5IBfll000 service=storage-engine engine=tsm1 db_shard_id=45776 May 13 12:09:21 h2934423 influxd[4399]: ts=2021-05-13T10:09:21.708594Z lvl=info msg="Reindexing WAL data" log_id=0U5IBfll000 service=storage-engine engine=tsm1 db_shard_id=45776 May 13 12:09:21 h2934423 influxd[4399]: ts=2021-05-13T10:09:21.711758Z lvl=info msg="Write failed" log_id=0U5IBfll000 service=storage-engine service=write shard=45802 error="[shard 45802] open /var/lib/influxdb/engine/data/6e477acd0cde6465/autogen/45802/index/2/MANIFEST: too many open files" May 13 12:09:21 h2934423 influxd[4399]: ts=2021-05-13T10:09:21.712599Z lvl=info msg="Write failed" log_id=0U5IBfll000 service=storage-engine service=write shard=45829 error="[shard 45829] open /var/lib/influxdb/engine/data/6e477acd0cde6465/autogen/45829/index/0/MANIFEST: too many open files"

What is wrong? How can I avoid this error? Reducing data is not an option.

yareblo avatar May 13 '21 10:05 yareblo

does ur issue solved ??? and how did u solved that??

musthaq16 avatar Jan 18 '24 12:01 musthaq16