influxdb
influxdb copied to clipboard
Too many open files -> Write failed
Running 2.06 on Ubuntu 18.04 (virtual environment, 6 CPUs, 16GB RAM)
ulimit -n set to 65535 via /etc/security/limits.conf
Cardinality does not seem to be too big (I have no reference):
Tag Value bucket1 Value Bucket 2 Value Bucket 3
A 0 4 4 B 6 37 37 C 0 4 4 _field 5 199 199 _measurement 1 3 3 _start 0 0 0 _stop 0 0 0
plus a telegraf bucket (normal configuration).
Writing data via python-influx-client synchronously.
-> Log shows tons of Reindexing-Messages -> After saving approx 20.000 values, write fails with "too many open files"-errors:
May 13 12:09:21 h2934423 influxd[4399]: ts=2021-05-13T10:09:21.670017Z lvl=info msg="index opened with 8 partitions" log_id=0U5IBfll000 service=storage-engine index=tsi May 13 12:09:21 h2934423 influxd[4399]: ts=2021-05-13T10:09:21.680668Z lvl=info msg="Reindexing TSM data" log_id=0U5IBfll000 service=storage-engine engine=tsm1 db_shard_id=45774 May 13 12:09:21 h2934423 influxd[4399]: ts=2021-05-13T10:09:21.681093Z lvl=info msg="Reindexing WAL data" log_id=0U5IBfll000 service=storage-engine engine=tsm1 db_shard_id=45774 May 13 12:09:21 h2934423 influxd[4399]: ts=2021-05-13T10:09:21.706835Z lvl=info msg="index opened with 8 partitions" log_id=0U5IBfll000 service=storage-engine index=tsi May 13 12:09:21 h2934423 influxd[4399]: ts=2021-05-13T10:09:21.707965Z lvl=info msg="Reindexing TSM data" log_id=0U5IBfll000 service=storage-engine engine=tsm1 db_shard_id=45776 May 13 12:09:21 h2934423 influxd[4399]: ts=2021-05-13T10:09:21.708594Z lvl=info msg="Reindexing WAL data" log_id=0U5IBfll000 service=storage-engine engine=tsm1 db_shard_id=45776 May 13 12:09:21 h2934423 influxd[4399]: ts=2021-05-13T10:09:21.711758Z lvl=info msg="Write failed" log_id=0U5IBfll000 service=storage-engine service=write shard=45802 error="[shard 45802] open /var/lib/influxdb/engine/data/6e477acd0cde6465/autogen/45802/index/2/MANIFEST: too many open files" May 13 12:09:21 h2934423 influxd[4399]: ts=2021-05-13T10:09:21.712599Z lvl=info msg="Write failed" log_id=0U5IBfll000 service=storage-engine service=write shard=45829 error="[shard 45829] open /var/lib/influxdb/engine/data/6e477acd0cde6465/autogen/45829/index/0/MANIFEST: too many open files"
What is wrong? How can I avoid this error? Reducing data is not an option.
does ur issue solved ??? and how did u solved that??