taosdata
telegraf did not complete within its flush interval
did not complete within its flush interval 通过telegraf 读取数据库数据插入TDengine报错。 插入时返回日志只消耗了40ms左右。 有时能坚持3、4天。。有时3、4个小时就出现类似“假死现象”，之后restart taosd 无响应。可能在落盘。。只能重启服务器 间歇性也会出现报错。但是taosd不会出现“假死现象” 还能继续插入。 期间经历4个版本 3.0.0.12、3.0.0.13、3.0.0.14、3.0.0.15 多次修改metric_batch_size： 200，500，1000，2000 及outputs.http timeout：5s，10s，20s，30s 自动建表也修改为固定表名了。 请教如何排查处理该问题。 相关配置如下 24CPU-128GMem-1000GDisk centos7.6 TDengine：3.0.0.15 telegraf:v1.24 postgresql,code=SYGL038,tname=SYGL038 pv=0 1666834560000000000 postgresql,code=SYGL029,tname=SYGL029 pv=1 1666834560000000000 postgresql,code=SYGL031,tname=SYGL031 pv=1 1666834560000000000 共6000条
-
telegraf config
[agent]interval = "60s"round_interval = truemetric_batch_size = 500metric_buffer_limit = 2000collection_jitter = "0s"flush_interval = "2s"flush_jitter = "5s"precision = ""log_with_timezone = "Asia/Shanghai"hostname = ""omit_hostname = true[[outputs.http]]url = "http://192.168.10.22:6041/influxdb/v1/write?db=iotdb"method = "POST"timeout = "20s"username = "root"password = "taosdata"data_format = "influx"influx_max_line_bytes = 250[[inputs.postgresql_extensible]]address = "postgres://postgres:[email protected]/iotdb"[[inputs.postgresql_extensible.query]]sqlquery="select gathertime,pv ,code tname,code from syncdataairiot where code != ''"[[inputs.postgresql_extensible.query]]sqlquery="select gathertime,pv ,code tname,code from syncdata where code != ''"[[processors.tag_limit]]limit = 2keep = ["tname","code"][[processors.converter]][processors.converter.fields]tag = ["tname","code"] -
telegraf error info
Oct 27 07:57:40 timingdata.novalocal telegraf[11759]: 2022-10-27T07:57:40+08:00 E! [agent] Error writing to outputs.http: Post "http://192.168.10.22:6041/influxdb/v1/write?db=iotdb": context deadline exceeded (Client.Timeout exceeded while awaiting headers)Oct 27 07:57:43 timingdata.novalocal telegraf[11759]: 2022-10-27T07:57:43+08:00 W! [agent] ["outputs.http"] did not complete within its flush intervalOct 27 07:57:46 timingdata.novalocal telegraf[11759]: 2022-10-27T07:57:46+08:00 W! [agent] ["outputs.http"] did not complete within its flush intervalOct 27 07:57:50 timingdata.novalocal telegraf[11759]: 2022-10-27T07:57:50+08:00 W! [agent] ["outputs.http"] did not complete within its flush intervalOct 27 07:57:53 timingdata.novalocal telegraf[11759]: 2022-10-27T07:57:53+08:00 W! [agent] ["outputs.http"] did not complete within its flush intervalOct 27 07:57:57 timingdata.novalocal telegraf[11759]: 2022-10-27T07:57:57+08:00 W! [agent] ["outputs.http"] did not complete within its flush intervalOct 27 07:57:59 timingdata.novalocal telegraf[11759]: 2022-10-27T07:57:59+08:00 W! [agent] ["outputs.http"] did not complete within its flush intervalOct 27 07:58:00 timingdata.novalocal telegraf[11759]: 2022-10-27T07:58:00+08:00 E! [agent] Error writing to outputs.http: Post "http://192.168.10.22:6041/influxdb/v1/write?db=iotdb": context deadline exceeded (Client.Timeout exceeded while awaiting headers) -
LOCAL VARIABLES;
taos> SHOW LOCAL VARIABLES;name | value |==================================================================firstEp | timingdata.novalocal:6030 |secondEp | timingdata.novalocal:6030 |fqdn | timingdata.novalocal |serverPort | 6030 |tempDir | /tmp/ |minimalTmpDirGB | 1.000000 |shellActivityTimer | 3 |compressMsgSize | -1 |compressColData | -1 |queryPolicy | 1 |querySmaOptimize | 0 |queryPlannerTrace | 0 |queryNodeChunkSize | 32768 |queryUseNodeAllocator | 1 |keepColumnName | 0 |smlChildTableName | tname |smlTagName | _tag_null |smlDataFormat | 0 |maxMemUsedByInsert | 1024 |numOfTaskQueueThreads | 12 |configDir | /etc/taos/ |scriptDir | /etc/taos/ |logDir | /var/log/taos/ |minimalLogDirGB | 1.000000 |numOfLogLines | 10000000 |asyncLog | 1 |logKeepDays | 0 |debugFlag | 0 |simDebugFlag | 143 |tmrDebugFlag | 131 |uDebugFlag | 131 |rpcDebugFlag | 131 |jniDebugFlag | 131 |qDebugFlag | 131 |cDebugFlag | 131 |timezone | Asia/Shanghai (CST, +0800) |locale | en_US.UTF-8 |charset | UTF-8 |enableCoreFile | 1 |numOfCores | 24.000000 |openMax | 1024 |streamMax | 16 |pageSizeKB | 4 |totalMemoryKB | 131861632 |os sysname | Linux |os nodename | timingdata.novalocal |os release | 3.10.0-957.el7.x86_64 |os version | #1 SMP Thu Nov 8 23:39:32 U... |os machine | x86_64 |version | 3.0.1.5 |compatible_version | 3.0.0.0 |gitinfo | d58f7a62f72e95161068c2fa9c9... |buildinfo | Built at 2022-10-19 17:06 |` -
taosadapter_2022_10_27_00_00.log
10/27 10:20:01.561087 00015939 TAOS_ADAPTER debug "finish insert influxdb cost: 12.409063ms" model=influxdb sessionID=760 10/27 10:20:01.561193 00015939 TAOS_ADAPTER info "| 204 | 12.647394ms | 192.168.10.22 | POST | /influxdb/v1/write?db=iotdb " model=web sessionID=760 10/27 10:20:04.197600 00015939 TAOS_ADAPTER debug "start insert influxdb: pl1c,code=SSC131 pv=0.03515625 1666837160000000000 pl1c,code=SSC141 pv=0.01953125 1666837160000000000 pl1c,code=SSC151 pv=73.8671875 1666837160000000000 plc,code=SSC13 pv=0.03515625 1666837160000000000 plc,code=SSC14 pv=0.01953125 1666837160000000000 plc,code=SSC15 pv=73.8671875 1666837160000000000 " model=influxdb sessionID=761 10/27 10:20:04.207189 00015939 TAOS_ADAPTER debug "finish insert influxdb cost: 9.56198ms" model=influxdb sessionID=761 10/27 10:20:04.207352 00015939 TAOS_ADAPTER info "| 204 | 9.761369ms | 192.168.10.47 | POST | /influxdb/v1/write?db=iot " model=web sessionID=761 10/27 10:20:15.340927 00015939 TAOS_ADAPTER debug "start insert influxdb: pl1c,code=SSC131 pv=0.02734375 1666837170000000000 pl1c,code=SSC141 pv=0.017578125 1666837170000000000 pl1c,code=SSC151 pv=74.4140625 1666837170000000000 plc,code=SSC13 pv=0.02734375 1666837170000000000 plc,code=SSC14 pv=0.017578125 1666837170000000000 plc,code=SSC15 pv=74.4140625 1666837170000000000 " model=influxdb sessionID=762 10/27 10:20:15.349004 00015939 TAOS_ADAPTER debug "finish insert influxdb cost: 8.047348ms" model=influxdb sessionID=762 10/27 10:20:15.349129 00015939 TAOS_ADAPTER info "| 204 | 8.267572ms | 192.168.10.47 | POST | /influxdb/v1/write?db=iot " model=web sessionID=762 10/27 10:20:25.823794 00015939 TAOS_ADAPTER debug "start insert influxdb: pl1c,code=SSC131 pv=0.03125 1666837180000000000 pl1c,code=SSC141 pv=0.017578125 1666837180000000000 pl1c,code=SSC151 pv=74.53125 1666837180000000000 plc,code=SSC13 pv=0.03125 1666837180000000000 plc,code=SSC14 pv=0.017578125 1666837180000000000 plc,code=SSC15 pv=74.53125 1666837180000000000 " model=influxdb sessionID=763 10/27 10:20:25.832259 00015939 TAOS_ADAPTER debug "finish insert influxdb cost: 8.433258ms" model=influxdb sessionID=763 10/27 10:20:25.832388 00015939 TAOS_ADAPTER info "| 204 | 8.683968ms | 192.168.10.47 | POST | /influxdb/v1/write?db=iot " model=web sessionID=763 10/27 10:20:38.767790 00015939 TAOS_ADAPTER debug "start insert influxdb: pl1c,code=SSC131 pv=0.03125 1666837190000000000 pl1c,code=SSC141 pv=0.015625 1666837190000000000 pl1c,code=SSC151 pv=74.3359375 1666837190000000000 plc,code=SSC13 pv=0.03125 1666837190000000000 plc,code=SSC14 pv=0.015625 1666837190000000000 plc,code=SSC15 pv=74.3359375 1666837190000000000 " model=influxdb sessionID=764 10/27 10:20:38.775442 00015939 TAOS_ADAPTER debug "finish insert influxdb cost: 7.621327ms" sessionID=764 model=influxdb 10/27 10:20:38.775558 00015939 TAOS_ADAPTER info "| 204 | 7.843585ms | 192.168.10.47 | POST | /influxdb/v1/write?db=iot " model=web sessionID=764 10/27 10:20:48.952652 00015939 TAOS_ADAPTER debug "start insert influxdb: pl1c,code=SSC131 pv=0.02734375 1666837200000000000 pl1c,code=SSC141 pv=0.015625 1666837200000000000 pl1c,code=SSC151 pv=74.921875 1666837200000000000 plc,code=SSC13 pv=0.02734375 1666837200000000000 plc,code=SSC14 pv=0.015625 1666837200000000000 plc,code=SSC15 pv=74.921875 1666837200000000000 " model=influxdb sessionID=765 10/27 10:20:48.965678 00015939 TAOS_ADAPTER debug "finish insert influxdb cost: 12.995303ms" model=influxdb sessionID=765 10/27 10:20:48.965797 00015939 TAOS_ADAPTER info "| 204 | 13.222315ms | 192.168.10.47 | POST | /influxdb/v1/write?db=iot " model=web sessionID=765 10/27 10:21:00.047615 00015939 TAOS_ADAPTER debug "start insert influxdb: postgresql,code=LY52003_FI002,tname=LY52003_FI002 pv=2813.368 1666837260000000000 postgresql,code=LY52003_FQ002,tname=LY52003_FQ002 pv=165784672 1666837260000000000 postgresql,code=LY52003_LI002,tname=LY52003_LI002 pv=3.749 1666837260000000000 postgresql,code=LY52003_PI002,tname=LY52003_PI002 pv=0.367 1666837260000000000 postgresql,code=LY52006_FI001,tname=LY52006_FI001 pv=760.766 1666837260000000000
