NIFI-10210 Processor supporting Apache iotdb database

[CloudWise-Lukemiao]

Iotdb process supports SQL insertion, session insertion and SQL query

[CloudWise-Lukemiao]

Thanks for the contribution @CloudWise-Lukemiao!

The implementation appears to use both the JDBC and Native Java library for interacting with IoTDB. JDBC access is already possible using existing components and leveraging the DBCPConnnectionPool or HikariCPConnectionPool services.

The IoTDB documentation for JDBC notes that using JDBC is not recommended for high throughput, so there could be value in new NiFi components that leverage the Java Native API.

With that background, the PutIoTDB processor includes a direct dependency on JSON input, which is problematic in terms of flexibility. It would be much better to evaluate an approach that leverages record-oriented processing using NiFi Record Reader services. It may also be useful to create a Controller Service that provides reusable IoTDB Sessions, which could support multiple processors.

If you are willing to work through more of these details, discussing the options and general design on the associate Jira issue would be a great next step.

Thanks for the contribution @CloudWise-Lukemiao!

The implementation appears to use both the JDBC and Native Java library for interacting with IoTDB. JDBC access is already possible using existing components and leveraging the DBCPConnnectionPool or HikariCPConnectionPool services.

The IoTDB documentation for JDBC notes that using JDBC is not recommended for high throughput, so there could be value in new NiFi components that leverage the Java Native API.

With that background, the PutIoTDB processor includes a direct dependency on JSON input, which is problematic in terms of flexibility. It would be much better to evaluate an approach that leverages record-oriented processing using NiFi Record Reader services. It may also be useful to create a Controller Service that provides reusable IoTDB Sessions, which could support multiple processors.

If you are willing to work through more of these details, discussing the options and general design on the associate Jira issue would be a great next step.

Thanks for pointing out the problem. Next, I will change the query interface mode of JDBC to the session mode of iotdb, and I will remove the JDBC insertion mode. I hope to discuss the options and general design on the associate JIRA issue would be a great next step

[exceptionfactory]

Thanks for pointing out the problem. Next, I will change the query interface mode of JDBC to the session mode of iotdb, and I will remove the JDBC insertion mode. I hope to discuss the options and general design on the associate JIRA issue would be a great next step

Thanks for the response @CloudWise-Lukemiao, that sounds like a good next step. Including unit tests for a new processor is also important.

[CloudWise-Lukemiao]

Thanks for pointing out the problem. Next, I will change the query interface mode of JDBC to the session mode of iotdb, and I will remove the JDBC insertion mode. I hope to discuss the options and general design on the associate JIRA issue would be a great next step

Thanks for the response @CloudWise-Lukemiao, that sounds like a good next step. Including unit tests for a new processor is also important.

Thanks for pointing out the problem. Next, I will change the query interface mode of JDBC to the session mode of iotdb, and I will remove the JDBC insertion mode. I hope to discuss the options and general design on the associate JIRA issue would be a great next step

Thanks for the response @CloudWise-Lukemiao, that sounds like a good next step. Including unit tests for a new processor is also important.

hi,There is a problem I want to discuss. In addition to the data source, the putiotdb processor interface also needs a devicesid configuration (string), measurements (array), datatypes (array) and measurements, and batchsize (submitted quantity). As shown in the figure。

[exceptionfactory]

@CloudWise-Lukemiao There is some discussion on NIFI-10234, so working through the details on that issue would be helpful.

[exceptionfactory]

@CloudWise-Lukemiao as mentioned earlier, there is more implementation discussion on NIFI-10234 and ongoing work for a PutIoTDB Processor in PR #6416.

With that background, it seems best to close this PR for now. Feel free to provide feedback on the other PR, as there are still some issues to be resolved in terms for record field mapping.