NIFI-10234 Implement PutIoTDB

[xuanronaldo]

PutIoTDB

Properties of PutIoTDB

property	description	default value	necessary
Host	The host of IoTDB.	null	true
Port	The port of IoTDB.	6667	true
Username	Username to access the IoTDB.	null	true
Password	Password to access the IoTDB.	null	true
Record Reader	Specifies the type of Record Reader controller service to use for parsing the incoming data and determining the schema.	null	true
Schema	The schema that IoTDB needs doesn't support good by NiFi. Therefore, you can define the schema here. Besides, you can set encoding type and compression type by this method. If you don't set this property, the inferred schema will be used. It can be updated by expression language.	null	false
Aligned	Whether using aligned interface? It can be updated by expression language.	false	false
MaxRowNumber	Specifies the max row number of each tablet. It can be updated by expression language.	1024	false

Inferred Schema of Flowfile

There are a couple of rules about flowfile:

1. The flowfile can be read by Record Reader.
2. The schema of flowfile must contains a field Time, and it must be the first.
3. The data type of time must be STRING or LONG.
4. Fields excepted time must start with root..
5. The supported data types are INT, LONG, FLOAT, DOUBLE, BOOLEAN, TEXT.

Convert Schema by property

As mentioned above, converting schema by property which is more flexible and stronger than inferred schema.

The structure of property Schema:

{
	"timeType": "long",
	"field": [{
		"tsName": "root.sg.d1.s1",
		"dataType": "INT32",
		"encoding": "RLE",
		"compressionType": "GZIP"
	}, {
		"tsName": "root.sg.d1.s2",
		"dataType": "INT64",
		"encoding": "RLE",
		"compressionType": "GZIP"
	}]
}

Note

1. The first column must be Time. The rest must be arranged in the same order as in field of JSON.
2. The JSON of schema must contain timeType and fields.
3. There are only two options long and string for timeType.
4. The columns tsName and dataType must be set.
5. The tsName must start with root..
6. The supported dataTypes are INT32, INT64, FLOAT, DOUBLE, BOOLEAN, TEXT.
7. The supported encoding are PLAIN, DICTIONARY, RLE, DIFF, TS_2DIFF, BITMAP, GORILLA_V1, REGULAR, GORILLA.
8. The supported compressionType are UNCOMPRESSED, SNAPPY, GZIP, LZO, SDT, PAA, PLA, LZ4.

Relationships

relationship	description
success	Data can be written correctly or flow file is empty.
failure	The shema or flow file is abnormal.

[exceptionfactory]

One additional note, Apache NiFi just released version 1.17.0, so the current main branch is now on 1.18.0-SNAPSHOT. Please rebase and update to the current version.

Thanks!

[xuanronaldo]

Ok, I got it. Thanks for your suggestions. I'm working on it.

[xuanronaldo]

I have pushed the code, and all checks have passed. What should I do next to merge the branch?

[exceptionfactory]

Thanks for making the updates @xuanronaldo. Merging requires approval from at least one committer. I will try to take a look at the latest version soon.

[xuanronaldo]

Thanks for making the initial changes @xuanronaldo.

I took a closer look at the implementation and noted a number of areas for improvement. The division of the abstract class and concrete class is helpful to indicate the core functionality. The core logic in onTrigger was difficult to follow at points because of the layout and nesting, so some general refactoring would be helpful.

Thanks for your suggestions. I will update the code asap.

[xuanronaldo]

I have merged the main branch to resolve the problem which happened in workflow. Please approve running workflows agin. @exceptionfactory

[xuanronaldo]

All checks have passed, could you please take a look at my latest commit? @exceptionfactory

[xuanronaldo]

Okay, I got it. I will update the code asap. Thanks for your suggestions.

[xuanronaldo]

Hi, @exceptionfactory . Sorry about committing the code so late. I got lots of work to do, recently. Besides, I have updated a document about IoTDB Schema Template in our website as you requested. The link of it was added into the description of Schema Template.

[xuanronaldo]

Yep, you are rigth @exceptionfactory . The template in IoTDB is a structure to reduce the memory usage. However, the Schema Template here is just an adapter to define the schema in IoTDB. If you want to create schema by native way, the DDL or coding is necessary. Therefore, I designed the Schema Template to make it.

[xuanronaldo]

I have reverted these changes. However, several files were still marked for modification. I have no idea what to do. @exceptionfactory

[exceptionfactory]

I have reverted these changes. However, several files were still marked for modification. I have no idea what to do. @exceptionfactory

I see the files are still marked as changed. If it is necessary to squash all current changes in order to remove the merge commits and avoid including these files marked as changed, that would be fine too. Either way, the latest version should just reflect the new components.

[xuanronaldo]

I have reverted these changes. However, several files were still marked for modification. I have no idea what to do. @exceptionfactory

I see the files are still marked as changed. If it is necessary to squash all current changes in order to remove the merge commits and avoid including these files marked as changed, that would be fine too. Either way, the latest version should just reflect the new components.

I have no idea about squash all current changes. Or I can create a new branch and submit another PR?

[exceptionfactory]

Thanks for following up @xuanronaldo. Sure, if it is easier for you to create a new branch with the current version, that's fine as well. It is easy to link to this PR for background, and we can work from the new PR.