inlong
inlong copied to clipboard
[INLONG-10109][SDK] Support to transform from Json protocol to CSV/KV protocol by single SQL
[SDK] Support to transform from Json protocol to CSV/KV protocol by single SQL
Fixes #10109 Parent issue: #10022
Motivation
Support to transform from Json protocol to CSV/KV protocol by single SQL
Modifications
- Support to parse the following json string { "sid":"value1", "packageID":"value2", "msgs":[ {"msg":"value4","msgTime":1713243918000}, {"msg":"v4","msgTime":1713243918000} ] }
- Generate the final data by the following SQL select $root.sid, $root.packageID, $child.msgTime, $child.msg from source
- The final data is the following string value1|value2|1713243918000|value4 value1|value2|1713243918000|v4
Verifying this change
(Please pick either of the following options)
-
[ ] This change is a trivial rework/code cleanup without any test coverage.
-
[ ] This change is already covered by existing tests, such as: (please describe tests)
-
[ ] This change added tests and can be verified as follows:
(example:)
- Added integration tests for end-to-end deployment with large payloads (10MB)
- Extended integration test for recovery after broker failure
Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
- If a feature is not applicable for documentation, explain why?
- If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation