[Bug] Unexpected behavior when creating Doris routine load with whitespace in jsonpaths
Search before asking
- [X] I had searched in the issues and found no similar issues.
Version
1.2.7.1
What's Wrong?
When creating a Doris routine load, if the jsonpaths in the PROPERTIES section is written with whitespace around the items, the routine job can be created successfully. However, it would be changed to the PAUSED status because of an error shown by the show routine load for xxx_job command: Reason: All fields is null or not matched, this is an invalid row.. src line. The error message is a bit confusing and not quite helpful to locate the problem.
What You Expected?
The expected behavior should be either to raise an error while creating the routine load job, or to trim the whitespace in the jsonpaths items automatically which would not cause the error: Reason: All fields is null or not matched, this is an invalid row.. src line..
How to Reproduce?
Here is the SQL statement to reproduce the issue:
CREATE ROUTINE LOAD condition_log_job ON condition_log WITH APPEND COLUMNS TERMINATED BY ",",
COLUMNS(
col1,
col2,
col3,
col4,
col5,
col6
) PROPERTIES (
"desired_concurrent_number" = "5",
"max_error_number" = "0",
"max_batch_interval" = "10",
"max_batch_rows" = "200000",
"max_batch_size" = "104857600",
"format" = "json",
"jsonpaths" = "[\" $.col1 \",\" $.col2 \",\" $.col3 \",\" $.col4 \",\" $.col5 \",\" $.col6 \"]",
"strip_outer_array" = "false",
"num_as_string" = "false",
"fuzzy_parse" = "false",
"strict_mode" = "false",
"timezone" = "Asia/Shanghai",
"exec_mem_limit" = "2147483648"
)
FROM
KAFKA (
"kafka_broker_list" = "kafkahost:port",
"kafka_topic" = "some_kafka_topic",
"property.kafka_default_offsets" = "OFFSET_BEGINNING",
"kafka_partitions" = "0, 1, 2"
);
Anything Else?
While troubleshooting this issue, I found that the error message Reason: All fields is null or not matched, this is an invalid row.. src line was not very helpful in identifying the problem. It would be beneficial if the error messages could provide more specific information about what caused the issue. In this case, indicating that the jsonpaths items should not contain leading or trailing whitespace would have been very helpful.
Additionally, it would be great if the system could automatically trim the whitespace in the jsonpaths items during the creation of the routine load job. This would prevent such issues from occurring in the future and improve the user experience.
I hope these suggestions are helpful. Thank you for your attention to this matter.
Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
Code of Conduct
- [X] I agree to follow this project's Code of Conduct