intel
Getting "Invalid UTF-8 character for UTF control characters'
Hi, While compiling expression along with utf 8 chars getting below error. If I remove HS_FLAG_UTF8 flag then it compiles fine. IS there any restriction for utf8 control characters? "bob logged in from "
code snippet if (hs_compile(test1, HS_FLAG_DOTALL|HS_FLAG_UTF8, HS_MODE_BLOCK, NULL, &database, &compile_err) != HS_SUCCESS) { fprintf(stderr, "ERROR: Unable to compile pattern "%s": %s\n", test1, compile_err->message); hs_free_compile_error(compile_err);
ERROR: Unable to compile pattern "bob logged in from ": Expression is not valid UTF-8.
Hyperscan supports UTF8 patterns in 2 ways: has utf8 flag set, or has (*UTF8) control verbs at the beginning of a pattern. Both ways will let Hyperscan know user intends to compile a UTF8 pattern. However, only the expression itself is a valid UTF8 string, can Hyperscan finally handles it in UTF8.
The following code just checks UTF8 validity of an expression body. FYI. https://github.com/intel/hyperscan/blob/64a995bf445d86b74eb0f375624ffc85682eadfe/src/compiler/compiler.cpp#L171
Thank you for the quick response! I have added HS_FLAG_UTF8 flag , and the UTF-8 control character, whether at the end, middle, or end, is giving me an invalid expression error.
if (hs_compile(test1, HS_FLAG_DOTALL|HS_FLAG_UTF8, HS_MODE_BLOCK, NULL, &database,
Can you please suggest how to proceed? Thank you!
Can you provide us the full test code? Better in .txt attachment. By simply copying the expression in your original question seems cannot produce the error.
Hi Can you please provide your comments over the sample code, the first byte of the char string is the utf 8 control-del character?
Hey sorry for being late. Your code should be fine, because the error comes from a bug in our utf8 validity function, where we mistreat 0x7f as an invalid one-byte utf8 case:
https://github.com/intel/hyperscan/blob/64a995bf445d86b74eb0f375624ffc85682eadfe/src/parser/utf8_validate.cpp#L74-L76 Should be "s[i] <= 0x7f" here.
Your first byte of char string happens to fall into the corner cases. We'll push the fix recently. You might currently do manually modification if needed.