tomlplusplus
tomlplusplus copied to clipboard
Non-ascii character may trigger "unreachable" parser state
Environment
toml++ version and/or commit hash:
toml++ v3.3.0 commit c635f218c0aefc801d9748841930365e54fe3089
Compiler:
g++ 10.2.1
C++ standard mode:
-std=c++17
Target arch:
x86_64
Library configuration overrides:
none
Relevant compilation flags:
-std=c++17 -O2 -fsanitize=undefined
Describe the bug
In certain very specific cases, a non-ascii character can force the parser into the TOML_UNREACHABLE
state in function is_non_ascii_horizontal_whitespace()
.
The conditions that cause this seem to be:
- TOML file contains a multi-line basic string
- and the multi-line string contains a line-ending-backslash
- and the first non-whitespace character on the next line is a non-ascii character in a certain codepoint range.
Steps to reproduce (or a small repro code sample)
The following TOML document triggers this issue:
s = """\
§"""
Note that the last character on the first line is a backslash. The first character on the second line is U+00A7.
The following C++ program also demonstrates this:
#include <string>
#include <iostream>
#include <toml++/toml.h>
int main()
{
std::string doc = "s = \"\"\"\\\n\xc2\xa7\"\"\"\n";
try
{
auto data = toml::parse(doc, std::string("tricky"));
std::cout << "great!!" << std::endl;
}
catch (const toml::parse_error& err)
{
std::cerr << "\n\n" << err << "\n";
return 1;
}
catch (...)
{
std::cerr << "\n\nAn unspecified error occurred.\n";
return 1;
}
return 0;
}
If I compile with -fsanitize=undefined
, this program prints
include/toml++/impl/unicode_autogenerated.h:44:13: runtime error: execution reached an unreachable program point
Without the -fsanitize
flag, the parser runs without errors and appears to parse the document correctly.
Additional information
Apart from the specific issue in the implementation of is_non_ascii_horizontal_whitespace()
, I also doubt whether it is in principle even correct to assign special significance to non-ascii whitespace. The TOML 1.0.0 specification talks about "trimming whitespace", which is open for interpretation. However, the ABNF grammar unambiguously specifies whitespace as being ASCII space and ASCII tab and nothing else. And this narrow interpretation of whitespace is applicable to the definition of mlb-escaped-nl
in the ABNF.
For example, in the following TOML document, toml++ parses s
as an empty string, while I think the correct answer is that s
contains a single character U+00A0:
s = """\
<U+00A0>"""
I'm experimenting with a TOML fuzzer which revealed this issue. Out of 1000 randomly generated TOML documents, toml++ now parses 985 documents correctly. The remaining 15 documents seem to fail on this issue. (This is a pretty good score; many other TOML parsers fail on the majority of random files.)
Sorry to bother you again with an obscure case. I understand if this gets low priority. But it may still be worthwhile to fix it eventually.