Define a sentence as a word* without a NEW_LINE at the end.

[benignbala]

Using 'phrase' for the header causes errors because the phrase itself has a NEW_LINE and the header was a phrase followed by a NEW_LINE. So, header became word* NEW_LINE NEW_LINE which is incorrect. Now, header is sentence NEW_LINE, which is then word* NEW_LINE.

[abelsromero]

I don't see any problem with it.

[benignbala]

Without this PR, it doesn't parse the header for simple_example.adoc. Thanks

PS: I did reply earlier through email, it somehow didn't reach here. :(

[abelsromero]

Without this PR, it doesn't parse the header for simple_example.adoc. Thanks

I was that, did you have a change to look into other issues? I really have problems understanding why words ending with punctuation are not correctly processed?

[benignbala]

@abelsromero Sure, I'll have a go at that :). Thanks

[benignbala]

A collection of changes that seem to work now :) Punctuations work, at least to some extent. Thanks

[abelsromero]

Some improvements really! I see that headers and author sections are parsed now, the simple-document.adoc is almost parsed :smiley: However, the punctuation issues persist, words ending in punctuation signs fail. Note: by now, ignore the issue in haven't

I think it is because the tokenizer symbol WS : [ \t\r]+ -> skip ;, this is just wrong word : CHARS PUNCTUATION? | CHARS SPLPUNCTUATION CHARS; It expects a space between the letters and the trailing punctuation. I fear we need to define a special symbol that concatenates letters and sings.

[benignbala]

@abelsromero Thanks :) BTW, is this word: CHARS PUNCTUATION? | CHARS SPLPUNCTUATION CHARS; is needed because of words that have a punctuation like "haven't" ,"it's" and "cross-legged" that have a punctuation in the middle of a word. And like you noticed, it seems to be parsing punctuations fine if there is a space between the word and the punctuation. Will check what's going wrong there. Thanks