initial rework of syntest to be usable internally

[keith-hall]

this is an initial rework of the syntest example, to make it usable internally, as it would be useful for parser tests (see https://github.com/trishume/syntect/pull/180#discussion_r198685953). I think it will need some more work to properly support assertions that extend over the end of the line (https://github.com/trishume/syntect/issues/175) and spill over onto not only the next line, but also the line afterwards etc. although such a scenario may never come up in real life use.

[keith-hall]

I have moved it to a new syntax_tests module and added a doc comment :+1: The main function has been modified; it used to read line by line, parsing as it went - now it reads all the syntax test assertion positions/details first, then parses line by line afterwards as a separate step and compares the results. In terms of using io::Write, great idea. I think it's a little beyond my skillset and free time at the moment, so it'd be great if someone else wouldn't mind contributing this. It will be necessary to also modify Util::debug_print_ops - perhaps it'd make sense to rename it debug_write_ops, take an io::Write as a parameter, and create a new debug_print_ops which will just pass in stdout() (or https://doc.rust-lang.org/std/io/struct.Stdout.html#method.lock (I'm not sure which is better)).

[robinst]

Nice, thanks for picking this up :)!

In the current form, I don't think we should make it a public API. An API with the only output being some printed text doesn't seem very useful generally.

So I would do either one of these two things:

• Change it to pub(crate)
• Move the printing parts back to the syntest example, and change the API to return plain structs with the information about the assertions instead.

(With the latter, we could do some nice things such as formatting as HTML so you can more easily see the actual scopes, show the results in a text editor instead, etc.)

[keith-hall]

Printed text isn't the only output - the API returns the successful/failed assertion count too ;)

I would guess that changing the API to return structs with the information about the assertions, and printing them from the syntest example would mean that we would lose the "instant" nature of the results. i.e. currently, one would see in the output about failures at the top of the syntax test file without having to wait for the whole rest of the file to be evaluated. I'd be keen to keep this, if possible. I like the idea of formatting the output, although I don't have any concrete ideas of how it could look at the moment. (Currently I just pipe the output of syntest into a file and review it in Sublime.) Maybe outputting to HTML (like synhtml), using a specified color scheme, and then showing the failures on top of that somehow?

[robinst]

Hehe :). Is it possible to make it return an Iterator with results? Then you could just iterate over that and print as the assertions come back.

[keith-hall]

I think that returning an iterator with the results is a cool idea, it would allow a consumer of the API to fail fast when a failure occurs, if so desired. Though I guess it would likely prevent the use cases we mentioned above, whereby the (optionally colored) output could be combined with the syntax test results, as it would only iterate over those results and then the file would have to be parsed again to achieve an integrated view, which isn't ideal. Also, it would mean all the consumers would have to track the number of failures and thus could cause code duplication when all they care about is the totals (for summary purposes). But using io:Write instead wouldn't magically enable this either, so maybe it's not a realistic goal at the moment?

Marking the API for internal use only would solve needing to make further complicated changes at the moment. But changing the method to pub(crate) seems to prevent the syntest example from seeing it, and I get warnings about dead code. Any ideas? :)

[trishume]

Okay I'll try and give the new implementation a real review then, likely this weekend. If you'd like I can probably also do the io::Write stuff after I do that.

@robinst I agree it's not the greatest thing to have as a public API but:

• there's unfortunately no pub(examples)
• making it be a nice proper API may be a decent amount of work.
• I've never been too stingy with public APIs in syntect and in this case if we implement a better API it should be pretty easy to reimplement it in terms of the new API, or do a major release and remove the old version (I doubt anyone will use it)

[trishume]

Sorry I'm so late on reviewing this, I wanted to use my limited syntect bandwidth for stuff that needed to be in the 3.0 release. This is still on my TODO list.

[keith-hall]

No worries, the work you did on the color schemes was awesome and indeed more important ;)

[Enselic]

@keith-hall Is this PR something we still want to try to get in?

In general I think it is undesired to add public API only for the purpose of testing, because each increase in API surface makes all subsequent bugfixes and features more complicated to do due to the mere fact that there are more things to take into account, so in general a minimal API surface is desirable.

Couldn't we simply keep the tests as mod tests that have access to internal interfaces?

I apologize in advance for any ignorance on my part.

[keith-hall]

although not mentioned in the original post, the idea behind moving the code from an example to be part of the public API is because running syntax tests isn't just useful for testing syntect itself, but also for people writing sublime-syntax files. If syntect were to be used in a text editor, for example, then users/syntax authors might expect to be able to test their sublime-syntax/syntax_test_ files directly, and it wouldn't make sense for projects depending on syntect to copy/paste or recreate that logic when it is ST compatible and thus IMO belongs in syntect.

[Enselic]

Thanks for the clarification, that makes sense. I will try to find the time to help review this in a not too distant future.