Design Suggestion: Reconsider eml.ml approach

[sidkshatriya]

(Interestingly on the issue with ocaml-lsp not working well in Dream, someone else has already filed a ticket https://github.com/aantron/dream/issues/155 )

Thank you for this important and substantial open source contribution to the OCaml ecosystem! Thanks especially for the great documentation and many many examples too!

I had some comments on the *.eml.ml in the Dream framework:

• I personally place a lot of emphasis on the LSP server “understanding” everything in the ml file. I will gladly pay a small price in using a templating language that places nice with the OCaml LSP server. My guess is that a lot of beginners would prefer to have a LSP server that does not stop working in certain sections of their code – that is likely to be confusing and a net negative for them. Yes, using something like tyxml is a few extra steps but to me is a still worth the effort. The average OCaml beginner tends to be reasonably sophisticated and should be OK with tyxml. As you already would know – tyxml [%html] section allows variable interpolation and ocaml lsp server continues to work properly. Even in tyml jsx, ocaml-lsp-server works.
• While you gain some simplicity in source code using Dream specific templating you need to have a special dune rule to convert the eml.ml → ml which affects some of the beginner friendliness that I guess you’re aiming for

In brief, while I see your point in using the eml.ml approach I think the gains are mixed at best. I also notice that you have painstakingly pointed out that the eml.ml approach is purely optional and never compulsory anywhere. But the point is that this pattern is everywhere on the Dream website so it seems like it is the recommended approach.

My personal suggestion would be to put the tyxml approach front and center in the Dream framework examples. It plays nice with the ecosystem and does not add a “quirk” or unnecessary customization. People value cohesiveness with the broader OCaml ecosystem. Alternatively, dream framework can add its own tyxml like ppx.

What is your opinion about this design suggestion? Would be awesome to understand your opinion on this issue.

BTW Its cool if you feel that the current approach makes sense. In that case please feel free to close this ticket. I just wanted to spark a discussion as I feel that it might be too late to make such a significant change in the future.

P.S. I also added this point on https://discuss.ocaml.org/t/dream-a-simple-yet-feature-complete-web-framework/7909/8 but that was an old-ish thread so its possible you haven't seen it.

[aantron]

I personally place a lot of emphasis on the LSP server “understanding” everything in the ml file.

The main suggestion here in the short term is to use .eml.html files, as I also just suggested in the related issue https://github.com/aantron/dream/issues/155. I don't know yet how to make this more prominent in the examples, as .eml.ml files are just too convenient and concise not to use them in most examples :)

The average OCaml beginner tends to be reasonably sophisticated and should be OK with tyxml.

I've had feedback both in agreement and not with this statement, including some from beginners that would have been intimidated if examples used mainly TyXML. The examples are also not aimed at an average OCaml beginner as at least I imagine them to be today. The examples are partially aimed at beginners that OCaml might have, if OCaml libraries were easier to use than they tend to be, and required less sophistication to get started with (I don't fully agree with the choice of "sophistication" for the meaning of my sentence here, it's a bit of a different meaning from that in the sentence I am quoting).

In addition, the examples are aimed at users like me (not beginners). I prefer to use "literal" templates to TyXML. Note that I am not in any way "against" TyXML, and have been a user of it and contributor to it as well.

While you gain some simplicity in source code using Dream specific templating you need to have a special dune rule to convert the eml.ml → ml which affects some of the beginner friendliness that I guess you’re aiming for

This is true. In comparison with using TyXML, however, TyXML still requires a (preprocess ...) stanza. It's not as ugly as the eml rule, but it partially offsets the annoyance of editing dune files for eml.

[cemerick]

This is the sort of thing that is particularly susceptible to bikeshedding — everyone has at least 3 opinions about [HTML] templating — but after spending some nontrivial amount of time doing a little bakeoff of my own between the various approaches (the "endorsed" eml option, tyxml "classic" (i.e. direct usage of its API in my OCaml sources), the tyxml ppx, and the tyxml jsx ppx) in the process of porting a small rails app over to dream, I thought I'd put my 2¢ in anyway.

At the end of the day, I can't say enough good things about using tyxml via its jsx ppx. While it does unfortunately imply mixing reason syntax into what is otherwise an OCaml project, it does avoid the pain I feel in pushing HTML into strings (as demanded by the "regular" tyxml ppx), and makes it incredibly easy to move markup around between prototyping tools, existing content, online documentation/stack overflow, etc. Meanwhile, it is all just regular reason/OCaml, and so vscode facilities like code completion, navigation, refactoring, doc lookup, etc all work as expected.

In the end, while I appreciate the ease of eml, its tooling incompatibility is basically a showstopper IMO and the interleaved-code approach encouraged by % lines is a trap I don't intend on falling into again (did I mention I was porting from a rails app with erb templates?). In contrast, the tyxml jsx ppx gives me all the goodness of "literal" templates as you point out, but without limiting access to quality tooling, and preserving all the structured programming facilities I expect in OCaml/reason.

Really appreciate all the spectacular work here, and your interest in engaging with feedback. <3

</2¢>

[sidkshatriya]

Thanks for documenting your experience!

I assume you're referring to https://ocsigen.org/tyxml/latest/manual/jsx right? Just so I understood you correctly, one would need to write your template code into a .re file, correct? Or are you envisaging another way of using the jsx ppx?

(And yes, the jsx syntax is very nice and intuitive. However I've only seen it be used in reason files).

[cemerick]

@sidkshatriya Yes, sorry, I should have linked to the jsx ppx in my message.

And yes, the JSX does need to be in .re reason files. Thankfully, you can freely mix reason and OCaml files within the same project (at least when using dune, I'm not familiar with other toolchains), so aside from the one additional build-time dependency to pull in refmt (which dune uses automatically in the course of its coordination of builds), there is no special configuration or friction doing so.

I do keep those reason files pretty strictly templating-only, leaving all of the types, app functionality, and even view-related utilities in OCaml. This naturally led to an almost controller/view feel to the codebase, e.g. for a blog, I might have a blog_html.re file containing all of the JSX templates, and then a blog.ml file containing everything else. In a couple of spots where there are some custom types that templates needed to be aware of, I have pushed those into a separate file (e.g. blog_types.ml), but that's a wrinkle I can live with.

At the end of the day, I would love to have JSX syntax available in OCaml files, but outside of falling back to cppo (which is probably a dead end), that's probably never going to happen. Though I never tried it (again, really not a fan of HTML in strings in general), https://github.com/cxa/ppx_bsx does exist if that's something that works for you.

[tmattio]

Hi!

I'm a maintainer of the OCaml VSCode extension and a contributor of the LSP server, so I wanted to give some perspective on supporting .eml files from there.

While we can certainly add syntax support in the extension for .eml files (to have the syntax highlighting for both OCaml and HTML, not one or the other as is currently the case with the extension workaround), having full support for .eml file, with error reporting, is unlikely to happen as it would require support for the syntax all the way down to Merlin. The good news is, once Dune 3.0 is release, it will be integrated in the LSP server, so Dune errors (as opposed to Merlin errors) will be reported in the source code as well. Unfortunately, it won't be as exhaustive as Merlin: only the first error of the file will be reported, as Dune (and the compiler) are only able to give the first error.

I believe the experience once the syntax is supported and Dune 3.0 is integrated in the LSP will be acceptable. But of course, it will be missing important features such as code completion, code actions, etc.

If we want to have these features, we have two options:

• Using PPX for .eml files instead of a custom file generator.
• Adding support for eml syntax in Merlin.

Note that we are currently going through the application period for the Winter Outreachy session. One of the project is to add support for .eml in the extension (along with other syntaxes if time allows). So if we find an intern, this should come soon.

[sidkshatriya]

@tmattio Thanks for detailed note.

Unfortunately, it won't be as exhaustive as Merlin: only the first error of the file will be reported, as Dune (and the compiler) are only able to give the first error.

Can you give an example of this? Lets say I have a snippet of eml in my OCaml file. Right now what merlin does is that it gives me a lot of errors where the eml is but then "recovers" and does an acceptable job for the code following the eml snippet. This makes emls painful but not entirely a showstopper.

Once dune 3.0 comes out will this behavior regress? Will I continue getting helpful pointers from the the lsp later in my file? In other words, the new behavior of dune 3.0 + merlin is not clear to me in your note. Also isn't dune already integrated with merlin via the dot merlin reader?

[tmattio]

Once dune 3.0 comes out will this behavior regress?

No, the integration with Dune 3.0 will only provide additional diagnostics (coming from the compiler), but will not interfere with the current Merlin diagnostics.

I'll let LSP maintainers communicate on this when they are ready, I just wanted to give some ideas of what the solutions could be to support .eml files in the editors going forward, and a heads up that the syntax will probably come in VSCode in the coming months 🙂

[faldor20]

Would it perhaps be possible to split the file before serving to the LSP/merlin? I believe this is how Razor pages in c# work as described here https://devblogs.microsoft.com/aspnet/new-experimental-razor-editor-for-visual-studio/

[faldor20]

I'm working on a solution to this, as far as language server support. Should have a prototype available soon.