Roadmap for the next release

[Emm]

• What we have so far:
  • OCaml 4.1.0 compatibility
  • array of tables support
  • parsing error location fix (31df3b9c793e9015dab1a5eb76b3e12a93e26734)
  • convenience functions
• What isn't quite finished:
  • Oasis support (looks like the PR is conflicting)

As it is right now (whether or not Oasis support goes in) we could release as 2.2.0, since AFAIK there is no API breakage. Is there anything else we're planning on?

[mackwic]

Great initiative !

To answer your points:

• I like the "release early and often" moto, so I'm fully ok for the 2.2.0 release
• Oasis can wait. It still has some issues, I will fix them. It shouldn't be a new version as it's only a dev change
• I don't know how good we are relatively to the last Toml version support... does this point needs to improve ?
• I don't think that we can do much more without user feedback, which is another piece of cake

---

So, as there's few things to do in this roadmap, another question here is what's the vision of the project. Not a very easy one as toml is only a small useful little thing around a larger app code base. Thought, I want TOML to be the language of configuration of choice, and I want Toml to be the lib de facto installed by any smart ocaml programmer. I am tired of the hard ecosystem Ocaml and I see Toml as a wonderful opportunity to show the way of what a great lib could be (well, Ocsigen being another great one, thought we don't hear of them very much).

So, the goals as I see them are:

1. To prove that we are reliable with a nice test suite
2. To make the project attractive with a good documentation and nice stylesheets
3. To improve the project with user feedback so that we really become unavoidable

1. Reliability: Better testing

• scrape a large base of toml files on github and throw them at the parser as part of the test suite
• use the torture suite for toml to detect the most nasty errors
• Related: improve the displaying of the tests in OUnit2, to make them more like this, no reason that the node.js cool kids keep the nice terminal formatting for their own eyes. :)

2. Attractiveness: Better documentation

Now, the documentation is hosted at sagotch's home, with the ugly base ocaml stylesheet.

• using the github pages will be far better (adapted domain + versioned + handle the load very well)
• using a better stylesheet like the ocsigen one
• point to try ocaml by ocamlpro and other tutorials in the README: don't assume that the reader is an Ocaml expert
• provide a bit more code samples and How-to (especially how to load Toml in the REPL), a FAQ, all with a call to ask. Though there's not much need for that two last, it creates a link with the user, brings confidence and help the user getting in touch with us.
• A section They use Toml will be also good. It's a win-win for the guys featured and for us (us: credibility, them: visibility, awareness)

3. Being the best: Improving via feedback

This is the trickier one. Especially in the ocaml world. Here are my proposals

• first, @Emm, i should add you to the maintainers ML. Will do at the end of this post.
• stackoverflow ? I don't know if ocamlers use SO a lot. But stalking for the combination of Toml and Ocaml tags should point bad errors. The kind we really don't want to happens.
• tell the user we love him: everywhere ! Add something like "You use Toml ? contact us ! Tell us what you like, what you dislike, or just say hello !" on source file header, on the README, on the description of the project. We can't be good if we don't know what's good or wrong for the user.
• communication: post to ML/news portals to say "hey there ! We exist !"
  • What do you think of the Ocaml ML ?
  • Is Ocaml Weekly still a thing ?
  • How can we be featured by github on the ocaml language ?
  • Facebook like a lot Ocaml and are kind OSS fellow, maybe we can get in touch ?
  • I will do a "Show HN" soon on Hacker news, they like Ocaml.
  • LinuxFr is also a nice place, with known french people like gasche.
  • I any more ideas of where to post, please suggest.

4. Next step: the killer features

Killer features takes time to implement and can be hard to test. That's why we should ask the users about what would be the real killer feature for them, so that we can priorize, or completely avoid an idea which seemed nice but was in fact useless (or dumb, it happens often with me).

Some ideas:

• A fully-featured configuration module deeply integrated with Toml. Basically it handles configuration, and has two method load and dump which use Toml to write or parse the backing file.
• Integration with Batteries:
  • types: provide a #to_enum function ? Ability to use the batteries type rather than the ocaml primitives ? Store in a BatMap (I love this name) rather than in a hash ?
  • implement the base interface for collections ? (it seems to have changed to a monadic interface)
• Integration with Core lib:
  • Core types ? I more a Batteries user, but there should be some type to integrate with
  • Core Async IO ? should not be difficult to integrate with...
• Automatic toml serialization/deserialization: in the vein of what yojson do coupled with atdgen
• By the way there should be some ideas to steal to the great Yojson project

---

Yeah, so that was my brain dump of my great plans for this project. It's obviously a long term vision, we don't have to do it now. You will note that there's few feature request, it's because we already have most of them and it's more priority to gains visibility and traction that add feature blindly (unless of course, you need this feature. In this case it's not blind anymore ;) ). It's also a bit megalomaniac, but it's part of the fun. Nothing to lose here, but attracting people to a more dynamic ocaml would really be a great win for everyone.

Like always, comments, suggestions, critic, crazy ideas most welcome.

[mackwic]

@Emm you should have received the ML mail by now, tell me if it didn't worked or if you want to change the email. It's rarely used anyway.

[Emm]

@mackwic, I have rescued the email from the spam folder and added a filter on the two addresses, so yes, it's all good.

I think we are already pretty much unavoidable if you want to do at least semi-complex configuration handling in OCaml and you don't like XML. The other options I'm aware of are config-files (which looks OK for simple cases) and ocaml-inifiles (the least said about it the better). That said, more traction is always good.

Reliability

• I'm all for having more test cases, especially for parsing. So yes, let's scrape it all.
• I wasn't aware of the "Toml torture suite". Is this https://github.com/BurntSushi/toml-test? It looks like it reads/writes JSON, which would mean writing code to convert our internal structures to/from JSON. It may be a worthwhile project on its own, but this is probably something you'd want to spin off in a separate library, and then plug the acceptance tests on this new "toml-json" library.
• Regarding OUnit2, I'm not sure what options we have to make them look prettier. I'd like to add that I have in fact contributed to worsen the reporting of failed tests, by refactoring the tests to use the public interface (which, as the types are abstract, prevents the use of the TomlInternal.Dumper pretty printer). I really think we should keep using the public interface, but we should do something about the dumper.

Attractiveness

• I agree on the use of Github pages (more centralization is good), at least until documentation gets integrated with Opam.
• I'd like to have a good way to version the documentation, though. It's handled automatically on Hackage, and it's very convenient. In any case, we should start using @since tags in the documentation (or at least whoever makes the release should do a quick pass to add them, since it can be tricky to know in advance what version number will be used).
• Related: we should add a changelog ("use git log" is not a changelog).
• Stylesheet; I'm partial to the stylesheet used by Daniel Buenzli - see here for instance: http://erratique.ch/software/xmlm/doc/Xmlm. What do you think?
• Documentation: I've tried to improve it, but I agree it's not perfect. In particular, it's weak on tricky things like operations on arrays.
• A short introduction to OCaml with links would be a good thing
• "Who uses Toml?" Well, good question. I use it in a side project, but it's not really something worth opening. On Opam, there is a reverse dependency on ketrew, which is apparently a workflow engine. Oh, and Simon Cruanes is apparently using it.

Improving via feedback

• I'm fine with encouraging the user to get in touch
• HN: good idea, but it's a bit of a gamble. Getting to the front page is hard. Otherwise, there is reddit. r/ocaml is... well, much like the mailing list, low traffic. Otherwise, maybe r/programming, but it's also a gamble
• Facebook: could be an idea, but I don't know who we should get in touch with. Otherwise, maybe Jane Street, for a future version of RWO? It would fit in the "cookbook" part of the book.
• Stackoverflow: doesn't seem to be that much activity (certainly nothing tagged both ocaml and toml)
• Haven't been on linuxfr for years, but I could do a writeup on Toml and how to read it on OCaml. It's the kind of thing the French Linux Magazine may be interested in, too. Maybe Linux Voice as well?

Killer features

• Automatic serialization/deserialization: have you gotten a look at https://github.com/whitequark/ppx_deriving_yojson ? I think ppx_deriving is the future.
• Integrating with $standard_libraries: right now we integrate with the "default" stdlib. I'm not sure we would gain that much in practice by adopting one of the two main standard libraries (or containers), considering that the v2 interface hides everything away from the user. Even with functorizing a hard dependency away, I'm not convinced it's worth it.
• What I've been thinking about:
  • ppx_deriving_toml
  • The ability to do something like Toml.as_float_array (Toml.path ("foo" </> "bar")) (could be tricky if there are arrays in the way)
  • A zipper interface to navigate and manipulate Toml data (probably tricky since our data structures mix maps and lists, but I haven't thought about it much)

That said, we're now feature complete. Which doesn't mean there are no corner cases, but we may not want to complicate the current library too much and instead have toml-based projects as separate Opam projects. However, it would require (IMHO) exposing TomlInternal. Without this, it's impossible to do introspection on a Toml data structure without try-catching each method in Toml.Value.To.

[mackwic]

Quick reply

reliability / tests

• Scraping: see Issue #18
• BurtSushi/toml-lang test suite: Yes it's this one. Doesn't seems too scary, right ? :) Issue #19
• nice test reporting: I will contribute to OUnit to add different reporters. The work is already half-done.
• What's your exact issue with the dumper ? Could it be fixed with ppx_deriving_toml deriving show ?

Attractiveness

• Github pages: see Issue #20
• Pushing the doc on the gh-pages branch would make it versioned, right ? Is it ok ? Agreed for the @since (issue #21)
• Changelog: Issue #22
• Stylesheet: it's too tiny, to blank, to sad ! At least for my taste. Don't you like the colorfulness of the Ocsigen project ?
• Doc: herm... An issue would be an understatement as we don't know the what and the how. I let the discussion open here for now.
• Beginers friendliness: see Issue #23
• Who use Ocaml: see Issue #24

Improving via feedback

• reddit is a good idea ! I think we're not ready for HN or reddit yet, but linuxfr could do it. No idea for the other linux mediums. I have also an account, we can co-redact a news on the 2.2.0 release. ;)
• facebook: I think we could just drop a mail to some ocamlers at facebook and simply ask them. A bit too soon maybe, but not too much.
• Jane Street: excellent idea ! I don't know for RWO, though. Maybe just ask them what they think of the lib on irc would be good for now.

killer features

• Automatic serialization/deserialization: Is that what's used by Eliom and Core ? But yeah, it's the idea. Seems easy to work with. See Issue #25
• Integration with $standard_library: I sightly disagree, it would be very nice to integrate with Core's Async IO. Also containers are a great thing. But yeah, very low priority.
• Toml.as_float_array: I didn't understood what you want to do.
• zipper interface: very nice idea ! Low priority, though. I think it's nice but I don't see the real world use case... Do you think of any ?

[mackwic]

Which doesn't mean there are no corner cases, but we may not want to complicate the current library too much and instead have toml-based projects as separate Opam projects. However, it would require (IMHO) exposing TomlInternal. Without this, it's impossible to do introspection on a Toml data structure without try-catching each method in Toml.Value.To.

I was thinking of a hidden lib with only TomlInternal and frontend lib (Std, Deriving, Bat....) which do the correct mapping... Maybe too hard to maintain but I like the idea...

[sagotch]

I answered on some opened issues, but for the points that do not have an issue tracking them:

Reliability

Better looking tests reports does not really matter to me, but why not.

Documentation

I still prefer the ocamldoc base stylesheet. Also, it looks familiar to OCaml users.

Short tutorial on how to use the library could be useful, and will not be a bad thing anyway. Maybe it could be extracted from a real project, or we could create a dummy one just to use the API.

Communication

• stack overflow stalking? If there is a possibility to automatically track SO with these tags with the support ML, we should do it.
• Tell the user we love him everywhere. Everywhere is to much for me. ("LEAVE ME ALONE!"). README is a good place to put this information, but I really think source files are not.

Killer features

• Zipper: what is the point of using zippers instead of navigating with the Map interface?
• Batteries/Core support: add maintainer work, no idea if it worst it. Not opposed to this idea, not convinced by it either.

TOML v0.3

Before adding new stuff to the library, we should handle changes in TOML standard since its 0.2 version. Don't you think?

[mackwic]

Great input !

Toml v0.3: it seems like few parsers have implemented it. Yeah, it should be a priority as it bring very nice things. Do you think we should/can keep the dates v0.2 compatible ? The date thing is going to change a lot, RFC 8601 dates are not the best for a "minimal conf language"...

Reliability

• Yeah, it's really because I want to make things nicer. Hey, everyone use its free time like they want, righ t? ;)

Documentation

• stylesheet: wow ! Didn't thing you'd thought that way. What do you think of the alternative brought by @Emm ?
• Little project giving example about how to use: good idea !

Communication

• So tracking is possible. ref, but no toml tag is available. Ocaml with manual filtering will be a hassle :(
• Everywhere is too much => :) . That's why we are 3 on the ML. And if things improve, we'll be even more. You don't have to do support if you don't want to, no problem.

Killer Features

-> That's why we should ask the user about it.

[Emm]

Reliability

• Regarding the dumper, it was used as pretty-printer for OUnit, and I assume printf debugging for Toml developers. At the moment, I unplugged it from OUnit (which therefore has no pretty printer anymore) since the public API (which we now use in our tests) does everything with abstract types, while the dumper expects our internal types (from TomlInternal). So at the moment it's essentially useless, and we don't have a working pretty-printer

Attractiveness

• The Ocsigen CSS is way too flashy for me, indeed. But it's not something I want to argue about a lot

Killer features

• Zippers: @sagotch, there is a shining example of why "just use the Map interface" isn't that great in practice in our code. It's the convert function in the parser. It turns the Hashtbl.t we use to build the structure when parsing the code into proper Map structures. The Map interface is sufficient to manipulate a simple map. Unfortunately, we have potentially nested maps and lists, that is, a hierarchy. Which means that changing a deeply nested value is bloody annoying: you need to put the matryoshka dolls back into each other by hand. On the other hand, with ppx_deriving_toml, it may not be a big issue, as the user could convert Toml structure back and from business objects and deal with the headaches herself.
• Toml.as_float_array: @mackwic, the idea would be to have a kind of very simple, type-safe "xpath" expression to get a nested value without all the piping we do at the moment. Again, may not be necessary with ppx_deriving_toml.

[sagotch]

The Everywhere is to much for me. ("LEAVE ME ALONE!") point was from a user point of view, not from a maintainer one. It makes me feel like begging for some affection. But, after all, I will probably don't even notice the sentence in source files, so I am not really opposed to it. :)

Which means that changing a deeply nested value is bloody annoying

Yes and now. The add method remove any previous binding of the key, so it is only boring in the way that you need to make multiple add calls to replace a value. Is it that bad? How would zipper help us here? (We should handle it for the user: split the key, and make nested calls as needed).

For the CSS thing. I find the original one less attractive but a lot more efficient for reading.

[Emm]

Let's assume we have a foo.bar.barbar table, and we want to update its key to new_value.

Zipper-less solution:

let barbar_table = Toml.get_table (Toml.key "foo") root_table |> Toml.Table.find (Toml.key "bar") |> Toml.to_table |> Toml.Table.find (Toml.key "barbar") |> Toml.to_table
let barbar_table' = Toml.Table.add (Toml.key "key") (Toml.of_string "new_value")
let root_table_with_new_value = ... (* enjoy putting everything back in the root table - backwards, obviously *)

Zipper solution (with an imaginary interface):

let zipper_on_barbar = Toml.Zipper.zipper root_table |> Toml.Zipper.to (Toml.key "foo") |> Toml.Zipper.to (Toml.key "bar") |> Toml.Zipper.to (Toml.key "barbar")
let zipper_with_updated_key = zipper_on_barbar |> Toml.Zipper.replace (Toml.key "key") (Toml.of_string "new_value")
(* All done, go back to the top and get a Toml table back from the zipper *)
let toml_with_new_value = zipper_with_updated_key |> Toml.Zipper.root |> Toml.Zipper.to_table

[sagotch]

I did not even tried to compile, but I think the following function would handle addition/replacement in a nested table (modulo some corrections).

(** split : string -> Toml.key list *)
let deep_add table (keys : key list) value =
  let rec aux acc = match key with
    | [ key ]  ->
       List.fold_right
     (fun (k, v) (tk, tv)  -> (tk, Toml.add tv k v) )
     (List.tl acc)
     (fst (List.hd acc), Toml.add (snd (List.hd acc)) key value)
       |> snd
    | key :: q -> aux ((key, Toml.get_table key (List.hd acc)) :: acc) q
  in aux [("", table)] keys

[Emm]

Sure, you can do that via help functions, using a zipper is not the only way. The big advantage with a zipper is that you can easily work on a specific node (eg, to change several values in the same table) at little runtime cost and verbosity. But anyway, let's see if we can't have ppx_deriving_toml first. If the user turns the configuration file into a business object she will be responsible for updating this object, not us. I guess it would help with validation (an important topic we haven't talked about yet).

[cljoly]

@mackwic Another solution, maybe easier to implement than Atdgen would be to inspirate from MassD/bson like this

let user = {
  name = "Joe"
  value = 5;
} deriving (Bson_ext);

[redianthus]

I'm closing as this is old and most stuff discussed here is not applicable to the current state of the library. Feel free to open separated issues for specific improvements.