seed
seed copied to clipboard
tindzk
YAML support?
I really like Seed (although I can't use it without #18), however I'm not sure I can use it with TOML. Especially with #18 and #19 the configuration forms a deep tree, and especially with many modules (which the README says it's designed for) it can be very wide too. TOML is nice for some things but I can't see it scaling well for me.
As an example, GitLab CI uses TOML for configuring the runner binary, but YAML for defining a project's various jobs and their configuration, and I think they both make sense. The former is basically a few sections of flat config, with a bit of nesting. The latter is a much more complicated tree of settings. You can have many jobs, each of which can have many settings, each of which might itself need to be defined with sub-properties.
I admit the TOML generated by seed init overwhelmed me more than it might have, because some of what it generated may have been unnecessary.
I found something online to convert TOML to YAML and was much happier with the result. So what I started doing is using yaml2toml (on Ubuntu, sudo snap install marshal) to translate my config for seed. Specifically, I ran echo seedbuild.yml | entr -rs "yaml2toml --preserve-key-order seedbuild.yml build.toml && seed bloop && bloop compile -w (bloop projects)" (this will run the part in quotes whenever seedbuild.yml changes via my shell, Fish).
I didn't get all that far in converting my decently-sized sbt project to seed (I stopped mainly due to #18), but here is how the two ways of defining it compare:
TOML
[project]
scalaVersion = "2.12.8"
scalaJsVersion = "0.6.28"
scalaOptions = ["-encoding", "UTF-8", "-unchecked", "-deprecation", "-Xfuture"]
testFrameworks = ["minitest.runner.Framework"]
[resolvers]
maven = ["https://repo1.maven.org/maven2", "https://jcenter.bintray.com", "https://jitpack.io"]
[module]
[module.sharedCommon]
root = "shared_common"
targets = ["jvm", "js"]
sources = ["shared_common/src/main/scala"]
compilerDeps = [["org.scalamacros", "paradise", "2.1.1", "full"]]
scalaDeps = [["cc.co.scala-reactive", "reactive-routing", "0.6.4"], ["com.github.cornerman.sloth", "sloth", "34b09cdccb"], ["com.github.julien-truffaut", "monocle-macro", "1.5.1-cats"], ["com.github.krzemin", "octopus", "0.3.3"], ["com.lihaoyi", "sourcecode", "0.1.7"], ["io.circe", "circe-core", "0.11.1"], ["io.circe", "circe-generic", "0.11.1"], ["io.circe", "circe-parser", "0.11.1"], ["io.github.nafg", "slick-additions-entity", "0.9.1.1"]]
[module.sharedCommon.test]
sources = ["shared_common/src/test/scala"]
scalaDeps = [["io.monix", "minitest", "2.5.0"]]
[module.volunteerShared]
root = "volunteer_shared"
targets = ["jvm", "js"]
sources = ["notes_shared/src/main/scala"]
moduleDeps = ["sharedCommon"]
[module.notesShared]
root = "notes_shared"
targets = ["jvm", "js"]
sources = ["notes_shared/src/main/scala"]
moduleDeps = ["volunteerShared"]
[module.doctorsShared]
root = "doctors_shared"
targets = ["jvm", "js"]
sources = ["doctors_shared/src/main/scala"]
moduleDeps = ["sharedCommon", "notesShared"]
[module.util]
root = "util"
targets = ["jvm"]
sources = ["util/src/main/scala"]
moduleDeps = ["volunteerShared"]
scalaDeps = [["net.liftweb", "lift-util", "3.3.0"], ["org.scala-lang.modules", "scala-xml", "1.2.0"]]
[module.modelsCommon]
root = "models_common"
targets = ["jvm"]
sources = ["models_common/src/main/scala"]
moduleDeps = ["volunteerShared"]
javaDeps = [["org.flywaydb", "flyway-core", "5.2.4"]]
scalaDeps = [["com.github.tminglei", "slick-pg", "0.17.3"], ["com.github.tminglei", "slick-pg_circe-json", "0.17.3"], ["io.github.nafg", "slick-additions", "0.9.1.1"], ["net.liftweb", "lift-util", "3.3.0"]]
[module.jsCommon]
root = "js_common"
targets = ["js"]
sources = ["js_common/src/main/scala"]
moduleDeps = ["sharedCommon"]
scalaOptions = ["-P:scalajs:sjsDefinedByDefault"]
scalaDeps = [["com.github.japgolly.scalajs-react", "ext-monocle-cats", "1.4.2"], ["io.github.nafg.css-dsl", "bootstrap3", "0.4.0"], ["io.github.nafg.scalajs-facades", "react-select_2-1-2", "0.6.0"], ["io.github.nafg.scalajs-react-util", "core", "0.7.0"]]
YAML
project:
scalaVersion: 2.12.8
scalaJsVersion: 0.6.28
scalaOptions:
- '-encoding'
- UTF-8
- '-unchecked'
- '-deprecation'
- '-Xfuture'
testFrameworks:
- minitest.runner.Framework
resolvers:
maven:
- https://repo1.maven.org/maven2
- https://jcenter.bintray.com
- https://jitpack.io
module:
# SHARED MODULES
sharedCommon:
root: shared_common
targets: [jvm, js]
sources: [shared_common/src/main/scala]
compilerDeps:
- ["org.scalamacros", "paradise", "2.1.1", "full"]
scalaDeps:
- ["cc.co.scala-reactive", "reactive-routing", "0.6.4"]
- ["com.github.cornerman.sloth", "sloth", "34b09cdccb"]
- ["com.github.julien-truffaut", "monocle-macro", "1.5.1-cats"]
- ["com.github.krzemin", "octopus", "0.3.3"]
- ["com.lihaoyi", "sourcecode", "0.1.7"]
- ["io.circe", "circe-core", "0.11.1"]
- ["io.circe", "circe-generic", "0.11.1"]
- ["io.circe", "circe-parser", "0.11.1"]
- ["io.github.nafg", "slick-additions-entity", "0.9.1.1"]
test:
sources: [shared_common/src/test/scala]
scalaDeps:
- ["io.monix", "minitest", "2.5.0"]
volunteerShared:
root: volunteer_shared
targets: [jvm, js]
sources: [notes_shared/src/main/scala]
moduleDeps: [sharedCommon]
notesShared:
root: notes_shared
targets: [jvm, js]
sources: [notes_shared/src/main/scala]
moduleDeps: [volunteerShared]
doctorsShared:
root: doctors_shared
targets: [jvm, js]
sources: [doctors_shared/src/main/scala]
moduleDeps: [sharedCommon, notesShared]
# JVM MODULES
util:
root: util
targets: [jvm]
sources: [util/src/main/scala]
moduleDeps: [volunteerShared]
scalaDeps:
- ["net.liftweb", "lift-util", "3.3.0"]
- ["org.scala-lang.modules", "scala-xml", "1.2.0"]
modelsCommon:
root: models_common
targets: [jvm]
sources: [models_common/src/main/scala]
moduleDeps: [volunteerShared]
javaDeps:
- ["org.flywaydb", "flyway-core", "5.2.4"]
scalaDeps:
- ["com.github.tminglei", "slick-pg", "0.17.3"]
- ["com.github.tminglei", "slick-pg_circe-json", "0.17.3"]
- ["io.github.nafg", "slick-additions", "0.9.1.1"]
- ["net.liftweb", "lift-util", "3.3.0"]
# JS MODULES
jsCommon:
root: js_common
targets: [js]
sources: [js_common/src/main/scala]
moduleDeps: [sharedCommon]
scalaOptions: ["-P:scalajs:sjsDefinedByDefault"]
scalaDeps:
- ["com.github.japgolly.scalajs-react", "ext-monocle-cats", "1.4.2"]
- ["io.github.nafg.css-dsl", "bootstrap3", "0.4.0"]
- ["io.github.nafg.scalajs-facades", "react-select_2-1-2", "0.6.0"]
- ["io.github.nafg.scalajs-react-util", "core", "0.7.0"]
Thanks for your thorough analysis!
In TOML, you can write inline tables such as:
module = {
sharedCommon = {
root = "shared_common",
# ...
test = {
# ...
}
}
}
With this syntax, the example is quite similar to your YAML file.
I chose TOML because it has a simple specification and is used by other build tools like Cargo. I fear that having an additional configuration format will burden development, as we would have to offer the same codecs twice and make sure errors are handled in the same way. Also, there has to be a native Scala implementation such that we can support Scala Native in the future.
As for seed init, feel free to open another issue with your suggestions how the output could be improved. I do agree with you it is a bit verbose.
Regarding #18, this will get implemented soon after I am done with some other improvements and once #17 has been merged in.
Are you sure that's valid? The docs say
Inline tables are intended to appear on a single line. No newlines are allowed between the curly braces unless they are valid within a value. Even so, it is strongly discouraged to break an inline table onto multiples lines. If you find yourself gripped with this desire, it means you should be using standard tables.
Have you considered using HOCON? That has the advantage of having ways to DRY things, as well as built in includes, support for environment variables, and other really useful features.
And there are pure scala implementations: https://github.com/akka-js/shocon and https://github.com/ekrich/sconfig
Are you sure that's valid? The docs say
Inline tables are intended to appear on a single line. No newlines are allowed between the curly braces unless they are valid within a value. Even so, it is strongly discouraged to break an inline table onto multiples lines. If you find yourself gripped with this desire, it means you should be using standard tables.
It appears that the TOML library implements a language extension that allows multi-line tables. The following build file loads fine:
project = {
scalaVersion = "2.13.0",
scalaJsVersion = "0.6.28",
scalaOptions = [
"-encoding",
"UTF-8",
"-unchecked",
"-deprecation"
],
testFrameworks = ["minitest.runner.Framework"]
}
module = {
example = {
root = "shared",
sources = ["shared/src"],
targets = ["js"],
js = {
root = "js",
sources = ["js/src"]
},
test = {
js = {
jsdom = true,
sources = ["js/test"],
scalaDeps = [["io.monix", "minitest", "2.5.0"]]
}
}
}
}
HOCON looks like a promising contender. I would have preferred if they offered fewer syntactic features to express the same concept, but the format is still much simpler than YAML. As I see it, HOCON has three main advantages over TOML which are JSON compatibility, value substitutions and access to environment variables.
Alternatively, we could use JSON directly. JSON has the advantage that there are decent Scala libraries, is in wide use and can be easily manipulated with tools like jq.
If you would like to implement additional support to load HOCON or JSON configuration files, I'd be happy to provide some guidance.
I'm playing with this locally. I basically just converted the circe Json AST to the toml.Value AST and keep everything else basically them same. It's not PR quality but I can show a diff.
HOCON and JSON could be done just by using circe-core or https://github.com/circe/circe-config instead of circe-yaml.
That's a nice solution since it acts directly on the AST level, so we get to re-use all of the TOML codecs. However, I'd prefer if we used HOCON (e.g. via sconfig or circe-config) since this format is more common in the JVM ecosystem and has a simpler specification.
I can do that when it comes to making a PR, like I said it doesn't change much about the approach. However I would want to experiment with using hocon first, however my experimenting is now kind of blocked on something, which I opened a different issue for (how to go about compiling twirl templates).
On Tue, Aug 20, 2019, 6:04 AM Tim Nieradzik [email protected] wrote:
That's a nice solution since it acts directly on the AST level, so we get to re-use all of the TOML codecs. However, I'd prefer if we used HOCON (e.g. via sconfig or circe-config) since this format is more common in the JVM ecosystem and has a simpler specification.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/tindzk/seed/issues/23?email_source=notifications&email_token=AAAYAUBNHXTZU7KIX3A3LSTQFO6SPA5CNFSM4IDT7MP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4VYMPQ#issuecomment-522946110, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAYAUEONR6BAFVWNW3D4C3QFO6SPANCNFSM4IDT7MPQ .
My personal opinion: TOML is the way to go, friendly, simple and easy-to-learn. I'd see supporting YAML as a regression:
TOML and YAML both emphasize human readability features, like comments that make it easier to understand the purpose of a given line. TOML differs in combining these, allowing comments (unlike JSON) but preserving simplicity (unlike YAML).
The simpler this tool is, the better. That means, less configuration and more opinionated workflows (IMO) :smile:
How heavily have you used Seed?
On Thu, Sep 12, 2019, 5:47 AM Jorge [email protected] wrote:
My personal opinion: TOML is the way to go, friendly, simple and easy-to-learn. I'd see supporting YAML as a regression:
TOML and YAML both emphasize human readability features, like comments that make it easier to understand the purpose of a given line. TOML differs in combining these, allowing comments (unlike JSON) but preserving simplicity (unlike YAML).
The simpler this tool is, the better. That means, less configuration and more opinionated workflows (IMO) 😄
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/tindzk/seed/issues/23?email_source=notifications&email_token=AAAYAUCWBJAIL76L4HX6SDTQJIF2VA5CNFSM4IDT7MP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6RJ6PQ#issuecomment-530751294, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAYAUE5YXAUPDSQFPITOELQJIF2VANCNFSM4IDT7MPQ .
Let me propose two more languages:
Dhall
Dhall appears to be a good fit for a build tool since it allows users to define 'variables' and functions. Furthermore, the language was designed not to be Turing-complete. The authors claim:
The language aims to support safely importing and evaluating untrusted Dhall code, even code authored by malicious users. We treat the inability to do so as a specification bug.
Dhall could allow us to avoid boilerplate as in #48. It supports importing Dhall code from URLs, so we can provide default configurations for common generators (such as Twirl templates or Play routes). Another advantage over the other formats so far is that it is fully typed.
I experimented with it a bit here: https://github.com/tindzk/seed-dhall-experiments/
Jsonnet
Jsonnet is already popular in DevOps. Its syntax is more approachable than Dhall's and there is a even a recent implementation for Scala. It comes with a standard library that contains common functions like split, join, length etc.
Unlike Dhall it is not statically typed, but since we already perform type checking within Seed, I do not think this is a problem.
Conclusion
Both languages look promising. They both generate JSON, so I think the best way forward is to add JSON support to Seed. While trying out Dhall, I did notice that our current schema is not the best fit for JSON, so we will need to optimise it without breaking compatibility with existing TOML builds.
I really like how simple https://github.com/tindzk/seed-dhall-experiments/blob/master/build.json reads :+1: Both of those languages look a little bit complicated IMO, the benefit of using something like TOML and JSON out-of-the-box is that people should be more familiar with that syntax than Dhall's or Jsonnet's.
I personally believe JSON is the way to go, trivially all JS build tools use it and they have proven how comfortable using JSON can be as configuration for any tool. However, if we want to be as similar to cargo as possible we should just continue using TOML and work on adding good VS Code support with completions. Possibly create a language server.
That being said, if you believe something like Dhall might be a good fit, I'm good with that: what's important for me is that it generates JSON. However note that at the moment you say that you can no longer call seed simple, at least from an aesthetics POV.
After giving it more thought, I'm no longer sure something like Dhall or Jsonnet would make the build tool easy to use for users.
Edited: Changed my opinion on whether Dhall/Jsonnet is a good way forward, they seem too niche and too flexible. I would go for something simpler for which we can easily create editor support and that has a lower cognitive overhead (people should be able to change settings using their intuition instead of checking the docs).
What JS build tools use it? There's package.json but that's usually very simple. Webpack uses an actual JS file, not JSON (although typically it consists mostly of JSON).
Almost all devops tools use YAML. Off the top of my head, almost all CIs, docker-compose, kubernetes, ansible and salt.
If you've never used YAML before I admit there's a bit of a learning curve, and some of the more advanced features like reuse features also have to be learned, but at the end of the day it's the least noisy and results in the simplest files.
Anyway it's great to experiment but can we discuss what are the metrics for determining success?
Personally my goal would be what build file is the easiest to maintain, which means easy to read, and as little code as possible.
On Thu, Sep 12, 2019 at 8:53 AM Jorge [email protected] wrote:
I really like how simple https://github.com/tindzk/seed-dhall-experiments/blob/master/build.json reads 👍 Both of those languages look a little bit complicated IMO, the benefit of using something like TOML and JSON out-of-the-box is that people should be more familiar with that syntax than Dhall's or Jsonnet's.
I personally believe JSON is the way to go, trivially all JS build tools use it and they have proven how comfortable using JSON can be as configuration for any tool. That being said, if you believe something like Dhall might be a good fit, I'm good with that: what's important for me is that it generates JSON. However note that at the moment you say that you can no longer call seed simple, at least from an aesthetics POV.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/tindzk/seed/issues/23?email_source=notifications&email_token=AAAYAUBI2VR5Z6ZVYGFS2OTQJI3TRA5CNFSM4IDT7MP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6RYRXY#issuecomment-530811103, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAYAUCFCZ3A2VHNXITIJRLQJI3TRANCNFSM4IDT7MPQ .
@tindzk I have edited my comment above to reflect some of my thoughts after sleeping on the Jsonnet/Dhall's proposal. I also left you some messages in Gitter in case you wanna chat :smile:.
@tindzk My Scala port of lightbend/config, ekrich/sconfig has been improving slowly and I have been attempting to support many versions including currently compiling with Dotty although there are too many problems to pass the tests due to Dotty not the library. I should have Scala.js 1.0.0-RC1 support when scala-collection-compat becomes available as well to finish off full cross platform support.
I think when you get down to enterprise grade configuration, value substitutions are a big deal. The library has a lot of history which could be considered legacy but also could be considered battle tested. I think the only user I have is scalafmt but that is held up because of Scala Native, 0.4.0 for now.
Sorry for the radio silence and thanks for all your suggestions!
After discussing this issue more with @jvican, we came to the conclusion that the best option is to only support TOML for now. I do agree with some of the issues raised here, but feel that overall TOML is the most satisfying solution in terms of readability and simplicity. For the missing features like variables, I would be happy to implement them as language extensions in toml-scala.
If the motivation for changing the configuration format is to avoid boilerplate, I propose we first explore how the schema could be improved. For instance, Play projects are not as well-supported as in sbt, but we can introduce abstractions to handle them better.
If TOML support in IntelliJ is lacking, it would be best to improve the plug-in. This will benefit other projects too. Features particularly useful for Seed would be to enforce the schema and to auto-complete dependencies (see also #60).
At the moment, my focus is to finalise Seed's core functionality (publishing artefacts, running test suites, Play support etc.) Afterwards, we can consider more elaborate use cases such as additional configuration formats.