Removing obsolete grammars, and naming new libraries

[tedinski]

I wanted to write down naming intentions here.

We have a number of libraries that should be considered obsolete and removed:

grammars/silver/util/treemap  (use raw:treemap)
grammars/silver/util/fixedmap  (ditto)
grammars/silver/util/command  (use cmdargs)
grammars/lib/errors  (obsolete by langutil)
grammars/lib/system  (??!)
grammars/lib/stringmap  (obsolete by raw:treemap)

This library is one that we preserve only because it was used by the adventium grammars, which we don't want to break:

grammars/lib/xml

These should be mined of whatever value they have, and then removed as grammars, imo:

grammars/lib/extcore
grammars/silver/util (the files directly in this grammar, not the grammars below it)

We have a slight problem with the following grammars:

grammars/lib/json
grammars/lib/monto
grammars/lib/monto/helpers
grammars/ide
grammars/silver/definition/regex

I believe these are mis-placed. I'm not sure what a good naming scheme would be. I would consider it reasonable to perhaps place them all under langutil. But other suggestions very welcome! (Perhaps a "silver:languages:" grammar prefix for trivial library items like regex and json? Not sure if I like that idea though... gives the impression that's where languages should go, and it's really only special cases...)

The following are all fine: (Although I kinda regret the raw name. Perhaps we should hope to migrate away from that? And perhaps not have tree in the name for map/set either.)

grammars/silver/util/cmdargs
grammars/silver/util/deque
grammars/silver/util/raw/treemap
grammars/silver/util/raw/graph
grammars/silver/util/raw/treeset
grammars/silver/langutil
grammars/silver/langutil/pp

These are... maybe okay. I think it should be renamed silver:util:testing. (And they need cleanup.)

grammars/silver/testing
grammars/silver/testing/sampleTests
grammars/silver/testing/bin

Finally, I'll obsolete the old issue #12 and restate intentions for these grammars:

These should be corralled under silver.compiler:

grammars/silver/definition
grammars/silver/modification
grammars/silver/extension
grammars/silver/host
grammars/silver/composed
grammars/silver/analysis
grammars/silver/driver

And these should be renamed silver.lang:

grammars/core
grammars/core/monad

Including the runtime, currently named common which should become silver.lang.runtime perhaps. The reason for this combination is the need to pair together these inter-related sources into one "package". (And this would simplify bootstrapping, too.) Otherwise we distribute a SilverRuntime.jar that depends on things that are provided by whatever jar you build, which means we never have anything but a circular dependency mess.

In the end, this would see us distributing pre-compiled artifacts:

silver.compiler-1.0.0.jar
silver.lang-1.0.0.jar
silver.util-1.0.0.jar
silver.langutil-1.0.0.jar

[krame505]

Do we really want to have libraries commonly used by extension writers under silver:? It seems to me that the silver top-level grammar should just be for the actual implementation of silver itself (and maybe for sub-languages useable by other extensions like regex and json - maybe - but I would prefer to have these somewhere else as well), and that things like langutil belong under lib:.

Similar for the raw libraries - don't these belong under lib as well? Since we have decided we don't like the name 'raw', these could get moved somewhere else, and export whatever the new location is, with the addition of a deprecated message.

Another thing that should probably be fixed is core:monad. I originally put this here and not in core to try and keep things seperate, but still have it exported by core. This doesn't actually work due to how we handle core, and needing to import this manually hasn't actually been a huge pain. So maybe this should also go under lib.

[tedinski]

The analog here is Java and other JVM hosted languages. java.lang is visibile by default. Then you import java.util.datastructure or java.lang.io or java.time etc.

It's a question of namespace ownership. It's probably okay for us to claim a top level name like silver for the language, but doing more than that is unwise (and why AbleC and Copper are all under edu.)

Also, the other bit there would have us putting the silver implementation under silver:compiler. So it does get grouped sensibly.

I didn't realize that exports didn't work for core. I've actually thought about breaking some things out of core (well, I guess just IO?) So maybe having to import it isn't so bad?

[remexre]

I'd be in favor of:

• Deleting old stuff
• Keeping "general" libraries in lib
• Splitting into more granular jars
• Splitting out IO from core
• Moving core:monad

I'm against silver:lang; I'm not sure what's wrong with the name core (other than the exports issue).

[krame505]

The analogy here is to languages like java - we have java.lang imported by default. lib: really should go away, since we should only be 'claiming' the top-level grammar namespace silver:.

[remexre]

Hi, I'm going over the Silver code for Monto right now, and I'm probably going to move everything to the "right place" as defined by this and #175; should lib:json be moved to silver:json?

Also, should the part of the reorganization for moving all the grammars that "should" be in silver:compiler to there be done for the 0.4 release?

[tedinski]

Getting that kind of change in for 0.4 is unlikely. We're mostly last-minute scrambling to get the bare minimum done for it as usual.

In the other referenced thread, I think we decided on having that sort of grammar be under silver:support:json? (To restate the reasoning, silver:util and silver:langutil would be versioned artifacts/libraries in the long run when we have such a thing, but each thing under silver:support would be its own versioned artifact. So things could optionally depend on silver:support:json or not, and it doesn't get taken along for a ride for other things that don't need it.)

I guess I don't have a strong reason why silver:json isn't an acceptable approach, except that it seems less nice to put things in the top-level silver namespace like that. But maybe it's nice to have shorter names? After all, there's functionally no difference between silver:json as an artifact and silver:support:json. Our documentation should be perfectly able to distinguish the "important" things from the lesser libraries without having to do that with namespaces, too... IDK. Naming things is hard.

[remexre]

I talked to @ericvanwyk earlier and he thought that it'd make sense to keep silver:support for things that are for silver development, whereas something like JSON might be used as a language in its own right (e.g. in an extension).

It'd be nice to have something like silver:helperlangs (or maybe just silver:langs?) for things like JSON, regexes, or other things that might be used as "sublanguages."

[krame505]

Is there really a good reason to seperate these? To me it seems that these are just two different types of libraries that can be used by other extensions, and I would favor minimizing the number of 'arbitrary' top level category grammars that we have (e.g. a regex library that also included an evaluation feature, or something else that involves concrete syntax that is useful outside of importing for use as a DSL.)

Lucas Kramer

On Wed, Sep 27, 2017 at 4:32 PM, Nathan Ringo [email protected] wrote:

I talked to @ericvanwyk https://github.com/ericvanwyk earlier and he thought that it'd make sense to keep silver:support for things that are for silver development, whereas something like JSON might be used as a language in its own right (e.g. in an extension).

It'd be nice to have something like silver:helperlangs (or maybe just silver:langs?) for things like JSON, regexes, or other things that might be used as "sublanguages."

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/melt-umn/silver/issues/152#issuecomment-332646243, or mute the thread https://github.com/notifications/unsubscribe-auth/AIE1iixeeA9gVjCy4fmcDjGlnjwwQHcsks5smrDFgaJpZM4NJnTs .

[tedinski]

I think I'm swayed over to just silver:json.

1. It avoids having to come up with some extra naming scheme.
2. It shouldn't be too hard to emphasize the things under silver that are more important in our documentation.
3. They're still separate artifacts in future-someday-versioning system.
4. I guess it's sorta what java does. e.g. java.time
5. support, lib, langs, helperlangs, etc are all pretty unhelpful names.
6. If we get things wrong, it's not harder to fix than any other wrong choice of name,

The main disadvantages is pretty much just:

1. It'd be nice if the list of things under grammars/silver was aesthetically pleasing, instead of just all the libraries we ship.

Oh well.

[krame505]

I see how having everything in one grammar could be simpler, but my concern is that people would find it seriously confusing to have the compiler implementation itself be one of many things all in the same grammar, alongside what is now core, and numerous other libraries of varying importance. What about just having silver:compiler, silver:lang which would be imported by default, and silver:lib where we would dump 100% of everything else? With this we would get a slightly-less-confusing organization, but still wouldn't be trying to categorize things into different sub-grammars.

Lucas Kramer

On Wed, Sep 27, 2017 at 5:41 PM, Ted Kaminski [email protected] wrote:

I think I'm swayed over to just silver:json.

1. It avoids having to come up with some extra naming scheme.
2. It shouldn't be too hard to emphasize the things under silver that are more important in our documentation.
3. They're still separate artifacts in future-someday-versioning system.
4. I guess it's sorta what java does. e.g. java.time
5. support, lib, langs, helperlangs, etc are all pretty unhelpful names.
6. If we get things wrong, it's not harder to fix than any other wrong choice of name,

The main disadvantages is pretty much just:

1. It'd be nice if the list of things under grammars/silver was aesthetically pleasing, instead of just all the libraries we ship.

Oh well.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/melt-umn/silver/issues/152#issuecomment-332664065, or mute the thread https://github.com/notifications/unsubscribe-auth/AIE1ijiwsyHwPCIGy_3swphA6FxwViu3ks5smsD_gaJpZM4NJnTs .

[ericvanwyk]

The JSON grammar is different in that it is its own little language - not a library like core that is used in Silver code. The plan is to use in the monto grammar that, rightly I think, lives under support.

To some extent, silver:json is just hijacking the top-level silver grammar name - it could also be edu:umn:cs:melt:json. But having some of these little languages that are used by some part of Silver but could also be imported by other grammars (a JSON ableC extension for example) just tucked under silver seems fine to me.

Eventually all the Silver implementation gets moved under compiler - but this wouldn't be imported by default. Only grammars under lang - that is my understanding. lib could then have libraries that are shipped with Silver but are not imported by default.

[krame505]

My point is that a DSL package such as json can still be considered a library, regardless of whether it contains functions, abstract productions, or concrete productions. json isn't something that is compiled and used by itself, it is only used in other Silver code, correct? Is there really anything wrong then with calling it silver:lib:json?

[ericvanwyk]

I think we want to separate names from where they are in a repo. If we think users would prefer to write silver:json then that is what we should name it. This grammar is used in Silver but one might also use it as the abstract syntax for an ableC extension that add JSON syntax to C. It could also have concrete syntax for JSON.

No, nothing wrong with silver:lib:json but is this name driven by a desire to have a better organized directory? That isn't as important to me - we'll have (maybe) documentation that guides the users through all of this - no just looking at a silver directory structure and trying to make sense of it.

We could have these living some place other than in the silver/grammars/silver directory of the Silver repo too. I just don't see the need for the lib bit.

[krame505]

My concern is more about how things are logically arranged then about where they physically live. The goal should be to design things such that they don't require documentation to make sense, in my opinion. When a user sees something being imported from the silver: namespace, they would logically to think that anything in silver: is something to be imported, when in reality some things like compiler and lang will have special importance. However as Ted said this is sorta what java does, and I suppose I could live with this situation...

Another thing to at least think about now when designing this is how to do things such as #194, where we may want to use ableC in a silver extension that ships with silver. Any thoughts on this?

[ericvanwyk]

We should resolve this if possible. It seems that we agree on the following.

1. silver:compiler contains the implementation of the Silver language.
2. silver:lang contains code that is automatically imported by Silver.

and maybe we agree that

3. silver:lang:X is code that needs to be explicitly imported. I presume that this is code that extends or builds on something already in silver:lang and not some new thing.

Lucas is arguing that other new libraries be under silver:lib and others are arguing that they can just live under silver. The silver:lib approach has the advantage of simplicity, but the disadvantage of adding lib to many things and to also not following the Java/JVM approach.

The advantage of the silver approach is short names, it follows the Java/JVM approach, but uses must understand that lang has special meaning.

Since lang does have special meaning, Silver can be made to report an error if one imports silver:lang that explains the situation. There is nothing inherently wrong with importing something from silver:compiler and it is what one would do for writing Silver extensions.

Also, there is no reason to not nest things under something under silver. For example, for #194 we might have a silver:cs grammar that has basic bits of this and a silver:cs:ableC extension that has the rest that is needed for ableC. This would let us organize things in a way that the naming hierarchy is used in understanding.

Thus, I propose that we move forward with the silver approach and thus name the JSON grammar silver:json.

Any strong objections? Can we all agree on this?

[krame505]

I can live with this. I would say that calling this 'the Java approach' seems a bit of a misnomer as Java does seem a bit more structured than this, however.

I don't see a reason why we would start putting Silver extensions in a grammar other than silver:compiler:extensions - but think I see your point.

I was wondering in the context of #194 more about how we would go about adding a dependency on ableC to Silver, since it was mentioned at one point that we may introduce a way of moving silver components into seperate repos? Or was that something unrelated?

[ericvanwyk]

OK, great. @remexre you can rename the JSON stuff at your convenience.

@krame505 - I think the #194 issues are separate from this.

I'll leave this open until we do the renaming in Silver - but for now the name for JSON is set.

[tedinski]

Some thoughts on dependencies. I was envisioning these artifacts:

silver.compiler-1.0.0.jar
silver.lang-1.0.0.jar
silver.util-1.0.0.jar
silver.langutil-1.0.0.jar

Each of these artifacts can ship multiple grammars, of course, beneath the top-level namespace the artifact name is claiming.

We need to ensure we manage dependencies appropriately, and to think clearly, we have to distinguish between artifact dependencies and grammar dependencies.

• silver.lang should have no outside dependencies at all. Well, except for the copper runtime.
  • Within itself, there should be no dependencies originating from the grammar corresponding to core (right now we do have one for reflect, but this wouldn't be necessary in the future, as it currently exists only to trigger the build, and once we have artifacts like this, they'd build all parts of themselves automatically.)
  • This artifact also includes the Silver runtime java code, if that wasn't clear.
  • There must not be any dependencies from runtime or generated code on things outside of the silver.lang artifact. This runtime should be the only requirement for compiled Silver binaries.
  • But there may be dependencies from such things outside of the core equivalent. e.g. the grammar silver:lang:reflect or similar things in the future.
• silver.util and silver.langutil should likewise have no dependencies, except for on silver.lang of course.
  • Together with being simple & general, the lack of outside dependencies is what qualifies something to be lumped into these artifacts, instead of getting their own artifacts.
  • We could consider allowing langutil to use util if we find a good reason, I see nothing wrong with that.
• silver.compiler is an application not a library, so it's dependencies maybe don't matter as much. However, this is some merit to the idea of "compiler as library" so perhaps we should try to avoid going too wild.

[krame505]

When you mention reflect here, are you referring to what is now core:reflect or what is now silver:reflect?
core:reflect is a runtime dependency with the AST nonterminal, but we don't really want it imported by default.
silver:reflect is the actual Silver-side library that is not needed by the runtime or anything currently in core. This is not build-triggered by core.

The only reason silver:langutil currently would depend on silver:reflect is to avoid flow errors due to an orphaned occurs for pp, since these equations really belong with the definition of the AST nonterminal.

[krame505]

Also, are you envisioning silver:json, silver:reflect, etc. all being distributed as seperate artifacts in addition to those listed, or would they be under silver.util?

[tedinski]

When you mention reflect here, are you referring to what is now core:reflect or what is now silver:reflect?

Once we have an artifact system like this, they could likely be merged.

Right now we have them separate (as I understand it?) simply because one needs to be built by default because generated code depends on it, and the other is code we don't want getting built and depended upon all the time when it's not being used.

Once we have artifacts, generated code can happily reference silver:lang:reflect without needing to induce any dependencies on additional artifacts.

Also, are you envisioning

Case by case basis, I suppose.

[krame505]

But you mentioned above

There must not be any dependencies from runtime or generated code on things outside of the silver.lang artifact.

Since what is now a part of core:reflect is needed by both the runtime and generated code, I don't see how this could be merged into a seperate artifact?

[tedinski]

Ah. Not separate artifact. Merged into the same grammar in the silver.lang artifact.

[krame505]

I guess I still don't understand what you are proposing. Do you mean merging both core:reflect and silver:reflect into a silver:lang:reflect grammar that is distributed as part of the silver.lang artifact? That would make silver.lang depend on silver.langutil, and besides I kind of think reflection should live as its own grammar anyway.

Or do you just mean merge core:reflect into the top-level silver:lang grammar, while leaving silver:reflect alone? I don't see how this is really any better than just leaving core:reflect as silver:lang:reflect, since we don't really want AST imported by default.

[tedinski]

Do you mean merging both core:reflect and silver:reflect into a silver:lang:reflect grammar that is distributed as part of the silver.lang artifact?

Yep.

That would make silver.lang depend on silver.langutil

We can cross this bridge when we come to it, but maybe we could move anything that uses anything from langutil into a silver:langutil:reflect helper grammar or something like that.

AFAICT, this is just the code that pretty prints an AST right? Perhaps we should just have a String serilizer in the reflection grammar and put the Document variant into silver:langutil:reflect now.

I kind of think reflection should live as its own grammar anyway.

Nah, it's a core part of the language, it belongs in silver:lang because anything generated code & runtime references should be there, that's pretty much the definition of a core part of the language.

[krame505]

Hmm, I guess I mostly don't have a problem with this. The only question is where does the deserialization code live? This involves some concrete syntax and building a parser. Is having this live in silver:lang a problem? Putting this in a library by itself might be kinda weird, and it doesn't really belong in silver:langutil.

[tedinski]

With the artifacts approach, it wouldn't be a problem for that parser to be in silver:lang:reflect since it'd be distributed in already compiled form, and it doesn't create any extra dependencies or anything. (We'd probably put the syntax in silver:lang:reflect:whatever of course.)

[krame505]

Also, hackUnparse would want to use the Document pp variant, since the String serialization will fail on anything that isn't anyAST.

[krame505]

Although I guess we could move hackUnparse to langutil...

[krame505]

I would like to try and actually do this refactoring before the hackathon next week, since these changes would likely entail significant merge conflicts with whatever else is being worked on then. My plan is to basically do this as Ted plotted out above, although I will probably wait on actually splitting the build out into seperate artifacts.

There's been some bikeshedding of what to call the standard library once this is moved under the silver namespace. Any strong objections to silver:lang, as Ted originally suggested?

This refactor is somewhat blocked by #333, as there will be changes made to deep-rebuild that require bootstrapping, and also to avoid even more merge conflicts. @602p are you available to chat about this in the next couple of days?