Scoped and managed dependencies

[maxandersen]

This issue is intended to be the parent of all the various issues related to the features needed for more fine grained dependency mechanisms than jbangs current bruteforce //DEPS which is basically one classpath for both build/compile/boot/runtime....which works in about 90% of cases.

And that latter is to be remembered - whatever we do here I do not think we should change the working default behaviour - its easy to understand and use. I will rather not have any finegrained controls than give up easy to use in 90% of cases.

That said - there are starting to be more cases where some finer control is needed, lets gather the ones we know of:

• [ ] define managed dependencies - in maven speak: dependency management pom import - to control what specific version of a library to be used. We have @pom today but it is problematic as it prevents normal pom dependency used in projects like graalvm truffle #1683
• [ ] being able to exclude (transitive) dependencies - useful to leave out i.e. optional log4j dependencies included by some libraries #150
• [ ] be able to mark dependencies as buildtime only (graalvm "compileOnly", maven "provided"); useful for things like annotationprocessors and usecases like jreleaser extension where the runtime classpath really shouldn't include jreleaser itself (see PR discussion)
• [ ] be able to define module deps(//MDEPS) as well as claspath deps (//DEPS) #559
• [ ] consider how boot-module-path and agent-module-path fits in this #618 and #1662

Currently the above I'm leaning towards the following principles/conclusions (mainly here to validate if things stays consistent, but curious to hear feedback on it):

• scopes feels very much like "tags" on a dependency type - i.e. one way to express it could be asciidoc block property //DEPS[build] or //DEPS{build} to make parsing less ambigious as [] is valid in version ranges.
• test dependency / test scope is NOT currently in the cards as that would be more a separate project/script that uses existing jbang script. The tests has the other script as a dependency; the main project doesn't know about the tests...we might want to make that a thing later but then thats really a different kind of relationship - not a classpath defined dependency scope.
• exclusions in maven is AFAICS done only on transitive dependencies on a specific explicit dependency, need to use enforce rules for global exclusions. "provided" does it partially feature requests exist for global definable - gradle has global exclude options but seem to vary over time - i think a first step is to define global excludes...and that it will be useful being able to do group excludes .. like //!DEPS org.jreleaser:* but also be able to do it at buildtime...//!DEPS[build] org.jreleaser
• we'll need to find out how users/tools get the resolved classpaths/scopes.

🧪 Proposed Design Directions

To support these features incrementally and maintain JBang’s focus on simplicity, the following ideas are proposed for review and refinement:

🎯 Scoped Dependencies

Introduce optional scopes (tags) on dependencies to control when they are available in the lifecycle:

//DEPS{compile} com.example:lib:1.0
//DEPS{runtime} com.example:lib:1.0

• Shorthand style ({scope}) or key=value style ({scope=compile}) allowed.
• Multiple scopes can be combined: {compile,runtime} (equivalent to today’s default).
• This allows fine-grained control over classpath assembly for compile and runtime phases without affecting current behavior (scopeless //DEPS would still default to both).

📦 Module Dependencies (//MDEPS)

Introduce //MDEPS for declaring JPMS module dependencies which are resolved not through Maven but used to build the JPMS module-path or --add-modules settings:

//MDEPS java.sql
//MDEPS{static} jdk.crypto.ec

This supports modular Java applications, GraalVM agent injection, etc., aligning with long-standing requests.

🚫 Global Exclusions (//!DEPS)

Instead of requiring Maven-style per-dependency exclusions, JBang can introduce a simpler global exclusion directive:

//!DEPS commons-logging:commons-logging
//!DEPS org.slf4j:*

• Excludes matching dependencies from transitive resolution.
• Explicitly declared //DEPS still take precedence (i.e. explicit includes override global excludes).
• Wildcards (*) allowed in artifactId or groupId for convenience.

We can also support scope-specific exclusions:

//!DEPS{runtime} org.jreleaser:*

This example removes org.jreleaser dependencies only from the runtime classpath, while keeping them available for compile as needed.

🧩 Resolution Rules

A few behavioral principles underpin the above:

• //!DEPS exclusions apply only to transitive dependencies by default.

• Explicit //DEPS always wins over a global exclusion.

• Scopes define when a dependency is added to the classpath:

  • compile = available only during compile
  • runtime = available only at runtime

• Wildcard support in exclusions (group:* or group:artifact*) gives lightweight power without full dependency graph introspection.

---

Open questions/concerns

• Is there a usecase for exclusions only defined on certain dependencies?
• if we only have global exclusions these would have to be added on all deps in maven export
• are there more properties to "add" to a dependency than scopes?
• should "agent" and "boot" be a scope ?
• how is managed deps expressed if @pom not allowed? //DEPS[managed=true, scope=compile]?
• is //DEPS equal to //DEPS[scope=runtime,compile] ?
• how should tools/humans get these different classpaths?
• anything missed here?

📌 Next Steps

Each of the checklist items in this parent issue can be implemented incrementally. The above proposals aim to balance power and flexibility without compromising JBang’s zero-config default UX.

Feedback on naming, syntax, parsing, and implementation trade-offs is very welcome!

[maxandersen]

@quintesse @cstamas - curious about feedback on this one - its a bit complicated but its important to get somewhat right thus curious about questions/clarifications and if you think there are some usecases this suggested approach can't handle....also interested if you think its perfect :)

[aalmiray]

Does this mean that configuring an annotation processor would have to use the following?

//DEPS{compile} org.kordamp.jipsy:jipsy-processor:1.2.0

[aalmiray]

There’s perhaps a bit of dissonance with compile as laid out in this RFE when compared with how Maven and Gradle handle scopes.

Both Maven and Gradle treat compile as the provider of classes for both compilation and runtime. Gradle went one step further and added compileOnly to signal which classes are needed during compilation but won’t be added to runtime.

CompileOnly makes sense for behavior provided by annotation processors for example.

Coming from Maven or Gradle, using compile as explained by OP may cause confusion as it would work differently in JBang. Essentially, Maven and Gradle treat compile as the union of compile and runtime.

That’s how it’s been done so far. Question is, should JBang do the same or break with tradition?

• Is it weird to have a composite scope (compile + runtime) without a proper name?
• if it had a proper name, is compile the correct one to coincide with Maven and Gradle?
• if so, another scope for just compilation would be needed, like Gradle’s compileOnly, which JBang could call build

[cstamas]

Just FYI:

https://github.com/apache/maven/blob/master/api/maven-api-core/src/main/java/org/apache/maven/api/DependencyScope.java#L62

[maxandersen]

Does this mean that configuring an annotation processor would have to use the following?

//DEPS{compile} org.kordamp.jipsy:jipsy-processor:1.2.0

that was my thinking yes.

[maxandersen]

That’s how it’s been done so far. Question is, should JBang do the same or break with tradition?

• Is it weird to have a composite scope (compile + runtime) without a proper name?

my thinking was that scope gets treated as a "tag" and a dependency can be added to multiple of them. I.e. I can state "compile,agent" to state that I want it when compiling and on agent class path. won't be on any other.

thus "compile,runtime" made sense to me (at least for now :) as the implied default.

• if it had a proper name, is compile the correct one to coincide with Maven and Gradle?

reason I used compile and runtime is that we have --compile-option and --runtime-option

• if so, another scope for just compilation would be needed, like Gradle’s compileOnly, which JBang could call build

I do admit it could be problematic if too much dissonance between the various build systems. I'll read up on them and figure out if different names needed.

[maxandersen]

Just FYI:

https://github.com/apache/maven/blob/master/api/maven-api-core/src/main/java/org/apache/maven/api/DependencyScope.java#L62

so maven and Gradle calls what I want to call "compile" for "compile-only" because maven 2.5 decades ago decided to use "compile" to mean everything ? is that a good summary ? :)

[maxandersen]

if we did use "compile-only" and "runtime-only" should it then also be "agent-only" and "boot-only" ...? or are agent and boot not good for scopes?

I'm leaning towards that its more sane to use scopes as tags which can then be combined...there is at least a natural mapping between this and gradle/maven approaches afaics.

[cstamas]

so maven and Gradle calls what I want to call "compile" for "compile-only" because maven 2.5 decades ago decided to use "compile" to mean everything ? is that a good summary ? :)

I think yes

Here is an old draft what maven4 does today (handle with "grain of salt", may have typo or mistakes and is not official in any way): https://gist.github.com/cstamas/3dfe50390f2e7a00c4e4ffc364e245e6

[maxandersen]

so maven and Gradle calls what I want to call "compile" for "compile-only" because maven 2.5 decades ago decided to use "compile" to mean everything ? is that a good summary ? :)

I think yes

Here is an old draft what maven4 does today (handle with "grain of salt", may have typo or mistakes and is not official in any way): https://gist.github.com/cstamas/3dfe50390f2e7a00c4e4ffc364e245e6

Yeah - it's that kind of insane matrix i want to avoid dealing with if can be avoided.

Of course I'll need to as will have to resolve maven transitive dependencies but I don't spot something treating scopes as tags can't capture in that table...

[quintesse]

I'm generally on board with most things described in the proposal (understanding that things like @aalmiray 's comments might still change the final exact details).

The only thing I'm wary about is the //MDEPS. They might cause a confusion of expectations. We deliberately chose to exclude path to jars from being useable with //DEPS (eg you can't do //DEPS path/to/my.jar https://github.com/foo/bar/my.jar) and the reasoning behind is that //DEPS is meant for dependency management, it does lookups, updates versions, it brings in dependencies, etc etc. All things we can't do with JARs.

So //MDEPS as described here also wouldn't be the same as DEPS-but-for-modules. There's no downloading of dependencies. It functions more as a permission system where you tell the JVM you want access to, for example java.sql. But, and that's very important, if the JBang script itself is a module (using the //MODULE <name> directive) then we must list all modules that we want to access, this includes any modules defined by Maven artifacts!

This could mean that you might need to do something like:

//DEPS org.acme:mymodule:1.2.3
//MDEPS org.acme.modules.mymodule
//MODULE demo

... <code here> ...

Where //DEPS brings in the module and //MDEPS tells the JVM that we want to use the module (the names of the two do NOT need to coincide! There are many examples of modules with different names than the Maven artifact they are part of. The names are often similar, but not always.)

But on top of that //MDEPS is just too simple, in many cases you will need to add additional information about for example the services that you want imported or what packages you want accessible for introspection etc. Which is the reason that many module info files are longer than a single line.

So my first objection is that //MDEPS is not a good name, but that would be easy enough to fix. My second objection is that //MDEPS is way too simple a system to be able to properly use modules.

[aalmiray]

Pinging @sormuras for //MDEPS

I wish his database of gav <=> module covered all publicly available artifacts. Still automagically figuring out a matching gav for a given module would fall flat when dealing with non public jars/modules.

[maxandersen]

@quintesse we don't need to handle what is better done in a users module info file.

I was thinking more as way to define things like boot module path or separate compile / runtime rules.

[quintesse]

@maxandersen ok, but what use would the //MDEPS have then? Because the example you gave with //MDEPS java.sql is exactly what you'd put in a module info file. It would then only work for very simple cases.

And separate compile/runtime rules... do you have examples how you would see that used?

So my suggestion would be to split this off from this issue into its own issue. Because although they seem somewhat related I feel they are sufficiently different to a) need more discussion on //MDEPS where we seem to be on the same wavelength for the //DEPS changes and b) not be implementable in similar ways, which most likely means it's a quite a bit more work

[maxandersen]

See https://github.com/jbangdev/jbang/issues/618

And yes each of these are probably separate issues that we can implement separately but wanted to check if overall approach would work.

[quintesse]

Oh sure, the extension of the directives format seems to be okay. (Personally I'm more a fan of //DEPS[xxx] than //DEPS{xxx} but that's minor)

The parsing and the "model" will become quite a bit more complicated, though.

I'm also wondering if //!DEPS is necessary? Couldn't we do //DEPS !xxx?

Which could then be extended to the scopes as well: an alternative implementation could be: //DEPS xxx, yyy{build}, zzz{run}. This is of course a lot less nice if you have many dependencies that don't follow the default scope but it has the advantage that our directives parser and model don't need to become more complex. (Why? Because we can treat the value of //DEPS as a black box and pass it as-is to the dependency parsing code. If it's at the level of the directives we need to do the parsing before that. Which means already doing some pre-parsing and passing those values as a set or map to the dependency parser. All doable of course but less "nice")

[quintesse]

Another idea that follows more the original suggestion but doesn't affect the names of the directives themselves would be to use:

//DEPS {build} xxx, yyy
//DEPS {run} zzz

And perhaps also allow

//DEPS {build} xxx, yyy {run} zzz

Complexity-wise it would still be worse than the //DEPS xxx, yyy{build}, zzz{run} option though. (Well... if we'd only allow a single {...} and only at the start of the line we could simplify it quite a bit)

[aalmiray]

I'd prefer having each dependency in its own line, rather than grouping them by scope.

Also agree that //DEPS !xyz reads better than //!DEPS xyz. After all, both instructions are the entry point for dependency management, it should be consistent whether it's inclusion or exclusion.

[maxandersen]

Reason for one {} is that then we can agressive grab that as there is no other } to be found.

Also if user want it it per dependency they can put one per line.

Also having it at start makes it work same way on command line.

That's at least was my thinking on why.

Having it per dependency feels ... Verbose. Remember it could be multiple tags and if other info that would be repeated too.

[aalmiray]

Right, but who is the main character? Is it the scope/tags or the gav? 😉

[maxandersen]

I'd prefer having each dependency in its own line, rather than grouping them by scope.

Also agree that //DEPS !xyz reads better than //!DEPS xyz. After all, both instructions are the entry point for dependency management, it should be consistent whether it's inclusion or exclusion.

Not sure if this is going too much perl :) but could make ! a shorthand for exclude modifier/property so //DEPS {!,compile} is shorthand for //DEPS {exclude=true,scope=compile}

...og and as I type it out I realize why it's a different directive. //!DEPS org.jreleaser:* should be allowed. Ie. IT defines A pattern rather than an explicit GAV.

We could go verbose and call it //EXCLUDEDEPS ...but boy that's a mouthful :)

[aalmiray]

Yes, but both //DEPS !com.acme:anvil:1.2.3 and //DEPS !com.acme:* could be valid.

One says, I don't want that explicit dep, the other says I don't want all matching deps.

FWIW Maven supports both types of exclusions.

[maxandersen]

But exclusions arent dependencies. Can't depend on org.jreleaser:*

[quintesse]

Also having it at start makes it work same way on command line.

I'm not sure I understand what you are referring to here?

I would say that having --deps xxx{build} would be much more similar to //DEPS xxx{build} (in fact it could most likely be the same parser) while //DEPS{build} xxx would need special handling for both cases. What would be the format on the command line? --deps{build} xxx? Besides it probably be very hard to make picocli understand that it would also be totally weird. SO what would we use? --deps {build} xxx? That can't be, we'd get all kinds of problems with spaces and quoting.

To me personally the postfix solution sounds easiest to implement. It would be verbose, yes, but people aren't supposed to use this anyway except in emergencies.

The other thing is ! being the same as [exclude=true], I had actually thought the same thing yesterday. But pretty soon I told myself: "don't over-engineer things!!!".

[maxandersen]

Agreed, lets put modifier on GAV syntax: //DEPS g:a:v{run,build} that does greatly simplify things internally. If we find cases where it gets too verbose we can consider adding //DEPS {run,build} g:a:v q:x:v as a "modifier"

Both of these would be possible to use on command line - latter needs ""'s :)

[maxandersen]

FWIW Maven supports both types of exclusions.

It actually does not support these - maven exclusions are filters and only applied to an actual GAV transitive deps. They just look similar but aren't :)

[maxandersen]

Thinking to not call it scopes but to allow to declare when to use it.

Ie. //DEPS A:b:1.2{when=boot}

And use when needing to refer to it in i.e. jbang info classpath --when=boot my app.java

Question is what Default would be.

Today it's build,run combined.

[quintesse]

Today it's build,run combined.

I think today it's at least "build,run,agent" (if you're still thinking about introducing "agent")

~Btw, I don't think we should complicate this, but if the agent is a jbang script it has its own build phase, so theoretically at least there could be a "agentbuild" (not suggesting this name, it's just to make it obvious what it is).~

Edit: sorry, scratch that last bit, if it's a script it will have its own DEPS during build. Those are separate from the DEPS from the parent script.

[maxandersen]

Agent is not taken from //deps today so it's only build,run today.

[quintesse]

You're right, but I knew there was something: it does actually take any additional dependencies added on the commandline with --deps.