Honor Maven repositories specified in .m2/settings.xml

[apupier]

Is your feature request related to a problem? Please describe.

Currently the Maven repositories specified in the .m2/settings.xml are not picked up to resolve dependencies

Describe the solution you'd like

Honor Maven repositories specified in .m2/settings.xml

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context

When using a JBang application, it might not be possible to specify the Maven repository.

Specific case here: due to https://issues.apache.org/jira/browse/CAMEL-21283 , there is no way to use the specific 4.8.0.redhat-xxx version for soem commands

Steps to reproduce:

• edit .m2/settings.xml, to add:

    <profiles>
    	<profile>
    		<id>Red Hat</id>
    		<activation>
    			<activeByDefault>true</activeByDefault>
    		</activation>
    		<repositories>
    			<repository>
    				<id>redhat.ga</id>
    				<url>https://maven.repository.redhat.com/ga</url>
    			</repository>
    		</repositories>
    		<pluginRepositories>
    			<pluginRepository>
    				<id>redhat.ga</id>
    				<url>https://maven.repository.redhat.com/ga</url>
    			</pluginRepository>
    		</pluginRepositories>
    	</profile>
    </profiles>
  

• jbang '-Dcamel.jbang.version=4.8.1' camel@apache/camel init 'demo.camel.yaml'

• jbang '-Dcamel.jbang.version=4.8.1' --verbose camel@apache/camel run 'demo.camel.yaml' --dev --logging-level=info --camel-version=4.8.0.redhat-00017 --local-kamelet-dir=.

  [jbang] [0:948] Using JDK: 21 (21.0.3+9-LTS, current, /home/apupier/.sdkman/candidates/java/21.0.3-tem)
  [jbang] [0:951] Resolving artifact(s): org.apache.camel:camel-bom:4.8.1@pom, org.apache.camel:camel-jbang-core:4.8.1, 
  org.apache.camel.kamelets:camel-kamelets:4.9.0
  [jbang] [0:953] Repositories: central=https://repo1.maven.org/maven2, apache-snapshot=https://repository.apache.org/content/groups/snapshots/
  

  [jbang] [ERROR] Could not resolve dependencies: The following artifacts could not be resolved: org.apache.camel:camel-core:jar:4.8.0.redhat-00017 (absent), org.apache.camel:camel-core-engine:jar:4.8.0.redhat-00017 (present, but unavailable), org.apache.camel:camel-main:jar:4.8.0.redhat-00017 (present, but unavailable), org.apache.camel:camel-java-joor-dsl:jar:4.8.0.redhat-00017 (present, but unavailable), org.apache.camel:camel-kamelet:jar:4.8.0.redhat-00017 (present, but unavailable): Could not find artifact org.apache.camel:camel-core:jar:4.8.0.redhat-00017 in central (https://repo1.maven.org/maven2)
  

[quintesse]

Sorry for dragging you into this issue @cstamas :-) But with MIMA I'm not 100% sure anymore how this should work. Is this because if you specify //REPO lines in the script they override anything that the user might have in their settings.xml?

(Their script uses //REPOS central=https://repo1.maven.org/maven2,apache-snapshot=https://repository.apache.org/content/groups/snapshots/)

[cstamas]

@quintesse See here: https://github.com/jbangdev/jbang/blob/main/src/main/java/dev/jbang/dependencies/ArtifactResolver.java#L154-L155

For some reason JBang opts to REPLACE user settings.xml reposes with own "partials".

[quintesse]

@cstamas look at the "blame" on those lines ;-)

But I'm guessing you implemented that to reproduce what we were already doing before.

Thing is, I think what we were trying to do is to replace any defaults (so you can say that you do not want central, by not including it in your list of repos), but not so much any user-defined ones.

Is there a way to do that with MIMA right now? So being able to override Maven defaults but leaving anything in settings.xml alone?

We'd then have to see if that is something we want to enable, because I think the reason we did it this way was to make sure that the results of a resolve are always predictable. Which might mean that we'd need a special option that the user could add to indicate that they want JBang to pick up on the settings.xml repos. (Not sure I'm very fond of that idea, but it's an option)

[cstamas]

MIMA by default will add Central only. Then, if context override is built with withUserSettings(true), and settings.xml is found, and there are profiles that are active and define repositories, they will be added to set of remote repositories on context.

So one thing you could do is create a context to get "user env" related to RemoteRepositories (not defining JBang ones yet, just to pick up user env), toss that context away and create new one as today w/ REPLACE op, as you should prepare list of remote reposes using list from previous context, and JBang one... Unsure is this clear.

[quintesse]

Yeah, it's somewhat clear. But wouldn't that temporary context also include the default Central one?

If so, we'd have to ignore that one, and merge the remining ones with the ones JBang wants to add. Dunno, seems all a bit complicated. Not sure if we should simply go the way of not replacing repos. Let's see what @maxandersen thinks about all this,

[cstamas]

Sure.

Just one tiny detail re differences between Maven3 and Maven4:

• Maven3 had a "super POM" that every built model inherited, see https://github.com/apache/maven/blob/maven-3.9.x/maven-model-builder/src/main/resources/org/apache/maven/model/pom-4.0.0.xml but the "root" env never contained central, so MIMA is adding Central explicitly (as here we do not deal with MavenProject that would have effective model with Central defined).
• Maven4 reworked this: there the "installation settings" (and in Maven4 you have "installation", "user" and "project" settings, all 3 used to build effective settings) define the Central remote repository, see https://github.com/apache/maven/blob/master/apache-maven/src/assembly/maven/conf/settings.xml

In short: Maven3 must have Central added "to start from somewhere" (otherwise by def env has no remote repositories defined, unless user have some in user settings.xml), while Maven4 by def have Central in "root" (unless installation settings modified).

[quintesse]

Ok, thanks for the explanation! 👍

[maxandersen]

A) I dont understand the issue in camel jira. Sounds like camel-jbang changed some behavior ?

B) we don't (and someone has to explain why we should) default honor .m2/settings.xml same reason gradle doesn't. Your local maven setup shouldn't affect jbang execution.

That's why we default to include central, but if you specify any other repos we expect you to specify all of them.

You can use multiple --repos to add any combo - why is that not working with camel-jbang?

Could we honor .m2/settings.xml somewhow ? Maybe - but it would have to be some explicit flag or env variable. Can't/shouldn't be the default IMO.

Don't we honor some .m2/settings.xml ? Yes, we do get matching security creds if the same id is in .m2/settings.XML. But it's also not default - you have to align the repo name/urls.

[apupier]

A) I dont understand the issue in camel jira. Sounds like camel-jbang changed some behavior ?

yes, it was changed in an incompatible way. It is fixed (reverted) in 4.8.1 but unfortunately, the productized version is based on 4.8.0 And it is touching a part which is not completely productized, thus not possible to backport.

You can use multiple --repos to add any combo

Do you mean something like that should work? jbang '-Dcamel.jbang.version=4.8.1' --verbose --repos=https://maven.repository.redhat.com/ga camel@apache/camel run 'fr.camel.yaml' --dev --logging-level=info --camel-version=4.8.0.redhat-00017 --local-kamelet-dir=. By placing the --repos in the jbang parameters

why is that not working with camel-jbang?

that's part of the mystery I try to solve. I'm wondering if in fact the Camel JBang is not using yet another dependency resolution as a second step and it is this one which is affected

B) we don't (and someone has to explain why we should) default honor .m2/settings.xml same reason gradle doesn't. Your local maven setup shouldn't affect jbang execution.

I think lot of users will already have it configured for Maven or Gradle. it would avoid to have them to configure also something different for JBang.

[quintesse]

I'm wondering if in fact the Camel JBang is not using yet another dependency resolution as a second step and it is this one which is affected

I think that's actually the issue here. Because you can run

jbang build '-Dcamel.jbang.version=4.8.1' camel@apache/camel

Without problem. So the first step builds (and runs) correctly.

It's the second step where the camel script runs jbang again that is missing the necessary repository,

If I run

jbang '-Dcamel.jbang.version=4.8.1' camel@apache/camel

the output shows that there is a [--repos=<repositories>] option but when I try to add that option, like this:

jbang '-Dcamel.jbang.version=4.8.1' --verbose camel@apache/camel run --repos=https://maven.repository.redhat.com/ga 'fr.camel.yaml' --dev --logging-level=info --camel-version=4.8.0.redhat-00017 --local-kamelet-dir=.

I get an error that the option doesn't exist:

Unknown option: --repos=https://maven.repository.redhat.com/ga
Possible solutions: --repository, --reload

What? --repository ? That doesn't appear in the help?? Well , let's try that option.

jbang '-Dcamel.jbang.version=4.8.1' --verbose camel@apache/camel run --repository=https://maven.repository.redhat.com/ga 'fr.camel.yaml' --dev --logging-level=info --camel-version=4.8.0.redhat-00017 --local-kamelet-dir=.

Nope:

Unknown option: --repository=https://maven.repository.redhat.com/ga
Possible solutions: --repos, --reload

(And now it says --repos again??)

I'm giving up, the CLI is pretty weird :-)

[maxandersen]

A) I dont understand the issue in camel jira. Sounds like camel-jbang changed some behavior ?

yes, it was changed in an incompatible way. It is fixed (reverted) in 4.8.1 but unfortunately, the productized version is based on 4.8.0 And it is touching a part which is not completely productized, thus not possible to backport.

Ok, but that isn't something jbang can "fix", right? You'll have to fix camel-jbang afaics?

You can use multiple --repos to add any combo

Do you mean something like that should work? jbang '-Dcamel.jbang.version=4.8.1' --verbose --repos=https://maven.repository.redhat.com/ga camel@apache/camel run 'fr.camel.yaml' --dev --logging-level=info --camel-version=4.8.0.redhat-00017 --local-kamelet-dir=. By placing the --repos in the jbang parameters

That is how jbang works - I don't know how camel-jbang handles its settings.

I'm wondering if this is first time camel-jbang is being used with productized bits and thus first time it needs more than one repository defined?

We'll need to figure out a way camel-jbang can know what settings it is running with because it is not jbang but camel-jbang doing "something" here afaics?

why is that not working with camel-jbang?

that's part of the mystery I try to solve. I'm wondering if in fact the Camel JBang is not using yet another dependency resolution as a second step and it is this one which is affected

I think that is most likely; but can't tell as I don't know how camel-jbang does it...it for sure need to somehow know about multiple repositories.

B) we don't (and someone has to explain why we should) default honor .m2/settings.xml same reason gradle doesn't. Your local maven setup shouldn't affect jbang execution.

I think lot of users will already have it configured for Maven or Gradle. it would avoid to have them to configure also something different for JBang.

Maven is configured in your pom.xml or ~/.m2/settings.xml ...gradle is always in build.gradlle. With jbang its based on your script + command line.

There isn't "one global" location for repositories.

Does camel-jbang not honor //REPOS in its .java files either? really should get those aligned so we don't create a firth way of doing resolution.