ceylon-herd
ceylon-herd copied to clipboard
eclipse-archived
Interoperability with Ivy and Maven dependency resolvers
This enhancement would allow to create a flat classpath of Ceylon CARs for Java projects using Maven, Ivy or Gradle. This is possible by offering appropriate repository "facades" through the Herd repository server.
Maven repository structure
Maven expects the following layout (see Maven Repository Layout - Final) for primary artifacts:
/$groupId[0]/../${groupId[n]/$artifactId/$version/$artifactId-$version.$extension
and for secondary artifacts:
/$groupId[0]/../$groupId[n]/$artifactId/$version/$artifactId-$version-$classifier.$extension
Ivy repository structure
Ivy is more flexible and allows to specify custom patterns for artifact resolution (see Ivy Documentation - Main Concepts). The default patten that is used by Gradle is the following (see Gradle DSL Reference - IvyArtifactRepository):
Artifacts: $baseUri/[organisation]/[module]/[revision]/[type]s/[artifact](.[ext])
Ivy module descriptors: $baseUri/[organisation]/[module]/[revision]/[type]s/[artifact](.[ext])
Meta information
To resolve transitive dependencies both repository types require meta information. Maven uses pom.xmls. Ivy uses ivy.xmls, but can also process pom.xmls. Those files must be accessible via HTTP requests.
Meta information augmentation When the repository server responds to a "foreign" meta data request for a Ceylon module, it should automatically add all implicit dependencies of the Ceylon language to the response. For interoperability these Ceylon language modules should be published to the Herd repository so that Java projects that depend on a Ceylon library do not have to provide them by themselves.
Artifact aliases For interoperability it would be very useful when CAR files are also available under the same name but with JAR file extension when accessed through a facade.
Many IDEs automatically link source and javadoc JARs to the downloaded artifacts by searching in the module cache for files like $artifactId-$version-$classifier-sources.$extension (IDE dependent see NetBeans DependencyNode). Ceylon source artifacts should be made available in a way that this resolution works out of the box. This means that the artifacts are made available under another name than they are normally accessible in Herd.
I had the same thought the other day, I think that'd be really handy indeed. Would you be willing to test this for me?
So Maven repo docs fail to mention these:
- http://central.maven.org/maven2/last_updated.txt (simple enough timestamp)
- view-source:http://central.maven.org/maven2/archetype-catalog.xml (not applicable: archetype list)
- http://central.maven.org/maven2/org/hibernate/maven-metadata.xml (not applicable: maven plugin list)
- http://central.maven.org/maven2/org/hibernate/hibernate-core/maven-metadata.xml (list of versions per groupId:artifactId)
The biggest problem would be that Ceylon modules don't have a group/artifact split. We can emulate one, by splitting on the last ., but it may be weird.
That's already what happens in the pom.xml stored in every car file:
<?xml version="1.0" ?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>ceylon.interop</groupId>
<artifactId>java</artifactId>
<version>1.2.2</version>
<name>ceylon.interop.java</name>
<dependencies>
<dependency>
<groupId>ceylon</groupId>
<artifactId>collection</artifactId>
<version>1.2.2</version>
</dependency>
</dependencies>
</project>
Alternatively, we could add an annotation in module.ceylon, something like:
mvn("ceylon", "interop.java")
module ceylon.interop.java 1.0.0 {
...
}
We can emulate one, by splitting on the last
., but it may be weird.
Why can't the group and artifact be identical: the module name?
That's a slippery slope right there -- adding Maven specific annotations to language module...
Maybe have a group annotation instead. This could be an informative tag for Herd, that could be used as a grouping of similar modules together.
And additionally, this would be used for mvn pom generation.
Lacking group annotation, group and artifact id could very well be module's full name
An example:
group("ceylon.sdk")
module ceylon.time 1.2.2 { ...}
Would generate following POM file:
<?xml version="1.0" ?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>ceylon.sdk</groupId>
<artifactId>ceylon.time</artifactId>
<name>ceylon.time</name>
<version>1.2.2</version>
<dependencies>
...
</dependencies>
</project>
Usually the groupid refers to the project. See http://central.sonatype.org/pages/choosing-your-coordinates.html
The groupId identifies your project uniquely across all projects and you control this section of the overall name-space.
I think a better default would be: groupId = Ceylon module name artifactId = last component of the Ceylon module name
Why can't the group and artifact be identical: the module name?
This might be the best default, because then the artifact would have already the right name for resolution.
Would you be willing to test this for me?
Yes. I will try to create some kind of integration test for this.
Note that if we change the group/artifact mapping in Herd, we will also want to change it in the generated pom.xml in the .car files…
I think Herd should simply extract the pom.xml from the car file, and build the correct hierarchy of folders + generate checksums, that should be enough to expose a Maven repo, right?
That is indeed another option. Except that:
- some files (jars) may not have any
pom.xml - Herd has more info than the
pom.xmlfiles, although cosmetic, such as authors, urls, licence, description, etc…
So it should be a mix of both, as we do when generating OSGI metadata in manifest: reuse information specified in the internal POM, and add any additional information that can be provided by Herd.
reuse information specified in the internal POM, and add any additional information that can be provided by Herd
Looks like a good compromise.
authors, urls, licence, description, etc…
Technically, poms can also contain such data, so we could fill it from module.ceylon if present (is Herd already doing this?).
I can, that's why I said that it's a bit richer if I generate the pom.xml rather than use the one inside the .car.
https://modules.ceylon-lang.org/maven/1/ceylon/language/1.2.1/language-1.2.1.pom https://modules.ceylon-lang.org/maven/1/ceylon/language/maven-metadata.xml
Can you guys try it out?
Supports .jar, .jar.sha1, -sources.jar, .-sourcesjar.sha1, .pom, .pom.sha1 and maven-metadata.xml.
Does not support javadoc yet but I'm pretty sure the Java tools would not be able to make sense of ceylondoc anyway.
Wow so fast! I did a small test with Gradle. It works like a charm!
plugins { id 'groovy' }
repositories {
maven {
name = 'ceylon-herd'
url = 'https://modules.ceylon-lang.org/maven/1/'
}
}
dependencies {
compile 'ceylon:language:1.2.1'
compile 'ceylon.interop:java:1.2.1'
compile 'ceylon:collection:1.2.1'
}
The sources are parsed as Java files, but this should be an Eclipse problem. In Netbeans sources are not shown for Ceylon sources. The Java source of Array.class in ceylon.language is shown, so I think Netbeans looks only for java files as sources :/.
I will try to proxy the Ceylon Herd server with a local Sonatype Nexus server as a second test.
I will have more time tomorrow then I will give you more detailed feedback. Do you think it's useful to have some kind of Gradle/Maven integration tests that can be run against the Herd server?
When I see those artifacts, I really think a fully qualified artifact id would be better.
having an annotation to specify a group id : can raise a problem if you want to deduce de GAV from the Ceylon module name because you don't have the information
having a scheme with multiple group ids can make a problem later if you want to put the same deps on maven central because usually you owns a single group id
at the same time, maven does not manage cyclic dependencies, so I don't see how you would publish ceylon cyclic dependencies in a maven repo.
ATM @vietj has parts of the distrib published at http://mvnrepository.com/artifact/org.ceylon-lang under the org.ceylon-lang group.
Using ceylon.language:ceylon.language as coordinates would make publishing to Maven Central harder has every groupId has to be registered (same as on Herd BTW, and we don't see that as a big problem although we do have an issue open to claim domains or wildcards).
Another option is to use a common groupId for every Ceylon module: ceylon:ceylon.language.
But what is about other projects? Let's assume somebody wants to publish to Maven Central and Herd. In this situation it would be very important to be able to control artifactid and groupid. Otherwise the dependencies might be downloaded from the Maven repository and the Herd repository and both would get in the classpath.
I begin to think that the best idea is to have an annotation that ensures compatibility. Ivy, Maven and many other tools use at least group id and artifact id for resolution. For Ceylon modules an annotation for group id might be enough, but when people want to upload JARs to Herd and Maven they may want to use their existing groupid and artifact-id-scheme.
So maybe an annotation that can override group id and artifact id for repositories that support both coordinates would be the most interoperable way. The developer has to decide if she wants to use "Herd-mode" (Module-Version) or "Maven-mode" (Group-Module-Version) for module resolution.
why not use the same coordinates that already are in maven central?
I agree that org.ceylon-lang feels natural.
Here's another idea: how about org.ceylon-lang.modules? i.e. the address of modules in Herd?
If you don't change the group, you don't need to go through the sonatype process again to determine if you own the groupId or not.
Here's another idea: how about org.ceylon-lang.modules? i.e. the address of modules in Herd?
Imagine a big project that uses a Ceylon module which itself depends on a Java-Jar module uploaded to Herd (possible?). The Ceylon module is only published in Herd, so it refers to the Jar in the pom as org.ceylon-lang.modules:com.company.project.module. The big project uses the same library itself, but uses the normal Maven project coordinates. Now the same module is mentioned twice: com.company.project:module and org.ceylon-lang.modules:com.company.project.module.
edit: Oh this could happen the other way around already, am I right?
With an repository interoperability annotation this problem would not occur: When Herd simulates a Maven repository it would always use the coordinates specified by the annotation (also for transitive dependencies). The CMR could also use the annotation for duplicate detection.
The only question that would remain is, how the groupid is, when the module does not use the interoperability annotation.
I think in the future in the light of Jigsaw Maven and Ivy artifact resolvers will support multi-version resolution of dependencies. So I think finding a good solution for mapping artifacts between Herd and Maven/Ivy is crucial.
It just occured to me that this can also become a very important security problem. Bad users could use the Herd repository to inject manipulated Jar dependencies in projects that use Maven and Herd at the same time. You would always have to check Maven Central for similar groupids when accepting a new module into Herd...
With this in mind using "org.ceylon-lang.modules" as group id for all modules provided by Herd is probably the most safe option. Duplicate dependencies would have to be solved per project, which is an acceptable mid-term solution.
A possible long term solution would be, that when you want to link an artifact between Herd and Maven the POM at the Maven repository must contain a key generated by Herd that proves ownership of the group id... tedious :(.