coursier
coursier copied to clipboard
coursier
[Question] - Retrieving sbt plugin versions from hosted Sonatype Nexus
At my work we use Scala Steward to update all of our internal dependencies. We use a hosted Sonatype Nexus to both have a mirror of Maven Central and also to have our own internal releases Maven repo. From looking through the Scala Steward code it seems that it relies on Coursier to retrieve the list of versions available for the artifact.
https://github.com/scala-steward-org/scala-steward/blob/a2cea0f1f72f13b38f3b5cb01ae3cad04a58aff0/modules/core/src/main/scala/org/scalasteward/core/coursier/VersionsCache.scala#L54
If I understand it right, this is done for artifacts that have a maven-metadata.xml file, but as you know, sbt plugins that are published Maven style don't have one. Which again, to my understanding because of this, Coursier then relies on the directory listing? This is how it seems to work for our maven-central mirror with the sbt-scalafmt plugin. By looking at it with a path like this, I can see an html directory listing of all versions:
https://nexus-redacted.com/repository/maven-public/org/scalameta/sbt-scalafmt_2.12_1.0
Scala Steward (Coursier) is able to correctly gather the versions (again I assume from this and sounds like that from this thread in Steward). However, I'm finding a lot of contradictory information about whether or not this is available on different versions of Nexus3. Plus those that manage our Nexus is saying there is no option to turn this on.
- Community post saying that it exists but is at a different path now <-- This seems to be the case
- Sonatype ticket to create this feature
- Another ticket for the raw tree view
- Announcement saying it's now supported in 3.7
- SO post saying it's in a different location now
From what I can tell, for our own internal nexus where I'd expect the directory listing to be like the sbt-scalafmt example above, it just 404's.
https://nexus-redacted.com/repository/redacted-releases/redacted/redacted/sbt-redacted_2.12_1.0
There is an html index of sorts at this url:
https://nexus-redacted.com/service/rest/repository/browse/redacted-releases/redacted/redacted/sbt-redacted_2.12_1.0/
However, looking through the code here, I don't believe that will work: https://github.com/coursier/coursier/blob/59bf7739079f6b906ecb77435c8afce2d3658f2c/modules/core/shared/src/main/scala/coursier/maven/MavenRepository.scala#L208-L252
And when Scala Steward attempts to get the available versions of the plugin, it gets none and therefore doesn't send in prs for our internal sbt plugins. Which sort of leads me to believe that because of the lack of directory listing there, Coursier isn't getting the versions? Do you know if it's possible for Coursier to get the versions of an sbt plugin without the directory listing, or does that have to be there? Or could there be another check possible to capture this alternative html view the same way the other is captured?
EDIT: For the time being we just add in a custom release step to create / update the maven-metadata.xml file, but it'd still be great to see if what I mentioned above is the case or not.
@ckipp01 Could you share the code of your custom release step for maven-metadata.xml? struggling with the same problem currently
@ckipp01 We are also encountering this issue when trying to utilize scala-steward for our internal sbt plugins. We tried to use arktekk/sbt-aether-deploy to generate/update the maven-metadata.xml, but Nexus3 complains that the file is invalid when publishing.
Could you briefly outline how you were able to generate the correct maven-metadata.xml?
So this is a pretty hacky solution, but we essentially just included this as a custom release step (if you use sbt-release).
import java.nio.charset.StandardCharsets
import java.nio.file.Paths
import java.text.SimpleDateFormat
import java.util.Base64
import sbtrelease.ReleasePlugin.autoImport.ReleaseStep
import sbt.Project
import sbt.Keys
import sbt.internal.util.ManagedLogger
import Env._
import scala.xml.Elem
import scala.xml.Node
import scala.xml.NodeSeq
import scala.xml.XML
object MetaDataUpdate {
/** An extra release step that is sort of hack specifically for an sbt plugin.
* This step will look to see if there is an existing `maven-metadata.xml` file
* (which sbt plugins don't produce when published maven style) and add the new
* version to it or create a new one and push it up.
*/
val step: ReleaseStep = ReleaseStep { st =>
val logger = st.log
val extracted = Project.extract(st)
val name = extracted.get(Keys.name)
val org = extracted.get(Keys.organization)
val version = extracted.get(Keys.version)
val target = extracted.get(Keys.target)
val metadataName = "maven-metadata.xml"
val creds = Base64.getEncoder
.encodeToString(s"${nexusUsername}:${nexusPassword}".getBytes(StandardCharsets.UTF_8))
val metadataLoc =
s"${nexusAddress}/${nexusRepositoryPath}${org.replace(".", "/")}/${name}_2.12_1.0/${metadataName}"
val outputPath = target.getPath() + metadataName
val existingResponse = requests.get(
metadataLoc,
headers = Map(
"Authorization" -> s"Basic: ${creds}"
),
check = false
)
if (existingResponse.is2xx) {
val rawText = existingResponse.text()
val existingMetadata = XML.loadString(rawText)
val versions = existingMetadata \\ "version"
val metadata = createMetadata(org, name, version, Some(versions))
logger.info("Existing maven-metadata.xml found and being updated...")
saveAndSend(metadataLoc, metadata, outputPath, creds, logger)
} else if (existingResponse.statusCode == 404) {
logger.warn("Existing maven-metadata.xml file not found. Attempting to create a new one...")
val metadata = createMetadata(org, name, version, None)
saveAndSend(metadataLoc, metadata, outputPath, creds, logger)
} else {
logger.error(s"${existingResponse.statusCode.toString()}: ${existingResponse.statusMessage}")
// If something goes wrong here we just blow up
sys.error("Something went wrong when requesting maven-metaldata.xml.")
}
st
}
private def createMetadata(
org: String,
name: String,
newVersion: String,
existingVersions: Option[NodeSeq]
) = {
val newVersionNode = <version>{newVersion}</version>
val versions = existingVersions match {
case Some(existing) => existing ++ newVersionNode
case None => newVersionNode
}
<metadata modelVersion="1.1.0">
<groupId>{org}</groupId>
<artifactId>{name}_2.12_1.0</artifactId>
<versioning>
<latest>{newVersion}</latest>
<release>{newVersion}</release>
<versions>
{versions}
</versions>
<lastUpdated>
{new SimpleDateFormat("yyyyMMddHHmmss").format(new java.util.Date)}
</lastUpdated>
</versioning>
</metadata>
}
private def saveAndSend(
metadataLoc: String,
metadata: Elem,
path: String,
creds: String,
logger: ManagedLogger
): Unit = {
XML.save(path, metadata)
val putReponse = requests.put(
metadataLoc,
headers = Map(
"Authorization" -> s"Basic: ${creds}"
),
data = Paths.get(path)
)
if (!putReponse.is2xx) {
logger.error(s"${putReponse.statusCode}: ${putReponse.statusMessage}")
sys.error("Unable to update metadata")
}
}
}
It's pretty self-explanatory, but it will just check to see if there is a metadata file, and if not create one. If it does fine one, it just updates it. It's using requests-scala for the calls.
Which sort of leads me to believe that because of the lack of directory listing there, Coursier isn't getting the versions?
I'm a bit late to the party here, but yes, for sbt plugins, we rely on directory listings. As the plugin artifacts' paths aren't standard (the _2.12_1.0 appears in a directory name, but not in the filenames), it seems Sonatype doesn't generate or update maven-metadata.xml files itself.
Do you know if it's possible for Coursier to get the versions of an sbt plugin without the directory listing, or does that have to be there? Or could there be another check possible to capture this alternative html view the same way the other is captured?
If you push a maven-metadata.xml file yourself, as you do, there should be a way to have it picked… It should actually be checked if getting the directory listing fails (via the orElse). Maybe this code doesn't look for maven-metadata.xml at the right URL? It could be fixed then.
If you push a
maven-metadata.xmlfile yourself, as you do, there should be a way to have it picked… It should actually be checked if getting the directory listing fails (via theorElse). Maybe this code doesn't look formaven-metadata.xmlat the right URL? It could be fixed then.
Just to clarify, yes, it does work now with us creating the maven-metadata.xml file, but was curious if there was a solution where having that step to manually create one for sbt plugins on self-hosted nexus would no longer be necessary.
thanks for the great workaround, on my end I had to add check = false for this to work otherwise scala-requests 0.6.5 would throw
val existingResponse = requests.get(
metadataLoc,
headers = Map(
"Authorization" -> s"Basic: ${creds}"
),
check = false
)
Hi! I came here from https://github.com/scala-steward-org/scala-steward-action/issues/209#issuecomment-799687563 and then https://github.com/scala-steward-org/scala-steward/issues/1628.
I have the same problem but for libraries (not sbt-plugins) published maven-style. There is no maven-metadata.xml and Scala Steward (using Coursier) can't find any updates.
Is there a way without the workaround with manually creating maven-metadata.xml?
@ckipp01 based upon the coursier code path, it looks like if you set: val sbtModule: ModuleID = Defaults.sbtPluginExtra("com.example" % "sbt-plugin-example" % "0.1.0", scalaVersion.value, scalaBinaryVersion.value) these two attributes will enable: https://github.com/coursier/coursier/blob/master/modules/core/shared/src/main/scala/coursier/maven/MavenRepository.scala#L284
Do you know if this was being set to true and still failing?
Hey @er1c, ironic you are commenting on this now 😆 I was actually just looking at something related because I'd really love to get complete working on a private nexus like it does on maven central, and also for snapshots. There is a related issue to this one here: https://github.com/coursier/coursier/issues/1700 where it gives some more details, but all my digging sort of shows that much of this is related.
To answer your question, yes, even with that it won't work. It will work if it's a mirrored sbt plugin, or an sbt plugin on maven central. Mainly because in both of those cases the metadata file is where it's supposed to be, or there is an html index that can be scraped in the expected place. Once you have a self-hosted nexus or you are trying to complete snapshots, this all sort of goes out the window. Also somewhat related there are some crazy hacks people are coming up with to try to get around all this such as https://discord.com/channels/632150470000902164/635669047588945930/897544818878709841
@ckipp01 thanks for the update, I'm playing with the s3 plugin, and got parts of it coursier-friendly, but the scala steward is still a problem: https://gist.github.com/er1c/f149b74673e4a1e2120c7f5b06483e91#file-build-sbt-L89 & https://github.com/EVENFinancial/fm-sbt-s3-resolver/blob/master/src/sbt-test/fm-sbt-s3-resolver/scala-steward/src/test/scala/ScalaStewardTest.scala#L23
I've also had to (not published yet) split out the java dependency so it can be added to the csrConfiguration's extra class path: https://github.com/coursier/sbt-coursier/commit/92e40c22256bea44d1e1befbef1cb2a627f8b155#diff-910da7b2f4f62901095520e1d89c9f9b2fae0b2a61e39a15a6fde3890495d482R181 but that doesn't seem to get it to work actually calling the directory listing either (although my Cache/config still might be not setup properly)
I've changed one of our libraries to publish using https://github.com/arktekk/sbt-aether-deploy rather than sbt publish, to see if Steward could then see the updates- but no difference. the aether deploy does seem to be pushing metadata.
Looking around other threads I wonder if the issue is that CodeArtifact doesn't allow directory listings, and Coursier needs that. (I thought it was supposed to fall back to metadata though, but perhaps I've misunderstood that)
I came some years late to the party with the same problem, did someone find/implement a permanent fix?
