sbt Can't use Microsoft Azure DevOps Maven Repository as Resolver

Ref https://github.com/sbt/sbt/issues/2366 Ref https://github.com/sbt/sbt/issues/5264

steps

Add a Microsoft Azure DevOps Maven repository as resolver with Coursier throws:

[warn]         Note: Unresolved dependencies path:
[error] stack trace is suppressed; run last update for the full output
[error] (update) sbt.librarymanagement.ResolveException: Error downloading com.foo:bar_2.11:1.0.0
[error]   Not found
[error]   Not found
[error]   not found: /Users/mantovani/.ivy2/local/com.foo/bar_2.11/1.0.0/ivys/ivy.xml
[error]   not found: https://repo1.maven.org/maven2/com/foo/bar_2.11/1.0.0/bar_2.11-1.0.0.pom
[error]   checksum format error: /Users/mantovani/Library/Caches/Coursier/v1/https/pkgs.dev.azure.com/Foo-backend/Foo/_packaging/Foo-releases/maven/v1/com/foo/bar_2.11/1.0.0/bar_2.11-1.0.0.pom.sha1

ThisBuild / name := "baz"
ThisBuild / organization := "com.foo"
lazy val scala211 = "2.11.12"

crossScalaVersions := Seq(scala211)
ThisBuild / scalaVersion := scala211

libraryDependencies ++= Seq(
  "foo" %% "bar" % 0.1 % "provided"
)

ThisBuild / useCoursier := true

resolvers +=   "nabu-releases" at "https://pkgs.dev.azure.com/foo/Foo/_packaging/foo-releases/maven/v1"
credentials += Credentials(Path.userHome / ".sbt" / ".credentials")

sbt version

[info] This is sbt 1.3.9

sbt:foo> show credentials
[info] * FileCredentials("/Users/mantovani/.sbt/sonatype_credential")
[info] * FileCredentials("/Users/mantovani/.sbt/.credentials")

cat /Users/mantovani/.sbt/.credentials

realm=foo
host=pkgs.dev.azure.com
user=foo
password=xxx

The checksum actually has a HTML from Azure:

cat /Users/mantovani/Library/Caches/Coursier/v1/https/pkgs.dev.azure.com/nabu-backend/Nabu-Bots/_packaging/nabu-releases/maven/v1/com/modak/spark-profiler_2.11/1.0.6-2.4.0/spark-profiler_2.11-1.0.6-2.4.0.pom.sha1


<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">




<html lang="en-US">
<head><title>

            Azure DevOps Services | Sign In

</title><meta http-equiv="X-UA-Compatible" content="IE=11;&#32;IE=10;&#32;IE=9;&#32;IE=8" />
    <link rel="SHORTCUT ICON" href="/favicon.ico"/>

    <link data-bundlelength="508502" data-bundlename="commoncss" data-highcontrast="/_static/tfs/M166_20200303.4/_cssbundles/HighContrast/vss-bundle-commoncss-vJ2fST3DWywo5reQpe4kluM0dIKSswAv-7hLhXVoyoEU=" data-includedstyles="jQueryUI-Modified;Core;Splitter;PivotView" href="/_static/tfs/M166_20200303.4/_cssbundles/Default/vss-bundle-commoncss-vRG44Xj08_ItEwhXa3JU14uvR-wUjc7PMTHxSYSSCaHw=" rel="stylesheet" />
<link data-bundlelength="116152" data-bundlename="viewcss" data-highcontrast="/_static/tfs/M166_20200303.4/_cssbundles/HighContrast/vss-bundle-viewcss-vdlgmvK3r7d2NJT2HxmrnSW5uGherQCq6dVLcVJGo_uE=" data-includedstyles="VSS.Controls" href="/_static/tfs/M166_20200303.4/_cssbundles/Default/vss-bundle-viewcss-vu75e1r3CYt-i_9HTv9byA67VrRg063mttS0GgT3kje8=" rel="stylesheet" />

If I turn off CoursierThisBuild / useCoursier := false I have the following error from ivy:

sbt:foo> test
[error] Unable to find credentials for [https://pkgsprodcus1.pkgs.visualstudio.com/ @ pkgs.dev.azure.com].
[error]   Is one of these realms misspelled for host [pkgs.dev.azure.com]:
[error]   * Foo-releases
[error] Unable to find credentials for [https://pkgsprodcus1.pkgs.visualstudio.com/ @ pkgs.dev.azure.com].
[error]   Is one of these realms misspelled for host [pkgs.dev.azure.com]:
[error]   * Foo-releases
[error] Unable to find credentials for [https://pkgsprodcus1.pkgs.visualstudio.com/ @ pkgs.dev.azure.com].
[error]   Is one of these realms misspelled for host [pkgs.dev.azure.com]:
[error]   * Foo-releases
[error] Unable to find credentials for [https://pkgsprodcus1.pkgs.visualstudio.com/ @ pkgs.dev.azure.com].
[error]   Is one of these realms misspelled for host [pkgs.dev.azure.com]:
[error]   * Foo-releases
[error] Unable to find credentials for [https://pkgsprodcus1.pkgs.visualstudio.com/ @ pkgs.dev.azure.com].
[error]   Is one of these realms misspelled for host [pkgs.dev.azure.com]:
[error]   * Foo-releases

Apr 01 '20 21:04 mantovani

For the record, I think the publishing side will be fixed by https://github.com/sbt/ivy/pull/36. The resolving side would need to be handled by Coursier, so this should be reported to coursier/coursier.

Apr 05 '20 19:04 eed3si9n

@eed3si9n I open a ticket with as well Coursier https://github.com/coursier/coursier/issues/1649

Apr 06 '20 10:04 mantovani

In order to make it work with sbt 1.3.10 you have to create a null realm credential and disable Coursier like:

ThisBuild / useCoursier := false
credentials += Credentials(null, "my.artifact.repo.net", "admin", "****")

If you do use a file just let realm blank.

Apr 16 '20 08:04 mantovani

@mantovani checked out the url response headers and the realm is reported as www-authenticate →Basic realm="https://pkgsprodeus21.pkgs.visualstudio.com/" not foo

Apr 27 '20 21:04 gilandose

I attempted to tackle this issue and found that even if the HTTP realm is an Option in sbt/coursierint there was still room for an NPE, #5526 seems to fix it.

I'm still unsure it needs any work on upstream coursier as stated in coursier/coursier#1649, see my latest comment in #5483... There might be an User Agent issue here as well. I tried logging UA from coursier HTTP requests, it sets the UA header with an empty value, I'll try to set it to Apache Maven or unsetting it to see what happens.

May 03 '20 14:05 3rwww1

Still have this problem for sbt > 1.2.8

Feb 01 '21 10:02 agolovenko

The problem seems to be in the default coursier configuration.

csrConfiguration := {
    val conf = csrConfiguration.value
    val resolvers = csrResolvers.value
    val adoRepoHost = "pkgs.dev.azure.com"
    val adoCredentialsOpt = credentials.value.collectFirst { case creds: DirectCredentials if creds.host == adoRepoHost => creds }
    val newConfOpt = adoCredentialsOpt.map { adoCredentials =>
      val auths =
        resolvers
          .collect {
            case repo: MavenRepository if repo.root.startsWith(s"https://$adoRepoHost/") =>
              repo.name -> Authentication(adoCredentials.userName, adoCredentials.passwd)
          }
      auths.foldLeft(conf) { case (conf, (repoId, auth)) => conf.addRepositoryAuthentication(repoId, auth) }
    }
    newConfOpt.getOrElse(conf)
  }

This snippet manually adds authentication information to resolution requests towards Azure Artifacts servers. After that the artifacts are resolved as usual.

The checksum error occurs because Azure Artifacts sometimes returns an HTTP 203 response. The payload of the response is not the original pom.xml though. It is an html page telling you to login somewhere somehow.

Coursier treats 2XX response codes as successes so even if you disable the checksum checking you still get some SAX exception with no stacktrace in the error log.

May 28 '21 21:05 markoutso

I'm dealing with this problem myself. Unfortunately I tried the snippet above with sbt 1.5.5, and it didn't seem to help.

I do notice that my organization has a "(orgname).pkgs.visualstudio.com" host instead, and swapped it out as the repo host, but no joy.

Jul 23 '21 20:07 SattaiLanfear

I'm dealing with this problem myself. Unfortunately I tried the snippet above with sbt 1.5.5, and it didn't seem to help.

I'm dealing with right now as well. But for me update works and updateClassifiers attempts to download the same POMs as update and then tries to download it again and it will fail with the unauthorized html in the pom/sha.

edit: We have resolved this issue by applying the csrConfiguration to the updateClassifiers task as well as it is specifically determined in Defaults.scala.

.settings(
  csrConfiguration := { ..
  },
  updateClassifiers / csrConfiguration := csrConfiguration.value
}

Aug 17 '21 10:08 spockz

Hello,

This issue is still happening for us, and it's really annoying.

What fixes it for us is including this sbt-coursier version:

addSbtPlugin("io.get-coursier"  % "sbt-coursier" % "1.0.3")

Before sbt-coursier version 1.0.3 we were doing this:

coursierChecksums := Nil,
coursierArtifactsChecksums := Nil,

But depending on sbt-coursier creates other problems and I'd rather not have it in plugins.sbt.

Sep 30 '21 12:09 alexandru

To add extra details, this solution DOES NOT work:

// Does not work for us
.settings(
  csrConfiguration := {
    // ...
  },
  updateClassifiers / csrConfiguration := csrConfiguration.value
}

Oct 19 '21 13:10 alexandru

One more — adding the sbt-coursier plugin fixes the problem, but there's an extra problem ...

The .sha1 and .pom get downloaded from Azure DevOps on every sbt update. This makes the update, a normally slow operation, to be even slower. In a multi-project setup, I'm seeing update take 35+ seconds.

Oct 19 '21 13:10 alexandru

This is still a problem for us. Unable to publish jar to Azure DevOps from SBT>

Any alternative ways to deal with it?

Dec 24 '21 14:12 purijatin

.settings( csrConfiguration := { .. }, updateClassifiers / csrConfiguration := csrConfiguration.value }

Using the above I'm able to successfully download artefacts from an ADO Maven repository.

edit: We have resolved this issue by applying the csrConfiguration to the updateClassifiers task as well as it is specifically determined in Defaults.scala.

The addition to the updateClassifiers key is necessary, but somehow blocks sources from being fetched when running sbt updateClassifiers

EDIT: Fetching sources was fixed by adding the following to the Coursier config:

.withClassifiers(Vector("sources"))
.withHasClassifiers(true)

Jan 10 '22 13:01 Roymprog

The addition to the updateClassifiers key is necessary, but somehow blocks sources from being fetched when running sbt updateClassifiers

That is interesting because that is exactly how we use it to make it able to download sources. Without it it lacks the proper settings.

Jan 10 '22 20:01 spockz

The below worked for me in the interim: https://github.com/arktekk/sbt-aether-deploy

# Use 0.26.0 for JDK8. I think the versions above that use JDK11
addSbtPlugin("no.arktekk.sbt" % "aether-deploy" % "0.26.0")

Able to push using the above plugin and following settings:


    publishMavenStyle := true,
    publishTo := {
      val url = "https://pkgs.dev.azure.com/msazure/One/_packaging/AIOps-Core-Model/maven/v1/"
      Some("releases".at(url))
    },
    credentials += Credentials(Path.userHome / ".sbt" / ".credentials")

Jan 11 '22 07:01 purijatin

I am having issues publishing to an azure feed, but different. I get

[error] (publish) java.net.ProtocolException: Server redirected too many times (20)

I can publish a regular maven project just fine, but sbt is giving me problems.

ThisBuild / version := "0.1.0-SNAPSHOT"
ThisBuild / scalaVersion := "2.13.10"
ThisBuild / versionScheme := Some("early-semver")

lazy val root = (project in file("."))
  .settings(
    name := "untitled",
    publishMavenStyle := true,
    publishTo := Some("tsmaven" at "https://[SELF HOSTED ADDRESS]/[ORG]/_packaging/tsmaven/maven/v1"),
    credentials += Credentials (Path.userHome / ".sbt" / "azureArtifactsFeed.credentials")
  )

Dec 20 '22 14:12 schragnasher

Can we get help on this, please? Still unable to publish the Azure DevOps Maven repository and seeing the below error: [error] (publish) java.net.ProtocolException: Server redirected too many times (20)

Dec 22 '22 16:12 KeerthiYandaOS

This still keeps happening.

The solution seems to be to set the realm to null. Since we're using a credentials file, we had to resort to doing this something like this:

ThisBuild / credentials += {
  // https://github.com/sbt/sbt/issues/5492#issuecomment-614492826
  val file = Path(sys.env.getOrElse("SBT_CREDENTIALS_PATH", Path.userHome.toString)) / ".sbt" / ".credentials"
  val props = new java.util.Properties()
  props.load(new java.io.FileInputStream(file))
  Credentials(
    realm = null,
    host = props.getProperty("host"),
    userName = props.getProperty("user"),
    passwd = props.getProperty("password")
  )
}

Jan 17 '23 11:01 alexandru

This still keeps happening.

The solution seems to be to set the realm to null. Since we're using a credentials file, we had to resort to doing this something like this:

ThisBuild / credentials += {
  // https://github.com/sbt/sbt/issues/5492#issuecomment-614492826
  val file = Path(sys.env.getOrElse("SBT_CREDENTIALS_PATH", Path.userHome.toString)) / ".sbt" / ".credentials"
  val props = new java.util.Properties()
  props.load(new java.io.FileInputStream(file))
  Credentials(
    realm = null,
    host = props.getProperty("host"),
    userName = props.getProperty("user"),
    passwd = props.getProperty("password")
  )
}

This does not help with my issue, I am using a null/empty realm in the creds file. Still get too many redirects, and aether deploy continually give unauthorized even when i knew the creds are good from publishing other maven projects.

Jan 31 '23 15:01 schragnasher