sbt
sbt copied to clipboard
sbt
Consider bumping `scala-xml` to 2.x
Problem outline
Not really a bug, just something a bunch of people are going to hit on. I've reported this to scala/bug, but these updates may have to go hand in hand if we do them. I'll copy over what I posted there since the reasoning is the same:
I know this was a conversation a long time ago in https://github.com/scala/scala/pull/9743 but recently this conversation has surfaced again. Recently in
sbt-scoverageI had some reports of conflicts since I was still on 1.x, but some other libraries like scalatest bumped to 2.x making the problem hit people a bit harder. Due to that, I went ahead and updated to 2.x. Due to me updating, we got even more reports of conflicts with sbt-native-packager, which caused them to update to 2.x as well. Now we're in a position where people are getting hit with stuff like this:[error] * org.scala-lang.modules:scala-xml_2.12:2.1.0 (early-semver) is selected over {1.3.0, 1.0.6, 1.2.0} [error] +- org.scoverage:scalac-scoverage-reporter_2.12:2.0.2 (depends on 2.1.0) [error] +- com.github.sbt:sbt-native-packager:1.9.10 (sbtVersion=1.0, scalaVersion=2.12) (depends on 2.1.0) [error] +- org.scala-sbt:testing_2.12:1.7.1 (depends on 1.3.0) [error] +- org.scala-sbt:sbinary_2.12:0.5.1 (depends on 1.0.6) [error] +- org.scala-sbt:main_2.12:1.7.1 (depends on 1.3.0) [error] +- org.scala-sbt:librarymanagement-core_2.12:1.7.0 (depends on 1.2.0) [error] +- org.scala-lang:scala-compiler:2.12.16 (depends on 1.0.6) [error] +- io.get-coursier:lm-coursier-shaded_2.12:2.0.10 (depends on 1.3.0)Where common sbt plugins are conflicting with sbt and Scala 2.12 itself. Looking back at past conversations I understand the breaking changes between 1.x and 2.x of scala-xml are extremely small so doing the following is pretty safe:
in your plugins.sbt
ThisBuild / libraryDependencySchemes ++= Seq( "org.scala-lang.modules" %% "scala-xml" % VersionScheme.Always )But this isn't ideal to have this all over the place, and will cause confusions for users. Is it time to update scala-xml in 2.12? Is there something holding this back? We're in a weird position now where have the ecosystem is update for 2.12 and half isn't, causing issues.
Related to this issue, is there any plan to update to Scala 2.13? 🤔
From my understanding of past discussions, no. Updating to 2.13 would have huge ramifications to the sbt plugin ecosystem causing all plugins to have to update to 2.13. So I believe the plan is for sbt 2.x to be on Scala 3, so the plugin ecosystem would only need to go through that migration once. @eed3si9n can correct me if I'm wrong on that.
Commented on https://github.com/scala/bug/issues/12632#issuecomment-1209906363
Yea. Unless something prevents us from going to Scala 3, sbt 2.x should be on 3 (see https://eed3si9n.com/sudori-part3/ etc).
would declaring the dependency as "[1.3,3.0[" maybe work as intended?
Well, this issue is blocking Twirl (see https://github.com/playframework/twirl/pull/548#issuecomment-1278520071), which is blocking Caliban (see https://github.com/ghostdogpr/caliban/pull/1472)
As suggested by @mkurz in his comment in the Twirl repo, can we maybe get a 1.8.0-M1 release using Scala 2.12.17 to be able to unblock everyone, maybe?
To be honest that whole scala-xml upgrade thinig is a mess. The fact that sbt fails with an error on startup just by upgrading a librarie's patch version because it upgraded scala-xml is very very annoying.
Next time this has should be coordinated somehow (next candidate is scala-parser-combinators...
Next time this has should be coordinated somehow
I'm not against but, IMO, it's quite hard to coordinate all these projects maintained by many different people unless one person or a small group of persons takes the responsibility to coordinate everything.
The few of us who are involved in this scala-xml migration in Scala 2.12 and sbt are making PRs here and there to update things. It takes time, it's open-source work made in people's free time. Without this person (or small group) coordinating and making the required work, I don't see how it can be done more quickly.
True, but bumping in patch releases is also not the best idea. That crashes sbt just because upgrading sbt-native-packager from 1.9.9 to 1.9.10... That's what's really bothers me.
True, but bumping in patch releases is also not the best idea. That crashes sbt just because upgrading sbt-native-packager from 1.9.9 to 1.9.10... That's what's really bothers me.
And suddenly I have to talk to 5 other maintainers to convice them to upgrade scala-xml because I can't, in a clean way, upgrade a patch version...
For what it's worth, I understand your frustration, but this thread on sbt probably isn't the correct place to air it as it's not the fault of sbt at all.
There is more context in here but this wasn't the first time scala-xml updates have caused issues. Projects have bumped back and forth to handle this, published multiple versions, etc. I totally agree that it's a bit of a nightmare. I'll even take part of the blame as I'm the one who bumped scoverage. However, I bumped because someone else bumped and I got complaints about it. That project probably bumped because someone else bumped. Thus is the nature of developing with Scala, the JVM, semver, etc.
As @guizmaii mentioned these things are really hard to coordinate without a single person at the helm guiding it, but even with that, you'll have projects that won't bump or respond, etc. There are various open prs and issues around the ecosystem to bump. I encourage people to continue to push for this, comment on issues, and lend a hand where they can. This will hopefully get everything over the tipping point and we can forget about scala-xml 1.x and this issue for good.
This will hopefully get everything over the tipping point and we can forget about scala-xml 1.x and this issue for good.
scala-parser-combinators like: hold my beer :wink:
@eed3si9n do you have any timeline on when you'd like to get 1.8.x out? I noticed that with the latest 1.7.3 even just using addDependencyTreePlugin globally (which I normally do) causes issues:
[error] java.lang.RuntimeException: found version conflict(s) in library dependencies; some are suspected to be binary incompatible:
[error]
[error] * org.scala-lang.modules:scala-xml_2.12:2.1.0 (early-semver) is selected over {1.3.0, 1.2.0, 1.0.6}
[error] +- org.scala-lang:scala-compiler:2.12.17 (depends on 2.1.0)
[error] +- org.scala-sbt:testing_2.12:1.7.3 (depends on 1.3.0)
[error] +- org.scala-sbt:sbinary_2.12:0.5.1 (depends on 1.0.6)
[error] +- org.scala-sbt:main_2.12:1.7.3 (depends on 1.3.0)
[error] +- org.scala-sbt:librarymanagement-core_2.12:1.7.1 (depends on 1.2.0)
[error] +- io.get-coursier:lm-coursier-shaded_2.12:2.0.12 (depends on 1.3.0)
I've seen others report this as well. However it's not clear where 2.12.17 is coming in from in this scenario. This also made me realize it wasn't bumped in lm-coursier-shaded, so I sent in https://github.com/coursier/sbt-coursier/pull/427.
Either way the ball is sort of rolling, so how do we think it's best to proceed, and how can I help?
Either way the ball is sort of rolling, so how do we think it's best to proceed, and how can I help?
Maybe get sbt 1.7.4 out fast out with a coursier version that does not pull in Scala 2.12.17 yet?
Maybe get sbt 1.7.4 out fast out with a coursier version that does not pull in Scala 2.12.17 yet?
Why roll back when we can move forward? Honestly the faster all of this is addressed and pushed forward the better. I sort of consider Scala itself moving to 2.x in 2.12.x as the official flag waving to the rest of the ecosystem to move forward. That's why I ended up doing https://github.com/coursier/coursier/pull/2548 since I figured "let's follow Scala". Plus, it's a Milestone release. So in Semver terms I figured it was safe to do with at the time. Maybe should't have assumed that seeing coursier pretty much only releases milestones and that could cause confusion when bumping.
Even if we roll back, there are still tons of users out there with a variation of
ThisBuild / libraryDependencySchemes += "org.scala-lang.modules" %% "scala-xml" % VersionScheme.Always
Somewhere in their build or metabuild. So imo rolling back elongates the pain. That's why I mention, what can we collectively do to push this forward? I'd strongly prefer a 1.8.x with scala-xml 2.x than a new 1.7.x rolling back coursier/scala-xml.
@eed3si9n do you have any timeline on when you'd like to get 1.8.x out?
I don't think we are waiting on anything, so we could potentially ship RC-1 as soon as this week. /cc @adpi2
It's weird that setting VersionScheme.Always for scala-xml doesn't work for my local machine (Apple M1 Max, macOS Monterey v12.6) if I use sbt 1.7.3. It's fine in GitHub Actions (Ubuntu).
If I use sbt 1.7.2, VersionScheme.Always works for both my local one and GHA.
It's weird that setting VersionScheme.Always for scala-xml doesn't work for my local machine (Apple M1 Max, macOS Monterey v12.6) if I use sbt 1.7.3
Sounds like a separate bug report, if you can manage to come up with reproduction steps that work for others
@SethTisue I just tested it with another Apple M1 (Pro) and it's fine there. 😵💫 I'll try to find difference between them and will report it if I find.
It's weird that setting VersionScheme.Always for scala-xml doesn't work for my local machine (Apple M1 Max, macOS Monterey v12.6) if I use sbt 1.7.3
Sounds like a separate bug report, if you can manage to come up with reproduction steps that work for others
For reproduction clone: https://github.com/ajozwik/quill-generic And change build.properties to 1.7.3
@SethTisue I just tested it with another Apple M1 (Pro) and it's fine there. 😵💫
This sounds like it could be something in your global ~/.sbt/1.0/plugins.sbt or something. For example if you're using addDependencyTreePlugin locally it will fail for any project on your machine using 1.7.3. So fon't forget to check there.
. For reproduction clone: https://github.com/ajozwik/quill-generic And change build.properties to 1.7.3
Locally with this repo bumping to 1.7.3 gives me no issues. Same as above, it could be something in your global config actually causing an issue here.
. For reproduction clone: https://github.com/ajozwik/quill-generic And change build.properties to 1.7.3
Locally with this repo bumping to 1.7.3 gives me no issues. Same as above, it could be something in your global config actually causing an issue here.
Yes, I have: addDependencyTreePlugin in ~/.sbt/1.0/plugins/plugins.sbt
I am guessing that in some situations sbt-coursier's Scala version (2.12.17) is bubbling up. I've opened https://github.com/coursier/coursier/issues/2573 to request 2.12.16 version of sbt-coursier for 1.7.x branch.
@SethTisue I just tested it with another Apple M1 (Pro) and it's fine there. 😵💫
This sounds like it could be something in your global
~/.sbt/1.0/plugins.sbtor something. For example if you're usingaddDependencyTreePluginlocally it will fail for any project on your machine using 1.7.3. So fon't forget to check there.
@ckipp01 Yeah! I should have checked your comment early! 😂
I just found that disabling all plugins I have at ~/.sbt/1.0/plugins/sbt-plugins.sbt solves the issue. I'll check one by one and share what caused it later.
I have 7 plugins in ~/.sbt/1.0/plugins/sbt-plugins.sbt and only addDependencyTreePlugin caused the problem.
So it's the same as Andrzej's one.
In case anyone here missed it and is following along, v1.8.0-RC1 was realeased which does bring in the 2.x version of scala-xml. Please do test it out and report back if you're hitting on any scala-xml problems. I'm assuming there may be some, and then we can chase them down in the various repos.
I still had problems with version conflicts when trying to use addDependencyTreePlugin in plugins.sbt (and sbt.version=1.8.0-RC1), but removing metals.sbt and .bloop/ folders did help (which weren't touched upon for a while).
