Relocating scala classes

[jongwook]

Relocating scala classes that contain some non-java features like package objects and trait compositions doesn't seem to work with Gradle shadow plugin.

Can it be considered as an enhancement to be included in the future?

[johnrengelman]

I don't have any Scala experience so I can't comment on this intelligently. Shadow uses the ASM library to do the class relocation. If some other support is necessary for Scala classes then I'd be happy to look at a pull request.

[jnadler]

We're seeing the same problem in a mixed Scala+Java project. Not sure why ASM would treat Scala classes differently. If I can get some time, I'll dig in and see if I can figure this out.

This plugin is the key part of what makes it possible for us to build Apache Spark fat jars - thanks John!

[dainkaplan]

Also seeing this issue with scala classes. This thread is pretty old... @jnadler or anyone else find a solution to this?

[asm0dey]

1. It still doesn't work
2. Interesting facts 2.1 We're shading org.json package 2.2 We use it inside scala trait 2.3 bytecode is being changed, but 2.4 scalap shows old signatures which means it's stores somewhere inside scala-specific annotation (for example @ScalaSignature)

❯ scalap CatalogMetadataProvider
package org.jetbrains.ztools.spark
trait CatalogMetadataProvider extends scala.AnyRef {
  def listTables(dbName: scala.Predef.String): org.json.JSONArray
  def listFunctions(dbName: scala.Predef.String): org.json.JSONArray
  def listColumns(dbName: scala.Predef.String, tableName: scala.Predef.String): org.json.JSONArray
  def listDatabases: org.json.JSONArray
  def toJson: org.json.JSONObject
}

❯ javap CatalogMetadataProvider
Warning: File ./CatalogMetadataProvider.class does not contain class CatalogMetadataProvider
Compiled from "CatalogMetadataProvider.scala"
public interface org.jetbrains.ztools.spark.CatalogMetadataProvider {
  public abstract org.jetbrains.bigdataide.shaded.org.json.JSONArray listTables(java.lang.String);
  public abstract org.jetbrains.bigdataide.shaded.org.json.JSONArray listFunctions(java.lang.String);
  public abstract org.jetbrains.bigdataide.shaded.org.json.JSONArray listColumns(java.lang.String, java.lang.String);
  public abstract org.jetbrains.bigdataide.shaded.org.json.JSONArray listDatabases();
  public abstract org.jetbrains.bigdataide.shaded.org.json.JSONObject toJson();
}

[johnrengelman]

Looks like a problem with Maven Shade too: https://issues.apache.org/jira/browse/MSHADE-345

[asm0dey]

Known to work in sbt-assembly tho: https://github.com/sbt/sbt-assembly#shading

[johnrengelman]

@asm0dey Yeah, but not natively through ASM. See https://github.com/sbt/sbt-assembly/blob/db69ab53cdbf0bfb3cff07183355a74c6ce49f14/src/main/scala/sbtassembly/Shader.scala https://github.com/sbt/sbt-assembly/blob/master/src/main/scala/org/pantsbuild/jarjar/JJProcessor.scala There's a bunch of custom handling for processing Scala bytecode. So, this isn't a bug, it's a new feature and would require someone to write all the processing/tracking logic for Scala bytecode to be processed.

Contributions welcome.

[KazankovMarch]

Workaround, worked in my case: use https://github.com/google/jarjar for relocating

// Kotlin DSL
val jarJar = task<JavaExec>("jarjar") {
    val finalJarFile = "$buildDir/libs/${project.name}-${rootProject.version}.jar"
    main = "-jar"
    args = listOf("path/to/jarjar.jar", "process", "path/to/relocating_rules.txt", finalJarFile, finalJarFile)
}
tasks.withType<ShadowJar> {
    //...//
    finalizedBy(jarJar)
}

[shawjef3]

I have a possible implementation for this. https://github.com/johnrengelman/shadow/pull/756

[tribbloid]

hmm, this is interesting. So sbt works but both gradle & maven fails?

I encounter the same problem on scala 2.12.16 & gradle 7.4.2, when relocating json4s

[asm0dey]

Yes, because sbt is scala-aware.

вс, 12 июн. 2022 г., 05:03 Peng Cheng @.***>:

hmm, this is interesting. So sbt works but both gradle & maven fails?

I encounter the same problem on scala 2.12.16 & gradle 7.4.2, when relocating json4s

— Reply to this email directly, view it on GitHub https://github.com/johnrengelman/shadow/issues/146#issuecomment-1153045445, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ4XAVLT3WRVD4RTWRBG3LVOVAPRANCNFSM4BKLAU5A . You are receiving this because you were mentioned.Message ID: @.***>

[shawjef3]

This is not a feature of SBT or Gradle. It's a feature of the shadowing plugins. In principle there's no need to change build tools to relocate Scala classes.

[tribbloid]

This is not a feature of SBT or Gradle. It's a feature of the shadowing plugins. In principle there's no need to change build tools to relocate Scala classes.

Totally agree, our next patch should be able to introduce the correct behaviour of this feature.

For a minimalistic comparative test case, see https://stackoverflow.com/questions/73768130/within-the-ecosystem-of-java-scala-or-kotlin-is-there-a-reliable-way-to-repack/74061407#74061407

[tribbloid]

it is not fixed?

looks like sbt has to hang around for a while

[johnrengelman]

Sorry was thinking about closing stale issues, did that and then backed away from it.

Same message as above. If someone wants to contribute support…thiugh id probably want to consider adding that support as an additional library rather than in the core code.

[tribbloid]

I see ... I may need to set up a bounty for it