java annotation and reflection: "illegal cyclic reference involving object InterfaceAudience"

[scabug]

See attached exception traces.

With Scala 2.11.8 and Hadoop CDH v5.4, calling typeOf on a type annotated with the Hadoop InterfaceAudience annotation causes a scala.reflect.internal.Symbols$CyclicReference: illegal cyclic reference involving object InterfaceAudience exception in the REPL. (This is a problem since I use the Ammonite REPL, and it always calls typeOf on types it wants to print to the console. But the issue exists independent of which REPL I am using, given the attached trace is in the regular REPL.)

The interesting thing is that calling typeOf with the fully-qualified reference causes a different exception: java.lang.AssertionError: assertion failed: no symbol could be loaded from interface org.apache.hadoop.classification.InterfaceAudience$Public in object InterfaceAudience with name Public and classloader scala.reflect.internal.util.ScalaClassLoader$URLClassLoader@536aaa8d

[scabug]

Imported From: https://issues.scala-lang.org/browse/SI-10129?orig=1 Reporter: Tom Dyas (tdyas) Affected Versions: 2.11.8 Attachments:

• trace.txt (created on Jan 1, 2017 4:13:37 PM UTC, 16986 bytes)

[tdyas]

I just wanted to ping this issue so see what it's status was. Any idea why this error occurs?

[PengleiShi]

Is there any update? i met with same error

[SethTisue]

a good next step would be for someone to construct a self-contained reproduction (preferably not involving any external libraries); nobody's likely to attempt a fix if we don't have one of those

[badrinathpatchikolla]

I met with the same error when using scala reflection to parse and execute multiple case classes. Any update on the issue?

[SethTisue]

@badrinathpatchikolla are you able to construct a self-contained reproduction?

[badrinathpatchikolla]

I am using Spark Standalone

Spark Version: 2.4.7 and Scala 2.11.12

Steps to reproduce :

spark-shell --packages "org.scala-lang:scala-compiler:2.11.12,org.scala-lang:scala-reflect:2.11.12"

import scala.reflect.runtime.currentMirror
import scala.tools.reflect.ToolBox

def compile[A](codeStr: String): A = {
  val toolbox = currentMirror.mkToolBox()
  val tree    = toolbox.parse(codeStr)
  toolbox.eval(tree).asInstanceOf[A]
}

val codeStr1=
  """
   import org.apache.spark.SparkConf
   import org.apache.spark.sql.SparkSession
   val sparkConf = new SparkConf
   val spark = SparkSession.builder().config(sparkConf).getOrCreate
   spark.sparkContext.hadoopConfiguration.set("fs.s3a.access.key", "tes")
    """.stripMargin

println(compile[Any](codeStr1))

Error :

scala.tools.reflect.ToolBoxError: reflective compilation has failed:

illegal cyclic reference involving object InterfaceAudience
  at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal.throwIfErrors(ToolBoxFactory.scala:316)
  at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal.wrapInPackageAndCompile(ToolBoxFactory.scala:198)
  at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal.compile(ToolBoxFactory.scala:252)
  at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$$anonfun$compile$2.apply(ToolBoxFactory.scala:429)
  at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$$anonfun$compile$2.apply(ToolBoxFactory.scala:422)
  at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$withCompilerApi$.liftedTree2$1(ToolBoxFactory.scala:355)
  at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$withCompilerApi$.apply(ToolBoxFactory.scala:355)
  at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl.compile(ToolBoxFactory.scala:422)
  at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl.eval(ToolBoxFactory.scala:444)
  at compile(<console>:34)
  ... 49 elided

This is coming when i add spark.sparkContext.hadoopConfiguration.set("fs.s3a.access.key", "test") hadoopConfiguration line to the code.

@SethTisue @scabug

[SethTisue]

what you've provided is a step in the right direction, but:

a self-contained reproduction would need to not involve Spark

it would also need to be on Scala 2.13 (or at least 2.12), not 2.11; 2.11 is no longer maintained

[nimatrueway]

Hey @SethTisue 👋

This is Nima from engineering team of Hootsuite. I spend a few days and created a minimal test case to reproduce this issue. I also did a tentative investigation on why this occurs.

I am very much willing to contribute to my lovely programming language 😍 though I need a bit of guidance before implementing any solution. Kindly please take a look at my draft PR here and share your thoughts with me.

https://github.com/nimatrueway/scala/pull/1

[som-snytt]

@nimatrueway Thanks for the repro. I tried them out as partest tests at https://github.com/scala/scala/pull/10078 but once they work they can run as in-process junit tests. I'll take a look at your diagnosis for fun; I don't know much about reflection; I know work has been done to support Java inner classes.

[SethTisue]

@lrytz maybe you could look at Nima's code...?

[badrinathpatchikolla]

@SethTisue This issue present in scala 2.12.x also, is this fixed in scala 2.13.x?

[som-snytt]

@badrinathpatchikolla my test linked in my previous comment still fails. That is Nima's test.

I had added some debug locally, but I don't remember whether I understood Nima's helpful comments on nimatrueway.

My comment "for fun" is a code phrase for "I'll run a test while doing something else [usually preparing food]."

[joroKr21]

It's fairly easy to reproduce. Just copy the annotation in a file. Both the runtime retention policy and the outer class being annotated are required.

package foo;

import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;

@InterfaceAudience.Public
public class InterfaceAudience {
    @Retention(RetentionPolicy.RUNTIME)
    public @interface Public {}
}

import foo.InterfaceAudience
import scala.reflect.runtime.universe._

object Annotated extends App {
  @InterfaceAudience.Public class Foo
  println(typeOf[Foo])
}

scala.reflect.internal.Symbols$CyclicReference: illegal cyclic reference involving class InterfaceAudience
Exception in thread "main" java.lang.RuntimeException: error reading Scala signature of Annotated: illegal cyclic reference involving class InterfaceAudience
	at scala.reflect.internal.pickling.UnPickler.unpickle(UnPickler.scala:48)
	at scala.reflect.runtime.JavaMirrors$JavaMirror.unpickleClass(JavaMirrors.scala:676)
	at scala.reflect.runtime.SymbolLoaders$TopClassCompleter.$anonfun$complete$3(SymbolLoaders.scala:37)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at scala.reflect.internal.SymbolTable.slowButSafeEnteringPhaseNotLaterThan(SymbolTable.scala:320)
	at scala.reflect.runtime.SymbolLoaders$TopClassCompleter.complete(SymbolLoaders.scala:34)
	at scala.reflect.internal.Symbols$Symbol.completeInfo(Symbols.scala:1572)
	at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1535)
	at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$8.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:206)
	at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol.info(SynchronizedSymbols.scala:158)
	at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol.info$(SynchronizedSymbols.scala:158)
	at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$8.info(SynchronizedSymbols.scala:206)
	at scala.reflect.internal.ReificationSupport$ReificationSupportImpl.select(ReificationSupport.scala:34)
	at scala.reflect.internal.ReificationSupport$ReificationSupportImpl.selectType(ReificationSupport.scala:25)
	at scala.reflect.internal.ReificationSupport$ReificationSupportImpl.selectType(ReificationSupport.scala:23)
	at Annotated$$typecreator1$1.apply(Annotated.scala:6)
	at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:238)
	at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:238)
	at scala.reflect.api.TypeTags.typeOf(TypeTags.scala:359)
	at scala.reflect.api.TypeTags.typeOf$(TypeTags.scala:359)
	at scala.reflect.api.Universe.typeOf(Universe.scala:73)
	at Annotated$.delayedEndpoint$Annotated$1(Annotated.scala:6)
	at Annotated$delayedInit$body.apply(Annotated.scala:4)
	at scala.Function0.apply$mcV$sp(Function0.scala:42)
	at scala.Function0.apply$mcV$sp$(Function0.scala:42)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
	at scala.App.$anonfun$main$1(App.scala:98)
	at scala.App.$anonfun$main$1$adapted(App.scala:98)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:575)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:573)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:933)
	at scala.App.main(App.scala:98)
	at scala.App.main$(App.scala:96)
	at Annotated$.main(Annotated.scala:4)
	at Annotated.main(Annotated.scala)