Scala Macros cause NullPointerException in com.sun.tools.javac.code.Type

[twalthr]

I'm not sure if this is the right place to report the issue, but maybe you can help me. We are using genjavadoc in Apache Flink (see pom.xml).

We recently upgraded to Scala 2.11 and cannot build out Javadocs anymore. It seems that genjavadoc is not producing correct Java files for Scala macros anymore.

This is a part of the log:

/Users/twalthr/flink/flink/flink-scala/target/genjavadoc/org/apache/flink/api/scala/codegen/TypeDescriptors.java:181: error: cannot find symbol
    public  scala.reflect.macros.blackbox.Context.universe getter ()  { throw new RuntimeException(); }
                                                 ^
  symbol:   class universe
  location: interface Context
/Users/twalthr/flink/flink/flink-scala/target/genjavadoc/org/apache/flink/api/scala/codegen/TypeDescriptors.java:182: error: cannot find symbol
    public  scala.reflect.macros.blackbox.Context.universe setter ()  { throw new RuntimeException(); }
                                                 ^
  symbol:   class universe
  location: interface Context
/Users/twalthr/flink/flink/flink-scala/target/genjavadoc/org/apache/flink/api/scala/codegen/TypeDescriptors.java:183: error: cannot find symbol
    public  scala.reflect.macros.blackbox.Context.universe tpe ()  { throw new RuntimeException(); }
                                                 ^
  symbol:   class universe
  location: interface Context
/Users/twalthr/flink/flink/flink-scala/target/genjavadoc/org/apache/flink/api/scala/codegen/TypeDescriptors.java:184: error: non-static type variable C cannot be referenced from a static context
    public  org.apache.flink.api.scala.codegen.TypeDescriptors<C>.UDTDescriptor desc ()  { throw new RuntimeException(); }
                                                               ^
java.lang.NullPointerException
	at com.sun.tools.javac.code.Type.map(Type.java:219)
	at com.sun.tools.javac.code.Types.erasure(Types.java:2246)
	at com.sun.tools.javac.jvm.ClassReader.isSameBinaryType(ClassReader.java:1348)
	at com.sun.tools.javac.jvm.ClassReader.findMethod(ClassReader.java:1323)
	at com.sun.tools.javac.jvm.ClassReader.readEnclosingMethodAttr(ClassReader.java:1274)
	at com.sun.tools.javac.jvm.ClassReader$12.read(ClassReader.java:1130)
	at com.sun.tools.javac.jvm.ClassReader.readAttrs(ClassReader.java:1381)
	at com.sun.tools.javac.jvm.ClassReader.readClassAttrs(ClassReader.java:1396)
	at com.sun.tools.javac.jvm.ClassReader.readClass(ClassReader.java:2226)
	at com.sun.tools.javac.jvm.ClassReader.readClassFile(ClassReader.java:2324)
	at com.sun.tools.javac.jvm.ClassReader.fillIn(ClassReader.java:2511)
	at com.sun.tools.javac.jvm.ClassReader.complete(ClassReader.java:2442)
	at com.sun.tools.javac.jvm.ClassReader.access$000(ClassReader.java:76)
	at com.sun.tools.javac.jvm.ClassReader$1.complete(ClassReader.java:240)
	at com.sun.tools.javac.code.Symbol.complete(Symbol.java:574)
	at com.sun.tools.javac.code.Symbol$ClassSymbol.complete(Symbol.java:1037)
	at com.sun.tools.javac.code.Symbol$ClassSymbol.flags(Symbol.java:973)
	at com.sun.tools.javadoc.ClassDocImpl.getFlags(ClassDocImpl.java:128)
	at com.sun.tools.javadoc.ClassDocImpl.isAnnotationType(ClassDocImpl.java:142)
	at com.sun.tools.javadoc.DocEnv.isAnnotationType(DocEnv.java:609)
	at com.sun.tools.javadoc.DocEnv.getClassDoc(DocEnv.java:582)
	at com.sun.tools.javadoc.PackageDocImpl.getClasses(PackageDocImpl.java:152)
	at com.sun.tools.javadoc.PackageDocImpl.addAllClassesTo(PackageDocImpl.java:168)
	at com.sun.tools.javadoc.RootDocImpl.classes(RootDocImpl.java:203)
	at com.sun.tools.doclets.internal.toolkit.AbstractDoclet.startGeneration(AbstractDoclet.java:127)
	at com.sun.tools.doclets.internal.toolkit.AbstractDoclet.start(AbstractDoclet.java:82)
	at com.sun.tools.doclets.formats.html.HtmlDoclet.start(HtmlDoclet.java:80)
	at com.sun.tools.doclets.standard.Standard.start(Standard.java:39)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.tools.javadoc.DocletInvoker.invoke(DocletInvoker.java:310)
	at com.sun.tools.javadoc.DocletInvoker.start(DocletInvoker.java:189)
	at com.sun.tools.javadoc.Start.parseAndExecute(Start.java:366)
	at com.sun.tools.javadoc.Start.begin(Start.java:219)
	at com.sun.tools.javadoc.Start.begin(Start.java:205)
	at com.sun.tools.javadoc.Main.execute(Main.java:64)
	at com.sun.tools.javadoc.Main.main(Main.java:54)

Command line was: /Library/Java/JavaVirtualMachines/jdk1.8.0_102.jdk/Contents/Home/jre/../bin/javadoc -Xdoclint:none @options @packages

Any help is highly appreciated as I already spent hours on this error. Do you know a way how I can at least skip the problematic files?

[twalthr]

@2m, @ktoso any idea?

[ktoso]

I'm unsure if it ever worked on macros to be honest. I don't think there's a way currently to filter out files - is it for a specific sub-project where you have macros?

How / where to reproduce this, could you submit a PR examplifying the problematic code and we could make it work with that?

[ktoso]

Also, do I understand correctly that is this for Flink itself? Was it on 2.10? What version of the plugin btw, latest?

[twalthr]

@ktoso Sorry, for replying so late. Yes, it is for Flink itself. You can reproduce the error if you run mvn javadoc:aggregate -DadditionalJOption="-Xdoclint:none" -Dmaven.javadoc.failOnError=false on the current master. It would be great if genjavadocs could simply skip scala.reflect.macros package somehow.

[twalthr]

We are thinking about dropping genjavadocs for now because of this bug. https://github.com/apache/flink/pull/5003

[ktoso]

Rather than dropping, we’d welcome some collaboration, it’s open source after all. The tool is something something we use ourselves, and maintain on a best effort basis.

Your proposal of skipping is fine, I think you’d be able to PR such a change -- should be pretty simple, WDYT?

[twalthr]

I will try to look into it. Would be happy about some hints nevertheless.

[ktoso]

Perhaps one could introduce an annotation to skip certain methods? When looking at what the method is then, one would check if it has the skip annotation and just skip it then. Would that work?

[twalthr]

I think the easiest solution would be to define entire exclusions packages in the pom.xml. If we could simply skip classes that use Scala macros like org.apache.flink.api.scala.codegen we would be happy.

[ktoso]

It's an sbt plugin so I don't know how you use it from maven actually? It could be passed through as an environment variable though, list of packages which should be skipped?

[twalthr]

We use it as described in your project description. Yes, the environment variable approach might work.

[ktoso]

Hah I see, which goes to show how often I look at this codebase... Forgot it's documented how to use from mvn ;)

Thanks! Stay in touch, happy to help if you get stuck

[2m]

It would be better to use scalac parameters instead of environment variables, as these are already used for the -P:genjavadoc:out=... parameter.