wala
Nullpointerexception in JDTJava2CAstTranslator
Hello,
I'm trying to create a call graph but encounter a null pointer exception. Depending on the java source I use as input the exception is thrown from different visitors, but it always involves "n.resolveMethodBinding()".
resolveMethodBinding returns null because it uses the class BindingResolver's implementation (org.eclipse.jdt.core.dom package) (line 194):
IMethodBinding resolveMethod(MethodInvocation method) {
return null;
}
It is likely that DefaultBindingResolver should be used instead of BindingResolver's which features an empty implementation for each method.
The exact same issue was described by someone else here: https://sourceforge.net/p/wala/mailman/message/34746738/
Hope you have any clue how to solve this issue, as I'm using this tool for research done with regard to my thesis.
Best regards, Edward
I don't know this code, nor do I understand Eclipse's AST data structures. @juliandolby any thoughts?
@edwardmp do you see the same stack as in https://sourceforge.net/p/wala/mailman/message/34746738/ ?
Thanks for the quick response. The stack changes a bit depending on which program I try to analyze, but resolveMethodBinding is the method being called which causes the nullpointerexception, all in invarious visitor methods in that class.
Example trace:
Exception in thread "main" java.lang.NullPointerException
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.visit(JDTJava2CAstTranslator.java:1480)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.visitNode(JDTJava2CAstTranslator.java:2901)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.visit(JDTJava2CAstTranslator.java:1462)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.visitNode(JDTJava2CAstTranslator.java:2887)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.visitNodeOrNodes(JDTJava2CAstTranslator.java:2959)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.createBlock(JDTJava2CAstTranslator.java:1275)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.visit(JDTJava2CAstTranslator.java:1280)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.visitNode(JDTJava2CAstTranslator.java:2861)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.visit(JDTJava2CAstTranslator.java:806)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.createClassDeclaration(JDTJava2CAstTranslator.java:485)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.visitTypeDecl(JDTJava2CAstTranslator.java:408)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.visit(JDTJava2CAstTranslator.java:2828)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.translateToCAst(JDTJava2CAstTranslator.java:261)
at com.ibm.wala.cast.java.translator.jdt.ejc.EJCSourceModuleTranslator$EjcAstToIR.acceptAST(EJCSourceModuleTranslator.java:92)
at org.eclipse.jdt.core.dom.CompilationUnitResolver.resolve(CompilationUnitResolver.java:1029)
at org.eclipse.jdt.core.dom.CompilationUnitResolver.resolve(CompilationUnitResolver.java:636)
at org.eclipse.jdt.core.dom.ASTParser.createASTs(ASTParser.java:990)
at com.ibm.wala.cast.java.translator.jdt.ejc.EJCSourceModuleTranslator.loadAllSources(EJCSourceModuleTranslator.java:181)
at com.ibm.wala.cast.java.loader.JavaSourceLoaderImpl.loadAllSources(JavaSourceLoaderImpl.java:505)
at com.ibm.wala.classLoader.ClassLoaderImpl.init(ClassLoaderImpl.java:529)
at com.ibm.wala.cast.java.loader.JavaSourceLoaderImpl.init(JavaSourceLoaderImpl.java:516)
at com.ibm.wala.cast.java.translator.jdt.JDTClassLoaderFactory.makeNewClassLoader(JDTClassLoaderFactory.java:69)
at com.ibm.wala.classLoader.ClassLoaderFactoryImpl.getLoader(ClassLoaderFactoryImpl.java:69)
at com.ibm.wala.ipa.cha.ClassHierarchy.<init>(ClassHierarchy.java:231)
at com.ibm.wala.ipa.cha.ClassHierarchy.<init>(ClassHierarchy.java:177)
at com.ibm.wala.ipa.cha.ClassHierarchy.make(ClassHierarchy.java:1230)
at com.ibm.wala.cast.java.client.JavaSourceAnalysisEngine.buildClassHierarchy(JavaSourceAnalysisEngine.java:135)
at eu.sig.aevas.callgraph.WalaStatic.run(WalaStatic.java:165)
at eu.sig.aevas.AeVas.main(AeVas.java:16)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Other input program stack trace:
Exception in thread "main" java.lang.NullPointerException
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator$TryCatchContext.<init>(JDTJava2CAstTranslator.java:3150)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.visit(JDTJava2CAstTranslator.java:2684)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.visitNode(JDTJava2CAstTranslator.java:2939)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.visitNodeOrNodes(JDTJava2CAstTranslator.java:2959)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.createBlock(JDTJava2CAstTranslator.java:1275)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.visit(JDTJava2CAstTranslator.java:1280)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.visitNode(JDTJava2CAstTranslator.java:2861)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.visit(JDTJava2CAstTranslator.java:806)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.createClassDeclaration(JDTJava2CAstTranslator.java:485)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.visitTypeDecl(JDTJava2CAstTranslator.java:408)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.visit(JDTJava2CAstTranslator.java:2828)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.translateToCAst(JDTJava2CAstTranslator.java:261)
at com.ibm.wala.cast.java.translator.jdt.ejc.EJCSourceModuleTranslator$EjcAstToIR.acceptAST(EJCSourceModuleTranslator.java:92)
at org.eclipse.jdt.core.dom.CompilationUnitResolver.resolve(CompilationUnitResolver.java:1029)
at org.eclipse.jdt.core.dom.CompilationUnitResolver.resolve(CompilationUnitResolver.java:636)
at org.eclipse.jdt.core.dom.ASTParser.createASTs(ASTParser.java:990)
at com.ibm.wala.cast.java.translator.jdt.ejc.EJCSourceModuleTranslator.loadAllSources(EJCSourceModuleTranslator.java:181)
at com.ibm.wala.cast.java.loader.JavaSourceLoaderImpl.loadAllSources(JavaSourceLoaderImpl.java:505)
at com.ibm.wala.classLoader.ClassLoaderImpl.init(ClassLoaderImpl.java:529)
at com.ibm.wala.cast.java.loader.JavaSourceLoaderImpl.init(JavaSourceLoaderImpl.java:516)
at com.ibm.wala.cast.java.translator.jdt.JDTClassLoaderFactory.makeNewClassLoader(JDTClassLoaderFactory.java:69)
at com.ibm.wala.classLoader.ClassLoaderFactoryImpl.getLoader(ClassLoaderFactoryImpl.java:69)
at com.ibm.wala.ipa.cha.ClassHierarchy.<init>(ClassHierarchy.java:231)
at com.ibm.wala.ipa.cha.ClassHierarchy.<init>(ClassHierarchy.java:177)
at com.ibm.wala.ipa.cha.ClassHierarchy.make(ClassHierarchy.java:1230)
at com.ibm.wala.cast.java.client.JavaSourceAnalysisEngine.buildClassHierarchy(JavaSourceAnalysisEngine.java:135)
at eu.sig.aevas.callgraph.WalaStatic.run(WalaStatic.java:165)
at eu.sig.aevas.AeVas.main(AeVas.java:16)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
By the way I use 1.3.9-SNAPSHOT versions of all WALA libraries, since I could not install some through maven (com.ibm.wala.cast.java and com.ibm.wala.ide.jdt)
Related: http://stackoverflow.com/questions/2017945/bindings-not-resolving-with-ast-processing-in-eclipse/5803778#5803778 http://stackoverflow.com/questions/2631981/why-does-resolvebinding-return-null-even-though-i-setresolvebindingstrue-on
I think I know why the issue arose:
I had to add jars of all libraries to by calling addSystemModule of class JavaSourceAnalysisEngine.
I needed to add rt.jar when my source files used no external libraries, and if they did .jars with all external libraries.
Thanks for digging in! We still shouldn't crash for missing jars. We should add some catch blocks that log a warning.
Yes I agree that a more helpful message would be convenient.
Also @msridhar, is there any way com.ibm.wala.cast.java and com.ibm.wala.ide.jdt could be made available in the maven repository?
Finally, if I add my source files through addSourceModule why doesn't the condition is this codeblock ever to false? Any clue?
if (!analysisScope.isApplicationLoader(c.getClassLoader())) {
continue;
}
I see it is a source loader and not an application loader. It works if I try to get the parent of the source loader, which is a application loader. But isn't there a better way?
@msridhar ideally, in my case, I would like to continue building the dependency graph when a lib jar is missing, just behave as if that library would have been added to the exclusions file. Is this possible?
@edwardmp that's what I was suggesting above. A hackish fix would be to add null checks and then just soldier on without the relevant method / field. WALA is generally robust to missing classes, etc., it's just this frontend that is falling down. I am very busy for the next few days, but I can try to take a look after that. Or, if you could attempt to throw together a patch and make a pull request, I can give comments.
Re: Maven Central, the problem is that these projects depend on Eclipse, and I'm not sure of the right way to specify such dependencies in a Maven jar. There must be some way to do it (after all, we can build WALA with Maven), I just don't know how to do it. Again, if someone could do some legwork here that would be helpful.
Re: addSourceModule, I don't know the answer. @juliandolby any thoughts?
@msridhar OK. Regarding the availability in the Maven repository, I am trying to add the missing projects but I am not that experienced with Maven and the setup is a bit complex, will definitely dive deeper into this later.
In the meantime, I want to use WALA in a production environment at the company I do an internship at, but they refuse to let me use the current 1.3.9-SNAPSHOT. Is there any way you could release 1.3.9 anytime soon?
Thanks in advance,
I'm fine with cutting a new release. Let me check with Julian. --Manu
On Mon, May 9, 2016 at 9:41 PM Edward Poot [email protected] wrote:
@msridhar https://github.com/msridhar OK. Regarding the availability in the Maven repository, I am trying to add the missing projects but I am not that experienced with Maven and the setup is a bit complex, will definitely dive deeper into this later.
In the meantime, I want to use WALA in a production environment at the company I do an internship at, but they refuse to let me use the current 1.3.9-SNAPSHOT. Is there any way you could release 1.3.9 anytime soon?
Thanks in advance,
— You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub https://github.com/wala/WALA/issues/99#issuecomment-217853334
I have tagged release 1.3.9 in the git repository (R_1.3.9). Still working on pushing new jars to Maven Central, should be done in the next day or so.
Hi @msridhar,
Did you have any opportunity yet to change how missing library jars are handled? This would be a great case for the use of Optionals in Java 8, although personally I have no experience in using them.
Haven't had time to look at this yet, and not sure when I will. At the least, anyone trying to fix will need a test case for reproducing the error, along with instructions on how to reproduce.
On Wed, May 18, 2016, 10:28 AM Edward Poot [email protected] wrote:
Hi @msridhar https://github.com/msridhar,
Did you have any opportunity yet to change how missing library jars are handled? This would be a great case for the use of Optionals in Java 8, although personally I have no experience in using them.
— You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub https://github.com/wala/WALA/issues/99#issuecomment-220099734
OK. I can certainly temporary fix it, although doing it in a proper way requires much knowhow of how the library operates. Tomorrow I will put some example code in a repository so it can be easily reproduced.
Hi @msridhar
Finally had some time to provide an example displaying the issue. This is the tiny system I want to create a callgraph for: https://github.com/edwardmp/wala-issue-example This tiny example depends on the Netty library.
Now I want to only use the source files to generate the callpgrah, using the Eclipse Java frontend.
This is the code used to generate the callgraph: https://github.com/edwardmp/wala-callgraph-generation-issue-example
The code in main first tries to generate the graph without adding the Netty.jar as a system module. Run it and you will see the null pointer exception. The commented out code in main shows that if you do run it and specify the jar, everything works OK.
Ideally, WALA should log an error but not throw an exception and keep generating the call graph but just exclude calls to that library.
Don't forget to change the hardcoded path in main to point to the the download location of the first repository I mentioned (hardcoded path in my example is /Users/Edward/Desktop/wala-issue-example).
Hope this helps!
@msridhar is it OK if a open a pull request in which some null checks are added?
Sure, that would be greatly appreciated! By the way, as you may have
noticed, I've done some refactoring on the master branch. From now on, you
should be able to use the wala.cast.java and wala.cast.java.ecj
projects using Maven Central jars. If you'd like to test it out, I can
create some snapshot jars; let me know. If this is what you needed, we can
cut a new release for it.
--Manu
On Mon, May 30, 2016 at 12:56 AM Edward Poot [email protected] wrote:
@msridhar https://github.com/msridhar is it OK if a open a pull request in which some null checks are added?
— You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/wala/WALA/issues/99#issuecomment-222434503, or mute the thread https://github.com/notifications/unsubscribe/AALyUSZMLwgxCnwR1RpvMDMBo_9vVb-aks5qGpgZgaJpZM4IS85m .
@msridhar That is very convenient, but what is more important is the problem mentioned above leading to a NPE.
I've been pulling my hair out the last few hours trying to run WALA on for example PMD (https://github.com/pmd/pmd, ran on pmd-core dir), but this again leads to a null pointer exception. But in this case, the reason doesn't seem to be related to missing jars from dependencies.
Caused by: java.lang.NullPointerException
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.createClassDeclaration(JDTJava2CAstTranslator.java:473)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.visitTypeDecl(JDTJava2CAstTranslator.java:408)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.visit(JDTJava2CAstTranslator.java:2828)
at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.translateToCAst(JDTJava2CAstTranslator.java:261)
at com.ibm.wala.cast.java.translator.jdt.ejc.EJCSourceModuleTranslator$EjcAstToIR.acceptAST(EJCSourceModuleTranslator.java:92)
at org.eclipse.jdt.core.dom.CompilationUnitResolver.resolve(CompilationUnitResolver.java:1029)
at org.eclipse.jdt.core.dom.CompilationUnitResolver.resolve(CompilationUnitResolver.java:636)
at org.eclipse.jdt.core.dom.ASTParser.createASTs(ASTParser.java:990)
at com.ibm.wala.cast.java.translator.jdt.ejc.EJCSourceModuleTranslator.loadAllSources(EJCSourceModuleTranslator.java:181)
at com.ibm.wala.cast.java.loader.JavaSourceLoaderImpl.loadAllSources(JavaSourceLoaderImpl.java:505)
at com.ibm.wala.classLoader.ClassLoaderImpl.init(ClassLoaderImpl.java:529)
at com.ibm.wala.cast.java.loader.JavaSourceLoaderImpl.init(JavaSourceLoaderImpl.java:516)
at com.ibm.wala.cast.java.translator.jdt.JDTClassLoaderFactory.makeNewClassLoader(JDTClassLoaderFactory.java:69)
at com.ibm.wala.classLoader.ClassLoaderFactoryImpl.getLoader(ClassLoaderFactoryImpl.java:69)
at com.ibm.wala.ipa.cha.ClassHierarchy.<init>(ClassHierarchy.java:231)
at com.ibm.wala.ipa.cha.ClassHierarchy.<init>(ClassHierarchy.java:177)
at com.ibm.wala.ipa.cha.ClassHierarchy.make(ClassHierarchy.java:1230)
at com.ibm.wala.cast.java.client.JavaSourceAnalysisEngine.buildClassHierarchy(JavaSourceAnalysisEngine.java:135)
This is caused because IVariableBinding fieldBinding = fieldFrag.resolveBinding(); has a value of null, thus it is again related to the resolving of bindings. It could be that the issue I opened this issue for isn't the real issue but just a symptom. Something is seriously wrong in resolving bindings.
Kind of frustrating as I really need to generate a call graph of various systems for research related to my thesis and I don't have much time to dive into this.
@edwardmp do you really need to use the Java source front end? All the problems here are in the source front end. The Java bytecode front end is much more robust.
@msridhar Unfortunately, the company I conduct my research at only has certain clients' systems source code available and (usually) not any binaries. I tried a few other call graph libraries before using WALA but those wouldn't work either.
Ok. What makes you think this is not due to missing jars? In any case, as a patch-up fix, my approach would be to add null checks around any relevant calls to resolveBindings(), and then "bail out" if the reset is null. E.g., for the stack you just posted, we just wouldn't add the FieldEntity on the next line. If you submit a PR that adds null checks of this form, I'll take a quick look at it and then merge.
@msridhar Well basically because I have checked that I have jars of all dependencies. Also, in the PMD case I deleted most source files and traced it to 1 specific file (that only had a dependency on a subdir).
OK I will try to add some null checks. With bailing out you mean just skipping the next line or returning from the method? Thanks for your cooperation! much appreciated
By bailing out I generally mean for whatever operation the binding is being used for (typically adding something to a class or class hierarchy), don't do that operation. But if you're a little over-aggressive in bailing out, that's fine for now (better than crashing).
@msridhar spent the last few hours adding checks but this is not really a solution; sometimes you can skip a part of a method but other times you can only return null from that method in case the binding is null, which just propagates the NPE to another place.. Digging in to find the real reason the bindings are not set but no luck so far
@msridhar
So far it boils down to this: setRecoveryBindings must be set to true when the ASTParser is created.
I have a small test case where a ClassInstanceCreation can not resolve the constructor bindings while all other bindings are actually present.
Example statement: new ArrayList<>(10); does have constructor bindings and new ArrayList<>(); (without a capacity) does have bindings . But when the capacity is dependent upon another list (e.g. new ArrayList<>(otherList.size()); it does not have bindings` Why is still the big question..
@msridhar Here is a more complete example:
public abstract class AbstractPropertySource implements PropertySource {
protected List<PropertyDescriptor<?>> copyPropertyDescriptors() {
List<PropertyDescriptor<?>> b = new ArrayList<>(); // works, resolveConstructorBindings() does not return null
List<PropertyDescriptor<?>> c = new ArrayList<>(1); // works fine
List<PropertyDescriptor<?>> copy = new ArrayList<>(b.size()); // null pointer exception because resolveConstructorBinding() returns null
copy.addAll(propertyDescriptors);
return copy;
}
}
Maybe its a bug in the JDT library?
Thanks for digging in! So is adding a call to setRecoveryBindings somewhere sufficient to fix this?
On Tue, May 31, 2016, 11:17 AM Edward Poot [email protected] wrote:
@msridhar https://github.com/msridhar Here is a more complete example:
public abstract class AbstractPropertySource implements PropertySource { protected List<PropertyDescriptor>> copyPropertyDescriptors() { List
> b = new ArrayList<>(); // works, v() does not return null List<PropertyDescriptor<?>> copy = new ArrayList<>(b.size()); // null pointer exception because resolveConstructorBinding() returns null copy.addAll(propertyDescriptors); return copy; } } — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/wala/WALA/issues/99#issuecomment-222774400, or mute the thread https://github.com/notifications/unsubscribe/AALyUeX46Vhb3eAui2Z3JGjwJxeFgJM-ks5qHHtDgaJpZM4IS85m .