terryyin
Recent lizard version broke Java parser
Hi! I'm using Lizard in my open source project Pydriller.
Recently it started failing on a test that I have that parses a Java file.
More specifically, it seems that it fails in parsing the number of methods of a Java file, and the complexity.
For example, take this file: https://pastebin.com/7FvSnFDD
With version 1.17.10
❯ pip install lizard==1.17.10
Successfully installed lizard-1.17.10
❯ lizard test-repos/lizard/FileJava.java
================================================
NLOC CCN token PARAM length location
------------------------------------------------
3 1 10 0 3 GitRepository::GitRepository@82-84@test-repos/lizard/FileJava.java
3 1 14 1 3 GitRepository::GitRepository@86-88@test-repos/lizard/FileJava.java
8 1 54 2 10 GitRepository::GitRepository@90-99@test-repos/lizard/FileJava.java
3 1 18 1 3 GitRepository::singleProject@101-103@test-repos/lizard/FileJava.java
3 1 23 2 3 GitRepository::singleProject@105-107@test-repos/lizard/FileJava.java
3 1 15 1 3 GitRepository::allProjectsIn@109-111@test-repos/lizard/FileJava.java
7 2 65 2 9 GitRepository::allProjectsIn@113-121@test-repos/lizard/FileJava.java
13 2 141 0 16 GitRepository::info@123-138@test-repos/lizard/FileJava.java
7 2 45 0 7 GitRepository::openRepository@140-146@test-repos/lizard/FileJava.java
3 1 20 1 3 GitRepository::discoverMainBranchName@148-150@test-repos/lizard/FileJava.java
13 2 92 0 15 GitRepository::getHead@152-166@test-repos/lizard/FileJava.java
10 3 63 0 11 GitRepository::getChangeSets@169-179@test-repos/lizard/FileJava.java
20 3 141 1 23 GitRepository::firstParentsOnly@181-203@test-repos/lizard/FileJava.java
7 2 54 1 8 GitRepository::getAllCommits@205-212@test-repos/lizard/FileJava.java
5 1 33 1 6 GitRepository::extractChangeSet@214-219@test-repos/lizard/FileJava.java
6 1 48 1 7 GitRepository::convertToDate@221-227@test-repos/lizard/FileJava.java
44 7 508 1 59 GitRepository::getCommit@239-297@test-repos/lizard/FileJava.java
10 2 103 2 12 GitRepository::getBranches@299-310@test-repos/lizard/FileJava.java
16 3 136 2 20 GitRepository::diffToModification@312-331@test-repos/lizard/FileJava.java
5 2 67 2 7 GitRepository::diffsForTheCommit@333-339@test-repos/lizard/FileJava.java
21 3 164 2 22 GitRepository::getDiffBetweenCommits@342-363@test-repos/lizard/FileJava.java
24 3 185 3 27 GitRepository::getDiffBetweenCommits@365-391@test-repos/lizard/FileJava.java
8 2 34 1 8 GitRepository::setContext@393-400@test-repos/lizard/FileJava.java
10 3 79 2 12 GitRepository::getSourceCode@402-413@test-repos/lizard/FileJava.java
14 3 84 2 16 GitRepository::getDiffText@415-430@test-repos/lizard/FileJava.java
10 2 104 1 11 GitRepository::checkout@432-442@test-repos/lizard/FileJava.java
9 3 71 1 9 GitRepository::deleteMMBranch@444-452@test-repos/lizard/FileJava.java
7 2 43 0 8 GitRepository::files@454-461@test-repos/lizard/FileJava.java
8 2 70 0 8 GitRepository::reset@463-470@test-repos/lizard/FileJava.java
3 1 13 0 3 GitRepository::getAllFilesInPath@472-474@test-repos/lizard/FileJava.java
3 1 14 0 3 GitRepository::totalCommits@477-479@test-repos/lizard/FileJava.java
3 1 30 3 3 GitRepository::blame@483-485@test-repos/lizard/FileJava.java
3 1 20 2 3 GitRepository::blame@487-489@test-repos/lizard/FileJava.java
28 5 264 3 31 GitRepository::blame@491-521@test-repos/lizard/FileJava.java
3 1 8 0 3 GitRepository::getMaxNumberFilesInACommit@523-525@test-repos/lizard/FileJava.java
12 3 105 1 17 GitRepository::getCommitFromTag@528-544@test-repos/lizard/FileJava.java
7 2 47 2 7 GitRepository::getActualRefObjectId@546-552@test-repos/lizard/FileJava.java
7 2 24 0 7 GitRepository::checkMaxNumberOfFiles@561-567@test-repos/lizard/FileJava.java
7 2 24 0 7 GitRepository::checkMaxSizeOfDiff@576-582@test-repos/lizard/FileJava.java
4 1 27 1 4 GitRepository::getSystemProperty@591-594@test-repos/lizard/FileJava.java
3 1 18 1 3 GitRepository::setPath@596-598@test-repos/lizard/FileJava.java
3 1 13 1 3 GitRepository::setFirstParentOnly@600-602@test-repos/lizard/FileJava.java
5 1 41 1 5 GitRepository::clone@605-609@test-repos/lizard/FileJava.java
10 3 64 0 11 GitRepository::delete@612-622@test-repos/lizard/FileJava.java
3 1 13 1 3 GitRepository::setDataToCollect@625-627@test-repos/lizard/FileJava.java
9 3 52 1 10 GitRepository::diffFiltersAccept@635-644@test-repos/lizard/FileJava.java
1 file analyzed.
==============================================================
NLOC Avg.NLOC AvgCCN Avg.token function_cnt file
--------------------------------------------------------------
466 9.0 2.0 70.9 46 test-repos/lizard/FileJava.java
===============================================================================================================
No thresholds exceeded (cyclomatic_complexity > 15 or length > 1000 or nloc > 1000000 or parameter_count > 100)
==========================================================================================
Total nloc Avg.NLOC AvgCCN Avg.token Fun Cnt Warning cnt Fun Rt nloc Rt
------------------------------------------------------------------------------------------
466 9.0 2.0 70.9 46 0 0.00 0.00
With the most recent version
❯ pip install lizard==1.17.18
Successfully installed lizard-1.17.18
❯ lizard test-repos/lizard/FileJava.java
================================================
NLOC CCN token PARAM length location
------------------------------------------------
3 1 10 0 3 GitRepository::GitRepository@82-84@test-repos/lizard/FileJava.java
3 1 14 1 3 GitRepository::GitRepository@86-88@test-repos/lizard/FileJava.java
8 1 54 2 10 GitRepository::GitRepository@90-99@test-repos/lizard/FileJava.java
3 1 18 1 3 GitRepository::singleProject@101-103@test-repos/lizard/FileJava.java
3 1 23 2 3 GitRepository::singleProject@105-107@test-repos/lizard/FileJava.java
3 1 15 1 3 GitRepository::allProjectsIn@109-111@test-repos/lizard/FileJava.java
7 2 65 2 9 GitRepository::allProjectsIn@113-121@test-repos/lizard/FileJava.java
13 2 141 0 16 GitRepository::info@123-138@test-repos/lizard/FileJava.java
1 file analyzed.
==============================================================
NLOC Avg.NLOC AvgCCN Avg.token function_cnt file
--------------------------------------------------------------
466 5.4 1.2 42.5 8 test-repos/lizard/FileJava.java
===============================================================================================================
No thresholds exceeded (cyclomatic_complexity > 15 or length > 1000 or nloc > 1000000 or parameter_count > 100)
==========================================================================================
Total nloc Avg.NLOC AvgCCN Avg.token Fun Cnt Warning cnt Fun Rt nloc Rt
------------------------------------------------------------------------------------------
466 5.4 1.2 42.5 8 0 0.00 0.00
As you can see, the latest version only identifies 8 methods in the file. Which is definitely wrong since there are 46 (which was the correct reply I was given by Lizard in version 1.17.10).
Thanks for the reporting. I do find a bug introduced by a previous fix for anonymous class. A new release is also created. It would be great if you can confirm if the problem is gone.
@terryyin I am still seeing this problem even in the latest version. Any plans to fix it?
@terryyin I am still seeing this problem even in the latest version. Any plans to fix it?
Yes it's still there. Waiting for a fix.
@terryyin I found that if .class is present (for example, in method call parameters), it causes issues with subsequent method resolution.
Test code:
def test_my_code(self):
code = """
public String funcA() {
return callMethod(1, Whatever.class);
}
public String funcB() {
try {
if (aVar != null) return true;
} catch (Exception e) {
return false;
}
}
"""
result = get_java_function_list(code)
for i in result:
print(i.long_name)
self.assertEqual('funcA()', result[0].long_name)
self.assertEqual('funcB()', result[1].long_name)
Running this test case produces the following output:
funcA()
if( aVar != null)
However, the correct output should be:
funcA()
funcB()
As shown in the figure below:
@liying2008 thanks for providing the example. Now I can reproduce the problem with additional test and the bug should be fixed now. I've made a new release, please check.
@liying2008 thanks for providing the example. Now I can reproduce the problem with additional test and the bug should be fixed now. I've made a new release, please check.
The new version (1.17.27) has been verified to work correctly. Many thanks to @terryyin for the efficient work.
Hi, @terryyin . I've found another issue. When ::new appears in a method, the parsing becomes incorrect, and the subsequent methods cannot be resolved.
This issue still exists in version 1.17.27 .
Test code:
def test_my_code(self):
code = """
public String[] funcA() {
return properties.stream().toArray(String[]::new);
}
public String funcB() {
return "something";
}
"""
result = get_java_function_list(code)
for i in result:
print(i.long_name)
self.assertEqual(2, len(result))
self.assertEqual('funcA()', result[0].long_name)
self.assertEqual('funcB()', result[1].long_name)
Running this test case only prints the funcA() method; funcB() is not printed.
As shown in the figure below:
@liying2008 thanks for report and it's so nice to see a test is provided. The problem is fixed with a new release.
@liying2008 thanks for report and it's so nice to see a test is provided. The problem is fixed with a new release.
Thanks to @terryyin — the issue is gone when using version 1.17.28 .