javasrc2cpg: missing inherited methods

[mpollmeier]

When creating a cpg for HSL, java2cpg includes the inherited methods of the interface CustomerRepository, while javasrc2cpg leaves them out.

This difference seems to account for a large number of missing flows in javasrc2cpg, in comparison to java2cpg. Can you adapt javasrc2cpg to ensure it creates separate method entries for the inherited methods?

java2cpg:

cpg.method.fullName(".*CustomerRepository.*").fullName.l
List(
  "io.shiftleft.repository.CustomerRepository.delete:void(java.io.Serializable)",
  "io.shiftleft.repository.CustomerRepository.exists:boolean(java.io.Serializable)",
  "io.shiftleft.repository.CustomerRepository.findAll:java.lang.Iterable()",
  "io.shiftleft.repository.CustomerRepository.findOne:java.lang.Object(java.io.Serializable)",
  "io.shiftleft.repository.CustomerRepository.save:java.lang.Object(java.lang.Object)",
  "io.shiftleft.repository.CustomerRepository.findByFirstName:java.util.List(java.lang.String)"
)

javasrc2cpg:

cpg.method.fullName(".*CustomerRepository.*").fullName.l
List(
  "io.shiftleft.repository.CustomerRepository.findByFirstName:<unresolved>.List<Customer>(java.lang.String)",
  "io.shiftleft.repository.CustomerRepository.<init>:void()"
)

2. Also note the difference in findByFirstName: java.util.List vs <unresolved>.List.
3. Not sure about this one, but I'd assume the constructor CustomerRepository.<init> isn't correct, given that this is just an interface...

[mpollmeier]

Similar case: java.io.BufferedWriter.write(String) is missing, which is being used here. Javadoc link: https://docs.oracle.com/javase/8/docs/api/java/io/BufferedWriter.html. Note that there are additional overloads of write, some of which are being inherited. Java2cpg seems to only include external methods that are being used.

java2cpg:

cpg.method.fullName(".*java.io.BufferedWriter.*").fullName.l
List(
  "java.io.BufferedWriter.<init>:void(java.io.Writer)",
  "java.io.BufferedWriter.write:void(java.lang.String)",
  "java.io.BufferedWriter.close:void()"
)

javasrc2cpg:

joern> cpg.method.fullName(".*java.io.BufferedWriter.*").fullName.l
List(
  "java.io.BufferedWriter.<init>:void(java.io.FileWriter)",
  "java.io.BufferedWriter.close:void()"
)

[mpollmeier]

maybe these groupCounts are helpful to pinpoint further differences:

cpg.graph.nodes.groupCount(_.label).sorted

// java2cpg:
  ("ANNOTATION", 97),
  ("ANNOTATION_LITERAL", 65),
  ("ANNOTATION_PARAMETER", 63),
  ("ANNOTATION_PARAMETER_ASSIGN", 63),
  ("ARRAY_INITIALIZER", 48),
  ("BINDING", 1020),
  ("BLOCK", 332),
  ("CALL", 1573),
  ("CONFIG_FILE", 162),
  ("DEPENDENCY", 79),
  ("FIELD_IDENTIFIER", 275),
  ("FILE", 182),
  ("FINDING", 2),
  ("FLOW", 2),
  ("IDENTIFIER", 1847),
  ("IOFLOW", 2),
  ("KEY_VALUE_PAIR", 26),
  ("LITERAL", 332),
  ("LOCAL", 897),
  ("MATCH_INFO", 56),
  ("MEMBER", 68),
  ("META_DATA", 1),
  ("METHOD", 332),
  ("METHOD_PARAMETER_IN", 580),
  ("METHOD_PARAMETER_OUT", 580),
  ("METHOD_RETURN", 332),
  ("MODIFIER", 798),
  ("NAMESPACE", 55),
  ("NAMESPACE_BLOCK", 183),
  ("PACKAGE_PREFIX", 1),
  ("PROGRAM_POINT", 2),
  ("RETURN", 172),
  ("SENSITIVE_DATA_TYPE", 6),
  ("SENSITIVE_MEMBER", 24),
  ("SENSITIVE_REFERENCE", 110),
  ("SENSITIVE_VARIABLE", 110),
  ("SINK", 2),
  ("SOURCE", 2),
  ("TAG", 1693),
  ("TYPE", 127),
  ("TYPE_DECL", 206),
  ("UNKNOWN", 124)


// javasrc2cpg:
  ("BINDING", 158),
  ("BLOCK", 382),
  ("CALL", 744),
  ("CONTROL_STRUCTURE", 38),
  ("FIELD_IDENTIFIER", 86),
  ("FILE", 24),
  ("IDENTIFIER", 787),
  ("LITERAL", 258),
  ("LOCAL", 77),
  ("MEMBER", 65),
  ("META_DATA", 1),
  ("METHOD", 300),
  ("METHOD_PARAMETER_IN", 593),
  ("METHOD_PARAMETER_OUT", 593),
  ("METHOD_RETURN", 300),
  ("MODIFIER", 34),
  ("NAMESPACE", 7),
  ("NAMESPACE_BLOCK", 24),
  ("RETURN", 87),
  ("TYPE", 117),
  ("TYPE_DECL", 118),
  ("TYPE_REF", 4)

[mpollmeier]

further potentially helpful debug info: typeDecls that only occur in java2cpg, not javasrc2cpg:

(cpgj2c.typeDecl.fullName.toSet &~ cpgjsrc2cpg.typeDecl.fullName.toSet).sorted

  "android.os.AsyncTask",
  "bottom_type",
  "byte",
  "char",
  "com.ulisesbocchio.jasyptspringboot.annotation.EnableEncryptableProperties",
  "float",
  "io.shiftleft.model.Constants$Type[]",
  "io.shiftleft.model.Customer[]",
  "io.shiftleft.model.Patient[]",
  "java.io.OutputStream",
  "java.io.Writer",
  "java.lang.AbstractMethodError",
  "java.lang.ArithmeticException",
  "java.lang.ArrayIndexOutOfBoundsException",
  "java.lang.ArrayStoreException",
  "java.lang.AssertionError",
  "java.lang.Boolean",
  "java.lang.Byte",
  "java.lang.CharSequence[]",
  "java.lang.Character",
  "java.lang.Class",
  "java.lang.ClassCastException",
  "java.lang.ClassCircularityError",
  "java.lang.ClassFormatError",
  "java.lang.ClassNotFoundException",
  "java.lang.Cloneable",
  "java.lang.Double",
  "java.lang.Enum",
  "java.lang.Error",
  "java.lang.ExceptionInInitializerError",
  "java.lang.Float",
  "java.lang.IllegalAccessError",
  "java.lang.IllegalMonitorStateException",
  "java.lang.IncompatibleClassChangeError",
  "java.lang.IndexOutOfBoundsException",
  "java.lang.InstantiationError",
  "java.lang.InternalError",
  "java.lang.InterruptedException",
  "java.lang.Iterable",
  "java.lang.LinkageError",
  "java.lang.Long[]",
  "java.lang.NegativeArraySizeException",
  "java.lang.NoClassDefFoundError",
  "java.lang.NoSuchFieldError",
  "java.lang.NoSuchMethodError",
  "java.lang.NullPointerException",
  "java.lang.Object[]",
  "java.lang.OutOfMemoryError",
  "java.lang.Runnable",
  "java.lang.Short",
  "java.lang.StackOverflowError",
  "java.lang.StringBuffer",
  "java.lang.StringBuilder",
  "java.lang.System",
  "java.lang.Thread",
  "java.lang.ThreadDeath",
  "java.lang.Throwable",
  "java.lang.UnknownError",
  "java.lang.UnsatisfiedLinkError",
  "java.lang.VerifyError",
  "java.lang.Void",
  "java.lang.invoke.LambdaMetafactory",
  "java.lang.ref.Finalizer",
  "java.nio.charset.Charset",
  "java.nio.charset.StandardCharsets",
  "java.util.Base64",
  "java.util.HashSet",
  "java.util.Iterator",
  "java.util.List",
  "java.util.Map",
  "java.util.Set",
  "java.util.concurrent.TimeoutException",
  "javax.annotation.PostConstruct",
  "javax.persistence.Entity",
  "javax.persistence.GeneratedValue",
  "javax.persistence.Id",
  "javax.persistence.OneToMany",
  "javax.persistence.OneToOne",
  "javax.persistence.Table",
  "javax.servlet.ServletOutputStream",
  "javax.servlet.http.Cookie",
  "javax.servlet.http.HttpServletRequest",
  "javax.servlet.http.HttpServletResponse",
  "javax.servlet.http.HttpSession",
  "null_type",
  "org.apache.commons.codec.digest.DigestUtils",
  "org.apache.http.Header",
  "org.apache.http.HttpEntity",
  "org.apache.http.HttpRequest",
  "org.apache.http.StatusLine",
  "org.apache.http.auth.AuthenticationException",
  "org.apache.http.auth.Credentials",
  "org.apache.http.auth.UsernamePasswordCredentials",
  "org.apache.http.client.ClientProtocolException",
  "org.apache.http.client.methods.CloseableHttpResponse",
  "org.apache.http.client.methods.HttpPost",
  "org.apache.http.client.methods.HttpUriRequest",
  "org.apache.http.entity.StringEntity",
  "org.apache.http.impl.auth.BasicScheme",
  "org.apache.http.impl.client.CloseableHttpClient",
  "org.apache.http.impl.client.HttpClients",
  "org.apache.http.protocol.HttpContext",
  "org.jasypt.encryption.StringEncryptor",
  "org.jasypt.encryption.pbe.StandardPBEStringEncryptor",
  "org.jasypt.properties.EncryptableProperties",
  "org.joda.time.DateTime",
  "org.slf4j.Logger",
  "org.slf4j.LoggerFactory",
  "org.springframework.beans.factory.annotation.Autowired",
  "org.springframework.boot.CommandLineRunner",
  "org.springframework.boot.SpringApplication",
  "org.springframework.boot.autoconfigure.SpringBootApplication",
  "org.springframework.boot.autoconfigure.web.ErrorAttributes",
  "org.springframework.boot.autoconfigure.web.ErrorController",
  "org.springframework.context.ConfigurableApplicationContext",
  "org.springframework.context.annotation.Configuration",
  "org.springframework.context.annotation.PropertySource",
  "org.springframework.core.env.Environment",
  "org.springframework.core.env.SimpleCommandLinePropertySource",
  "org.springframework.core.io.ClassPathResource",
  "org.springframework.data.repository.CrudRepository",
  "org.springframework.expression.Expression",
  "org.springframework.expression.ExpressionParser",
  "org.springframework.expression.spel.standard.SpelExpressionParser",
  "org.springframework.http.HttpStatus",
  "org.springframework.http.ResponseEntity",
  "org.springframework.stereotype.Component",
  "org.springframework.stereotype.Controller",
  "org.springframework.util.FileCopyUtils",
  "org.springframework.web.bind.annotation.ControllerAdvice",
  "org.springframework.web.bind.annotation.ExceptionHandler",
  "org.springframework.web.bind.annotation.GetMapping",
  "org.springframework.web.bind.annotation.PostMapping",
  "org.springframework.web.bind.annotation.RequestMapping",
  "org.springframework.web.bind.annotation.ResponseBody",
  "org.springframework.web.bind.annotation.ResponseStatus",
  "org.springframework.web.bind.annotation.RestController",
  "org.springframework.web.context.request.RequestAttributes",
  "org.springframework.web.context.request.ServletRequestAttributes",
  "org.springframework.web.context.request.WebRequest",
  "org.springframework.web.servlet.ModelAndView",
  "org.springframework.web.util.HtmlUtils",
  "org.zeroturnaround.exec.InvalidExitValueException",
  "org.zeroturnaround.exec.ProcessExecutor",
  "org.zeroturnaround.exec.ProcessResult",
  "org.zeroturnaround.exec.stream.slf4j.Slf4jOutputStream",
  "org.zeroturnaround.exec.stream.slf4j.Slf4jStream",
  "short"

[johannescoetzee]

With some fixes over time, all of the required methods are now found and https://github.com/joernio/joern/pull/1681 finally fixes the extra default constructor for interfaces.