Performance Issue with file

[anon767]

Hi everybody,

I encountered This C File which takes an incredible amount of time until parsed.

Regards, Tom

[anon767]

This one and this one too :(

[oxisto]

ffserver.c: 87021 ms. ControlFlowSensitiveDFGPass takes 84965 ms mpegvideo.c: Parses in 614 ms overall on my M1, although I get a LOT of problem declaration nodes there so not sure now much of it is really parsed h264.c: Took 97127 ms on my M1. ControlFlowSensitiveDFGPass takes 67280 ms, so room for improvement there as well I would say.

[anon767]

Thanks for checking, could you give me your translation configuration? Seems like this is then a problem on my side

[oxisto]

Thanks for checking, could you give me your translation configuration? Seems like this is then a problem on my side

I was doing it as part of a unit test - I was parsing it without any heads, just the single file. this might make a difference.

@Test
  void testFF() throws Exception {
    File file = new File("src/test/resources/ffserver.c");
    TranslationUnitDeclaration tu =
        TestUtils.analyzeAndGetFirstTU(List.of(file), file.getParentFile().toPath(), true);

    assertNotNull(tu);
  }

Internally the following config is used for the unit tests:

val builder =
            TranslationConfiguration.builder()
                .sourceLocations(files)
                .topLevel(topLevel.toFile())
                .loadIncludes(true)
                .disableCleanup()
                .debugParser(true)
                .failOnError(true)
                .typeSystemActiveInFrontend(false)
                .useParallelFrontends(true)
                .defaultLanguages()
        if (usePasses) {
            builder.defaultPasses()
        }

[anon767]

Wow interesting. h264.c takes 22s for me on my MacBook Pro 2019 when using your configuration. It takes 3min and 10s when Im using my Configuration:

        val translationConfiguration = TranslationConfiguration
            .builder()
            .sourceLocations(paths)
            .defaultPasses()
            .defaultLanguages()
            .processAnnotations(true)
            .codeInNodes(true)
            .debugParser(false)
            .failOnError(false)
            .useParallelFrontends(true)
            .typeSystemActiveInFrontend(false)
            .loadIncludes(false)
            .build()

Is there something I have overlooked?

[konradweiss]

akes 67280 ms

I will look into this as I am already doing stuff on that pass.

[oxisto]

Wow interesting. h264.c takes 22s for me on my MacBook Pro 2019 when using your configuration. It takes 3min and 10s when Im using my Configuration:

        val translationConfiguration = TranslationConfiguration
            .builder()
            .sourceLocations(paths)
            .defaultPasses()
            .defaultLanguages()
            .processAnnotations(true)
            .codeInNodes(true)
            .debugParser(false)
            .failOnError(false)
            .useParallelFrontends(true)
            .typeSystemActiveInFrontend(false)
            .loadIncludes(false)
            .build()

Is there something I have overlooked?

Hmm codeInNodes and processAnnotations seem to be deactivated by default. Although I am not sure if codeInNodesreally has any effect. But it could potentially make a difference, the CXX implementation of that is not the best.

[oxisto]

Maybe this helps. I have implemented a way to read consolidated benchmark information. I will push the branch in a minute (Update: its here https://github.com/Fraunhofer-AISEC/cpg/pull/680)

Benchmark run 177c0350-88a8-48a0-bd40-18d5dbc52cac

Metric	Value
Translation config	{"debugParser": true, "loadIncludes": true, "includePaths": [], "includeWhitelist": [], "includeBlacklist": [], "frontends": {"class de.fraunhofer.aisec.cpg.frontends.java.JavaLanguageFrontend": [".java"], "class de.fraunhofer.aisec.cpg.frontends.cpp.CXXLanguageFrontend": [".c", ".cpp", ".cc", ".h", ".hpp"]}, "disableCleanup": true, "codeInNodes": true, "processAnnotations": false, "failOnError": true, "symbols": {}, "sourceLocations": ["src\/test\/resources\/ffserver.c"], "topLevel": "src\/test\/resources", "useUnityBuild": false, "useParallelFrontends": true, "typeSystemActiveInFrontend": false, "passes": ["de.fraunhofer.aisec.cpg.passes.TypeHierarchyResolver@1bdf4139", "de.fraunhofer.aisec.cpg.passes.JavaExternalTypeHierarchyResolver@5cf81961", "de.fraunhofer.aisec.cpg.passes.ImportResolver@1374d107", "de.fraunhofer.aisec.cpg.passes.VariableUsageResolver@2d9ac90a", "de.fraunhofer.aisec.cpg.passes.CallResolver@18acaed9", "de.fraunhofer.aisec.cpg.passes.EvaluationOrderGraphPass@3c70be10", "de.fraunhofer.aisec.cpg.passes.TypeResolver@5efa7435", "de.fraunhofer.aisec.cpg.passes.ControlFlowSensitiveDFGPass@3a7452f0", "de.fraunhofer.aisec.cpg.passes.FilenameMapper@1f979c6f"], "inferenceConfiguration": {"guessCastExpressions": false, "inferRecords": false}}
Number of files translated	1
Translated file(s)	[ffserver.c]
TranslationManager: Executing Language Frontend	994 ms
TypeHierarchyResolver: Executing Pass	65 ms
JavaExternalTypeHierarchyResolver: Executing Pass	0 ms
ImportResolver: Executing Pass	9 ms
VariableUsageResolver: Executing Pass	429 ms
CallResolver: Executing Pass	236 ms
EvaluationOrderGraphPass: Executing Pass	147 ms
TypeResolver: Executing Pass	41 ms
ControlFlowSensitiveDFGPass: Executing Pass	68926 ms
FilenameMapper: Executing Pass	3 ms
TranslationManager: Translation into full graph	70862 ms

Benchmark run 41836601-0f5e-45bd-89af-0ed1f62364e2

Metric	Value
Translation config	{"debugParser": true, "loadIncludes": true, "includePaths": [], "includeWhitelist": [], "includeBlacklist": [], "frontends": {"class de.fraunhofer.aisec.cpg.frontends.java.JavaLanguageFrontend": [".java"], "class de.fraunhofer.aisec.cpg.frontends.cpp.CXXLanguageFrontend": [".c", ".cpp", ".cc", ".h", ".hpp"]}, "disableCleanup": true, "codeInNodes": true, "processAnnotations": false, "failOnError": true, "symbols": {}, "sourceLocations": ["src\/test\/resources\/mpegvideo.c"], "topLevel": "src\/test\/resources", "useUnityBuild": false, "useParallelFrontends": true, "typeSystemActiveInFrontend": false, "passes": ["de.fraunhofer.aisec.cpg.passes.TypeHierarchyResolver@65add5c9", "de.fraunhofer.aisec.cpg.passes.JavaExternalTypeHierarchyResolver@2b8e63ef", "de.fraunhofer.aisec.cpg.passes.ImportResolver@60462928", "de.fraunhofer.aisec.cpg.passes.VariableUsageResolver@25249056", "de.fraunhofer.aisec.cpg.passes.CallResolver@852b03d", "de.fraunhofer.aisec.cpg.passes.EvaluationOrderGraphPass@40a8508a", "de.fraunhofer.aisec.cpg.passes.TypeResolver@369ced13", "de.fraunhofer.aisec.cpg.passes.ControlFlowSensitiveDFGPass@48012497", "de.fraunhofer.aisec.cpg.passes.FilenameMapper@29e562ef"], "inferenceConfiguration": {"guessCastExpressions": false, "inferRecords": false}}
Number of files translated	1
Translated file(s)	[mpegvideo.c]
TranslationManager: Executing Language Frontend	568 ms
TypeHierarchyResolver: Executing Pass	1 ms
JavaExternalTypeHierarchyResolver: Executing Pass	0 ms
ImportResolver: Executing Pass	0 ms
VariableUsageResolver: Executing Pass	7 ms
CallResolver: Executing Pass	4 ms
EvaluationOrderGraphPass: Executing Pass	8 ms
TypeResolver: Executing Pass	1 ms
ControlFlowSensitiveDFGPass: Executing Pass	2 ms
FilenameMapper: Executing Pass	0 ms
TranslationManager: Translation into full graph	601 ms

Benchmark run 7e464dca-9c18-45f5-8a5d-219808470783

Metric	Value
Translation config	{"debugParser": true, "loadIncludes": true, "includePaths": [], "includeWhitelist": [], "includeBlacklist": [], "frontends": {"class de.fraunhofer.aisec.cpg.frontends.java.JavaLanguageFrontend": [".java"], "class de.fraunhofer.aisec.cpg.frontends.cpp.CXXLanguageFrontend": [".c", ".cpp", ".cc", ".h", ".hpp"]}, "disableCleanup": true, "codeInNodes": true, "processAnnotations": false, "failOnError": true, "symbols": {}, "sourceLocations": ["src\/test\/resources\/h264.c"], "topLevel": "src\/test\/resources", "useUnityBuild": false, "useParallelFrontends": true, "typeSystemActiveInFrontend": false, "passes": ["de.fraunhofer.aisec.cpg.passes.TypeHierarchyResolver@7c194bef", "de.fraunhofer.aisec.cpg.passes.JavaExternalTypeHierarchyResolver@57a40e23", "de.fraunhofer.aisec.cpg.passes.ImportResolver@4b3e679", "de.fraunhofer.aisec.cpg.passes.VariableUsageResolver@4343b059", "de.fraunhofer.aisec.cpg.passes.CallResolver@5889eb2e", "de.fraunhofer.aisec.cpg.passes.EvaluationOrderGraphPass@1e7b1afa", "de.fraunhofer.aisec.cpg.passes.TypeResolver@20c91522", "de.fraunhofer.aisec.cpg.passes.ControlFlowSensitiveDFGPass@24325bf3", "de.fraunhofer.aisec.cpg.passes.FilenameMapper@2bf40c94"], "inferenceConfiguration": {"guessCastExpressions": false, "inferRecords": false}}
Number of files translated	1
Translated file(s)	[h264.c]
TranslationManager: Executing Language Frontend	5792 ms
TypeHierarchyResolver: Executing Pass	209 ms
JavaExternalTypeHierarchyResolver: Executing Pass	0 ms
ImportResolver: Executing Pass	26 ms
VariableUsageResolver: Executing Pass	7518 ms
CallResolver: Executing Pass	6519 ms
EvaluationOrderGraphPass: Executing Pass	8523 ms
TypeResolver: Executing Pass	2174 ms
ControlFlowSensitiveDFGPass: Executing Pass	60506 ms
FilenameMapper: Executing Pass	25 ms
TranslationManager: Translation into full graph	91304 ms

[anon767]

Im thrilled 😄

[konradweiss]

Is this issue so far resolved or are we keeping it open as we have further optimizations that we plan to follow up on?

[KuechA]

Is this still an issue on the current main branch? I can't replicate those times on my machine at the moment (e.g. the ffserver.c took around 3 seconds). We completely changed the ControlFlowSensitiveDFGPass, so I hope that the issue is resolved.

[anon767]

Looks good :) thank you!