ORT hang on analyzer and reporter with regex resolution match

[AJDurant]

When running ORT recently I have found it hanging with a single thread on 100% CPU. This occurs on the analyze command after writing out the file. Also on the report stage when trying to generate StaticHTML or WebApp reports.

The both hangs seem dependent on trying to use a regex match in my resolutions in .ort.yml:

resolutions:
  issues:
   - message: "Resolving PIP dependencies for path (.|\n)*?ERROR: Invalid requirement:"
     reason: "SCANNER_ISSUE"
     comment: "Detecting non-requirements file as pip requirements."
   - message: "Resolving PIP dependencies for path 'test/end2end/product/(.|\n)*?ERROR: No matching distribution found"
     reason: "SCANNER_ISSUE"
     comment: "Detecting non-requirements file as pip requirements."

Running the reporter gives this output:

$ docker run -v $PWD:/project ort --debug report -i /project/evaluation-result.yml -o /project/ -f StaticHtml
13:43:56.785 [main] DEBUG org.ossreviewtoolkit.cli.OrtMain - Used command line arguments: [--debug, report, -i, /project/evaluation-result.yml, -o, /project/, -f, StaticHtml]
________ _____________________
\_____  \\______   \__    ___/ the OSS Review Toolkit, version bd1a5348a2.
 /   |   \|       _/ |    |
/    |    \    |   \ |    |    Running 'report' under Java 11.0.15 on Linux with
\_______  /____|_  / |____|    40 CPUs and a maximum of 29184 MiB of memory.
        \/       \/
Environment variables:
ORT_CONFIG_DIR = /root/.ort/config
ORT_DATA_DIR = /root/.ort
JAVA_HOME = /opt/java/openjdk
ANDROID_HOME = /opt/android-sdk
GOPATH = /tmp/go

13:43:57.275 [main] DEBUG org.ossreviewtoolkit.cli.commands.ReporterCommand - Input ORT result file has SHA-1 hash ee51958b9a8ff7192432a54a0a1ddeb9d585d9de.
13:44:03.831 [main] PERFORMANCE org.ossreviewtoolkit.cli.commands.ReporterCommand - Read ORT result from 'evaluation-result.yml' (24.69 MiB) in 6.542539186s.
13:44:03.844 [main] DEBUG org.ossreviewtoolkit.utils.core.storage.LocalFileStorage - Creating directory '/root/.ort/scanner/archive' for local file storage.
Generating the 'StaticHtml' report in thread 'DefaultDispatcher-worker-2'...
13:44:03.959 [DefaultDispatcher-worker-2] PERFORMANCE org.ossreviewtoolkit.model.OrtResult - Computing excluded packages...
13:44:03.969 [DefaultDispatcher-worker-2] PERFORMANCE org.ossreviewtoolkit.model.OrtResult - Computing excluded projects...
13:44:03.975 [DefaultDispatcher-worker-2] PERFORMANCE org.ossreviewtoolkit.model.OrtResult - Computing excluded projects took 4.611928ms.
13:44:03.979 [DefaultDispatcher-worker-2] PERFORMANCE org.ossreviewtoolkit.model.OrtResult - Computing excluded packages took 19.294654ms.

The evaluation-report.yml file is 25278KB, so it is big, but the other reports I've tried work fine: GitLabLicenseModel,HtmlTemplate,NoticeTemplate

[sschuberth]

Thanks for the report @AJDurant. While not a solution to the root cause, you should be able to work around the issue by simplifying the regex, as messages are "sanitized" before being compared. Meaning that multiple consecutive whitespace (incl. newlines) is collapsed to a single space (" ").

[sschuberth]

you should be able to work around the issue by simplifying the regex

Did that actually work @AJDurant to avoid the hanging?

[AJDurant]

I commented out those resolutions, and haven't re-introduced them yet. But doing so did resolve the hang, so it's definitely the regex that is causing the problem.

[sschuberth]

Would you be able to share the scan result @AJDurant?

[AJDurant]

I can't share the full result, but may be able to share a cut down version. I'm not currently working on the same project though, so won't be any time soon.

[sschuberth]

I can't share the full result, but may be able to share a cut down version.

Any update here @AJDurant? Or if the issue is not reproducible anymore, can we close this?

[AJDurant]

Sorry @sschuberth, the project I was working on tailed off, and I was avoiding using the regex when I was last working on it. If I pick it up again then I'll try and find a workaround and update here.