[Bug]: Correlation with many items causes engine restart, no user output

[richlv]

JASP Version

0.18.3

Commit ID

No response

JASP Module

Regression

What analysis are you seeing the problem on?

Correlation

What OS are you seeing the problem on?

macOS Intel

Bug Description

Running a correlation with a large number of items seems to hang and produces no results. At first, spinners are shown in the result section. Later only grey area is displayed. Otherwise results are displayed without any issues (that is, this does not seem to be a display issue).

While loading that many items is likely unwise, it would preferably not break JASP without any user-visible indicator.

Expected Behaviour

a) If feasible, results should be displayed.

b) In any case, either showing status (at least as spinners) or error message, if available, would be useful not to leave users confused.

Steps to Reproduce

1. Load data.
2. Start correlation.
3. Add many items.

Log (if any)

Logfiles (as they contain various parts of data) will be compressed and emailed to [email protected] as per the instructions in this form.

Final Checklist

• [ ] I have included a screenshot showcasing the issue, if possible.
• [ ] I have included a JASP file (zipped) or data file that causes the crash/bug, if applicable.
• [X] I have accurately described the bug, and steps to reproduce it.

[github-actions[bot]]

@richlv, thanks for taking the time to create this issue. If possible (and applicable), please upload to the issue website (https://github.com/jasp-stats/jasp-issues/issues/2746) a screenshot showcasing the problem, and/or a compressed (zipped) .jasp file or the data file that causes the issue. If you would prefer not to make your data publicly available, you can send your file(s) directly to us, [email protected]

[richlv]

Link to logs sent in email.

Edit: email delivery failed. Attempting a resend to the expanded address.

[richlv]

126 variables, 55 rows, 16GB (available memory might be half or less at times). As mentioned, this was just a test - and if something is actually unfeasible (likely), providing some feedback to the user would be useful :)

[tomtomme]

Interesting. Tested with 0.18.3 flatpak

I could compute this for 128 variables even... BUT only if I added them step by step. I started with 100 variables and then added 10 more step by step. If I start with roughly >110 then I can reproduce the bug (only grey area in output window)

Data test.jasp.zip

[JohnnyDoorn]

I think this issue is better suited for @RensDofferhoff, @JorisGoosen, or @boutinb - I can forward the logs to whoever can help here.

[boutinb]

On my Mac (32GB, 10 Cores M1 Pro), the test.jasp takes a bit long to refresh (~5 minutes), but it works...

[boutinb]

It works also if I add all 128 vars in one go.

What strikes me is that the Desktop log file is huge (207 MB), and not the Engine log file (just 17 KB). JASP 2024-05-29 09_58_07 Desktop.log.zip

This is also the case in the log files of @richlv

When I have a look at the log file, I see that the Desktop receives many temporary answers from the Engine. Just at the end, the Engine sends a JSON string of 3.1 MB. If I export the result in HTML, it is 26,7 MB, for a table of 128 * 2 rows and 128 columns Maybe this transformation of such a JSON string to a HTML string takes so long... But if I have a look at my CPU, I see that that nearly most of the time is taken by the Engine: when the Engine stops, the result is really quickly displayed. That means that the JSON transformation was really quick.

I see also in the log files of @richlv, that it takes 11 minutes before the Engine is ready, at the end it generates a JSON string of 6.4MB, the double of what I get, so I can't imagine that it is the problem.

[JorisGoosen]

string of 6.4MB, the double of what I get, so I can't imagine that it is the problem.

Did you check ram usage while this was being calculated?

[tomtomme]

@boutinb Opening the file, then refreshing does work for me too! But closing the analysis and starting it from scratch with 128 vars does not!

@JorisGoosen RAM usage of JASP peaked at about 4GB and I had plenty free memory space left (of 32GB)

[JorisGoosen]

The RAM usage of 4GB was when it worked or when it did not work? Or perhaps both?

On the other hand, whenever a new vriable gets dragged into the list it asks the engine to restart, maybe something in that process is breaking this.

[tomtomme]

Both.

Still valid for 0.95.4 - note to myself: retest when updated Qt in jasp is available. But only if you start the analysis from scratch. Refreshing it works fine.

Testfile test.jasp.zip

Logs JASP 2024-12-07 15_58_14 Engine 0.log JASP 2024-12-07 15_58_14 Desktop.log

From terminal:

flatpak run --branch=beta --devel org.jaspstats.JASP
Desktop:        QML cache directory: /home/tom/.var/app/org.jaspstats.JASP/cache/JASP/JASP/qmlcache_4f0565493a81ccc4b90f19682ac948ddc39027b7
Making new argument list for Application startup: JASP
Starting JASP 0.19.2 from commit 4f0565493a81ccc4b90f19682ac948ddc39027b7 and branch HEAD
QtWebEngineQuick initialized
Application initialized
Application init entered
MainWindow constructor started
TempFiles::createSessionDir(): '/home/tom/.JASP/temp/3' about to be (removed and re)created.
Going to construct the necessary models for JASP to function.
Desktop:        Checking qm file: /app/Resources/Translations/jaspDesktop-tr.qm
Desktop:        Invalid translation file found with name: jaspDesktop-tr.qm
Desktop:        Checking qm file: /app/Resources/Translations/jaspDesktop-ru.qm
Desktop:        Language (Russian) not registered in LanguageModel, adding it now
Desktop:        Checking qm file: /app/Resources/Translations/jaspDesktop-es.qm
Desktop:        Language (Spanish) not registered in LanguageModel, adding it now
Desktop:        Checking qm file: /app/Resources/Translations/jaspDesktop-pt_BR.qm
Desktop:        Invalid translation file found with name: jaspDesktop-pt_BR.qm
Desktop:        Checking qm file: /app/Resources/Translations/jaspDesktop-zh_Hans.qm
Desktop:        Language (Chinese) not registered in LanguageModel, adding it now
Desktop:        Checking qm file: /app/Resources/Translations/jaspDesktop-ja.qm
Desktop:        Language (Japanese) not registered in LanguageModel, adding it now
Desktop:        Checking qm file: /app/Resources/Translations/jaspDesktop-fr.qm
Desktop:        Language (French) not registered in LanguageModel, adding it now
Desktop:        Checking qm file: /app/Resources/Translations/jaspDesktop-nl.qm
Desktop:        Language (Dutch) not registered in LanguageModel, adding it now
Desktop:        Checking qm file: /app/Resources/Translations/jaspDesktop-zh_Hant.qm
Desktop:        Language (Chinese) not registered in LanguageModel, adding it now
Desktop:        Checking qm file: /app/Resources/Translations/jaspDesktop-pl.qm
Desktop:        Language (Polish) not registered in LanguageModel, adding it now
Desktop:        Checking qm file: /app/Resources/Translations/jaspDesktop-de.qm
Desktop:        Language (German) not registered in LanguageModel, adding it now
Desktop:        Checking qm file: /app/Resources/Translations/jaspDesktop-it.qm
Desktop:        Language (Italian) not registered in LanguageModel, adding it now
Desktop:        Checking qm file: /app/Resources/Translations/jaspDesktop-id.qm
Desktop:        Language (Indonesian) not registered in LanguageModel, adding it now
Desktop:        Checking qm file: /app/Resources/Translations/jaspDesktop-gl.qm
Desktop:        Language (Galician) not registered in LanguageModel, adding it now
Desktop:        Checking qm file: /app/Resources/Translations/jaspDesktop-pt.qm
Desktop:        Language (Portuguese) not registered in LanguageModel, adding it now
Desktop:        Checking qm file: /app/Resources/Translations/jaspDesktop-eo.qm
Desktop:        Invalid translation file found with name: jaspDesktop-eo.qm
Desktop:        Opened internal sqlite database for creation at '/home/tom/.JASP/temp/3/internal.sqlite'.
Desktop:        DataSet::DataSet(index=-1)
Desktop:        DataSetPackage::reset()
Desktop:        DataSet::DataSet(index=-1)
Desktop:        DataSetPackage::endLoadingData
Entering eventloop
Error in loadNamespace(x) : there is no package called ‘jaspBase’
Error in loadNamespace(x) : there is no package called ‘jaspBase’
Loading analysis based on different version of module(jaspRegression), but going ahead anyway. Analysis based on version: 0.18.3 and actual loaded version of module is: 0.19.2
QSGPlainTexture: Mipmap settings changed without having image data available. Call setImage() again or enable m_retain_image. Falling back to previous mipmap filtering mode.
Loading required namespace: ragg

Attaching package: ‘jaspBase’

The following objects are masked _by_ ‘.GlobalEnv’:

    %|%, betaDist, binomDist, BoxCox, chiSqDist, expDist, fDist, fishZ,
    gammaDist, geomDist, hasSubstring, ifElse, integerDist, invBoxCox,
    invFishZ, invLogit, logit, logNormDist, negBinomDist, normalDist,
    poisDist, powerTransform, replaceNA, rowMax, rowMaxNaRm, rowMean,
    rowMeanNaRm, rowMedian, rowMedianNaRm, rowMin, rowMinNaRm, rowSD,
    rowSDNaRm, rowSum, rowSumNaRm, rowVariance, rowVarianceNaRm, tDist,
    unifDist, weibullDist, YeoJohnson, zScores

[96:12:0100/000000.937957:ERROR:platform_shared_memory_region_posix.cc(47)] Descriptor access mode (2) differs from expected (0)