Performance drop loading test suite

[JFoederer]

I have been running some tests on RIDE version 2.0b2.dev11 and noted a significant performance drop when loading test suites. Opening a test suite consisting of approximately 2500 robot files previously took just seconds to load, now it takes more than 5 minutes.

The amount of keyword libraries also plays a role. There are some variations, but the majority uses a predefined set of approximately 50 keyword libraries, varying in size from a few to one hundred keywords each.

I am running on a Windows 10 machine, Intel i7 with 16GB RAM, no other load on the system. Python 3.8.7 and Robot framework 5.0.1.

I haven't been able to keep up with recent updates, so I can't really give an estimate on when the performance regression was introduced.

[HelioGuilherme66]

@JFoederer I have some ideas where the problem may be. I would like for to try to generate sample data, with similar conditions of your test suites. For that, we have a script that you can call. Here is some help:

invoke generate-big-project --help
Usage: inv[oke] [--core-opts] generate-big-project [--options] [other tasks here ...]

Docstring:
  Generate big test data project to help perf testing.

Options:
  -a STRING, --args=STRING
  -i, --install
  -u, --upgrade


C:\GitHub\RIDE>invoke generate-big-project
rfgen: Running under Python 3
Test project is created into directory (option 'd'): .\tmp\testdir
Generating test project with following settings
    5 test libraries (option 'l')
    10 keywords per test library (option 'k')
    1 test suites (option 's')
    10 tests per test suite (option 't')
    3 test steps per test case (option 'e')
    1 resource files (option 'f')
    0 external resource files (option 'g')
    30 resources per resource file (option 'r')

    Seed was 1657788693
    

I you create one, please post here the command you used (preferably with a fixed random seed). then we can benchmark with it. Thanks.

[JFoederer]

That looks like a very useful tool! What is the syntax for passing the options? It keeps telling me it doesn't know any of the options.

[HelioGuilherme66]

Sorry, I did tried today but seems that it is not working. I suspect changes on argsparser module.

[axiom41]

Hi all... Maybe i'm wrong but as far as i could see, the performance degradation when loading the test suites is related to the indentation problems (issue #2460), because RIDE (starting just from 2.0b2.dev11) is adding those extra spaces since the suite loading, even without making any change. The more test suites/test cases you have, the more you have to wait for RIDE to add such spaces. That's why i'm still on 2.0b2.dev10

[JFoederer]

I can also confirm that the performance drop is not yet present in 2.0b2.dev10. That does narrow down the search a lot.

[JFoederer]

Is there any news on this topic? For large projects anything later than dev10 is not workable due to excessive load times.

[emakaay]

👍 Projects that normally take 10 seconds to load. Now take several to 4 minuten to start. Not ideal

[HelioGuilherme66]

@JFoederer and @emakaay It will be my next priority to fix this, after my current task.

[emakaay]

👍

[emakaay]

Any progress on this ticket ?

[emakaay]

Thanks.... The b3 version has solved the problem. It feels even faster then before