roc Have reporting and solve tests be file-based

Let's update the roc_reporting and roc_solve tests to operate over files rather than by inlining all tests in a Rust file with a large number of macros. @bhansconnect did some analysis and doing so would avoid a large amount of compile+link time.

Proposed:

<test flags>
===
<roc source>
===
<reporting or solve output>

https://roc.zulipchat.com/#narrow/stream/231635-compiler-development/topic/rust.20compile.20times/near/340966165

Mar 13 '23 13:03 ayazhafiz

zulip gives an error that that link isn't publicly accessible. Can anyone with access copy+paste the context in this issue?

Dec 09 '24 21:12 shua

Here you go :)

8:58 PM

Oh, also, something that should be huge in CI. I am change release to not use lto. making a separate release-with-lto. That will only be used for running benchmarks and building nightlys. On my M1, removing lto doubles the speed of cargo test --no-run --release

<div class="unread_marker message_unread_marker"><div class="unread-marker-fill"></div></div>
<div class="messagebox">
    <div class="messagebox-content ">
        <span class="message_sender">

8:59 PM

Cause linking every test binary with lto is just a waste of time.

    </div>
</div>

9:31 PM

On M1:
~~1 minute longer test runs~~ for a 4 minute faster test build time. Yeah, that seems worth it to me.

    <div id="message-row-1-340959111" data-message-id="340959111" class="message_row  selectable_row" role="listitem"><div class="messagebox"><div class="messagebox-content "><div class="message_length_controller"></div>

    </div>
</div>

9:33 PM

Also, wow, LTO does more than I expected to the test execution time. Only ran things once, but that is this is like 25% faster test execution time with lto.

    </div>
</div>

9:43 PM

oh nvm, lto adds no test speed up. The speedup come from platforms being compiled.

    <div id="message-row-1-340961005" data-message-id="340961005" class="message_row  selectable_row" role="listitem"><div class="messagebox"><div class="messagebox-content "><div class="message_length_controller"></div>

    </div>
</div>

10:11 PM

Question for I assume Ayaz Hafiz , maybe Folkert de Vries . Could test reporting be changed to read in files and run test across all files in a folder like mono? The current test executable is so large that it takes 145s on my linux machine to link it (without lto). It very directly changes the build time of tests.

    <div id="message-row-1-340965220" data-message-id="340965220" class="message_row  selectable_row" role="listitem"><div class="messagebox"><div class="messagebox-content "><div class="message_length_controller"></div>

    </div>
</div>

10:15 PM

Yes that is true. I think the same holds for the solve tests

    </div>
</div>

10:16 PM

yeah, solve is pretty bad too at ~40s.

    </div>
</div>

10:17 PM

I am not opposed, but I think we should be a bit clever about how we do such a thing if we do it. The nice thing about the in-file tests is the localization of the tests and the output. With file goldens it's an easier to create duplicate tests IMO , and if we separate the test source from the output the experience is much worse. So my preference, if we go down this road, is to localize the tests and their output in the same file (in other compilers I've worked on this always works out the best)

    </div>
</div>

10:18 PM

But, I don't think I would have the time to do a migration like this soon myself

    </div>
</div>

10:27 PM

Also, running in ci right now. Compared to random passing PR. Some highlights:

name	old time	new time	old/new
Nix apple silicon cargo test	30	19	1.58
Macos x86-64 rust tests	20	13	1.54
Nix linux x86_64 cargo test	30	20	1.5

</div>

<div class="unread_marker message_unread_marker"><div class="unread-marker-fill"></div></div>
<div class="messagebox">
    <div class="messagebox-content ">
        <span class="message_sender">

10:28 PM

Ayaz Hafiz said:

But, I don't think I would have the time to do a migration like this soon myself

Do you think you could file an issue with the details just to track it for now?

    </div>
</div>

10:28 PM

yeah sure

    </div>
</div>

11:51 AM

we should have a build profile (and probably the default for now) for "build CLI without editor support"

:+1: Not using the editor feature by default is how I have things set up for the headless release

    </div>
</div>

1:55 PM

ah cool! Thoughts on making that the default in general for now?

    </div>
</div>

1:55 PM

I like the idea of getting to a world where people complain about that being an annoying default because so much work is being done on the editor, but obviously that is not the world of today :big_smile:

    </div>
</div>

2:06 PM

ah cool! Thoughts on making that the default in general for now?

:100: Yes, that's what I planned

<div class="unread_marker date_unread_marker"><div class="unread-marker-fill"></div></div>
<div class="unread_marker message_unread_marker"><div class="unread-marker-fill"></div></div>
<div class="messagebox">
    <div class="messagebox-content ">
            </div></div><span class="message_sender">
    <span class="sender_info_hover sender_name" role="button" tabindex="0">
        <span class="view_user_card_tooltip sender_name_text" data-is-bot="false">
            <span class="user-name">Ayaz Hafiz</span>
        </span>
                </span>
        </span>

11:08 PM

Brendan Hansknecht said:

Question for I assume Ayaz Hafiz , maybe Folkert de Vries . Could test reporting be changed to read in files and run test across all files in a folder like mono? The current test executable is so large that it takes 145s on my linux machine to link it (without lto). It very directly changes the build time of tests.

A first step in this direction: https://github.com/roc-lang/roc/pull/5242

Brendan Hansknecht 8:58 PM

Oh, also, something that should be huge in CI. I am change release to not use lto. making a separate release-with-lto. That will only be used for running benchmarks and building nightlys. On my M1, removing lto doubles the speed of cargo test --no-run --release 8:59 PM

Cause linking every test binary with lto is just a waste of time. 9:31 PM

On M1: 1 minute longer test runs for a 4 minute faster test build time. Yeah, that seems worth it to me. 9:33 PM

Also, wow, LTO does more than I expected to the test execution time. Only ran things once, but that is this is like 25% faster test execution time with lto. 9:43 PM

oh nvm, lto adds no test speed up. The speedup come from platforms being compiled. 10:11 PM

Question for I assume Ayaz Hafiz , maybe Folkert de Vries . Could test reporting be changed to read in files and run test across all files in a folder like mono? The current test executable is so large that it takes 145s on my linux machine to link it (without lto). It very directly changes the build time of tests. Ayaz Hafiz 10:15 PM

Yes that is true. I think the same holds for the solve tests Brendan Hansknecht 10:16 PM

yeah, solve is pretty bad too at ~40s. Ayaz Hafiz 10:17 PM

I am not opposed, but I think we should be a bit clever about how we do such a thing if we do it. The nice thing about the in-file tests is the localization of the tests and the output. With file goldens it's an easier to create duplicate tests IMO , and if we separate the test source from the output the experience is much worse. So my preference, if we go down this road, is to localize the tests and their output in the same file (in other compilers I've worked on this always works out the best) 10:18 PM

But, I don't think I would have the time to do a migration like this soon myself Brendan Hansknecht 10:27 PM

Also, running in ci right now. Compared to random passing PR. Some highlights: name old time new time old/new Nix apple silicon cargo test 30 19 1.58 Macos x86-64 rust tests 20 13 1.54 Nix linux x86_64 cargo test 30 20 1.5 10:28 PM

Ayaz Hafiz said:

But, I don't think I would have the time to do a migration like this soon myself

Do you think you could file an issue with the details just to track it for now? Ayaz Hafiz 10:28 PM

yeah sure Anton 11:51 AM

we should have a build profile (and probably the default for now) for "build CLI without editor support"

:+1: Not using the editor feature by default is how I have things set up for the headless release Richard Feldman 1:55 PM

ah cool! Thoughts on making that the default in general for now? 1:55 PM

I like the idea of getting to a world where people complain about that being an annoying default because so much work is being done on the editor, but obviously that is not the world of today :big_smile: Anton 2:06 PM

ah cool! Thoughts on making that the default in general for now?

:100: Yes, that's what I planned Ayaz Hafiz 11:08 PM

Brendan Hansknecht said:

Question for I assume Ayaz Hafiz , maybe Folkert de Vries . Could test reporting be changed to read in files and run test across all files in a folder like mono? The current test executable is so large that it takes 145s on my linux machine to link it (without lto). It very directly changes the build time of tests.

A first step in this direction: https://github.com/roc-lang/roc/pull/5242

Dec 10 '24 11:12 Anton-4