A Mechanism for Adding Hooks into CURI for Custom Installation Output

[Xliff]

Since its start, Rakudo has relied well upon CURI (ala CompUnit::Repository::Installation) for its installation purposes, and it has served its purpose well. However, as Rakudo grows older and Raku just grows the software CURI is responsible for will only grow more complex, it is only expected that CURI, in its current state, will take longer to execute.

Herein lies the problem. Here is a recent example of an recently performed install:

===> Searching for: LibXML
===> Updating fez mirror: https://360.zef.pm/
===> Updated fez mirror: https://360.zef.pm/
===> Updating cpan mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/cpan1.json
===> Updating p6c mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/p6c1.json
===> Updated p6c mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/p6c1.json
===> Updated cpan mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/cpan1.json
===> Searching for missing dependencies: File::Temp, W3C::DOM:ver<0.0.2+>, XML
===> Searching for missing dependencies: File::Directory::Tree
===> Building: LibXML:ver<0.6.10>:auth<cpan:WARRINGD>
===> Building [OK] for LibXML:ver<0.6.10>:auth<cpan:WARRINGD>
===> Testing: File::Directory::Tree
===> Testing [OK] for File::Directory::Tree
===> Testing: File::Temp:ver<0.0.10>:auth<zef:rbt>
===> Testing [OK] for File::Temp:ver<0.0.10>:auth<zef:rbt>
===> Testing: W3C::DOM:ver<0.0.2>:auth<cpan:WARRINGD>
===> Testing [OK] for W3C::DOM:ver<0.0.2>:auth<cpan:WARRINGD>
===> Testing: XML:ver<0.3.1>:auth<github:raku-community-modules>
===> Testing [OK] for XML:ver<0.3.1>:auth<github:raku-community-modules>

[ long wait, here ... ]

===> Testing: LibXML:ver<0.6.10>:auth<cpan:WARRINGD>

[ long wait, here ... ]

Where "long wait" means "anything over 2 minutes with no interaction with the user".

As the Raku user-base grows, I expect this type of behavior will prove to be very daunting, and my fear is that many of these users will abandon Raku, and do so quietly.

Suggestions like 'Use RAKUDO_MODULE_DEBUG' are not helpful, as they provide the user with too much information.

The recent addition of the RAKUDO_PRECOMPILATION_PROGRESS environment variable is a step in the right direction, however I feel it remains too technical as a practical solution.

What I would like to discuss is adding a mechanism, possibly triggered via META6.json to give the package maintainer a way to customize this output, so that users remain informed during the install process.

Something as simple as a console-bar graph, or a by-line message showing which compunits have been tested or installed would be enough to satisfy this issue.

Thank you in advance for your suggestions and comments.

[codesections]

This is potentially a naive question, but to what extent is this a CompUnit::Repository::Installation issue versus a Zef issue?

Running that same command with zef -v already produces a fairly reasonable (and, imo, non-overwhelming) amount of output. Even if we want to let package maintainers customize the default output via a setting in the META6.json file, I'd have thought that Zef could handle that. (We already have the concept of customary fields – which aren't required/spec'd but that are used by standard tools – for the META6.json format.

Am I missing a reason to address this issue at the CompUnit::Repository::Installation level instead of via Zef?

[Xliff]

@codesections - I've already had this discussion with ugexe. We both have agreed that this issue is best settled via CURI

[codesections]

That's totally fine. Out of curiosity, what makes it a better fit for CURI?

[Xliff]

@ugexe - I disagree. RAKUCO_PRECOMPILATION_PROGRESS is not sufficient. It can't be. It too does not know what the distributor wants to communicate to the user at install time. If this can be done without a hook into CURI, I'm all for it. Otherwise I don't see how this can be done without it.

[Xliff]

@ugexe,

This is because you still underestimate what I am saying. It's not just that the user remains informed, it is how.

Once again you take it upon yourself to redefine the issue.

That question is not for you to answer.

It is for the person writing the package.

[niner]

If I understand this issue correctly, what you are asking for is for a module to kind of plug into the installation process to output messages. If that's right, then the mechanism seems to be there already: BEGIN time code is executed during compilation, so also during precompilation. A BEGIN note ... will tell the user whatever you want.

I have to say though that personally as a user I'd not expect to get told arbitrary things during installation and may even get annoyed by messages that are not actually helping me and interfere with what I have practice with processing.

[Xliff]

@niner:

Thanks for the comment. From my prospective as a user, I'd want to know a few a few things:

1. What is going on
2. Where are we currently in the process
3. How much longer is this going to take

Given these criterion, Raku provides none of them.

A BEGIN note is fine, but it's CompUnit scoped for a reason. The solution to 1-3 implies something with a wider perspective.

[vrurg]

@niner The problem is about huge projects with hundreds+ modules where pre-compilation takes eons. Imagine any heavy GUI app, which loads tons of stuff on the start and displays a progress bar just to entertain you while it's heating up (GIMP? ;) ).

BTW, as far as I understand, this is not only the case of installing a module/app, but recompiling it if Rakudo version changes. It is rather scary to wait for minutes, sometimes, without knowing what's going on when an app is started for the first time after upgrade.

Theoretically, it is possible to use BEGIN for the purpose of reporting each module, but:

• this is a boilerplate to be added to each particular module
• basically, a module doesn't know how many was compiled already and how many is yet to compile
• the previous item could be solved, but the solution would only add extra to the complexity of the project
• and if any change is needed – go, do it for each individual module
• eventually, this all is going to be too error prone

Callbacks invoked before/after a unit compilation is the most clean solution here. A startup script would just pre-load META6 and monitor which modules are being/has been compiled.

Since Rakudo cannot cover all possible use cases, the best is to allow user's code to do what it needs to.

[vrurg]

BTW, it just crossed my mind that it must be a Supply, emitting pre-comp events.

[lizmat]

hmmm... that's an interesting thought!

[niner]

I noticed that the issue report says:

===> Testing [OK] for XML:ver<0.3.1>:auth<github:raku-community-modules>
[ long wait, here ... ]
===> Testing: LibXML:ver<0.6.10>:auth<cpan:WARRINGD>

This is not an issue with long installation time. It's long time of running tests! What's not being reported here is the tests' output. I have occasionally wondered about this design decision in zef. But that may be just because I am used to seeing the output from Perl module installers. This is not something that Rakudo can change at all, since it doesn't even know that tests are being run. In fact it doesn't know the concept of "test" at all.

However, since there was also a lot of discussion about the precompilation stage of the installation process, I may as well add my thoughts on that. Why do people seek information about progress in the first place? The answer is clear: because installation may just take a long time. While some form of progress report may keep users entertained or help preventing wrong assumptions about the program hanging, it wouldn't actually fix the underlying problem. Quite contrary it would only make that worse.

Progress reporting by itself consumes CPU time and will make installation slower. You might say that the impact is surely negligible and the increased user confidence worth it. But that's judging a solution that we do not even know how it will look like? Yes, just emitting events won't matter that much. But who will receive the events? If - as was being suggested - the receiving end is supplied by the distribution that gets installed, then this is code that itself has to be (pre-)compiled first. And that is certainly going to take a measurable and even noticeable amount of time.

I'd much rather we invest our time and effort into attacking the underlying problem itself. For some reason I don't see much happening in that regard, despite us all surely getting bored and annoyed by module installation times. My guess is that people think that the only way to improve this would be via major refactors deep inside the compiler, like RakuAST or super complicated spesh improvements that you'd need a PhD on compiler internals to understand. Nothing could be further from the truth.

There is quite a bit of low hanging fruit that could yield massive improvements. Right now for example every module gets compiled twice when installed with zef. Once for testing and then on installation. That's quite unnecessary! We have all the foundation we need to first install a module into a CompUnit::Repsoitory::Staging repository, use that for tests and if successful simply copy the resulting files from the Staging repository to the actual target. That's btw. exactly what happens when installing modules from RPM packages: the build process installs into a Staging repository and the resulting files are packaged up and on installation only extracted from the archive into their final location.

Right now, module installation uses a single one of the 16 CPU cores in my desktop computer. All other cores lie there dormant and underappreciated. That's because the precompilation part of CompUnit::Repository::Installation is a simple for loop iterating over the %provides hash from the distribution's meta data. Raku is a language that provides a lot of very nice high level support for parallelism. Xliff++ has already experimented with parallelized build, but unfortunately this work so far only made it into a custom build script.

Frankly, parallelizing installation looks like a beginner task to me. A good opportunity to explore Raku's concurrency support. And I happen to know that the author of the code is happy to answer questions and provide assistance.

There was also mention of upgrading Rakudo leading to prolonged application startup times. A script that precompiles all installed modules as (optional) part of Rakudo's installation process would again be something quite short and simple. This would basically be what Android does in the "optimizing apps" stage after updating the system. Of course this script would try to parallelize as much as possible. Parallelizing precompilation during application startup would be really hard and require something like heuristic look ahead parsing or plain guesses. Doing so for installed modules on the other hand is really quite trivial as all the information that's needed is there already and easily accessible.

That's all low hanging fruit that would cut down waiting time to a fraction of what it is now. Really, the 2 minutes could become 10 seconds on a beefy machine.

But the possibilities do not even stop there. If you really mean business, you start thinking about how to get rid of precompilation during installation altogether. MoarVM bytecode (and by extension precomp files) is 100 % platform independent. When I install a Java application, I do not wait for any compiler. I copy some jar files and be done with it. The same could be true for Raku applications. And indeed, it is actually true, if you install said application from RPM packages like we do on our servers.

Compared to Java the only disadvantage we have is that precomp files are dependent on the exact Rakudo version. But then, we expect most people to use the monthly releases. We could have modules precompiled for these releases by some central build server from which zef could download and deploy the precomp files. We already do build those modules with the Blin runs that precede every release. What if we just didn't throw away the results?

[lizmat]

===> Testing [OK] for XML:ver<0.3.1>:authgithub:raku-community-modules [ long wait, here ... ] ===> Testing: LibXML:ver<0.6.10>:authcpan:WARRINGD

This is not an issue with long installation time. It's long time of running tests! What's not being reported here is the tests' output. I have occasionally wondered about this design decision in zef. But that may be just because I am used to seeing the output from Perl module installers. This is not something that Rakudo can change at all, since it doesn't even know that tests are being run. In fact it doesn't know the concept of "test" at all.

That's why I love my "run-tests" script:

https://raku-advent.blog/2020/12/13/day-13-helping-the-github-action-elves/

It will output everything, even a --ll-exception backtrace, if a test failed or an error was thrown. And otherwise, it is (almost) silent.

[vrurg]

@niner In general, I agree with your point. BTW, the fact that we still don't really parallelize pre-comp came as a surprise to me. I remember there was a lot of work done to allow it. Or was it something different?

Anyway, yes, there is a lot can be done to pre-cache the prepared bytecode. But there are always ways to break this. It'd be good to think beyond zef and installation in general. In particular, even the development process is affected because often change in a module deep down the dependency chain leads to long re-compile where the dev would have no idea of wether it's done yet or his code is stuck.

But who will receive the events? If - as was being suggested - the receiving end is supplied by the distribution that gets installed, then this is code that itself has to be (pre-)compiled first.

There are options here. But normally there is some single point of start, either a script or the root module. Both can do the reporting by using BEGIN:

BEGIN {
    PROCESS::<$REPORT> = Supplier.new;
    start react whenever $*REPORT {
        note $_;
    }
}

for ^10 {
    $*REPORT.emit: "step $_";
    sleep .1;
}
$*REPORT.emit: "done";
sleep .3;

So, basically, I don't see why not to add a couple of lines for event emitting? I'd try it myself, but first I'd like to eventually be done with the current smartmatch project; and second, I don't know enough about CU internals to make proper decisions as to what data could be provided to the user land, and what is better be kept internal.

BTW, this could even be helpful if one wants to collect some statistics about the pre-comp stage.

A script that precompiles all installed modules as (optional) part of Rakudo's installation process would again be something quite short and simple.

There is possible problem with this: a project might use a dedicated module repository, not known to the compiler upgrade script.

[patrickbkr]

There is quite a bit of low hanging fruit that could yield massive improvements. Right now for example every module gets compiled twice when installed with zef. Once for testing and then on installation. That's quite unnecessary! We have all the foundation we need to first install a module into a CompUnit::Repsoitory::Staging repository, use that for tests and if successful simply copy the resulting files from the Staging repository to the actual target.

ugexe has already (pre-dating this issue) started work on this in the use-cur-staging-workflow branch.

[Xliff]

@niner - Thank you for you very insightful post. I would love to see more work in this direction and would be willing to devote some time to it. It would be interesting to see how I can shrink install times for the first few unreleased modules I have in the can.

Speaking of cans, though. I still feel this is kicking one particular can down the road. For particularly long installs, we still should be sending something to the user. The issue of precompiling the code that does the output is well received. In those cases we should be putting out caveats to those on the distribution end to "Keep it Simple!".

For what its worth, if you feel the time is better spend putting our efforts into speeding up installation time, then I am all for it. I am willing to do that and then revisit this issue to still see if it remains relevant.

[hythm7]

@niner

We have all the foundation we need to first install a module into a CompUnit::Repsoitory::Staging repository, use that for tests and if successful simply copy the resulting files from the Staging repository to the actual target.

For distributions that make use of %?RESOURCES for example, when copying the precompiled files from CompUnit::Repository::Staging to the actual target, these precompiled files still have the %?RESOURCES absolute path pointing to the Staging repository making it fails to obtain resources. can this be worked around somehow or for such distribution one need to re-install again to the target repository?

[niner]

For distributions that make use of %?RESOURCES for example, when copying the precompiled files from CompUnit::Repository::Staging to the actual target, these precompiled files still have the %?RESOURCES absolute path pointing to the Staging repository making it fails to obtain resources. can this be worked around somehow or for such distribution one need to re-install again to the target repository?

That's already a solved issue and the whole reason why e.g. %?RESOURCES<libraries/foo> returns a Distribution::Resource object that's mimicking an IO::Path instead of getting an IO::Path directly. The Distribution::Resource object knows the name of the repository it was initialized from and will update its path at runtime. Since the staging repository is pretending to be the target (e.g. you have a staging repository for "site"), the resource will look up the path of the correct repository.

And NativeCall also knows how to handle Distribution::Resource so no absolute path will be contained in precompiled files.

This is not just theoretical but used in practice when we create RPM packages for Raku modules: https://build.opensuse.org/project/show/devel:languages:perl6

Packaging follows the exact process I outlined here. The distribution will be installed into a staging repository for "vendor" which on the build service worker is somewhere like /home/abuild/rpmbuild/BUILDROOT/usr/share/perl6/ vendor. All generated files are then put into the package and on installation they are just extracted onto the file system into /usr/share/perl6/vendor.

[Xliff]

Why is the testing process silent?

CPAN doesn't do this.

Is it possible to at least emit the current test file being run and its results?

For example, this here is the relevant out for a module I just installed:

PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/AE.t .................. skipped: AnyEvent is not installed.
t/AETk.t ................ skipped: AnyEvent and/or Tk is not installed.
t/ReadLine.t ............ ok     
t/release-pod-syntax.t .. skipped: these tests are for release candidate testing
t/Tk.t .................. skipped: Tk is not installed.
All tests successful.
Files=5, Tests=15,  0 wallclock secs ( 0.01 usr  0.00 sys +  0.12 cusr  0.01 csys =  0.14 CPU)
Result: PASS

This is exactly what I would like to see when installing a package. I don't necessarily need to know what actual test is running, but a single test file and an PASS/FAIL on each line would be sufficient.

[lizmat]

@Xliff please privmsg me

[hythm7]

That's already a solved issue and the whole reason why e.g. %?RESOURCES<libraries/foo> returns a Distribution::Resource object that's mimicking an IO::Path instead of getting an IO::Path directly. The Distribution::Resource object knows the name of the repository it was initialized from and will update its path at runtime. Since the staging repository is pretending to be the target (e.g. you have a staging repository for "site"), the resource will look up the path of the correct repository. And NativeCall also knows how to handle Distribution::Resource so no absolute path will be contained in precompiled files. This is not just theoretical but used in practice when we create RPM packages for Raku modules: https://build.opensuse.org/project/show/devel:languages:perl6 Packaging follows the exact process I outlined here. The distribution will be installed into a staging repository for "vendor" which on the build service worker is somewhere like /home/abuild/rpmbuild/BUILDROOT/usr/share/perl6/ vendor. All generated files are then put into the package and on installation they are just extracted onto the file system into /usr/share/perl6/vendor.

Most likely I'm missing something, because that is not what I'm experiencing. currently when I install a distribution using install-dist.raku which uses CompUnit::Repository::Staging the installed precomp files try to access the Staging repository %?RESOURCES not the target repo %?RESOURCES.

Anyway I don't want to hijack this issue so I created a new issue in rakudo

EDIT: The behavior I described in this comment was actually a bug in Linenoise distribution (as pper niner in the mentioned issue).

[ugexe]

This is exactly what I would like to see when installing a package. I don't necessarily need to know what actual test is running, but a single test file and an PASS/FAIL on each line would be sufficient.

so pass zef the --verbose or --debug flag depending on how much you want to see