Interactive usage

[simonbyrne]

It would be great to have a way to use MPI.jl in parallel through the REPL, either 1 terminal session per rank, or a shared terminal session which would run commands on all ranks.

[jkozdon]

This gets you part of the way there: https://gist.github.com/DaoWen/2e8817c62f920d6a47f1

Doesn't quite work for me, but maybe some sort of riff on it would?

[sloede]

@simonbyrne Do you know if there is any news on this topic? Since we started adding MPI support to Trixi.jl, the lack of MPI support in the REPL has been bugging me considerably. It seems to be that many of Julia's strengths in interactive development now become a burden, since even a small test problem that has a serial runtime of <100ms now takes two minutes on 2 ranks due to significant startup and compilation times, the latter of which is not amortized as there is no caching.

If you from CliMA are interested in this, I'd be happy to kick around a few ideas to see if we can come up with a potential solution that would at least smoothen the development process (and ignoring for now that this is also a problem in production runs...). But maybe someone is already working on this?

[lcw]

If you want interactive use, tmpi works for me. It requires OpenMPI and tmux. So I use my system MPI install with

julia --project=. -e 'ENV["JULIA_MPI_BINARY"]="system"; using Pkg; Pkg.add("MPI"); Pkg.build("MPI"; verbose=true)

and then just open a repl with tmpi 2 julia --project=.. You now have a tmux session with all of your ranks (2 in this case). You can type in all of the ranks' repls at once (the default) or maximize a pane ("C-b + z" by default) and only type on one rank's repl.

[sloede]

Thanks a lot @lcw, this is already a great help for me personally! However, looking further down the road, it does not solve the issue in general, as it has several considerable restrictions:

• It requires the installation of tmux, which might not be possible (cluster?) or available (Windows?) everywhere
• It cannot be used with the MPI.jl-delivered MPI but requires to install MPI yourself
• It relies on OpenMPI, which might not be available everywhere (cluster, Windows)

I really feel like that there should be a Julian solution to this, especially when considering that Julia is often and repeatedly promoted as an "HPC" language... I wonder which features the tmpi script relies on that only OpenMPI provides, if there is a way around this, and how hard it would be to replicate something similar in Julia itself. Personally, I think something like being able to send commands to all ranks and collect unordered terminal output (just like a normal execution of mpiexec would do) would be sufficient for the majority of use cases.

[jkozdon]

seems to be that many of Julia's strengths in interactive development now become a burden, since even a small test problem that has a serial runtime of <100ms now takes two minutes on 2 ranks due to significant startup and compilation times, the latter of which is not amortized as there is no caching

In climate machine we have been using the PackageCompiler.jl to get around some of this.

You can see how we that here and in the docs

[sloede]

Thanks for the hint! I've looked at PackageCompiler.jl before but discarded it (for now), since we still have mostly "dev" runs and only very few "production" runs. If I understand correctly, PackageCompiler.jl will only really help if a lot of your code is matured and not changed/touched anymore during your dev cycle.

While I am on the topic: kudos to creating a Julia package as exceptional as ClimateMachine.jl! As far as I can tell, it's the biggest (only) game in town when it comes to "true" HPC CFD with Julia. Especially since starting to integrate MPI into Trixi.jl, I am happy that there is a more mature project that has already tackled many of the typical MPI issues such that most of the basic stuff just works...

Still, I am wondering if there isn't something that can be done for running the REPL natively with MPI. What I am thinking about is something like the tmpi approach but - for now - with even less functionality. I think what would cover 90% of the dev cycles with MPI would be something like this:

• Allow starting Julia with an additional flag that indicates parallel execution with the REPL, e.g., --parallel-interactive
• This flag should make Julia start a REPL on all ranks, in a true interactive mode on the MPI root and in a "headless" mode on all other ranks
• Obviously, all ranks need to be able to talk to root, so some sort of communication channel must be established (files? sockets? networking?)
• From now on, everything the users executes on his "root session" is sent to all non-root sessions and executed there as well
• Output on non-root sessions could either be gathered back on root and printed as received (hardcore MPI style ;-)) or prefixed by the rank (maybe nicer)

I don't know, maybe this is even possible without touching the Julia executable but somehow hooking into the REPL like OhMyREPL.jl? Please note that these are really only some initial thoughts (and they might not even work for some reason), but I'd be motivated to further investigate if there is interest from other parties such as CliMA. If not, that's ok as well of course, but in that case I think I'll be too much out of my depth to pursue this much further...

[simonbyrne]

Thanks for the hint! I've looked at PackageCompiler.jl before but discarded it (for now), since we still have mostly "dev" runs and only very few "production" runs. If I understand correctly, PackageCompiler.jl will only really help if a lot of your code is matured and not changed/touched anymore during your dev cycle.

Note that by default our script only compiles the dependencies (which change rarely), and not the ClimateMachine.jl package itself, which helps considerably with development.

I would definitely be interested in something along these lines: please let me know how you go. You might find https://github.com/MasonProtter/ReplMaker.jl useful for this.

[sloede]

Thanks for the hint! I've looked at PackageCompiler.jl before but discarded it (for now), since we still have mostly "dev" runs and only very few "production" runs. If I understand correctly, PackageCompiler.jl will only really help if a lot of your code is matured and not changed/touched anymore during your dev cycle.

Note that by default our script only compiles the dependencies (which change rarely), and not the ClimateMachine.jl package itself, which helps considerably with development.

Good to know! This definitely makes it more interesting for development - I'll have to check if and how much it helps with our (very few) dependencies.

I would definitely be interested in something along these lines: please let me know how you go. You might find https://github.com/MasonProtter/ReplMaker.jl useful for this.

Also good to know. I'll have a look at it over the next few days and ping you on it (either here or on Slack).

[lcw]

Michael Schlottke-Lakemper [email protected] writes:

Personally, I think something like being able to send commands to all ranks and collect unordered terminal output (just like a normal execution of mpiexec would do) would be sufficient for the majority of use cases.

Sounds cool! I think it would be a great addition to the ecosystem.

[simonbyrne]

One challenge is that mpiexec doesn't play nicely with interactive sessions (something to do with TTY/PTYs). It seems that tmpi gets around this by starting a tmux session outside mpiexec, and then uses mpiexec to launch extra windows.

[tomchor]

Thanks for the suggestion, @lcw, tmpi works like a charm on my laptop!

More generally, I'd like to voice my support for this feature and see if there are any news regarding this issue. While tmpi works well on my laptop, I haven't been able to use it on my server yet and it'd be nice to have a reliable julian solution for this. This would be an incredible advantage of using Julia+MPI! I found it super useful for debugging and exploration.

[simonbyrne]

Having thought about this a bit more, the only long-term solution I see is having a mechanism by where you run the REPL in a different (non-MPI) process, and it communicates with the MPI processes.

See also this Discourse thread

[JobJob]

In OpenMPI I just discovered (via mpirun --help output) this -xterm option which opens xterm windows for each specified rank, i.e. mpirun -xterm 0,1 -np 2 julia --project opens two functioning Julia REPLs for interactive MPI use.