require ++ppn with ++mpiexec-no-n for smp charmrun

[jcphill]

Let's suppose a user types "charmrun ++mpiexec-no-n ++remote-shell srun +p12 +ppn 6 namd2 ..." and then get this error:

Charmrun does not recognize the flag '+ppn'.
Charm++'s flags need to be placed *after* the program name.

So they change to "charmrun ++mpiexec-no-n ++remote-shell srun +p12 namd2 +ppn 6 ..." (or make this mistake in the first place), and try again to find that they get a hang during startup, and adding ++verbose only gives something like this:

Charmrun> scalable start enabled. 
Charmrun> charmrun started...
Charmrun> added host "127.0.0.1", IP:127.0.0.1
Charmrun> Charmrun = 10.0.0.8, port = 42037
Charmrun> Sending "$CmiMyNode 10.0.0.8 42037 24046 0" to client 0.
Charmrun> find the node program "/shared/home/jim/NAMD_2.14b1_Linux-x86_64-netlrts-smp-CUDA/namd2" at "/shared/home/jim" for 0.
Charmrun> Starting srun ./charmrun.220654 
Charmrun> mpiexec started
Charmrun> node programs all started
Charmrun> Waiting for 0-th client to connect.
Charmrun remote shell(127.0.0.1.0)> remote responding...
Charmrun remote shell(127.0.0.1.0)> remote responding...
Charmrun remote shell(127.0.0.1.0)> starting node-program...
Charmrun remote shell(127.0.0.1.0)> starting node-program...
Charmrun> Waiting for 1-th client to connect.

If they use ++mpiexec instead of ++mpiexec-no-n the behaviour is a little more reasonable:

Charmrun> scalable start enabled. 
Charmrun> charmrun started...
Charmrun> added host "127.0.0.1", IP:127.0.0.1
Charmrun> Charmrun = 10.0.0.8, port = 36637
Charmrun> Sending "$CmiMyNode 10.0.0.8 36637 24185 0" to client 0.
Charmrun> find the node program "/shared/home/jim/NAMD_2.14b1_Linux-x86_64-netlrts-smp-CUDA/namd2" at "/shared/home/jim" for 0.
Charmrun> Starting srun ./charmrun.220793 
Charmrun> mpiexec started
Charmrun> node programs all started
Charmrun> Waiting for 0-th client to connect.
srun: error: Unable to allocate resources: Requested node configuration is not available
Charmrun> Timeout waiting for node-program to connect

Also, the timeout actually works when the 0-th client fails to connect, while charmrun seems willing to wait forever for the 1-th client to connect.

Basically, if the user says "trust me, mpiexec/ibrun/srun/etc. will do the right thing" but they use +ppn instead of ++ppn, charmrun will wait forever for processes that will never launch because it is expecting one process per pe and the number of processes is derived via +p/++ppn rather than being specified directly. The easiest way to avoid exposing the user to this hang is to require an explicit ++ppn when ++mpiexec-no-n is used with an smp layer, and maybe to also have charmrun print the number of pes, nodes, ppn, etc. before attempting to launch them.

[jcphill]

This is not an issue if ++n is used instead of +p, so the rule should be to require ++ppn whenever ++mpiexec-no-n is used with +p on an smp layer.