forall with dynamic policy selection

[artv3]

Building on user feedback a first pass at introducing run-time policy selection to forall.

See example run-time-forall.cpp

Developers can provide any number of execution policies and then select at run time how to dispatch their code.

Example usage:

./run-time-forall pol_0
./run-time-forall pol_1
./run-time-forall pol_2

[artv3]

tag: @adayton1

[artv3]

TODO: strongly type policy selection.

[artv3]

TODO:
//User should provide a pair of policies and indices,
//camp tuple, generate if run-time == ....

[artv3]

TODO: Add coverage for versions of forall that take a resource.

[artv3]

@ajkunen @davidbeckingsale I made a clean up pass to this PR. I wasn't sure on how integrate creating a list of both policies and indices, but hopefully what I have here does that job.

Any feedback would be great, ping: @tomstitt @rcarson3 @adayton1

[artv3]

Some of the CI are failing due to time limits and connectivity issues.

[artv3]

Did anyone else want to take a look or are we good to merge?

[MrBurmark]

I'll take a look tomorrow, thanks.

[rhornung67]

I'll take a look tomorrow, thanks.

Ditto, for me.

[rhornung67]

@artv3 👍 for adding tests!

[artv3]

@artv3 👍 for adding tests!

Thanks, not ready yet though just wanted to commit initial progress.

[artv3]

I just pushed up some basic test for dynamic forall. If they pass this PR should be good to go. The testing may be a bit redundant because of the run-time switching capability.

[rhornung67]

@artv3 is there a (good) way to prevent the redundant tests? We don't want to have to deal with more timeouts than we already experience.

[artv3]

@artv3 is there a (good) way to prevent the redundant tests? We don't want to have to deal with more timeouts than we already experience.

I'm thinking if we have a way to detect what test the executable is supposed to run, for example openmp vs sequential could try to only only invoke that policy number. Cmake would have to pass in extra info into the executable to be more clear.

If there are concerns about timing out, I can shorted the policy list to just 1 host and 1 device and we can differentiate based the working resource. This would be simpler.

[rhornung67]

Let's let it go as is for now. We can address it later if it becomes a problem.

[artv3]

I hit the following errors: srun: Job 10457853 step creation still disabled, retrying (Requested nodes are busy)

Will try again in the evening.

[artv3]

@rhornung67 are we ready to merge this PR?