Support for LSF and OpenLava Array Jobs

[ssadedin]

See https://github.com/ssadedin/bpipe/pull/118 for discussion

[ssadedin]

I have made an initial attempt at array support in the lsf_array branch:

https://github.com/ssadedin/bpipe/tree/lsf_array

This has been based on OpenLava, via TeraProc's service (which is awesome). It would be great if anybody who has access to a "real" LSF installation could test this and let me know whether it works or not.

Implementation Note

The implementation uses the template based approach from our recent SGE work. This means that the job command is not launched directly, rather, it is written to a separate file which is invoked by a wrapper script. In the wrapper script, the SGE implementation creates an "exit file" to signal that the job is complete. The new LSF implementation does the same thing, but it creates n exit files, one for each component of an n-sized array job. This is done using the $LSB_JOBINDEX environment variable.

One problem with this approach is that I am not sure what will happen with LSF automatic assignment of inputs and output via %I "magic" variables. I think they will not work, because LSF does not actually see the command itself anymore, it only sees the wrapper script. I am curious if this is a problem for LSF users in practice or not.