seamless

seamless copied to clipboard

Logs, progress and preliminary results from assistants

3

Now that the communion protocol has been ripped, there is no way to get logs, progress or preliminary results from remote transformations. This needs to be tackled in four stages:...

sjdv1982

Probably, with buffer_info, the fingertipping and recomputation can be avoided in many cases

sjdv1982

Annotate transformations that invoke other transformers as "nesting" in the transformation metadata in the Seamless database. The main use case is to identify all buffers and database entries that are...

sjdv1982

Right now, the only resource that is really internal to Seamless is "ncores" (see "multi-core-transformations" test). "memory" should be added as well: - Use bytes2human to parse the user specification...

sjdv1982

With the recent update in scratch and fingertipping, it must be made sure that local transformations are indeed still executed locally. A low-level test must be added. In addition, it...

sjdv1982

```python ctx.tf = lambda a, b: a + b ctx.tf.a = 4 ``` will give a CacheMissError for "4" This may or may not be a bug. This can be...

sjdv1982

This can be disastrous if both processes are compiling the same C code, or building the same modules. Solution: look up tempfile.gettempdir() in the Seamless code and change it.

sjdv1982

Support name-of-conda-environment in /bin/seamless

1

To indicate the name of the conda environment in which the command is to be executed. Must be known to the assistant/deployment machine(s). Most assistants should be able to deal...

sjdv1982

Dealing with huge buffers inside transformers

2

"huge" as being too big to fit comfortably in memory. They are already well supported by bash transformers, because of file zones. For Python, there is currently no alternative for...

sjdv1982

Add config for buffer write server/dir for scratch buffers

1

Something like `SEAMLESS_SCRATCH_DIR=/scratch` . On the long term, perhaps multiple scratches controlled by `scratch_level`: pyramid of RAMdisk, SSD, rotating disk, etc.

sjdv1982