Documentation

[sjdv1982]

Status of the documentation web page

• Need to rebalance between "Seamless explained" and feature documentation for several features. The former should be more conceptual, the latter more practical, but this is difficult to separate for these features.
  • Mounting to the file system
  • HTTP shareserver
  • Structured cells (especially authority and the joining process). Schemas must be discussed in "validation"
  • Deep cells
• Need to rebalance between "Seamless explained" and feature documentation for several more features that are conceptual, but not as advanced (accessible for beginners too, and for sysadmins). To be balanced with seamless-tools docs too.
  • Visualization
  • Validation
  • Job management
  • Deployment
• For the rest, the missing / incomplete sections have now been highlighted
• Need to finalize "Seamless explained" and beginner's guide.

Missing sections

TODO fill in here

Specific todo

• Basic example notebook / README.md . Add "edit cell over HTTP" section. Simple index.html for cell ctx.a with two buttons (get value, set value) and two fields. Move section 3 and 4 to the end (de-emphasize)

• write a big data example, e.g. a hhblits search where the database path is a DeepFolder checksum.

General todo

• Make some example video, do some publicity online

• Expand man pages for seamless-cli

• Low-level API documentation (long term)

[sjdv1982]

TO DOCUMENT: To get Silk in your transformer:

• Declare the inputpin celltype as "silk"
• Declare an extra inputpin _SCHEMA If this extra inputpin is not there, an empty schema is applied. Pins ending with _SCHEMA are never provided as arguments to the transform() function (or lambda)

[sjdv1982]

Document stdlib and status graphs:

• Describe provenance for the stdlib help files (i.e. store the Git filename), so that people know where to push changes.
• Create help for the status graphs and add provenance as well.

[sjdv1982]

After feedback from @khinsen: add diagram to "Seamless explained"

[sjdv1982]

At some point, also describe how to install a development environment (seamless-devel Docker image)

[sjdv1982]

Add a section to describe how to install conda packages in a running container: if you need to be root, use seamless-shell-existing.

[sjdv1982]

Need a section to describe imperative mode. Add a section in the beginner's guide, pointing this as a good place to start. Also update the "Seamless explained" document when it is there.

[sjdv1982]

Update from 2023 overhaul

Features and fixes that were completed, but need to be documented

Things TODO regarding integration:

Stage -1: implementing proper buffer handling, fingertip and scratch (DONE)

- Rip the entire communion protocol, too much trouble with websockets,
  threads and event loops and imperative mode.
  Make a seamless-micro-assistant that needs a database and a buffer read/write server.
  Use aiohttp requests to contact the assistant.
  DONE
  
  Check a nested/fingertip issue where imperative-delegated works if you restart the micro-assistant.py
  (after cleaning up database and buffer directory) but not otherwise. 
  UPDATE: happens even for simple functions in imperative-database if you delete buffers but not database
  UPDATE2: the simple problem has been fixed, but it still hangs sometimes for nested... restart micro-assistant regularly?
  Should happen so often since new jobs are normally sent to a different micro-assistant
  UPDATE3: all fixed now.
  DONE

- Finish the scratch Seamless issue: DONE
- Add seamless-fingertip tool (default delegation level 1, can be increased): DONE


Stage 0: /bin/seamless (DONE)

See also [seamless/bin/README], which will be documentation also.

- When bin/seamless operates on a file "file.ext", be able to load its checksum from "file.ext.CHECKSUM" (if both exist, use checksum). DONE

- Create "file.ext.FUTURE" so that "file.ext" (or file.ext.CHECKSUM, or file.ext.FUTURE) can be referred to in a follow-up job. DONE

- Download of job/transformer results can be enforced or prevented. Otherwise, a heuristic based on result size (and possibly, connection speed). Manual command "seamless-download file.ext" that inspects file.ext.CHECKSUM and downloads it. DONE

- Tool "seamless-upload" for files and directories. Optional "move" flag that replaces "file". DONE


Stage 1:

- mini-assistant: have an internal lock system just like transformation.py
(in fact, re-use it). Acquire locks before launching run-transformation, and release them afterwards. TODO
   
- ***Dogma: env (recipe) != Transformation.__env__ (schema) !!!***
  __env__ (and __meta__, __compilers__, etc... generally known as "dunder") give hints to make
  the transformation work. They do *not* modify the result value. TO DOCUMENT.
  
- In addition, implement "contest" API for mini-assistant (tests/highlevel/contest.py can't work without). DONE

- Make sure that buffer_cache eviction works correctly in the light of buffer_remote known buffers, is_readable, etc. DONE.

- TO DOCUMENT: for now, deep buffers do not persistently incref their subchecksums,
i.e. you can't "fill up" a deep cell/deep folder from Seamless and expect the content
to be written to a database. Might need a cell property and/or manual API for this,
in particular for transformers that return a deep cell/folder.

[sjdv1982]

Also document ContextPool and TransformationPool