crs-tools
crsd, new worker concept
Issue migrated from trac ticket # 128
component: rpc | priority: major
2016-01-17 22:37:17: @jjeising created the issue
There will be a new REST/oAuth (bearer token) based API. Both APIs should exist concurrently, old worker assignment will still work (via traditionally assignment), but won't support new features.
Worker groups and workers will stay. A worker get's multiple jobs and configuration for allowed number of concurrent jobs (this also enables single worker pause/no new job assignment).
Jobs have additional metadata for the worker to ticket assignment, they will hold script output, timing and more. A job is related to an encoding profile (or an encoding block, later). This enables history of jobs for a given profile and similar enhancements.
crsd will be written in Go and works via the new API. Development of crsd will be started by providing jobcontrol (spawning and monitoring commands, gathering outputs) and API communication.
crsd should be a good citizen on the command line and work on all three plattforms. It may be helpful to provide to provide a menu like strukture to peek into process output and show current status (termbox and termui/gocui may help there).
A job in crsd won't be killed if the Tracker cannot be reached but the completion step (e.g. copying an encoded file) should be hold off until the Tracker is reachable again (see lock in #65).
2016-01-17 22:55:22: @jjeising commented
Additional development resources:
- http://www.cons.org/cracauer/sigint.html
- https://github.com/bradfitz/runsit (has a lot of process handling in Go, just ignore the forking part)
2016-09-05 00:32:41: @jjeising commented
Jobs are typed as service jobs (encoding, preparing…) and system jobs. Each type gets assigned a number of slot, system jobs are unlimited. Encoding Profiles may have a slot width. crsd announces slot capacity and is assigned jobs accordingly, slot-width may depend on CPU core count (e.g. for encoding). Slots can depend on each other (system job shutdown, after all encodings are finished).