titpetric
Capture stderr/stdout output separately
I may want to use taskfiles as an ETL pipeline API; it should be local filesystem aware, so it knows where to write out outputs if it's a "local" processing job (e.g. data extraction with parsing, jq, yq), however in some cases it makes sense to collect only the output directly (e.g. pipe workflow). Using CombinedOutput doesn't give us access to the stdout separately;
May want to reuse task-ui for this but adjust to capture the outputs separately, so i can distribute a job runner similar to a github matrix job, but sort of in a structured ETL framework.
Filing this in the idea pile, likely going to skip the distributed nature of some tasks :dagger: ; but i'll need to fan out pretty soon as a single threaded job runner won't work for the use case.
Ahh interesting
I have used Process-compose ( https://github.com/F1bonacc1/process-compose ) and task. They both use the exact same way to describe variables.
EX: https://github.com/F1bonacc1/process-compose/blob/main/process-compose.yaml#L123C9-L123C13
{{.VAR_NAME}}
It's not really like ETL use cases. PC is for running many binaries at the same time, like docker compose, but without docker. SO I use Task to Run PC, and also PC to run tasks which run binaries. You can use it for both.
For ETL I use benthos with Task files... Just saying because Benthos is so designed for this.
https://github.com/mihaitodor/benthos-playground shows it off.
Its got a data mapping languages called bloblang, and a WASM based editor.
I suspect there might be a way to use Task with benthos in a more tight integration. I still am playing around..
I mean technically could log the .Output at the end, but as it is task-ui doesn't have a "storage" function.
Could do --audit-log, or expand this into a storage driver to query history (sqlite, etc). It could support more api driven etl jobs, if someone would use task-ui as a dispatch center for automation or multi-user workloads. Database storage feels more natural, so some kind of metadata can be sidecarred together with the programs output (if capturing is enabled).
On Sun, May 4, 2025, 14:46 Joe @.***> wrote:
joeblew999 left a comment (titpetric/task-ui#6) https://github.com/titpetric/task-ui/issues/6#issuecomment-2849199997
Ahh interesting
I have used Process-compose ( https://github.com/F1bonacc1/process-compose ) and task. They both use the exact same way to describe variables.
{{.VAR_NAME}}
It's not really like ETL use cases. PC is for running many binaries at the same time, like docker compose, but without docker. SO I use Task to Run PC, and also PC to run tasks which run binaries. You can use it for both.
For ETL I use benthos with Task files... Just saying because Benthos is so designed for this.
https://github.com/mihaitodor/benthos-playground shows it off.
Its got a data mapping languages called bloblang, and a WASM based editor.
— Reply to this email directly, view it on GitHub https://github.com/titpetric/task-ui/issues/6#issuecomment-2849199997, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABY7EGDPK3VSC4EGOO2VLT24YD2VAVCNFSM6AAAAABXVZUDKCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNBZGE4TSOJZG4 . You are receiving this because you authored the thread.Message ID: @.***>
you could do that.
but benthos is designed to do this. you can use task with benthos, so why not ?
Because it's first party value. If I can use it for additional purposes like audit trail, execution history, performance analysis, and if it gives me that value out of the box, then I have less of a need to pull in something else into the fold. I sometimes have need to distribute jobs across a fleet of servers, and a POST to one instance becomes a POST to many, and would want to collect logs and errors across the board, meaning a pgsql/mysql instance can be the sink. I don't necessarily need to collect this data, but it becomes telemetry. Using the stdout for other purposes like additional job dispatch is possible, who's to judge, logs have saved me in some data corruption scenarios, scripting a data restore from that.
On Mon, May 5, 2025, 08:33 Joe @.***> wrote:
joeblew999 left a comment (titpetric/task-ui#6) https://github.com/titpetric/task-ui/issues/6#issuecomment-2850034900
you could do that.
but benthos is designed to do this. you can use task with benthos, so why not ?
— Reply to this email directly, view it on GitHub https://github.com/titpetric/task-ui/issues/6#issuecomment-2850034900, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABY7EFSPXGPQMG3E64EWBT244A2ZAVCNFSM6AAAAABXVZUDKCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNJQGAZTIOJQGA . You are receiving this because you authored the thread.Message ID: @.***>
Totally understand you pov .
Let’s just decide if we want to add tab completions ?
Filed a separate issue on stderr/stdout capturing upstreams. It seems like a limitation of ttyrec. ttyplay will output the history with colors and combined stdout/stderr. Maybe this needs just a lightweight recorder with .Output and not ttyrec. Closing this.