tensorlakeai
Append Function Executor startup stdout, stderr to its tasks' stdout, stderr
Append Function Executor startup stdout, stderr to its tasks' stdout, stderr. This is needed to make sure that customers can see what happened during FE creation, especially if it failed.
Alternatively Executor can upload FE stdout, stderr into separate blobs and report their data payloads in FE state (Executor state reports). Then Server UI can provide access to it cause iirc UI already made FEs first class citizens. I think this is a cleaner implementation. We can do any post aggregation/processing of task and FE stdouts, stderrs later after they are saved.
We'll align on what path to follow.
I really like the idea of uploading FE stdout/stderr/logger info somewhere that's not Indexify -- I think we should take traffic off our control plane anytime we can.
I tried to add a quickfix: add FE function initialization error into outputs of all its tasks but it turned out that Server unassignes each task of a terminated FE immediately after Server sees that the FE is terminated. So even though Executor reports all task results with FE function intiialization error messages in stderr Server starts ignoring them.
From Slack:
Eugene
Also fyi I tried today to implement uploading of customer FE startup error logs (aka exceptions raised during Function constuctor runs) into stderrs of tasks assigned to the failed to startup FE. But it failed because even though Executor uploads the logs as task outputs Server currently marks all tasks as failed if FE terminated with customer error. It's not easy to fix this because in this case Server needs to wait until all the tasks assigned to the terminated FE are reported by executor (with their stderrs) and only then Server can delete the FE from desired state. This is very different to the current logic and makes things quite more complicated on the Server. So I left this to the time when we are ready to address function log streaming and etc more properly.
The downside of this is that with dynamic scheduling we're not showing function constructor (init) errors in task stderr anymore, we only log it in Executor logs. So only our operators can see them.
Diptanu Gon Choudhury
7:05 PM
@Eugene
we could extend the server data model for FEs to have a diagnostic payload and a GRPC api to upload the blob store path no?
Eugene
[7:08 PM](https://tensorlakeai.slack.com/archives/C08MG9PHGJE/p1749492537009389)
Yes this will work. There're two things we can do:
Upload the log now the way you proposed and handle it on Server. The downside is that this is a short to mid term solution until we figure out and implement proper function log streaming and related things.
Leave it as is until we really really have to implement this or until we implement proper log streaming.
It'd be 1 day of work. I don't mind investing into this. It's the last feature regression I'm aware about with dynamic scheduling.
I think I can implement it after we setup the cluster and start getting feedback from it. Just to not delete it more for the feature which is not important right now.
I also was thinking as part of this issue to implement properly capturing of stdout, stderr on function init and give them to customers but I'm worried now that if we temoprarily override stdout, stderr file descriptors in FE process and libraries loaded by customer functions create loggers that log into stdout, stderr then it'd result in suppressing all their outputs after function initialization. We need to revise our output capturing solution I think before trying that.