feldera icon indicating copy to clipboard operation
feldera copied to clipboard

Compiler server isolation improvements

Open lalithsuresh opened this issue 4 months ago • 3 comments

The compiler server currently compiles one task at a time (from SQL to rust). This isn't great when there are multiple users actively trying to compile programs at the same time.

  • The SQL compiler can always be invoked even if there are active Rust compilations in progress. This is key to give users quick feedback about their programs.

  • We should also enable multiple rust programs to be compiled at the same time if it makes sense to do so. Especially when only compiling the program crate (and not dependencies). There are challenges in doing so with respect to handling errors (e.g., one program having a rust error would block all other programs from being compiled).

For the 2nd point, we could consider moving to a one-workspace-per-program model and using sccache to reuse the work of compiling dependencies across programs.

lalithsuresh avatar Mar 01 '24 22:03 lalithsuresh