Andrey Novitskiy
Andrey Novitskiy
Scaling: We want to see how well Volga scales in 2 modes 1. Streaming Workers scaling - check network tx/rx, memory usage, backpressure etc. 2. On-Demand scaling - measure latency,...
Offline/Online storage connectors: Scylla, Redis, MySQL, s3, data lakes
Master task to keep track of Streaming Engine Enhancements to feature parity with existing engines (e.g. Flink) to a prod-ready state. Streaming engine is still in dev state and misses...
KeyByOperator can be rustified with default dict-based key function
WindowOperator can be rustified if we do not pass custom Python-based aggregate functions
If we do not pass custom join functions to JoinOperator, it can be rewritten in Rust
Rewrite DataReader/DataWriter and all network IO related stuff in Rust with PyO3
Current FeatureRepository is pretty simple - inited as a global var, no persistance, etc. We need to properly design remote feature metadata storage, look up, authoring versioning, etc.
We need to store the state of the job and job metadata in a persistent manner. This should be local (in-memory) and remote (SQLAlchemy?)
Create a webserver/webapp to manage multiple jobs/ray clusters/tenants, etc., with proper API. This may have intersections with job metadata storage task.