Smart modules, data evolution and versioning

[pragmatrix]

I'm curious about if fluvio plans for or already has support for transparently handling the evolution of smart modules and the data processed by them.

From what I can see from the source code so far, is that the responsibility for type compatibility is completely left to the user of the API: record data must be convertible to bytes and type information is erased. So when - for example - an older smart module processes a record which would fail to deserialize (from bytes) into the type expected, it would just fail? Are there any consistency guarantees that smart modules only receive data from the same code basis?

[sehz]

Hi, @pragmatrix. That's a very good question.

Data evolution or Schema management is on the road map but we haven't started design work yet. But I agree, that we need to have an API to handle data evolution and that should be the simplest possible. This is the initial sketch

        /// simple smart module filter using String type
        /// if data is not string type, this will return invalid data type
         #[smartmodule(filter,type = String)]
        pub fn filter(record: &str) -> Result<bool> {
           Ok(record('a'))
        }
        
        /// using data converter
         #[smartmodule(filter,name = "uppercase", type = String)]
        pub fn filter(record: &str) -> Result<bool> {
           Ok(record('a'))
        }
        
         #[smartmodule(converter,name = "uppercase", type = String)]
       pub fn string_converter(record: &Record,version: DataVersion) -> Result<String> {
        std::str::from_utf8(record.value.as_ref())?;
      }

And we get make it fancy as possible.

[github-actions[bot]]

Stale issue message