DSL / Config for describing the Flow

[gedw99]

We could describe the flow using a file dsl / idl. A type of config fike.

Mythen the runner loads this config file.

It’s almost like how Benthos works though .

It would be useful to write a Comparison with Benthos in the readme too because there is Carey over in sone ways

[rupurt]

I feel like streaming SQL syntax like Flink, Materialize, RisingWave etc... would be a better fit as that is super common in the streaming space and already has pre-established syntax & constructs.

[gedw99]

That would be fantastic. With CDC pattern for data source change events

[gedw99]

Is there any start on this in the code ? Would be interested in helping.

checkout appcash pranadb on GitHub btw. Has sone similarities to your ideas .. gotta run

[gedw99]

Also if you have links to explain sql like streaming it would be cool.

materialize is a rust project that sort of does this against postresql db

[gedw99]

https://github.com/reugn/go-streams/blob/3771cd778875ede809125c8053be06406ad6fd95/examples/nats/main.go#L33

Would be good to see how that can be a SQL like thing .

the lever and parse could be like steam pipe project maybe that uses HCL