Storage-MQ listen to DatasourceImportSuccess Event

Open georg-schwarz opened this issue 4 years ago • 2 comments

Datsource imports should be persistent without pipeline execution.

Apr 22 '21 12:04 georg-schwarz

@georg-schwarz I am not sure if I understand this correctly, but isn't the data already saved in the adapter? https://github.com/jvalue/ods/blob/06625822af149a6589c1123e8c7219818d20ede1/adapter/src/main/java/org/jvalue/ods/adapterservice/datasource/DatasourceManager.java#L102-L121

Nov 11 '21 08:11 f3l1x98

Yes it is, but the adapter does not offer query functionality on the data. What if the user does not need a pipeline?

Right now the Storage service is not there yet, but the idea is everything in the storage-db will be queriable via GraphQL or a RESTful HTTP API.

If we want that, there are two options: (1) Create a default pipeline for every datasource, so that it is persisted in the storage db (2) The storage-mq also supports storing data from the datasource (adapter) service

This ticket describes the second option. But I guess is not high priority right now and should be discussed, if we really want this to be implemented.

Nov 11 '21 08:11 georg-schwarz