Draft - Issue 989 big data support scaling vuu how to abstract vuu interfaces to allow different implementations of key classes by adding features on construction

[chrisjstevo]

First example of a plugin structure for project.

[netlify[bot]]

✅ Deploy Preview for papaya-valkyrie-395400 canceled.

Name	Link
🔨 Latest commit	6a550ff29eb13fd1d2842cf5e43af6a5252639d6
🔍 Latest deploy log	https://app.netlify.com/sites/papaya-valkyrie-395400/deploys/65ae751267b00e0008938968

[rumakt]

I like the 'feature' and 'plugin' idea. This is very good 1st step towards making vuu a composable project

[naleeha]

I think this is good for understanding what it could potentially look like with the plugin concept & bootstraping relevant features etc. I'm not sure about the feature concepts and having factory for each at that level. I wonder we can make the construction of main concepts a bit simpler by making it less flexible - e.g. you always create a VirtualViewPort from a VirtualDataTable which knows to take in VirtualSort function etc Ensure we resolve the right implementation at creation using dependency injection

We may want to re-use the implementation in multiple plugin as well. E.g. VirtualDataTable for both Ignite and Hadoop. If thats the case, would it be better to tie the table creation by table type rather than plugin? For registering and resolving new types when you add a plugin, maybe we can have a simple DI Container, similar to what you have for features in this registry.register[DataTableFactory](VirtualDataTableFactory) registry.resolve[DataTableFactory](type = VirtualDataTable.type)

.addTable(
    TableDef(
      name = BasketTradingTable,
      type = VirtualDataTable,
      columns = Columns.fromNames(InstanceId.string(), BT.BasketId.string())
    ),
    (table, registry) => new BasketTradingProvider(table, registry.Resolve[DataSource](type = IgniteDataSource)),
  )

I think there is still a lot of context im not aware of & normally find things i didnt think of at implementation time. Having a second implementation would help a lot in visualizing the right solution. Would it make sense as next step to have a spike of implementing the Virtual/BigData vuu solution using the ignite & once we have the second example, try and come up with the exact interfaces and how we can wire them up correctly?

[rumakt]

Good step forward, lets keep the ball rolling.

[linux-foundation-easycla[bot]]

• :white_check_mark: login: rumakt / name: Tomasz Rumak (514a5310ea47df026416b1d2d89dad099c6c1a8d, 4e03230e8ee400457c85cf2b61ba77d80a825ad5, 9226e3304e3a51dee5c0dac73e16b10c6d7f2343, 61c709089153a37d192faff1125e17e8366073d2, f9e8f9500350a7575e02152d4caceb5df4a78840, fed77e47e96a497f9d26fdf0c0c70a124319c011, c4d2090596113627cfff4bab60356a8673920dd6)
• :white_check_mark: login: chrisjstevo (70e79addb7a93dd6e9402c54f8b1fd800586f23d, f04c0776abf091d5458d4abf4485fa363a6c5b86, cef0089677c852a9c1e93f799b0fd48f82f37b38, b09e9c683c2f270108324fbf43d04a04b200cf26, e8c5a7f42510c44c816a0715e41ce5337036d7a7, 1e9ca7b174b1ab7f33696e46e72609e96880a0eb, f6480a4db0e7619377fcc842f278b469ce473c8c, 80c2f1adc4ddee1d6802cbec6a33fda5b4b1dd5a, b037baaf6d0288f62876141882bd1ed797e54061, 868812596d99a73fab47b515b66e8344e83cfd18, 58a261d374e25f1b77a32e1e0ecf97af1a4bcc7b, 18bf20f0d0f5c9da7fceb732afb1e1b0f9bf3567, 857a681027922e62d2d50a080dc8df533e78258c, 913510b9792f72f7e905290476ded57a825a7726, 3f31f8307f036c2e1e1014047791cb94d7e8f346, a1bed3df2f2ba99f93c16c59332a6803ea452d30, 6a550ff29eb13fd1d2842cf5e43af6a5252639d6)
• :x: The email address for the commit (90b6a5a843b7c38b4e894bb6ab80d786bf17faef) is not linked to the GitHub account, preventing the EasyCLA check. Consult this Help Article and GitHub Help to resolve. (To view the commit's email address, add .patch at the end of this PR page's URL.) For further assistance with EasyCLA, please submit a support request ticket.