Multi Tenancy

[cedrics]

We are considering using turnilo for a client facing data exploration tool. For that we would need to add a filter clause based on one of the dimensions for each request sent out to druid. I haven't researched yet what would be the best way to add this to turnilo, but I think sth. like a callback that allows you to amend a query based on request data (for example, based on the subdomain) would suffice for our use case.

Have you thought about sth. similar already and I haven't seen is?If not, is this sth. you would be willing to add to turnilo if we provide the implementation?

[adrianmroz]

Doesn’t subsetFormula solve your problem? So each client would get different configuration that has access to subset of Druid data?

[cedrics]

@adrianmroz I didn't know about that option and it looks exactly what we want, but as far as I can tell it is a hard coded expression and it is not possible to change it based on request info. Correct me if I am wrong.

The specific implementation I would have in mind would be sth. like customer-123.turnilo.remerge.io where 123 is an ID we could parse out and then a filter by customer id 123. Would sth. like that be possible? That would be great 😸

[adrianmroz]

No, it's not possible. You would need to setup different turnilo instance for each ID and inside config you would add specific subsetFormula with condition customerID === 123 and so on.

Right now it is using plywood feature for Externals. It can't be dynamic.

[cedrics]

@adrianmroz thx for taking the time.

I am not yet super familiar with the inner workings of turnilo, so forgive my ignorance. Are you saying turnilo currently does not support a dynamically added filter clause as I described here or that it can't support it because of its design / limitations of plywood?

[adrianmroz]

Turnilo doesn't provide feature like you want (dynamic filtering). Right now it has static filter which is implemented via plywood external filter. External filter must be static.

So to implement your feature you would need to add something new, probably on backend you would have to check request and infer some customer ID, get query, modify it and send to druid. It is doable but I don't think we are interested in such feature and it sounds error prone. And sounds like slippery slope. Right now you would like to filter data based on customer id, later you would like to hide some metrics or dimensions ... I think multiple turnilo instances is better way to go.

[cedrics]

It is doable but I don't think we are interested in such feature

Ok, thanks for the feedback. If we decide to build it with turnilo we will keep it in mind and try to keep the adaptations as unobtrusive as possible

I think multiple turnilo instances is better way to go

I don't think this is really feasible for our use case. We would need to spin up hundreds of instances to get one instance per customer which is just a lot of wasted resources.

[adrianmroz]

I was thinking other day about this case. We have #521 right now, and here, proxy server passes some info to turnilo depending on user. So proxy server is responsible for auth. Maybe we could do something similar here?

I'm out of my depth, but consider this vague scenario:

1. Proxy based on user identity constructs header that encodes key-value pairs: userId: 123
2. You could define "dynamic" subsetFormula like $customer_id.in([{{params.userId}}])
3. Turnilo substitutes values and sends such filter to druid.

I don't know if it is possible to send map in http header and I don't know how to declare dynamic formula - we need something better than moustache template.

[cedrics]

Hey @adrianmroz, so we are currently working on this feature and we found it surprisingly hard to add this feature as plywood does not seem to be designed to support sth. like this. In the end I opted for patching plywood to allow passing a context to the external and therefore the requester per expression execution. We will test this a bit internally and try to get this merged into plywood (as the changes required are actually pretty simple). I am not very hopeful that my patch will be accepted as most of the issues and pull requests seem to go unanswered but we will try anyway.

[Laboltus]

As I understand at the time we can dynamically restrict access to datacubes (via "x-turnilo-allow-datacubes" header) and to rows (via queryDecorator). Is there a way to dynamically hide columns ?

[adrianmroz-allegro]

No, right now it is impossible to "hide" measures/dimensions dynamically. Config defines that statically.