Polymorphic relationships

[bcksl]

I've had polymorphic relationships on my wishlist, and looked into the Ash.Type.Union + calculation approach and thought maybe things could go a bit deeper. Realistically, this would combine well with resource behaviors, which is a topic unto itself.

defmodule App.User do
  @payment_method_types [
    App.Payment.Ideal
    App.Payment.Swift
    App.Payment.GooglePay
    App.Payment.ApplePay
    App.Payment.Stripe
  ]
	
  relationships do
    has_many :payment_methods, @payment_method_types
  end

  # ...
end

defmodule App.Payment.Ideal do
  relationships do
    belongs_to :user, App.User
  end

  # ...
end

In terms of an efficient AshPostgres implementation, it would not be dissimilar to existing relationships, but, in addition to payment_method_id, there would also be an enum payment_method_type to describe the possible types. load operations follow this pattern:

select * from users u
left join payment_ideal pi
  on u.payment_method_type = 'App.Payment.Ideal' and u.id = pi.user_id
left join payment_swift ps
  on u.payment_method_type = 'App.Payment.Swift' and u.id = ps.user_id
-- you get the idea
where u.id = 'a-nice-uuid'
   or u.id = 'or-this-one'
   or u.id = 'even-that-one';

The same transformation is applicable to many_to_many relationships, including those that are bidirectionally polymorphic.

Elixir is quite a good candidate for this kind of thing, because it is type-agnostic enough to allow you to return heterogeneous lists, but makes it easy to work with them by pattern-matching the struct. More than that, first-class support for polymorphism would be a huge boon to describing a lot of domain models.

Nested loads must be handled with care internally (resource behaviors addresses this), but the engine can complain if it doesn't have enough information to determine whether it can perform a further load on the result types, or different loads could be specified for different types, but this starts to get messy quickly.

A less-efficient (non-joined) approach when the relationship is polymorphic might be a first stage strategy.

In any case, I wanted to open the discussion on this and see what immediate challenges and benefits come to mind.

Other data-layers

• ETS doesn't support native joins anyways.
• Mnesia already supports 2-tuple identifiers which fit exactly the purpose.
• In general, when crossing data-layer boundaries the operations would need to be split anyways.

Alternative join approach

If desired, an alternative approach is to use a column for each type:

select * from users u
left join payment_ideal pi
  on u.payment_method_ideal_id = pi.user_id
left join payment_swift pi
  on u.payment_method_swift_id = pi.user_id
--- etc.

Which would require a constraint check (for allow_nil?: true, this would be 1 >=):

alter table users
add constraint check_one_payment_not_null check
(
  1 =
  (
    case when payment_method_ideal_id is null then 0 else 1 end
  + case when payment_method_swift_id is null then 0 else 1 end
  -- etc.
  )
)

[zachdaniel]

I think it would be really interesting to support this, but it is important to note that we would need to make some pretty serious changes to our expression language to support this. We'd need to support things like fragments that are only true on specific types, and we'd also lose a lot of the optimizations we might take for granted in some cases (that a given list is always homogenous on how it relates back to the thing that loaded it means we can Map.group_by(resource, :destination_attribute) to get the related values, for example.

Its worth exploring, but we may need to start small and work our way there in a branch to tease out all the hidden issues therein.

[bcksl]

How would these concerns be affected by first introducing the concept of resource behaviours/protocols, and allowing relationships to be polymorphic only according to a single one of those?

[zachdaniel]

Could you give me an example of what you have in mind?

[bcksl]

If we take the payments example from above, all of the App.Payment.* options would implement a App.Payment protocol that specifies the attributes, actions, etc. that the engine can expect to exist on each of them.

The relationship would then look like:

defmodule App.User do	
  relationships do
    has_many :payment_methods, App.Payment
  end

  # ...
end

[zachdaniel]

🤔 that would solve for some of it. But even then, we'd have to limit available filters to things contained in the the protocol until we figure out a way to sort of "switch on the type" in our expression language.

[zachdaniel]

I think its worth exploring though and could solve the bulk of the issues though.

[bcksl]

Absolutely. This covers a lot of the domain modeling use-cases pretty well, though. "If it's not something they all have in common, you can't filter on it," seems to be a good place to start. I'm not opposed to exploring options for type switching in expressions either, but I think it might make sense to start with limiting the scope first.

[zachdaniel]

Works for me!

[bcksl]

My initial thinking for the protocol specification would be writing a "bare" resource, with essentially the same DSL used for existing resources, and having a validator that ensures that all resources that implement it are isomorphic. Something cute to auto-register all the resources that implement a protocol so you don't have to list them would be nice as well :)

[bcksl]

Would it then be straightforward from an engine perspective to have it ignore struct tags and just pretend each thing it gets back is an instance of that bare resource for internal purposes?

[zachdaniel]

I think so? At least for most cases, and we can tackle the others one by one.

As for autmoatically registering them, honestly that kind of thing is almost universally harder than it seems like it should be (and may actually be impossible to solve all of the ergonomics issues around it). But power to you if you can find a way. Just know that others have tried and ultimately had to back down from that path.

[bcksl]

Ok, I was thinking that there would be the introduction of an implementations block:

implementations do
  implements App.Payment
  implements App.OtherFunStuff
end

And that this would essentially append those resources to the module being implemented, for purposes of constructing the tagged joins.

[zachdaniel]

We can do that :) But if we want to get the list of all things that implement App.Payment, that is a very hard problem.

[bcksl]

Can we push it up to the registry level to scan the resources and provide the...registry? :)

[zachdaniel]

Potentially, but they might be cross registry. They might even be cross API? I guess if we prevent that, then yes we can do it that way. Basically require that the api option be passed, and that all relevant resources be part of the provided api.

[bcksl]

Tbh I think it's probably ok most of the time to require that a protocol and all the resources that implement it are within the same registry, but I can imagine cases where it would be nice not to have the limitation. But in the same way it's required that cross-API relationships have an api specifier now, I don't see any issue with requiring that cross-API implements clauses have the same. How weird does that get on the engine side of things if a relationship is multiply-cross-API?

Performance optimizations can be eschewed a little bit at first, for sure. For example, I was also thinking that it might be an issue for the engine if some of the implementing resources are in different data layers. More efficient pathfinding for that kind of stuff can come later though, and we can just treat it like that whole relationship is in a different data layer and split.

[bcksl]

In the former case with api specifier it sounds like the implementation registry would need to be pushed up into the engine, like I guess is done with cross-API relationships.

[zachdaniel]

If we start with the restriction that all implementors must be in the API's registry, then its fine. Its easy to filter a list of modules to see which ones have a property. As long as the list is static somewhere (which it is in the APIs registry).

[zachdaniel]

The protocol (we should figure out what this kind of thing should be named, and really hash this aspect of it out, I'm not sure how I feel about it yet) wouldn't need to be in a registry, just the resources.

[bcksl]

Somehow this is a combination of structs and protocols in nature, if we consider those to be close to attributes and actions, respectively. "Behaviour" is a term I've used for this kind of thing in the past. "Structure" or "specification" are also descriptive of the situation.

I think it's no problem to make this single API-only at the start, particularly if the limitation is only on implementors being in the same registry, as I can see many cases where you might want to have a cross-API relationship to such a collection of implementors, but fewer where it's a big deal to have implementors of one "thing" across multiple APIs.

[bcksl]

Ash.Resource.Spec could work.

[zachdaniel]

Ash.Resource.Spec sounds like a good idea potentially. There is a lot of theory involved with doing this kind of thing declaratively, to the point that I wonder if we should just make it functional.

use Ash.Resource.Spec

def complies?(resource, opts) do
  ...
end

and just check it functionally. It would be an easy enough starting point, and then we could provide a built in spec like HasAttribute, name: :foo_id, type: type.

[bcksl]

That would certainly leave things maximally open-ended with minimal code for the validation aspect, so we could get going on the engine part ASAP. The Spec—or something—would still need to define an Ash.Resource according to the existing Ash.Resource DSL, so that the engine can figure out what it is allowed to do with the resource using the existing introspection methodology. We can simply have it feed implementing resources to complies? and generate explosive error messages at runtime if you didn't implement complies? strictly enough (;

The engine for now doesn't have to be particularly concerned about whether a given resource is compliant to a spec—it can be on you for the time being if things blow up—but it does need a way to do the things it already does re: figuring out how it can use/optimize interactions with that resource.

For a declarative validator, it's a matter of going through and figuring out which parts of the Ash.Resource DSL are conflicting/overridable. attribute.default might be a good example of something that is overridable, but attribute.type would not be. Stuff such as attribute.allow_nil? falls somewhere in the middle, but is probably only relevant to the parts of the engine that check it, so could potentially be allowed to be overridden.

[bcksl]

This also marks all the touch points in the engine that will need to be updated if a declarative validation strategy is defined, or equally well allow for that strategy to be a validation extension that auto-implements Spec, so I'm all for it.

[zachdaniel]

I think the declarative validator is even more complex than just overridable or not. It is for basic things but gets complex for more complex applications of this kind of pattern...but we probably won't find out without just going for it. I think we would just start off only supporting attributes, and even then only supporting attributes/types.

[bcksl]

For sure, attributes are the easy bit. I think we turn off all the other forms of optimization for these types of relationship to start.

Evaluating beforehand how complex it will become would begin with laying down the whole Ash.Resource DSL tree and just going through it. But a lot of that same information would come from starting with complete splitting and getting the various existing optimizations that are applicable working one by one and documenting what will break if various constraints aren't satisfied. They can grow together.

[zachdaniel]

I still think it's more complicated than just deciding what is/isn't mergeable, but that's besides the point. Attributes with exact matching types is good enough for now.

[bcksl]

No doubt, rather that each optimization that you are performing has a minimum set of constraints that satisfies its assumptions, which right now is checked by the engine using introspection on an individual resource. To understand whether a resource complies? to an implementation declaratively, with least strictness, we would need to determine what constraints each value of each DSL item actually implies to the engine.

Doing this might require introducing new DSL to Spec for hinting, which would then similarly need to be validated against implementors. It's entirely possible that there are optimizations that wouldn't be decidable from a "bare" resource definition as we've been discussing, in which case those need to be disabled for now.

I'd like to dig into some of the more complex compile-time optimizations that the engine is performing to get a lay of the land on what we'd be looking at going forward. Do you have an example of one of the optimizations you think would be challenging to evaluate fitness for against a declarative spec?

[zachdaniel]

🤔 Honestly we do very little compile time optimization. What I'm getting at with my comments is that we're talking about introducing a generic Spec concept, designed to ensure that some resource meets some required behavior. In basic cases, like "does it have an attribute with this type", we can define that like this:

attributes do
  attribute :name, :type
end

and that is enough. We can just see if every entity/option in the spec is also present in the resource in question. But imagine a spec that wanted to say "A create action called :create that accepts an input :foo".

Here are two different actions that match that requirement:

create :create do
  accept [:foo]
end

create :create do
  accept []
  argument :foo, :string
end

So what I'm saying is that the definition of a Spec will not always mean "do X items at X position in the spec DSL and the resource DSL match". And I think that may be problematic enough (i.e this method not being theoretically sound in advanced use cases) for us to need to come up with some other methodology for expressing this concept of "adoptable constraints on a resource". There is a lot of inspiration to look to for this kind of thing (protocols/behaviors/typeclasses), but Ash has hybrid characteristics of a type system and protocols and behaivours.

This is why I think it might be better to instead just have functional specs and/or an entirely different DSL for expressing these things.

dsl
|> require_attribute(:name, :type)
|> require_action(:create, inputs: [foo: :string])

For example. I'm still not sure how we'd hook it up, because the idea is that you'd want a relationship to be given a spec and to know that the spec enforces the things that the relationship needs, i.e

has_many :things, MyApp.Specs.SomeSpecOfPolymorphicTypes do
  destination_attribute :foo_id
end

we'd want the has_many relationship to raise if the spec given doesn't enforce the requirements of the relationship.

So that might lead us to a DSL:

use Ash.Spec

require_attributes do
  attribute :foo, :string
end

Wrapping this all up, though, I think the word we are looking for here is an Interface that defines a common set of behavior/shape of resources. Perhaps I'm wrong and we can do that by just defining a resource and like... "figuring it out". I.e if the spec says:

create :create do
  accept [:foo]
end

and the implementor does

create :create do
  argument :foo, :string
end

then we say "yes, these match".

[bcksl]

Nice, that matches well with what I was thinking, both the separate DSL or repurposing of Ash.Resource's DSL.

In the case where there are two things that are sufficiently isomorphic for the engine—your example of accept and argument—we could for sure make it less magical that they are equivalent by explicitly saying:

action :create do
  input :arg, :type
end

My thought to repurpose the Ash.Resource DSL had primarily two goals in mind:

1. Make it less of a learning curve for the Ash userbase to start using specs.
2. Leverage the existing engine code for determining how to interact with a resource.

It seems totally reasonable to sidestep the validation part at the start, so we could start with having the engine recognize tagged relationships (_id and _type), and choose the correct resource accordingly for return/further steps.