Mnesia DL: internal layout and in-mnesia Filter support

[zentourist]

This was a fun one! Compound Primary Keys almost broke me 😆 I started thinking this would take a day or two. I was wrong...

What this does:

• Changes the internal format of the Ash.Resource records when they get stored in Mnesia so that we can better query against the values using Matchspecs
• Builds MatchSpecs from Ash.Filter so that we can do in-Mnesia filtering. Today this is just using guards. I would like to eventually add Pattern Matching in the MatchHead for even more efficient queries.

It does NOT support all Filter functionality and if we get into any issues it will simply fallback to runtime filtering for now.

I added comprehensive tests for the Matchspec items. The tests were largely generated with Sonnet 4.5, but I reviewed them in depth.

Initially, this broke a bunch of tests in the suite, but I was able to get everything working again without touching any of the existing testing so feature-wise it should be backward compatible. NOTE: The internal format does change so if there are any persisted tables out there THIS WILL BREAK THEM.

Contributor checklist

Leave anything that you believe does not apply unchecked.

• [x] I accept the AI Policy, or AI was not used in the creation of this PR.
• [ ] Bug fixes include regression tests
• [ ] Chores
• [x] Documentation changes
• [x] Features include unit/acceptance tests
• [x] Refactoring
• [ ] Update dependencies

[zachdaniel]

🤔 NOTE: The internal format does change so if there are any persisted tables out there THIS WILL BREAK THEM.

We are going to need to figure something out here then, like making these new changes backwards compatible. Alternatively, we could break out a new data layer called AshMnesia for example, so that we can iterate on breaking changes.

[zentourist]

🤔 NOTE: The internal format does change so if there are any persisted tables out there THIS WILL BREAK THEM.

We are going to need to figure something out here then, like making these new changes backwards compatible. Alternatively, we could break out a new data layer called AshMnesia for example, so that we can iterate on breaking changes.

Yeah, I've been noodling on this. I don't hate the idea of breaking it out, but it is nice to have a good data_layer in the core for testing. I noticed a few tests use the mnesia DL. We could also just add a V2 namespace. That would also allow for easy benchmarks between the two :)

I also had a thought about introducing migrations using transform_table, but I haven't spent any time digging into that yet.

[zentourist]

I had one additional thought on the internal table format. I might be able to support both versions by introducing something like this:

mnesia do
  table :table_name
  version 1
end

If someone uses v1 for the format, it would work in a backward compatible way and allow for deprecation. I'd have to give this a go, but I think it's possible.

[zachdaniel]

Yep, I think versioning makes a lot of sense. We can potentially introduce a backwards compatibility configuration (see the docs on backwards compatibility configs) to make new apps default to the new better version, or make the option required (or warn when unset) so that users will have to configure it and can decide how to handle it.

[zentourist]

Hey @zachdaniel - Just wanted to let you know that I'm not abandoning this. I was on vacation and am just getting back. I should be able to get back to this soon. ✌️