YapDatabase relationship delete rules

[DesmanLead]

YDB_DeleteSourceIfAllDestinationsDeleted implementation counts only edges with the same name as the deleted one (even if there are edges with another name and same source in the database). Is it by design?

[robbiehanson]

Yes, this is by design. Some edges may signify ownership. While others only signify a "weak" relationship.

For example, there may be edges from "song" to "album". When the last song in an album is deleted, we want the album to be automatically deleted.

But there are also edges from "albumArtwork" to "album". Except that "albumArtwork" isn't meant to increment the "retainCount" of album.

Let me know if you have any other questions about this.

[DesmanLead]

Well, but sometimes more than one edge that signifies ownership required. Currently I don't see any clear solution to this task. Imagine the following structure: Collections: Artists, Albums, Singles. Edges: Artists -> Albums, Artists -> Singles. In this situation if one will delete a last album connected with an artist, the artist will be removed as well, and all the singles connected with this artist will be removed too. But in this case we need to delete an artist only if all the albums AND all the singles were deleted.

[robbiehanson]

Interesting point. Perhaps what we're getting at here is the need to specify ownership rules (in addition to the edge name) ...

[antonm76]

Robbie, is there any way to check whether a particular node is involved with another edges? According to DesmanLead example, YapDatabase has to check whether there are no edges from Artists to Singles when last Album is deleted and Yap is going to delete an Artist. Currently, in our project we handle such situation by having on edge for both relationships (i.e. one edge for Artists -> Albums and Artists -> Singles). Thus, when last Album is deleted there are still edges from Artist to Singles, so Artist is not deleted. But in this case we have to manually test for target collection when enumerating edges with source Artists as source collection.

[DesmanLead]

Actually, existing combinations of deletion rules and edge direction will cover all the use-cases in terms of ownership rules if the deletion rules will be implemented as antonm76 said.

[DesmanLead]

Any news about that?

[antonm76]

The issue still appears in 2.9.2. Is there any way to escalate the priority and make it finally fixed? So, the latest test case is the following: There are separate edges from one Parent collection two several Child collections. All edges are created with YDB_DeleteDestinationIfAllSourcesDeleted | YDB_DeleteSourceIfAllDestinationsDeleted rules. When last object in some Child collection is deleted, Yap deletes source from Parent, and this leads to cascading delete from all other Child collections.

[robbiehanson]

OK, I agree with you guys to a certain degree on this one.

The previous solution was sub-optimal, no matter how one looks at it.

Generally the DeleteXIfAllYsDeleted rule implies some kind of retainCount system. Which, in turn, implies that edges are either part of the retainCount, or they are not. This is generally solved by:

• having all edges be strong pointers (inflexible solution)
• or supporting both strong pointers and weak pointers (flexible solution)

However, the previous solution lacked any strong/weak metadata. In fact, it assumed every edge was strong. But it had a hidden gotcha: it only counted strong references that used the same name.

At the time I felt this was an optimal tradeoff that allowed me to skip supporting a strong/weak system. (I.E. Just be sure to use the same edge name.) However, this assumption was based on a subset of use cases that I was familiar with. And it has come to my attention that many people use relationships/edges as a general organizational system, and requiring matching edge names isn’t possible (as it would break the entire purpose of them creating the edges in the first place).

The example that you give is:

• Artists -> Albums
• Artists -> Singles

This seems to imply that the edge names MUST be different, as they are an inherent part of the organizational structure.

However, your push request (which I accepted) will now lead us down a road that necessitates that we more formally solve the retainCount problem.

The rules are now (after your push request) defined as:

• YDB_DeleteSourceIfAllDestinationsDeleted

  • retainCount = count(all edges where src matches)
  • delete node if retainCount drops to zero

• YDB_DeleteDestinationIfAllSourcesDeleted

  • retainCount = count(all edges where dst matches)
  • delete node if retainCount drops to zero

Let’s look at a concrete example of the complications that could arrive, and then I’ll propose a solution that may alleviate most of the complications.

Consider the following 3 edges

(foo1) ----------> (bar)
          @“foobar"[YDB_DeleteDestinationIfAllSourcesDeleted]

(foo2) ----------> (bar)
          @“foobar"[YDB_DeleteDestinationIfAllSourcesDeleted]

(buzz) ----------> (bar)
          @“buzzbar"[YDB_DeleteDestinationIfSourceDeleted]

If both foo1 & foo2 are deleted, should we delete bar? If buzz is deleted, should we delete bar?

In the current setup, the code says NO (to both questions). However, it’s not inconceivable to imagine a situation in which an alternative outcome is desired.

So here’s my proposal:

typedef NS_OPTIONS(uint16_t, YDB_NodeDeleteRules) {
    // notify only
    YDB_NotifyIfSourceDeleted      = 1 << 0,
    YDB_NotifyIfDestinationDeleted = 1 << 1,
    
    // one-to-one
    YDB_DeleteSourceIfDestinationDeleted = 1 << 2,
    YDB_DeleteDestinationIfSourceDeleted = 1 << 3,
    
    // one-to-many & many-to-many
    YDB_DeleteSourceIfAllDestinationsDeleted = 1 << 4,
    YDB_DeleteDestinationIfAllSourcesDeleted = 1 << 5,
    
    // options for:
    // - YDB_DeleteSourceIfAllDestinationsDeleted
    // - YDB_DeleteDestinationIfAllSourcesDeleted
    YDB_RequireNameMatch = 1 << 6,
    YDB_WeakReferenceSource = 1 << 7,
    YDB_WeakReferenceDestination = 1 << 8,
};

This would allow us to formally define the rules as such:

• YDB_DeleteSourceIfAllDestinationsDeleted

  • retainCount = count(all edges where src matches, excluding edges with WeakReferenceSource)
  • delete node if retainCount drops to zero

• YDB_DeleteSourceIfAllDestinationsDeleted | YDB_RequireNameMatch

  • retainCount = count(all edges where src & name matches, excluding edges with WeakReferenceSource)
  • delete node if retainCount drops to zero

• YDB_DeleteDestinationIfAllSourcesDeleted

  • retainCount = count(all edges where dst matches, excluding edges with WeakReferenceDestination)
  • delete node if retainCount drops to zero

• YDB_DeleteDestinationIfAllSourcesDeleted | YDB_RequireNameMatch

  • retainCount = count(all edges where dst & name matches, excluding edges with WeakReferenceDestination)
  • delete node if retainCount drops to zero

This may not solve every possible use case under the sun, but it certainly gets us a more powerful & flexible system. Plus, I think, simply adding the above documentation (the "formal" rules) to the code & wiki would alleviate much of the confusion that developers have encountered so far.

Let me know your thoughts.

[DesmanLead]

In the current setup, the code says NO (to both questions).

AFAIK, YES for the second one. YDB_DeleteDestinationIfSourceDeleted handling routine doesn't count edges. Correct me if I'm wrong.

Having both "weak" and "name match" rules seems redundant to me. "name match" can be achieved in terms of "weak" rules (only edges with the same name signify strong relationship, while others are weak).

I'm concerned about possible edge-cases with weak edges. E.g., how to handle an item with only weak edges pointing to it? Should it be removed when any weak edge removed, or not?

Another possible way is to count only edges with the appropriate rule. E.g., count only edges with YDB_DeleteSourceIfAllDestinationsDeleted in rules while processing deleted destination. This way edges with no rules will signify weak relationship. It may seem confusing at first, but it's consistent with the current rules model and covers most use-cases. What do you think about that?

We are going to discuss the problem more here and provide a feedback early next week.

[antonm76]

Hello, @robbiehanson and @DesmanLead !

@robbiehanson while set of rules you proposed is formally totally correct and covers all possible cases, at the same time, I just cannot imagine all possible combinations of YDB_RequireNameMatch, YDB_WeakReferenceSource and YDB_WeakReferenceDestination flags in one-to-many and many-to-many environment. And I also cannot imagine one would want to construct such data structures.

My own opinion (and it is for sure humble in this particular case -) ) is that any edge created with any of delete rules flags has to preserve (or retain, in your terminology) it's object i.e.:

YDB_DeleteSourceIfAllDestinationsDeleted and YDB_DeleteSourceIfDestinationDeleted should preserve source object

YDB_DeleteDestinationIfAllSourcesDeleted and YDB_DeleteDestinationIfSourceDeleted should preserve destination object.

So, it should behave like it behaves now, after @DesmanLead's pull request was approved.

The only question is how to handle the case you mentioned:

(foo1) ----------> (bar)
          @“foobar"[YDB_DeleteDestinationIfAllSourcesDeleted]

(foo2) ----------> (bar)
          @“foobar"[YDB_DeleteDestinationIfAllSourcesDeleted]

(buzz) ----------> (bar)
          @“buzzbar"[YDB_DeleteDestinationIfSourceDeleted]

I believe, one-to-one edges should win in such conditions, i.e. bar should be deleted if buzz is deleted despite it still have connections to foo, but at the same time, bar SHOULD NOT be deleted if all foo's are deleted. I believe if developer creates such one-to-one connection he should understand consequences.

Thanks