🛤️ tracking: Public Graph API

[runspired]

Tracking Issue for Improvements to the Performance, Reliability, and Encapsulation of the Relationship Layer.

Pre-Refactor

• [x] Performance Test App replicating expensive scenario(s)
• [x] Tracerbench integration for benchmarking the performance test app
• [x] (Stretch) CI Integration for Tracerbench

Stage 1

• [x] eliminate usage of recordData.isNew by the relationship state classes
• [x] eliminate usage of recordData methods/properties within the relationship/has-many/belongs-to state classes
• [x] add identifier / storeWrapper to the classes in the constructor by stealing them from recordData
• [x] eliminate usage of this.recordData in those classes
• [x] pass in identifier /storeWrapper to create and get vs stealing from RecordData
• [x] getBelongsTo/setBelongsTo/getHasMany/setHasMany/addToHasMany etc. methods should receive identifiers instead of recordDatas (likely internalModel would do this for now)
• [x] RecordData should convert the identifier back to recordData to give to the relationship layer so that this change is reviewable and landable on it's own

Stage 2

• [x] resource._relationships should be removed in favor of a WeakMap for the store/InternalModel to access RelationshipState
• [x] RecordData setBelongsTo/setHasMany/addToHasMany/push etc. methods should convert mutations / updates into operations and call relationships.perform with the operation on the relationship. This method will then translate the operation into the legacy method calls on the relationship in a switch statement.
• [x] RecordData getBelongsTo/getHasMany should convert to calling relationships.query
• [x] remove RecordData._relationships in favor of a WeakMap lookup for the graph for a given identifier e.g. graphFor(storeWrapper).<perform|query>(...)

Stage 3

• [x] we refactor to diff based storage :)

Stage 4

• [x] Refactor ManyArray to make better use of the relationship layer
• [x] Refactor InternalModel to use perform/query APIs (introduce private RecordData API for this)

Stage 5

• [x] Cleanup any store & InternalModel logic we can
• [x] Don't force InternalModel/RecordData creation for isEmpty check
• [x] Refactor to Singleton Graph

Stage 6

• [ ] Make Graph remoteState updates lazy

In most cases we can detect that a relationship has not been consumed yet and avoid doing work to set up initial state until it is

• [ ] Make local state updates lazy

We do not need to sync remote updates to local updates until the app has requested the current state. To achieve this we notify the change but do not actually do the work until pull.

• [ ] Make update propagation more efficient

Currently GC accounts for about 15% of the time spent populating the graph. This is due partly to the high temporary allocation overhead operations have and mostly to the temporary arrays and Sets created during the population process. Instead of decomposing operations into more operations we should find a simpler way of executing the instruction set. The allocation of temporary object should be reduced by the diff mechanism itself.

• [ ] Persistent Diff

The most common operations on the graph are membership checks (which we already optimize by use of a Set) and updates (which require performing a diff). Updates carry through the system: the changes applied to remote state will also need to be applied to local state, the changes applied to local state will need to be applied to the ManyArray, the changes applied to the ManyArray will need to be diffed by glimmer when updating the contents of #each. Our goal is to reduce how often we diff and store information such that the diff never needs to be recalculated.

• [x] Operations RFC to make perform/query public for relationships on RecordData
• [ ] Graph RFC to allow all RecordData implementations to make use of the relationship-graph if desired

[runspired]

Parallel Efforts that assist this effort (the first three are already tasks for Project Trim)

• [x] The Singleton RecordData Infrastructure
• [x] RecordData Documentation
• [x] A Refactor moving as much relationship UI logic out of internalModel/the store into the Model package as possible
• [x] A Singleton RecordData Implementation for EmberData

[runspired]

cc @sly7-7 This is an effort I'm pairing with @skaterdav85 on to improve relationships that should help you greatly :) If you have the ability to contribute to it to help resolve your performance issues with relationships that would be awesome!

[iamdtang]

👋

[runspired]

I removed the Diff util in #8550 which we may end up wanting later. It was currently unused so seemed better to drop for now.

[runspired]

The main change I want to see before marking the API as complete is a story around pagination. I began a spike for it in #9320.

The largest unanswered question is what the underlying page storage mechanism should be, and how (with it) to account for the potential for someone to fetch an arbitrary page of the relationship (where arbitrary means that a gap in pages exists between that page and any pages connected to the first or last page).

Pagination is so varied it is necessarily hard to support arbitrary pages (whereas it is relatively easy to support first/prev/next/last as pointers for iteration of a linked list).

Potential data structures include:

• a LinkedList
• a dictionary, holey array keyed by a signaled index
• a Map keyed by a signaled index
• something that keys by url but accounts for indexing