Improvements to enable pathfinder to handle large schemas.

[MTG2000]

Description

This PR contains updates to several components to make them better able to handle large schemas.

The changes include:

• Use virtualization for operations builder tab.
• Lazy render some components only when viewed.
• Change the way some components are passed & used so that they are not un-mounted & re-mounted again on re-renders
• Memoize some of the heavier calculations
• Add virtualization to schema documentation's tab sections lists.

The schema I used

I have generated a schema that is relatively big & used it while testing. The schema consists of 400 root queries, each query contains 3 fields & goes 5 levels deep. So it contains a total of around 150_000 types.

Here's the schema if you want to test with it: schema.txt

& Here's the script I used to generate the schema if you would like to try generating bigger schemas or adding different fields. generate-large-schema-script.zip

Note that the time complexity of the script above is O( numberOfTypes * numberOfFields ^ maxDepth ) So maxDepth value can't be more than 6-8 without reaching O(10^6) complexity.

And if you have other large schemas that you could test this PR with, it'd be great.

Some benchmarks

Testing with the schema above on my machine (a relatively powerful one), here are some before/after benchmarks for the time it takes between an interaction & the next paint:

	Before	After
Clicking on a field in operations builder	Minimum of 344ms	48ms
Switching between tabs	Minimum of 700ms	40ms
Opening a section with +100k fields in schema documentation tab	A few seconds	48ms

(And during development, the above values could be doubled due to double rendering in React)

Of course these values could change quite a bit across different devices & browsers.

Notes

• If I missed some section/component that is still affected by large schemas, let me know.
• If there are further/better optimizations to some parts, let me know.
• It should be mentioned that one of the drawbacks of virtualization is that CMD-F (or the native find-on-page) won't work anymore since the items are not even rendered, so we will likely need to introduce a custom search input in all the sections that got virtualized to keep a good user experience. Though I think we should do that in a separate PR.

Type of change

• [ ] 💔 Breaking
• [ ] 🚀 Feature
• [ ] 🐛 Fix
• [ ] 🛠️ Tooling
• [X] 🔨 Refactoring
• [ ] 🧪 Test
• [ ] 📦 Dependency
• [ ] 📖 Requires documentation update

[changeset-bot[bot]]

⚠️ No Changeset found

Latest commit: 03f7c6392c66fb2d22e676905a9bddc098b59891

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
pathfinder	⬜️ Ignored (Inspect)	Visit Preview		Oct 7, 2024 7:20am

[MTG2000]

I've added in-app search for the components that utilized virtualization.

I tried to keep it simple, but we can update its design if you want.

Also, for the searching algorithm, I just used javascript's include function. It was still instant with lists of 150_000 items.

So I didn't feel the necessity to add a specific searching library like fuse or minisearch, but we can do so if you think it's needed.

[MTG2000]

Actually, after thinking about it for a bit, I believe that literal searching won't be that useful if the user doesn't know the exact name of the field/type he is searching for, (and this is unlikely to happen especially with larger schemas.)

So I added fuzzysort library & used it in the searching.

[MTG2000]

I feel the search/filtering code & virtualization code is a bit duplicated across the 3 components where it's used, but there's a slight differences between the 3 usages that I wasn't sure of a clean abstraction.

If you have a suggestion for that, lmk.

[jonathanawesome]

Actually, after thinking about it for a bit, I believe that literal searching won't be that useful if the user doesn't know the exact name of the field/type he is searching for, (and this is unlikely to happen especially with larger schemas.)

So I added fuzzysort library & used it in the searching.

It's fantastic that you added the search bits, we've needed it for quite some time. There's a bit of design work necessary before we're ready to implement, though. We need to think about how we display results that are deep within selection sets, most critically. Let's pull this out for now and we'll get it in after we've gone through the design exercise.