ResourceSync resource lists should scale to > 50K items

[mjgiarlo]

Decomposed from projecthydra/sufia#2915.

Hyrax defines a simple ResourceSync implementation that provides an API for harvesting deposited works via sitemaps and content negotiation. It currently scales up to a maximum of 50K items (the upper bound for a sitemap), but it should scale beyond 50K items.

[atz]

@mjgiarlo: As written, this is more of a requirement than an implementable ticket. Please specify what implementation direction should be pursued.

[jcoyne]

@atz part of the sitemap spec says, "thou shalt not have more than 50k items in a sitemap". We are violating that. We need to partition it somehow (resource sync provides a mechanism for multiple resource lists), but we haven't decided on how to partition yet.

[tahirpoduska]

@atz @mjgiarlo @jcoyne We are seeing the effect of not scaling beyond 50k. Our repo now has more then 100k and using the changelist is putting a lot of stress on our servers.

I would appreciate any relevant information, and or the direction to process to address this issue.

[mjgiarlo]

Redirecting :point_up: to folks who are actively working on Hyrax: @no-reply @vantuyls @samvera/hyrax-code-reviewers

[no-reply]

I guess we had better slate this for work in the 3.x series.