Upcoming enforcing limits for SharePoint lists & libraries in Site Collection

[dennyregehr]

What type of issue is this?

Question

What SharePoint development model, framework, SDK or API is this about?

not applicable

Target SharePoint environment

SharePoint Online

What browser(s) / client(s) have you tested

• [ ] 💥 Internet Explorer
• [ ] 💥 Microsoft Edge
• [ ] 💥 Google Chrome
• [ ] 💥 FireFox
• [ ] 💥 Safari
• [ ] mobile (iOS/iPadOS)
• [ ] mobile (Android)
• [ ] not applicable
• [X] other (enter in the "Additional environment details" area below)

Additional environment details

• SPFx version: 1.15.2
• Node.js version: 16.13.2
• PNP/SP: 2.12

Issue description

On Jul 26, 2022, MC405984 was published that states Microsoft will start enforcing hard limits on the number of list and libraries a site collection can have instead of just a recommendation as it is now. I have developed a SPFx solution (single page web part) for my company's legal department that provides a UI for managing contract metadata (across several lists), and creates a library for each revenue generating client as soon as they have documents to upload. We currently have over 20,000 clients, and nearly 20,000 libraries in the site collection (single site). We were forced to use this model instead of a single library because the requirements demand unique permissions which in a single library would exceed the existing rule of 5,000 maximum unique permission per library. The SPFx web part is extremely fast, and creates libraries quickly with no throttling. The user's experience within the site are limited to the SPFx web part app which has been thoughtfully created to avoid throttling-prone queries. The one place that is slow is the Site Contents page, but users are never using it because their entire experience is within the UI of my app. So my question: does this change have to happen? My app was developed in 2020 and at the time this was not a rule, but a suggestion that was part of the Listview Threshold guidance. I went out of my way to craft an app that would work given the limits in the SharePoint online platform because it's great for document storage, identity management and security. This rule enforcement will cripple my application. Since the reason for the rule is to improve user experience (which I have to imagine is affecting Site Contents mostly), I wonder if it would make more sense to find a way to work with the few sites that contain a large number of lists/libraries rather than hamstringing the platform. Perhaps there could be an administrative setting to allow a site to opt-in to have more than 2000 lists/libraries with the default that it is not allowed. That would give me and other developers/admins a viable path forward, and would also enforce the 2000 limit for all other sites by default.

[ghost]

Thank you for reporting this issue. We will be triaging your incoming issue as soon as possible.

[CallumCrowley]

@dennyregehr is there a reason the libraries couldn't be spread across multiple sites? (site collections)

[dennyregehr]

While possible, the management of the app would be a nightmare. Off the top of my head, here are a few of the issues this would create:

• current libraries would have to be moved into 10 different site collections that would simply serve as document repositories, breaking thousands of unique permissions and requiring the existing links between contract records and the contract documents to be updated in multiple places (a pretty complicated migration for SharePoint online)
• to achieve the migration explained above would require an additional custom coded application, and IF we attempted to carry forward the existing unique permissions the complexity and run time would more than double
• app would have to be recoded to manage documents in an ever growing number of site collections
• each site collection would require a base set of permissions to be managed (in addition to unique permissions, we also use visitors, member/contributor/owner groups for some users of the app) by the app's owners (current site's site owners), and as the key stakeholders already struggle with this in a single site collection
• requests for permission would have to be granted across all 10+ site collections
• using multiple site collections increases the architectural complexity unnecessarily and leads to more difficulty generally for managing information, future migrations, ongoing support

My team and I have talked about this solution, but in the end it would be like having an app that uses 10+ instances of a SQL Server database instead of a single database. While it makes for a nice school project, it makes for a enterprise nightmare.

[CallumCrowley]

@dennyregehr I appreciate the difficulty of your migration, but the idea of 20k document libraries in one site collection gives me a shudder (given what must be going on at a SQL level to facilitate this). Particularly with each library having unique permissions.

While a SharePoint Online tenant can handle millions of sites (site collections), a single site collection doesn't perform well with thousands of libraries or subsites. This has always been the case, even in on-premises SharePoint. I honestly think the recommendation should have been more explicit if anything, particularly now Microsoft are thinking of enforcing the recommendation as a limit.

Many applications do use multiple databases, this is fairly normal and it's how SharePoint Online works which is most certainly an enterprise application. 10 SQL databases of reasonable size would perform far better than 1 enormous SQL database.

[dennyregehr]

Let me just hit the bullets here:

• This site collection is not used as a typical SharePoint site.
• It has nearly 20,000 libraries and it does perform quite well.
• In 2 years, we've had no performance issues. Not one.
• The solution I have built makes effective use of SharePoint for tabular data storage as well as document or binary data storage. I love it. It's great. It's working. Everyone is happy. The company is making money with this app.
• And the platform is changing something that will break my happy money making solution.
• I've presented what I think is a reasonable solution - an opt-in option for Site Collection admins so they can take on the risk of having more than 2,000 lists/libraries.
• I understand the overall reason for the rule - but I'm not using SharePoint in the typical way here. Users don't work directly in lists or libraries. They don't navigate into Site Contents. They stay in the SPFx app I have created.
• As I've watched developers (online community, conferences, SharePoint store, vendors who build SharePoint solutions, and in my own organization) for the past 10 years do wondrous and creative things with SharePoint I know I'm not alone in using SharePoint for solutions that are outside the box. And I've never seen someone try to make their custom SharePoint app use multiple Site Collections for storing documents for that app.

Thank you for your opinion @CallumCrowley. Does anyone else have some thoughts?