mimir icon indicating copy to clipboard operation
mimir copied to clipboard

[Docs] Add sharding/split-groups to Compactor capacity planning

Open eamonryan opened this issue 2 years ago • 10 comments

What this PR does

Adds documentation to the Planning Capacity page to include how to size -compactor.split-groups and -compactor.split-and-merge-shards.

Which issue(s) this PR fixes or relates to

Fixes #4620

Checklist

  • [ ] Tests updated
  • [X] Documentation added
  • [x] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

eamonryan avatar Mar 30 '23 00:03 eamonryan

The information that follows is an overview about the CPU, memory, and disk space that Grafana Mimir requires at scale.

I do see that, however, if you take a potential user who is looking to migrate to Mimir from their existing solution and they already operate at some specific high scale - this is the doc they will read since it helps them estimate resources and hopefully tells them anything else they need to pay attention to at a specific size.

Is this setting related to "capacity planning"? I think it depends on the viewpoint .

Will your installation struggle under certain conditions if you do not pay attention to this setting? It will (large tenant setup).

I don't necessarily think it's impossible to put the setting elsewhere though, but overall I think the options are:

  • Put it here and anywhere that is relevant
  • Link to the appropriate other page where it is mentioned from this page, such as Production Tips, as suggested (but is it just a "tip" if your installation will be heavily affected if this setting is not correct?)
  • Find some way for the setting to be auto-configured correctly for a given installation (much harder)

Happy to do whatever makes sense though.

eamonryan avatar Apr 06 '23 15:04 eamonryan

Will your installation struggle under certain conditions if you do not pay attention to this setting? It will (large tenant setup).

The settings you're talking about go way beyond the compactors. I 💯 with you the need of a doc page for that, which is what we attempted with the "production tips" doc page. I'm wondering if we could just link it from the capacity planning doc, and keep evolving the "production tips" (maybe even restructuring and renaming it).

pracucci avatar Apr 12 '23 07:04 pracucci

The CHANGELOG has just been cut to prepare for the next Mimir release. Please rebase main and eventually move the CHANGELOG entry added / updated in this PR to the top of the CHANGELOG document. Thanks!

lamida avatar Apr 17 '23 10:04 lamida

@eamonryan, mind resolving these conflicts?

osg-grafana avatar Jul 14 '23 14:07 osg-grafana

@eamonryan Have you considered this?

https://github.com/grafana/mimir/pull/4621#issuecomment-1504783389

pracucci avatar Jul 18 '23 13:07 pracucci

@cristiangsp, would you mind taking a look at this issue to gain technical context?

osg-grafana avatar Nov 16 '23 10:11 osg-grafana

@tacole02 - Would you mind checking this old PR for validity and either merging or closing?

GrafanaWriter avatar Oct 22 '24 21:10 GrafanaWriter

Hello @eamonryan, @pracucci , @pstibrany ! I'm just reviewing this very old PR. Would you mind letting me know if this is still relevant/accurate/something we should merge? Thanks! 😄

tacole02 avatar Oct 22 '24 23:10 tacole02

Will get to this tomorrow (hoping) and review new PR w/ Taylor.

eamonryan avatar Oct 25 '24 03:10 eamonryan

Thanks so much, @eamonryan . Feel free to ping me with the new PR.

tacole02 avatar Oct 25 '24 16:10 tacole02

We are going to close this one and revisit it in another PR (myself and @tacole02)

eamonryan avatar Oct 30 '24 22:10 eamonryan