Need guide on how to identify bottlenecks

[Laremere]

From several discussions in slack, identifying the source of a bottleneck when running Open Match is not trivial. We should write a guide on finding and resolving bottlenecks within the system, including any of the services not receiving enough resources:

• Frontend
• Backend
• Synchronizer
• Redis
• Evaluator
• MMF

Additionally, the guide should go into how to use tracing to see how long various operation are taking.

I think it may be worth building a tool which can download all of the currently searching tickets via query API, and then re-create them in another system. This would allow for testing various configurations without having to using a live production service. It could also download the tickets over time, so you can generate an artificially large number of tickets based on real data.