Add documentation/guide/tooling for query tracing - performance guiding

[jobergum]

Vespa has this awesome feature where you can trace the end to end execution of a query .

This used in combination with timestamp tracing can be used to fully understand the serving performance as it includes end to end tracing of query execution.

• This could help guiding on if threads per search can help on e.g latency, cost of first phase versus second phase ranking
• Time spent in the stateless container cluster
• Analysis of query Blueprint for matching, could fast-search improve matching

So some documentation would be a nice start, a UI tooling component would be even nicer:)

[jobergum]

I'm not sure if we need to treat this as a public API but I have a strong desire to rename the MTF parts to have better explanations before I document this more in-depth? Some inconsistent casing and I'm sure we can use a few bytes more to explain with higher-level abstractions? Thoughts @baldersheim and @bratseth on the API part?

[bratseth]

Agree! Often the best way to see that is to try to write doc for it.

I suggest just creating a list of desired changes, which then can be fixed in one go. It's obvious to us of course that MTF refers to Mordenkainen's Tome of Foes, but it may not be to everybody ... https://en.wikipedia.org/wiki/MTF

[jobergum]

I suggest that we use tag="query_setup" for the events with MTF in the event string, then trace events will all have a tag from the set

• query_setup
• blueprint => query_execution_plan
• match_threads => query_execution

For the MTF strings, I suggest making them more readable?

• MTF: Start" => "Read query from network, start query setup"
• "MTF: Build query" => "Building query tree"
• "MTF: reserve handles" => "..."
• "MTF: Fetch Postings" => "Performing dictionary lookups and posting list initialization"
• "MTF: Handle Global Filters" => ".."
• "MTF: prepareSharedState" => "..":
• "MTF: Complete" => "Completed query setup"

[jobergum]

What you think @baldersheim ? I think building tooling around this will be very useful for many

[geirst]

Query tracing was improved in terms of terminology and structure in: https://github.com/vespa-engine/vespa/pull/22217 https://github.com/vespa-engine/vespa/pull/22288

[geirst]

@jobergum The practical search performance guide can now be updated.

[baldersheim]

Soon is in the past now.