Adopt span tracing in hot code path

[waynexia]

What problem does the new feature solve?

Log and metrics are already integrated into our code base. Span tracing is another tool to further improve observability. We can start the adoption from insert/query/server modules first.

What does the feature do?

Add tracing::instrument to some critical paths

Implementation challenges

No response

[morigs]

Important notes about tracing crate:

1. It's not stable yet
2. It has relatively poor performance
3. It's not fully compatible with opentelemetry libraries

[v0y4g3r]

Maybe minitrace is also a good choice.

[waynexia]

Maybe minitrace is also a good choice.

100x❕

[tisonkun]

IIRC @zhongzc is the author of minitrace. Do you have some ideas we can move forward this issue?

Otherwise, I'd prefer move this issue to Ideas until we found concrete work items.

[waynexia]

We end up using tracing to instrument the code for better integration with existing tracking-logging infrastructure. But there are still some problems like post-sampling, multi-parent span etc.

[zhongzc]

there are still some problems like post-sampling, multi-parent span

This is precisely the realm of minitrace. If it is critical to the need, switching to minitrace is an option, and I would be happy to provide guidance on how to use it.