Add support for sampling rate in `streamlog`

[timvaillancourt]

Description

This PR implements the RFC https://github.com/vitessio/vitess/issues/15909 by adding a flag --querylog-sample-rate float to vtcombo, vtgate and vttablet in order for queries to be sampled randomly

This flag supports values between 0.0 (no logging) and 1.0 (log all queries) to match other "sample" flags

Related Issue(s)

Resolves https://github.com/vitessio/vitess/issues/15909

Checklist

• [ ] "Backport to:" labels have been added if this change should be back-ported to release branches
• [ ] If this change is to be back-ported to previous releases, a justification is included in the PR description
• [ ] Tests were added or are not required
• [ ] Did the new or modified tests pass consistently locally and on CI?
• [ ] Documentation was added or is not required

Deployment Notes

[vitess-bot[bot]]

Review Checklist

Hello reviewers! :wave: Please follow this checklist when reviewing this Pull Request.

General

• [ ] Ensure that the Pull Request has a descriptive title.
• [ ] Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

• [ ] Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

• [ ] Apply the release notes (needs details) label if users need to know about this change.
• [ ] New features should be documented.
• [ ] There should be some code comments as to why things are implemented the way they are.
• [ ] There should be a comment at the top of each new or modified test to explain what the test does.

New flags

• [ ] Is this flag really necessary?
• [ ] Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

• [ ] Each item in Jobs should be named in order to mark it as required.
• [ ] If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

• [ ] Protobuf changes should be wire-compatible.
• [ ] Changes to _vt tables and RPCs need to be backward compatible.
• [ ] RPC changes should be compatible with vitess-operator
• [ ] If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
• [ ] vtctl command output order should be stable and awk-able.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 90.90909% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 68.24%. Comparing base (f118ba2) to head (8f1dcd2). Report is 134 commits behind head on main.

Files	Patch %	Lines
go/streamlog/streamlog.go	90.90%	1 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #15919      +/-   ##
==========================================
- Coverage   68.40%   68.24%   -0.17%     
==========================================
  Files        1556     1541      -15     
  Lines      195121   197179    +2058     
==========================================
+ Hits       133479   134571    +1092     
- Misses      61642    62608     +966     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[vitess-bot[bot]]

Hello! :wave:

This Pull Request is now handled by arewefastyet. The current HEAD and future commits will be benchmarked.

You can find the performance comparison on the arewefastyet website.

[deepthi]

Flag docs are auto-generated, so we don't need any manual changes on the website for this. @frouioui can you confirm that I'm getting this right?

[frouioui]

Flag docs are auto-generated, so we don't need any manual changes on the website for this. @frouioui can you confirm that I'm getting this right?

@deepthi according https://github.com/vitessio/vitess-bot/blob/main/README.md#vitess-bot it should generate a PR upon merge to main and upon the next release. If it does not, I will investigate and in the meantime flags docs can be manually generated using the tool.

[frouioui]

After talking with @mattlord, I realized that actually the automation does not work. We should update the website docs manually at the same time as this PR.

[timvaillancourt]

Flag docs are auto-generated, so we don't need any manual changes on the website for this. @frouioui can you confirm that I'm getting this right?

Right, I forgot this is automatic now, thanks 👍

Also, based on the fact arewefastyet will not apply the flag that enables this I will remove the benchmark me label for now and add a Go benchmark instead

[deepthi]

After talking with @mattlord, I realized that actually the automation does not work. We should update the website docs manually at the same time as this PR.

@frouioui can you provide the instructions for that? Or should we simply plan to do that manually for v20 before release? I suspect there may be other PRs which are missing website documentation as well. One example: #16021 cc @shlomi-noach for v20 release decision.

EDIT: There's a Makefile target in the website repo to generate the docs. Do the following in a website branch, and make sure you have the main vitess repo checked out and accessible from there.

export COBRADOC_VERSION_PAIRS="<this_pr_branch_name>:20.0"
export VITESS_DIR=~/go/src/vitess.io/vitess
make generated-docs

It becomes easier if doing after merge into main, because we can use main:20.0

[frouioui]

Yeah exactly running VITESS_DIR=../vitess ./tools/sync_cobradocs.sh from inside the website repo works fine too to fill all the missed changes.

[timvaillancourt]

EDIT: replied to wrong PR/browser tab 🤦

[timvaillancourt]

To address my own paranoia regarding this thread I added a go benchmark:

BenchmarkShouldEmitLog
BenchmarkShouldEmitLog/default
BenchmarkShouldEmitLog/default-12         	327598167	         3.739 ns/op
BenchmarkShouldEmitLog/filter_tag
BenchmarkShouldEmitLog/filter_tag-12      	100000000	        11.67 ns/op
BenchmarkShouldEmitLog/50%_sample_rate
BenchmarkShouldEmitLog/50%_sample_rate-12 	79597881	        15.05 ns/op

Where default = no filter-tag or random sampling, filter_tag = query-comment filter tag, 50%_sample_rate is --query-sample-rate 0.5

I think I'm most surprised by filter_tag which essentially adds a string.Contains(sql), actually 😄

[shlomi-noach]

Re docs, good points. My suggestion: after code freeze, I'll generate vtctldclient docs manually. At any case we can decouple the docs discussion from this PR.