Improvements around queries

[dumbbell]

Currently, queries suffer several issues or limitations:

• They are executed remotely on the leader. This is a problem if the remote node has an incompatible version of Erlang/OTP or the module hosting the function, or even lacks the function entirely.
• Local queries solve this problem but they may be executed against ouf-of-date data when compared to updates that were previously submitted.

This collection of patches aims at fixing this with the following changes:

• The default options of updates changes to use the reply_from => local option.
• The default query type is now local (possible thanks to the change above).
• A new khepri:fence() API is introduced.

See individual commits to learn more.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 91.66667% with 6 lines in your changes missing coverage. Please review.

Project coverage is 89.38%. Comparing base (0f56ea9) to head (a379a73).

Files	Patch %	Lines
src/khepri_machine.erl	92.18%	5 Missing :warning:
src/khepri.erl	87.50%	1 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #260      +/-   ##
==========================================
- Coverage   89.57%   89.38%   -0.20%     
==========================================
  Files          21       21              
  Lines        3099     3080      -19     
==========================================
- Hits         2776     2753      -23     
- Misses        323      327       +4     

Flag	Coverage Δ
erlang-25	88.63% <91.66%> (-0.14%)	:arrow_down:
erlang-26	89.25% <91.66%> (-0.04%)	:arrow_down:
os-ubuntu-latest	89.38% <91.66%> (+0.03%)	:arrow_up:
os-windows-latest	88.60% <91.66%> (-0.17%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[dumbbell]

I renamed the existing testcase in test/fence.erl (too much copy-pasting :-) and added few more testcases.

[dumbbell]

This pull request is ready for review again. @the-mikedavis, compared to the last time you looked at it, I reintroduced the favor option but limited the values to low_latency and consistency. Regardless of the value, the query is always local. The difference is in the use of the fence mechanism of not.

[the-mikedavis]

I found a few typos in the commit messages:

• c5ea616 "loader" => "leader"
• 653ec8e "ais" => "is"
• b66fc4f "increaset" => "increase"
• b66fc4f "beraviors" => "behaviors"

[dumbbell]

Thank you! Thank was fast :-) I fixed the typos in the commit messages.

[dumbbell]

The use of reply_from => local for all commands is useless as it is in this patch because we try to send commands directly to the leader if we know it... I believe this explains the transient failure in cluster_SUITE we se in CI.

[the-mikedavis]

IIRC reply_from can also be member => RaServerId. We could pass the local node as that member

[dumbbell]

IIRC reply_from can also be member => RaServerId. We could pass the local node as that member

Good idea. I will restore the direct send to the leader if it is known and use the reply_from => {member, LocalRaServerId} option.

I'm in the process of removing the leader ID caching we do and use ra_leaderboard:lookup_leader/1 directly.

[dumbbell]

@the-mikedavis: I restored the send to the leader, but with reply_from => {member, LocalRaServer}. I also dropped the cached leader thing in favor of the use of ra_leaderboard:lookup_leader/1. I finally reorganized the commits a bit and reworded some commit messages. It's ready for review again and hopefully, it should be the last iteration :-)