snuba icon indicating copy to clipboard operation
snuba copied to clipboard

feat(rust): Connect consumer to clickhouse

Open dbanda opened this issue 2 years ago • 1 comments

This is an attempt to get the consumers to actually write to clickhouse on submit. A challenge here was our clickhouse client is async, so I had to modify the ProcessingStrategy to use an async submit function. This change had ripple effects all across the board.

The quick solution in many places where the submit function was called in sync functions was to use block_on but this would sometimes hang, so I avoided block_on and used async await + tokio::spawn.

In some parts of the code, I had to replace mutexes with async-aware mutexes to make sure that we are not holding the locks synchronously in the submit async function.

dbanda avatar Jun 28 '23 21:06 dbanda

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (46b90d2) 90.55% compared to head (ee7676d) 90.55%.

:exclamation: Current head ee7676d differs from pull request most recent head 8736a89. Consider uploading reports for the commit 8736a89 to get more accurate results

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #4437   +/-   ##
=======================================
  Coverage   90.55%   90.55%           
=======================================
  Files         806      806           
  Lines       39739    39742    +3     
  Branches      255      255           
=======================================
+ Hits        35985    35990    +5     
+ Misses       3720     3718    -2     
  Partials       34       34           
Impacted Files Coverage Δ
snuba/clusters/cluster.py 96.34% <100.00%> (+0.04%) :arrow_up:
snuba/consumers/consumer_config.py 98.85% <100.00%> (+0.01%) :arrow_up:

... and 1 file with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

codecov[bot] avatar Jun 29 '23 07:06 codecov[bot]

looks like this was completed in https://github.com/getsentry/snuba/pull/4490

MeredithAnya avatar Dec 09 '24 19:12 MeredithAnya