sentry icon indicating copy to clipboard operation
sentry copied to clipboard
Published 3 weeks ago • verified publisher icon getsentry

Bug: failure_rate / failure_rate_if causes 500 errors on Insights (dataset=spans)

Open piontas opened this issue 1 month ago • 10 comments

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

Steps to Reproduce

Summary

When calling the Organization Events API with dataset=spans and requesting the field:

failure_rate_if(is_transaction,equals,true)

(or implicitly: failure_rate())

the Sentry backend returns HTTP 500 Internal Server Error, and Snuba RPC logs show a corresponding 500 on EndpointTraceItemTable/v1.

Removing this single field makes the same query succeed (HTTP 200). This affects the Insights Frontend and Insights Backend landing tables, breaking UI functionality. This occurs on a fresh Sentry Self-Hosted 25.11.0 installation and also reproducible on production running the same version.

Impact

  • Insights → Frontend "Landing Table" fails to load
  • Insights → Backend "Landing Table" fails to load

All queries including failure_rate_if(...) on dataset=spans reliably return 500 All other fields (tpm(), p50_if, performance_score, etc.) work normally Removing only the failure-rate field makes the API return HTTP 200

Steps to Reproduce

Expected Result

✅ Working request (200)

GET /api/0/organizations/<org>/events/
  ?dataset=spans
  &field=is_starred_transaction
  &field=transaction
  &field=project
  &field=tpm()
  &field=p50_if(span.duration,is_transaction,equals,true)
  &field=p75_if(span.duration,is_transaction,equals,true)
  &field=p95_if(span.duration,is_transaction,equals,true)
  &field=failure_rate_if(is_transaction,equals,true)
  &field=performance_score(measurements.score.total)
  &field=count_unique(user)
  &field=sum_if(span.duration,is_transaction,equals,true)
  &query=(span.op:[ui.render,...] OR project.id:[2]) !span.op:http.server
  &statsPeriod=24h
  &sort=-sum_if(span.duration,is_transaction,equals,true)
  &referrer=api.insights.frontend.landing-table

Same request with only this field removed:

&field=failure_rate_if(is_transaction,equals,true)

returns 200

Actual Result

❌ Failing request (500)

GET /api/0/organizations/<org>/events/
  ?dataset=spans
  &field=is_starred_transaction
  &field=transaction
  &field=project
  &field=tpm()
  &field=p50_if(span.duration,is_transaction,equals,true)
  &field=p75_if(span.duration,is_transaction,equals,true)
  &field=p95_if(span.duration,is_transaction,equals,true)
  &field=failure_rate_if(is_transaction,equals,true)
  &field=performance_score(measurements.score.total)
  &field=count_unique(user)
  &field=sum_if(span.duration,is_transaction,equals,true)
  &query=(span.op:[ui.render,...] OR project.id:[2]) !span.op:http.server
  &statsPeriod=24h
  &sort=-sum_if(span.duration,is_transaction,equals,true)
  &referrer=api.insights.frontend.landing-table

Returns:

500 Internal Server Error (Sentry backend)

and Snuba logs:

POST /rpc/EndpointTraceItemTable/v1 ... 500 ... referrer=api.insights.frontend.landing-table

Product Area

Insights

Link

No response

DSN

No response

Version

25.11.0

piontas avatar Nov 25 '25 13:11 piontas

ENG-5970

linear[bot] avatar Nov 25 '25 13:11 linear[bot]

Assigning to @getsentry/support for routing ⏲️

getsantry[bot] avatar Nov 25 '25 13:11 getsantry[bot]

Logs from snuba-api:

"POST /rpc/EndpointTraceItemTable/v1 HTTP/1.1" 500 26897 "api.insights.frontend.landing-table" "python-urllib3/2.2.2"

piontas avatar Nov 25 '25 13:11 piontas

Logs from sentry-web:

[ERROR] django.request: Internal Server Error: /api/0/organizations/sentry/events/ (status_code=500 request=<WSGIRequest: GET '/api/0/organizations/sentry/events/?dataset=spans&environment=dev&field=is_starred_transaction&field=transaction&field=project&field=tpm%28%29&field=p50_if%28span.duration%2Cis_transaction%2Cequals%2Ctrue%29&field=p75_if%28span.duration%2Cis_transaction%2Cequals%2Ctrue%29&field=p95_if%28span.duration%2Cis_transaction%2Cequals%2Ctrue%29&field=failure_rate_if%28is_transaction%2Cequals%2Ctrue%29&field=performance_score%28measurements.score.total%29&field=count_unique%28user%29&field=sum_if%28span.duration%2Cis_transaction%2Cequals%2Ctrue%29&per_page=50&project=2&query=%28%20span.op%3A%5Bui.render%2Cinteraction%2Cui.interaction%2Cui.interaction.click%2Cui.interaction.hover%2Cui.interaction.drag%2Cui.interaction.press%2Cui.webvital.cls%2Cui.webvital.lcp%2Cui.webvital.fcp%2Cpageload%2Cnavigation%5D%20OR%20project.id%3A%5B2%5D%20%29%20%21span.op%3Ahttp.server&referrer=api.insights.frontend.landing-table&sampling=NORMAL&sort=-is_starred_transaction&sort=-sum_if%28span.duration%2Cis_transaction%2Cequals%2Ctrue%29&statsPeriod=24h'>)

piontas avatar Nov 25 '25 13:11 piontas

Routing to @getsentry/product-owners-insights for triage ⏲️

getsantry[bot] avatar Nov 27 '25 08:11 getsantry[bot]

This is only happening in the distributed setup with clickhouse cluster and kafka cluster. Just tested it in self-hosted docker-compose based version and it is working fine.

piontas avatar Nov 27 '25 20:11 piontas

@piontas we discussed internally, and our best initial guess is that a query is timing out:

  1. Does the error occur if you use smaller time ranges.
  2. Do you have any logs from the database layer itself about the error, rather than the web / snuba logs that seem a bit sparse.
  3. Are there any timeouts you can increase in your database configuration to test the timeout hypothesis?

bcoe avatar Dec 04 '25 16:12 bcoe

  1. yes, even for 1 minute time range
2025.12.04 23:28:07.164854 [ 828 ] {randomized-903c9712acc74eebbfe664f585fbd9d7} <Error> executeQuery: Code: 10. DB::Exception: Not found column sumOrNullIf(divide(1_UInt8, __table1.sampling_factor), and(mapContains(__table1.attributes_string_14, 'sentry.status'_String), or(not(in(if(mapContains(__table1.attributes_string_14, 'sentry.status'_String), arrayElement(__table1.attributes_string_14, 'sentry.status'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String)), __set_Nullable(String)_7344979180565805872_16682740436227664765)), and(isNull(if(mapContains(__table1.attributes_string_14, 'sentry.status'_String), arrayElement(__table1.attributes_string_14, 'sentry.status'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String))), _CAST(1_UInt8, 'UInt8'_String))))) in block. There are only columns: if(mapContains(__table1.attributes_string_4, 'sentry.request.method'_String), arrayElement(__table1.attributes_string_4, 'sentry.request.method'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String)), coalesce(if(mapContains(__table1.attributes_string_20, 'sentry.transaction'_String), arrayElement(__table1.attributes_string_20, 'sentry.transaction'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String)), if(mapContains(__table1.attributes_string_20, 'sentry.transaction'_String), arrayElement(__table1.attributes_string_20, 'sentry.transaction'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String)), if(mapContains(__table1.attributes_string_3, 'transaction'_String), arrayElement(__table1.attributes_string_3, 'transaction'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String))), if(mapContains(__table1.attributes_string_19, 'sentry.op'_String), arrayElement(__table1.attributes_string_19, 'sentry.op'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String)), transform(CAST(ifNull(CAST(__table1.project_id, 'Int64'_String), ''_String), 'String'_String), _CAST(['2']_Array(String), 'Array(String)'_String), _CAST(['python-flask']_Array(String), 'Array(String)'_String), 'unknown'_String), sumOrNullIf(divide(1_UInt8, __table1.sampling_factor), and(mapContains(__table1.attributes_float_25, 'sentry.exclusive_time_ms'_String), 1_Bool)), sumIf(minus(multiply(divide(1_UInt8, __table1.sampling_factor), divide(1_UInt8, __table1.sampling_factor)), divide(1_UInt8, __table1.sampling_factor)), and(mapContains(__table1.attributes_float_25, 'sentry.exclusive_time_ms'_String), 1_Bool)), countIf(if(mapContains(__table1.attributes_float_25, 'sentry.exclusive_time_ms'_String), arrayElement(__table1.attributes_float_25, 'sentry.exclusive_time_ms'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String)), and(mapContains(__table1.attributes_float_25, 'sentry.exclusive_time_ms'_String), 1_Bool)), sumIf(divide(1_UInt8, __table1.sampling_factor), and(mapContains(__table1.attributes_float_25, 'sentry.exclusive_time_ms'_String), 1_Bool)), quantileTDigestWeightedOrNullIf(0.5_Float64)(if(mapContains(__table1.attributes_float_5, 'sentry.duration_ms'_String), arrayElement(__table1.attributes_float_5, 'sentry.duration_ms'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String)), CAST(round(multiply(divide(1_UInt8, __table1.sampling_factor), 1000_UInt16)), 'UInt64'_String), and(mapContains(__table1.attributes_float_5, 'sentry.duration_ms'_String), 1_Bool)), quantilesTDigestWeighted(0.4_Float64, 0.405_Float64, 0.41_Float64, 0.415_Float64, 0.42_Float64, 0.425_Float64, 0.43_Float64, 0.435_Float64, 0.44_Float64, 0.445_Float64, 0.45_Float64, 0.455_Float64, 0.46_Float64, 0.465_Float64, 0.47_Float64, 0.475_Float64, 0.48_Float64, 0.485_Float64, 0.49_Float64, 0.495_Float64, 0.5_Float64, 0.505_Float64, 0.51_Float64, 0.515_Float64, 0.52_Float64, 0.525_Float64, 0.53_Float64, 0.535_Float64, 0.54_Float64, 0.545_Float64, 0.55_Float64, 0.555_Float64, 0.56_Float64, 0.565_Float64, 0.57_Float64, 0.575_Float64, 0.58_Float64, 0.585_Float64, 0.59_Float64, 0.595_Float64)(if(mapContains(__table1.attributes_float_5, 'sentry.duration_ms'_String), arrayElement(__table1.attributes_float_5, 'sentry.duration_ms'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String)), CAST(round(multiply(divide(1_UInt8, __table1.sampling_factor), 1000_UInt16)), 'UInt64'_String)), countIf(if(mapContains(__table1.attributes_float_5, 'sentry.duration_ms'_String), arrayElement(__table1.attributes_float_5, 'sentry.duration_ms'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String)), and(mapContains(__table1.attributes_float_5, 'sentry.duration_ms'_String), 1_Bool)), sumIf(divide(1_UInt8, __table1.sampling_factor), and(mapContains(__table1.attributes_float_5, 'sentry.duration_ms'_String), 1_Bool)), quantileTDigestWeightedOrNullIf(0.95_Float64)(if(mapContains(__table1.attributes_float_5, 'sentry.duration_ms'_String), arrayElement(__table1.attributes_float_5, 'sentry.duration_ms'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String)), CAST(round(multiply(divide(1_UInt8, __table1.sampling_factor), 1000_UInt16)), 'UInt64'_String), and(mapContains(__table1.attributes_float_5, 'sentry.duration_ms'_String), 1_Bool)), quantilesTDigestWeighted(0.85_Float64, 0.855_Float64, 0.86_Float64, 0.865_Float64, 0.87_Float64, 0.875_Float64, 0.88_Float64, 0.885_Float64, 0.89_Float64, 0.895_Float64, 0.9_Float64, 0.905_Float64, 0.91_Float64, 0.915_Float64, 0.92_Float64, 0.925_Float64, 0.93_Float64, 0.935_Float64, 0.94_Float64, 0.945_Float64, 0.95_Float64, 0.955_Float64, 0.96_Float64, 0.965_Float64, 0.97_Float64, 0.975_Float64, 0.98_Float64, 0.985_Float64, 0.99_Float64, 0.995_Float64)(if(mapContains(__table1.attributes_float_5, 'sentry.duration_ms'_String), arrayElement(__table1.attributes_float_5, 'sentry.duration_ms'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String)), CAST(round(multiply(divide(1_UInt8, __table1.sampling_factor), 1000_UInt16)), 'UInt64'_String)), sumOrNullIf(divide(1_UInt8, __table1.sampling_factor), and(mapContains(__table1.attributes_string_14, 'sentry.status'_String), or(not(in(if(mapContains(__table1.attributes_string_14, 'sentry.status'_String), arrayElement(__table1.attributes_string_14, 'sentry.status'_String), NULL_Nullable(Nothing)), __set_Nullable(String)_7344979180565805872_16682740436227664765)), and(isNull(if(mapContains(__table1.attributes_string_14, 'sentry.status'_String), arrayElement(__table1.attributes_string_14, 'sentry.status'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String))), _CAST(1_UInt8, 'UInt8'_String))))), sumIf(minus(multiply(divide(1_UInt8, __table1.sampling_factor), divide(1_UInt8, __table1.sampling_factor)), divide(1_UInt8, __table1.sampling_factor)), and(mapContains(__table1.attributes_string_14, 'sentry.status'_String), or(not(in(if(mapContains(__table1.attributes_string_14, 'sentry.status'_String), arrayElement(__table1.attributes_string_14, 'sentry.status'_String), NULL_Nullable(Nothing)), __set_Nullable(String)_7344979180565805872_16682740436227664765)), and(isNull(if(mapContains(__table1.attributes_string_14, 'sentry.status'_String), arrayElement(__table1.attributes_string_14, 'sentry.status'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String))), _CAST(1_UInt8, 'UInt8'_String))))), countIf(if(mapContains(__table1.attributes_string_14, 'sentry.status'_String), arrayElement(__table1.attributes_string_14, 'sentry.status'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String)), and(mapContains(__table1.attributes_string_14, 'sentry.status'_String), or(not(in(if(mapContains(__table1.attributes_string_14, 'sentry.status'_String), arrayElement(__table1.attributes_string_14, 'sentry.status'_String), NULL_Nullable(Nothing)), __set_Nullable(String)_7344979180565805872_16682740436227664765)), and(isNull(if(mapContains(__table1.attributes_string_14, 'sentry.status'_String), arrayElement(__table1.attributes_string_14, 'sentry.status'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String))), _CAST(1_UInt8, 'UInt8'_String))))), sumIf(divide(1_UInt8, __table1.sampling_factor), and(mapContains(__table1.attributes_string_14, 'sentry.status'_String), or(not(in(if(mapContains(__table1.attributes_string_14, 'sentry.status'_String), arrayElement(__table1.attributes_string_14, 'sentry.status'_String), NULL_Nullable(Nothing)), __set_Nullable(String)_7344979180565805872_16682740436227664765)), and(isNull(if(mapContains(__table1.attributes_string_14, 'sentry.status'_String), arrayElement(__table1.attributes_string_14, 'sentry.status'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String))), _CAST(1_UInt8, 'UInt8'_String))))), uniqIfOrNull(if(mapContains(__table1.attributes_string_31, 'sentry.user'_String), arrayElement(__table1.attributes_string_31, 'sentry.user'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String)), and(mapContains(__table1.attributes_string_31, 'sentry.user'_String), 1_Bool)), countIf(if(mapContains(__table1.attributes_string_31, 'sentry.user'_String), arrayElement(__table1.attributes_string_31, 'sentry.user'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String)), and(mapContains(__table1.attributes_string_31, 'sentry.user'_String), 1_Bool)), sumIf(divide(1_UInt8, __table1.sampling_factor), and(mapContains(__table1.attributes_string_31, 'sentry.user'_String), 1_Bool)), sumOrNullIf(multiply(if(mapContains(__table1.attributes_float_5, 'sentry.duration_ms'_String), arrayElement(__table1.attributes_float_5, 'sentry.duration_ms'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String)), divide(1_UInt8, __table1.sampling_factor)), and(mapContains(__table1.attributes_float_5, 'sentry.duration_ms'_String), 1_Bool)), sumIf(multiply(multiply(if(mapContains(__table1.attributes_float_5, 'sentry.duration_ms'_String), arrayElement(__table1.attributes_float_5, 'sentry.duration_ms'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String)), if(mapContains(__table1.attributes_float_5, 'sentry.duration_ms'_String), arrayElement(__table1.attributes_float_5, 'sentry.duration_ms'_String), _CAST(NULL_Nullable(Nothing), 'Nullable(Nothing)'_String))), minus(multiply(divide(1_UInt8, __table1.sampling_factor), divide(1_UInt8, __table1.sampling_factor)), divide(1_UInt8, __table1.sampling_factor))), and(mapContains(__table1.attributes_float_5, 'sentry.duration_ms'_String), 1_Bool)): While executing Remote. (NOT_FOUND_COLUMN_IN_BLOCK) (version 25.3.6.10034.altinitystable (altinity build)) (from 10.57.2.79:35040) (in query: SELECT (transform(CAST(ifNull((coalesce(if(mapContains(attributes_string_20, 'sentry.transaction'), arrayElement(attributes_string_20, 'sentry.transaction'), NULL), if(mapContains(attributes_string_20, 'sentry.transaction'), arrayElement(attributes_string_20, 'sentry.transaction'), NULL), if(mapContains(attributes_string_3, 'transaction'), arrayElement(attributes_string_3, 'transaction'), NULL)) AS `sentry.transaction_TYPE_STRING`), ''), 'String'), [], [], 'false') AS is_starred_transaction), (if(mapContains(attributes_string_4, 'sentry.request.method'), arrayElement(attributes_string_4, 'sentry.request.method'), NULL) AS `sentry.request.method_TYPE_STRING`), `sentry.transaction_TYPE_STRING`, (if(mapContains(attributes_string_19, 'sentry.op'), arrayElement(attributes_string_19, 'sentry.op'), NULL) AS `sentry.op_TYPE_STRING`), (transform(CAST(ifNull((cast(project_id, 'Int64') AS `sentry.project_id_TYPE_INT`), ''), 'String'), ['2'], ['python-flask'], 'unknown') AS project), (divide((round(sumIfOrNull(divide(1, sampling_factor), mapContains(attributes_float_25, 'sentry.exclusive_time_ms') AND true)) AS `epm().left`), 1.0) AS `epm()`), `epm().left`, (multiply(1.96, sqrt(sumIf(minus(multiply(divide(1, sampling_factor), divide(1, sampling_factor)), divide(1, sampling_factor)), mapContains(attributes_float_25, 'sentry.exclusive_time_ms') AND true))) AS `__snuba_custom_column__confidence_interval$epm().left$function_type:count`), (divide(countIf((if(mapContains(attributes_float_25, 'sentry.exclusive_time_ms'), arrayElement(attributes_float_25, 'sentry.exclusive_time_ms'), NULL) AS `sentry.exclusive_time_ms_TYPE_DOUBLE`), mapContains(attributes_float_25, 'sentry.exclusive_time_ms') AND true), sumIf(divide(1, sampling_factor), mapContains(attributes_float_25, 'sentry.exclusive_time_ms') AND true)) AS `__snuba_custom_column__average_sample_rate$epm().left`), (countIf(`sentry.exclusive_time_ms_TYPE_DOUBLE`, mapContains(attributes_float_25, 'sentry.exclusive_time_ms') AND true) AS `__snuba_custom_column__count$epm().left`), 1.0, (quantileTDigestWeightedIfOrNull(0.5)((if(mapContains(attributes_float_5, 'sentry.duration_ms'), arrayElement(attributes_float_5, 'sentry.duration_ms'), NULL) AS `sentry.duration_ms_TYPE_DOUBLE`), cast(round(multiply(divide(1, sampling_factor), 1000)), 'UInt64'), mapContains(attributes_float_5, 'sentry.duration_ms') AND true) AS `p50(span.duration)`), (quantilesTDigestWeighted(0.4, 0.405, 0.41, 0.415, 0.42, 0.425, 0.43, 0.435, 0.44, 0.445, 0.45, 0.455, 0.46, 0.465, 0.47, 0.475, 0.48, 0.485, 0.49, 0.495, 0.5, 0.505, 0.51, 0.515, 0.52, 0.525, 0.53, 0.535, 0.54, 0.545, 0.55, 0.555, 0.56, 0.565, 0.57, 0.575, 0.58, 0.585, 0.59, 0.595)(`sentry.duration_ms_TYPE_DOUBLE`, cast(round(multiply(divide(1, sampling_factor), 1000)), 'UInt64')) AS `__snuba_custom_column__confidence_interval$p50(span.duration)$function_type:p50,granularity:0.005,width:0.1`), (divide(countIf(`sentry.duration_ms_TYPE_DOUBLE`, mapContains(attributes_float_5, 'sentry.duration_ms') AND true), sumIf(divide(1, sampling_factor), mapContains(attributes_float_5, 'sentry.duration_ms') AND true)) AS `__snuba_custom_column__average_sample_rate$p50(span.duration)`), (countIf(`sentry.duration_ms_TYPE_DOUBLE`, mapContains(attributes_float_5, 'sentry.duration_ms') AND true) AS `__snuba_custom_column__count$p50(span.duration)`), (quantileTDigestWeightedIfOrNull(0.95)(`sentry.duration_ms_TYPE_DOUBLE`, cast(round(multiply(divide(1, sampling_factor), 1000)), 'UInt64'), mapContains(attributes_float_5, 'sentry.duration_ms') AND true) AS `p95(span.duration)`), (quantilesTDigestWeighted(0.85, 0.855, 0.86, 0.865, 0.87, 0.875, 0.88, 0.885, 0.89, 0.895, 0.9, 0.905, 0.91, 0.915, 0.92, 0.925, 0.93, 0.935, 0.94, 0.945, 0.95, 0.955, 0.96, 0.965, 0.97, 0.975, 0.98, 0.985, 0.99, 0.995)(`sentry.duration_ms_TYPE_DOUBLE`, cast(round(multiply(divide(1, sampling_factor), 1000)), 'UInt64')) AS `__snuba_custom_column__confidence_interval$p95(span.duration)$function_type:p95,granularity:0.005,width:0.1`), (divide(countIf(`sentry.duration_ms_TYPE_DOUBLE`, mapContains(attributes_float_5, 'sentry.duration_ms') AND true), sumIf(divide(1, sampling_factor), mapContains(attributes_float_5, 'sentry.duration_ms') AND true)) AS `__snuba_custom_column__average_sample_rate$p95(span.duration)`), (countIf(`sentry.duration_ms_TYPE_DOUBLE`, mapContains(attributes_float_5, 'sentry.duration_ms') AND true) AS `__snuba_custom_column__count$p95(span.duration)`), (divide((round(sumIfOrNull(divide(1, sampling_factor), mapContains(attributes_string_14, 'sentry.status') AND (not(in((if(mapContains(attributes_string_14, 'sentry.status'), arrayElement(attributes_string_14, 'sentry.status'), NULL) AS `sentry.status_TYPE_STRING`), ['ok', 'cancelled', 'unknown'])) OR isNull(`sentry.status_TYPE_STRING`) AND not(has(['ok', 'cancelled', 'unknown'], NULL))))) AS `failure_rate().left`), (round(sumIfOrNull(divide(1, sampling_factor), mapContains(attributes_float_25, 'sentry.exclusive_time_ms') AND true)) AS `failure_rate().right`)) AS `failure_rate()`), `failure_rate().left`, (multiply(1.96, sqrt(sumIf(minus(multiply(divide(1, sampling_factor), divide(1, sampling_factor)), divide(1, sampling_factor)), mapContains(attributes_string_14, 'sentry.status') AND (not(in(`sentry.status_TYPE_STRING`, ['ok', 'cancelled', 'unknown'])) OR isNull(`sentry.status_TYPE_STRING`) AND not(has(['ok', 'cancelled', 'unknown'], NULL)))))) AS `__snuba_custom_column__confidence_interval$failure_rate().left$function_type:count`), (divide(countIf(`sentry.status_TYPE_STRING`, mapContains(attributes_string_14, 'sentry.status') AND (not(in(`sentry.status_TYPE_STRING`, ['ok', 'cancelled', 'unknown'])) OR isNull(`sentry.status_TYPE_STRING`) AND not(has(['ok', 'cancelled', 'unknown'], NULL)))), sumIf(divide(1, sampling_factor), mapContains(attributes_string_14, 'sentry.status') AND (not(in(`sentry.status_TYPE_STRING`, ['ok', 'cancelled', 'unknown'])) OR isNull(`sentry.status_TYPE_STRING`) AND not(has(['ok', 'cancelled', 'unknown'], NULL))))) AS `__snuba_custom_column__average_sample_rate$failure_rate().left`), (countIf(`sentry.status_TYPE_STRING`, mapContains(attributes_string_14, 'sentry.status') AND (not(in(`sentry.status_TYPE_STRING`, ['ok', 'cancelled', 'unknown'])) OR isNull(`sentry.status_TYPE_STRING`) AND not(has(['ok', 'cancelled', 'unknown'], NULL)))) AS `__snuba_custom_column__count$failure_rate().left`), `failure_rate().right`, (multiply(1.96, sqrt(sumIf(minus(multiply(divide(1, sampling_factor), divide(1, sampling_factor)), divide(1, sampling_factor)), mapContains(attributes_float_25, 'sentry.exclusive_time_ms') AND true))) AS `__snuba_custom_column__confidence_interval$failure_rate().right$function_type:count`), (divide(countIf(`sentry.exclusive_time_ms_TYPE_DOUBLE`, mapContains(attributes_float_25, 'sentry.exclusive_time_ms') AND true), sumIf(divide(1, sampling_factor), mapContains(attributes_float_25, 'sentry.exclusive_time_ms') AND true)) AS `__snuba_custom_column__average_sample_rate$failure_rate().right`), (countIf(`sentry.exclusive_time_ms_TYPE_DOUBLE`, mapContains(attributes_float_25, 'sentry.exclusive_time_ms') AND true) AS `__snuba_custom_column__count$failure_rate().right`), (uniqIfOrNull((if(mapContains(attributes_string_31, 'sentry.user'), arrayElement(attributes_string_31, 'sentry.user'), NULL) AS `sentry.user_TYPE_STRING`), mapContains(attributes_string_31, 'sentry.user') AND true) AS `count_unique(user)`), (divide(countIf(`sentry.user_TYPE_STRING`, mapContains(attributes_string_31, 'sentry.user') AND true), sumIf(divide(1, sampling_factor), mapContains(attributes_string_31, 'sentry.user') AND true)) AS `__snuba_custom_column__average_sample_rate$count_unique(user)`), (countIf(`sentry.user_TYPE_STRING`, mapContains(attributes_string_31, 'sentry.user') AND true) AS `__snuba_custom_column__count$count_unique(user)`), (sumIfOrNull(multiply(`sentry.duration_ms_TYPE_DOUBLE`, divide(1, sampling_factor)), mapContains(attributes_float_5, 'sentry.duration_ms') AND true) AS `sum(span.duration)`), (multiply(1.96, sqrt(sumIf(multiply(multiply(`sentry.duration_ms_TYPE_DOUBLE`, `sentry.duration_ms_TYPE_DOUBLE`), minus(multiply(divide(1, sampling_factor), divide(1, sampling_factor)), divide(1, sampling_factor))), mapContains(attributes_float_5, 'sentry.duration_ms') AND true))) AS `__snuba_custom_column__confidence_interval$sum(span.duration)$function_type:sum`), (divide(countIf(`sentry.duration_ms_TYPE_DOUBLE`, mapContains(attributes_float_5, 'sentry.duration_ms') AND true), sumIf(divide(1, sampling_factor), mapContains(attributes_float_5, 'sentry.duration_ms') AND true)) AS `__snuba_custom_column__average_sample_rate$sum(span.duration)`), (countIf(`sentry.duration_ms_TYPE_DOUBLE`, mapContains(attributes_float_5, 'sentry.duration_ms') AND true) AS `__snuba_custom_column__count$sum(span.duration)`) FROM eap_items_1_dist WHERE in(project_id, [2]) AND equals(organization_id, 1) AND less(timestamp, toDateTime('2025-12-04 23:28:06')) AND greaterOrEquals(timestamp, toDateTime('2025-12-04 23:27:06')) AND ((notEquals(`sentry.op_TYPE_STRING`, 'pageload') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('pageload'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'navigation') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('navigation'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.render') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.render'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'interaction') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('interaction'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.action.swipe') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.action.swipe'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.action.scroll') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.action.scroll'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.action.click') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.action.click'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.action') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.action'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.load') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.load'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'app.lifecycle') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('app.lifecycle'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.interaction') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.interaction'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.interaction.click') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.interaction.click'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.interaction.hover') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.interaction.hover'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.interaction.drag') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.interaction.drag'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.interaction.press') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.interaction.press'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.webvital.cls') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.webvital.cls'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.webvital.lcp') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.webvital.lcp'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.webvital.fcp') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.webvital.fcp'))) OR equals(`sentry.op_TYPE_STRING`, 'http.server') OR isNull(`sentry.op_TYPE_STRING`) AND isNull('http.server')) AND (equals((cast(if(mapContains(attributes_float_26, 'sentry.is_segment'), arrayElement(attributes_float_26, 'sentry.is_segment'), NULL), 'Nullable(Boolean)') AS `sentry.is_segment_TYPE_BOOLEAN`), true) OR isNull(`sentry.is_segment_TYPE_BOOLEAN`) AND isNull(true)) AND (mapContains(attributes_float_5, 'sentry.duration_ms') OR mapContains(attributes_float_5, 'sentry.duration_ms') OR mapContains(attributes_string_31, 'sentry.user') OR mapContains(attributes_float_5, 'sentry.duration_ms')) AND equals(item_type, 1) GROUP BY is_starred_transaction, `sentry.request.method_TYPE_STRING`, `sentry.transaction_TYPE_STRING`, `sentry.op_TYPE_STRING`, project ORDER BY is_starred_transaction DESC, `sum(span.duration)` DESC LIMIT 52 OFFSET 0), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000d5af728
2025.12.04 23:28:07.167125 [ 900 ] {} <Error> TCPHandler: TCPHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket (peer: 10.57.1.59:50024, local: 10.57.2.78:9000). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):
2025.12.04 23:28:07.167282 [ 841 ] {} <Error> TCPHandler: TCPHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket (peer: 10.57.1.59:58768, local: 10.57.1.60:9000). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):


0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000d5af728
1. DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000d757248
2. DB::ReadBufferFromPocoSocketBase::socketReceiveBytesImpl(char*, unsigned long) @ 0x000000000d758bbc
3. DB::ReadBufferFromPocoSocketBase::nextImpl() @ 0x000000000d7592cc
4. DB::TCPHandler::runImpl() @ 0x00000000123e651c
5. DB::TCPHandler::run() @ 0x0000000012401e68
6. Poco::Net::TCPServerConnection::start() @ 0x000000001543bf58
7. Poco::Net::TCPServerDispatcher::run() @ 0x000000001543c474
8. Poco::PooledThread::run() @ 0x00000000154079bc
9. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015405d90
10. ? @ 0x000000000007d5b8
11. ? @ 0x00000000000e5edc
 (version 25.3.6.10034.altinitystable (altinity build))
  1. Don't see any timeouts

piontas avatar Dec 04 '25 23:12 piontas

👍 @Zylphrex does this stack trace help shine any light on this problem?

bcoe avatar Dec 05 '25 20:12 bcoe

@piontas How is your clickhouse cluster configured? I see references like __table1.attributes_string_14 which is not referring to the correct table.

Zylphrex avatar Dec 08 '25 19:12 Zylphrex

@Zylphrex here you go:

# config.xml
<?xml version="1.0"?>
    <yandex>
        <path>/var/lib/clickhouse/</path>
        <tmp_path>/var/lib/clickhouse/tmp/</tmp_path>
        <user_files_path>/var/lib/clickhouse/user_files/</user_files_path>
        <format_schema_path>/var/lib/clickhouse/format_schemas/</format_schema_path>

        <include_from>/etc/clickhouse-server/metrica.d/metrica.xml</include_from>

        <users_config>users.xml</users_config>

        <display_name>sentry-clickhouse</display_name>
        <listen_host>0.0.0.0</listen_host>
        <http_port>8123</http_port>
        <tcp_port>9000</tcp_port>
        <interserver_http_port>9009</interserver_http_port>
        <max_connections>4096</max_connections>
        <keep_alive_timeout>3</keep_alive_timeout>
        <max_concurrent_queries>100</max_concurrent_queries>
        <uncompressed_cache_size>8589934592</uncompressed_cache_size>
        <mark_cache_size>5368709120</mark_cache_size>
        <timezone>UTC</timezone>
        <umask>022</umask>
        <mlock_executable>false</mlock_executable>
        <remote_servers>
            <sentry-clickhouse>
                <shard>
                    <replica>
                        <internal_replication>true</internal_replication>
                        <host>sentry-clickhouse-0.sentry-clickhouse-headless</host>
                        <port>9000</port>
                        <user>default</user>
                        <compression>true</compression>
                    </replica>
                </shard>
                <shard>
                    <replica>
                        <internal_replication>true</internal_replication>
                        <host>sentry-clickhouse-1.sentry-clickhouse-headless</host>
                        <port>9000</port>
                        <user>default</user>
                        <compression>true</compression>
                    </replica>
                </shard>
                <shard>
                    <replica>
                        <internal_replication>true</internal_replication>
                        <host>sentry-clickhouse-2.sentry-clickhouse-headless</host>
                        <port>9000</port>
                        <user>default</user>
                        <compression>true</compression>
                    </replica>
                </shard>
            </sentry-clickhouse>
        </remote_servers>
        <merge_tree>
            <enable_mixed_granularity_parts>1</enable_mixed_granularity_parts>
            <!-- Number of parts after which INSERT into the table will be thrown -->
            <parts_to_throw_insert>600</parts_to_throw_insert>
            <!-- Number of parts after which INSERT into the table will be delayed -->
            <parts_to_delay_insert>200</parts_to_delay_insert>
            <!-- Max number of active parts in all partitions. Used to prevent out of disk space -->
            <max_parts_in_total>15000</max_parts_in_total>
            <max_suspicious_broken_parts>100</max_suspicious_broken_parts>
        </merge_tree>
        <macros incl="macros" optional="true" />
        <keeper_server replace="true">
            <tcp_port>9181</tcp_port>
            <server_id from_env="KEEPER_SERVER_ID" />
            <log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
            <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>
            <coordination_settings>
                <operation_timeout_ms>10000</operation_timeout_ms>
                <session_timeout_ms>30000</session_timeout_ms>
                <raft_logs_level>warning</raft_logs_level>
            </coordination_settings>
            <raft_configuration>
                <server>
                    <id>1</id>
                    <hostname>sentry-clickhouse-0.sentry-clickhouse-headless</hostname>
                    <port>9234</port>
                </server>
                <server>
                    <id>2</id>
                    <hostname>sentry-clickhouse-1.sentry-clickhouse-headless</hostname>
                    <port>9234</port>
                </server>
                <server>
                    <id>3</id>
                    <hostname>sentry-clickhouse-2.sentry-clickhouse-headless</hostname>
                    <port>9234</port>
                </server>
            </raft_configuration>
        </keeper_server>

        <builtin_dictionaries_reload_interval>3600</builtin_dictionaries_reload_interval>
        <max_session_timeout>3600</max_session_timeout>
        <default_session_timeout>60</default_session_timeout>
        <disable_internal_dns_cache>1</disable_internal_dns_cache>

        <distributed_ddl>
            <path>/clickhouse/task_queue/ddl</path>
        </distributed_ddl>
        <logger>
            <level>warning</level>
            <console>true</console>
        </logger>
        <query_thread_log remove="remove"/>
        <query_log remove="remove"/>
        <text_log remove="remove"/>
        <trace_log remove="remove"/>
        <metric_log remove="remove"/>
        <session_log remove="remove"/>
        <part_log remove="remove"/>
        <asynchronous_metric_log remove="remove"/>
        <allow_nullable_key>1</allow_nullable_key>
    </yandex>

# metrica.xml
<?xml version="1.0"?>
    <yandex>
        <clickhouse_remote_servers>
            <sentry-clickhouse>
                <shard>
                    <replica>
                        <internal_replication>true</internal_replication>
                        <host>sentry-clickhouse-0.sentry-clickhouse-headless</host>
                        <port>9000</port>
                        <user>default</user>
                        <compression>true</compression>
                    </replica>
                </shard>
                <shard>
                    <replica>
                        <internal_replication>true</internal_replication>
                        <host>sentry-clickhouse-1.sentry-clickhouse-headless</host>
                        <port>9000</port>
                        <user>default</user>
                        <compression>true</compression>
                    </replica>
                </shard>
                <shard>
                    <replica>
                        <internal_replication>true</internal_replication>
                        <host>sentry-clickhouse-2.sentry-clickhouse-headless</host>
                        <port>9000</port>
                        <user>default</user>
                        <compression>true</compression>
                    </replica>
                </shard>
            </sentry-clickhouse>
        </clickhouse_remote_servers>

        <macros>
            <replica from_env="HOSTNAME"></replica>
            <shard from_env="SHARD"></shard>
        </macros>
    </yandex>

# users.xml

<?xml version="1.0"?>
    <clickhouse>
      <profiles>
            <default>
                <distributed_ddl_task_timeout>180</distributed_ddl_task_timeout>
                <log_queries>0</log_queries>
                <log_query_threads>0</log_query_threads>
                <max_memory_usage>10000000000</max_memory_usage>
                <max_bytes_before_external_group_by>5000000000</max_bytes_before_external_group_by>
                <max_execution_time>600</max_execution_time>
            </default>
        </profiles>
       <users>
        <default>
          <password></password>
          <networks>
            <ip>::/0</ip>
          </networks>
          <profile>default</profile>
        </default>
      </users>
    </clickhouse>

# snuba settings.py - related Clishouse settings

SENTRY_DISTRIBUTED_CLICKHOUSE_TABLES = True

  # Clickhouse Options
  CLUSTERS = [
    {
      "host": env("CLICKHOUSE_HOST", "sentry-clickhouse"),
      "port": int(env("CLICKHOUSE_PORT", 9000)),
      "max_connections": int(os.environ.get("CLICKHOUSE_MAX_CONNECTIONS", 1)),
      "block_connections": bool(
          os.environ.get("CLICKHOUSE_BLOCK_CONNECTIONS", False)
      ),
      "user":  env("CLICKHOUSE_USER", "default"),
      "password": env("CLICKHOUSE_PASSWORD", ""),
      "database": env("CLICKHOUSE_DATABASE", "default"),
      "http_port": int(os.environ.get("CLICKHOUSE_HTTP_PORT", 8123)),
      "secure": os.environ.get("CLICKHOUSE_SECURE", "False").lower() in ("true", "1"),
      "ca_certs": os.environ.get("CLICKHOUSE_CA_CERTS"),
      "verify": os.environ.get("CLICKHOUSE_VERIFY"),
      "storage_sets": {
          "discover",
          "events",
          "events_ro",
          "metrics",
          "migrations",
          "outcomes",
          "querylog",
          "sessions",
          "transactions",
          "profiles",
          "functions",
          "replays",
          "generic_metrics_sets",
          "generic_metrics_distributions",
          "search_issues",
          "generic_metrics_counters",
          "spans",
          "events_analytics_platform",
          "group_attributes",
          "generic_metrics_gauges",
          "profile_chunks",
      },
      "single_node": False,
      "cluster_name": "sentry-clickhouse",
      "distributed_cluster_name": "sentry-clickhouse",
    },
  ]

piontas avatar Dec 17 '25 12:12 piontas

Since you're using distributed clickhouse, can you run the following query on your query nodes

SHOW CREATE TABLE eap_items_1_dist

and storage nodes:

SHOW CREATE TABLE eap_items_1_local

also, can you try running this query directly on your query node just to make sure we can reproduce the issue scoped to the clickhouse instance itself? (it is the query that is erroring out)

SELECT (transform(CAST(ifNull((coalesce(if(mapContains(attributes_string_20, 'sentry.transaction'), arrayElement(attributes_string_20, 'sentry.transaction'), NULL), if(mapContains(attributes_string_20, 'sentry.transaction'), arrayElement(attributes_string_20, 'sentry.transaction'), NULL), if(mapContains(attributes_string_3, 'transaction'), arrayElement(attributes_string_3, 'transaction'), NULL)) AS `sentry.transaction_TYPE_STRING`), ''), 'String'), [], [], 'false') AS is_starred_transaction), (if(mapContains(attributes_string_4, 'sentry.request.method'), arrayElement(attributes_string_4, 'sentry.request.method'), NULL) AS `sentry.request.method_TYPE_STRING`), `sentry.transaction_TYPE_STRING`, (if(mapContains(attributes_string_19, 'sentry.op'), arrayElement(attributes_string_19, 'sentry.op'), NULL) AS `sentry.op_TYPE_STRING`), (transform(CAST(ifNull((cast(project_id, 'Int64') AS `sentry.project_id_TYPE_INT`), ''), 'String'), ['2'], ['python-flask'], 'unknown') AS project), (divide((round(sumIfOrNull(divide(1, sampling_factor), mapContains(attributes_float_25, 'sentry.exclusive_time_ms') AND true)) AS `epm().left`), 1.0) AS `epm()`), `epm().left`, (multiply(1.96, sqrt(sumIf(minus(multiply(divide(1, sampling_factor), divide(1, sampling_factor)), divide(1, sampling_factor)), mapContains(attributes_float_25, 'sentry.exclusive_time_ms') AND true))) AS `__snuba_custom_column__confidence_interval$epm().left$function_type:count`), (divide(countIf((if(mapContains(attributes_float_25, 'sentry.exclusive_time_ms'), arrayElement(attributes_float_25, 'sentry.exclusive_time_ms'), NULL) AS `sentry.exclusive_time_ms_TYPE_DOUBLE`), mapContains(attributes_float_25, 'sentry.exclusive_time_ms') AND true), sumIf(divide(1, sampling_factor), mapContains(attributes_float_25, 'sentry.exclusive_time_ms') AND true)) AS `__snuba_custom_column__average_sample_rate$epm().left`), (countIf(`sentry.exclusive_time_ms_TYPE_DOUBLE`, mapContains(attributes_float_25, 'sentry.exclusive_time_ms') AND true) AS `__snuba_custom_column__count$epm().left`), 1.0, (quantileTDigestWeightedIfOrNull(0.5)((if(mapContains(attributes_float_5, 'sentry.duration_ms'), arrayElement(attributes_float_5, 'sentry.duration_ms'), NULL) AS `sentry.duration_ms_TYPE_DOUBLE`), cast(round(multiply(divide(1, sampling_factor), 1000)), 'UInt64'), mapContains(attributes_float_5, 'sentry.duration_ms') AND true) AS `p50(span.duration)`), (quantilesTDigestWeighted(0.4, 0.405, 0.41, 0.415, 0.42, 0.425, 0.43, 0.435, 0.44, 0.445, 0.45, 0.455, 0.46, 0.465, 0.47, 0.475, 0.48, 0.485, 0.49, 0.495, 0.5, 0.505, 0.51, 0.515, 0.52, 0.525, 0.53, 0.535, 0.54, 0.545, 0.55, 0.555, 0.56, 0.565, 0.57, 0.575, 0.58, 0.585, 0.59, 0.595)(`sentry.duration_ms_TYPE_DOUBLE`, cast(round(multiply(divide(1, sampling_factor), 1000)), 'UInt64')) AS `__snuba_custom_column__confidence_interval$p50(span.duration)$function_type:p50,granularity:0.005,width:0.1`), (divide(countIf(`sentry.duration_ms_TYPE_DOUBLE`, mapContains(attributes_float_5, 'sentry.duration_ms') AND true), sumIf(divide(1, sampling_factor), mapContains(attributes_float_5, 'sentry.duration_ms') AND true)) AS `__snuba_custom_column__average_sample_rate$p50(span.duration)`), (countIf(`sentry.duration_ms_TYPE_DOUBLE`, mapContains(attributes_float_5, 'sentry.duration_ms') AND true) AS `__snuba_custom_column__count$p50(span.duration)`), (quantileTDigestWeightedIfOrNull(0.95)(`sentry.duration_ms_TYPE_DOUBLE`, cast(round(multiply(divide(1, sampling_factor), 1000)), 'UInt64'), mapContains(attributes_float_5, 'sentry.duration_ms') AND true) AS `p95(span.duration)`), (quantilesTDigestWeighted(0.85, 0.855, 0.86, 0.865, 0.87, 0.875, 0.88, 0.885, 0.89, 0.895, 0.9, 0.905, 0.91, 0.915, 0.92, 0.925, 0.93, 0.935, 0.94, 0.945, 0.95, 0.955, 0.96, 0.965, 0.97, 0.975, 0.98, 0.985, 0.99, 0.995)(`sentry.duration_ms_TYPE_DOUBLE`, cast(round(multiply(divide(1, sampling_factor), 1000)), 'UInt64')) AS `__snuba_custom_column__confidence_interval$p95(span.duration)$function_type:p95,granularity:0.005,width:0.1`), (divide(countIf(`sentry.duration_ms_TYPE_DOUBLE`, mapContains(attributes_float_5, 'sentry.duration_ms') AND true), sumIf(divide(1, sampling_factor), mapContains(attributes_float_5, 'sentry.duration_ms') AND true)) AS `__snuba_custom_column__average_sample_rate$p95(span.duration)`), (countIf(`sentry.duration_ms_TYPE_DOUBLE`, mapContains(attributes_float_5, 'sentry.duration_ms') AND true) AS `__snuba_custom_column__count$p95(span.duration)`), (divide((round(sumIfOrNull(divide(1, sampling_factor), mapContains(attributes_string_14, 'sentry.status') AND (not(in((if(mapContains(attributes_string_14, 'sentry.status'), arrayElement(attributes_string_14, 'sentry.status'), NULL) AS `sentry.status_TYPE_STRING`), ['ok', 'cancelled', 'unknown'])) OR isNull(`sentry.status_TYPE_STRING`) AND not(has(['ok', 'cancelled', 'unknown'], NULL))))) AS `failure_rate().left`), (round(sumIfOrNull(divide(1, sampling_factor), mapContains(attributes_float_25, 'sentry.exclusive_time_ms') AND true)) AS `failure_rate().right`)) AS `failure_rate()`), `failure_rate().left`, (multiply(1.96, sqrt(sumIf(minus(multiply(divide(1, sampling_factor), divide(1, sampling_factor)), divide(1, sampling_factor)), mapContains(attributes_string_14, 'sentry.status') AND (not(in(`sentry.status_TYPE_STRING`, ['ok', 'cancelled', 'unknown'])) OR isNull(`sentry.status_TYPE_STRING`) AND not(has(['ok', 'cancelled', 'unknown'], NULL)))))) AS `__snuba_custom_column__confidence_interval$failure_rate().left$function_type:count`), (divide(countIf(`sentry.status_TYPE_STRING`, mapContains(attributes_string_14, 'sentry.status') AND (not(in(`sentry.status_TYPE_STRING`, ['ok', 'cancelled', 'unknown'])) OR isNull(`sentry.status_TYPE_STRING`) AND not(has(['ok', 'cancelled', 'unknown'], NULL)))), sumIf(divide(1, sampling_factor), mapContains(attributes_string_14, 'sentry.status') AND (not(in(`sentry.status_TYPE_STRING`, ['ok', 'cancelled', 'unknown'])) OR isNull(`sentry.status_TYPE_STRING`) AND not(has(['ok', 'cancelled', 'unknown'], NULL))))) AS `__snuba_custom_column__average_sample_rate$failure_rate().left`), (countIf(`sentry.status_TYPE_STRING`, mapContains(attributes_string_14, 'sentry.status') AND (not(in(`sentry.status_TYPE_STRING`, ['ok', 'cancelled', 'unknown'])) OR isNull(`sentry.status_TYPE_STRING`) AND not(has(['ok', 'cancelled', 'unknown'], NULL)))) AS `__snuba_custom_column__count$failure_rate().left`), `failure_rate().right`, (multiply(1.96, sqrt(sumIf(minus(multiply(divide(1, sampling_factor), divide(1, sampling_factor)), divide(1, sampling_factor)), mapContains(attributes_float_25, 'sentry.exclusive_time_ms') AND true))) AS `__snuba_custom_column__confidence_interval$failure_rate().right$function_type:count`), (divide(countIf(`sentry.exclusive_time_ms_TYPE_DOUBLE`, mapContains(attributes_float_25, 'sentry.exclusive_time_ms') AND true), sumIf(divide(1, sampling_factor), mapContains(attributes_float_25, 'sentry.exclusive_time_ms') AND true)) AS `__snuba_custom_column__average_sample_rate$failure_rate().right`), (countIf(`sentry.exclusive_time_ms_TYPE_DOUBLE`, mapContains(attributes_float_25, 'sentry.exclusive_time_ms') AND true) AS `__snuba_custom_column__count$failure_rate().right`), (uniqIfOrNull((if(mapContains(attributes_string_31, 'sentry.user'), arrayElement(attributes_string_31, 'sentry.user'), NULL) AS `sentry.user_TYPE_STRING`), mapContains(attributes_string_31, 'sentry.user') AND true) AS `count_unique(user)`), (divide(countIf(`sentry.user_TYPE_STRING`, mapContains(attributes_string_31, 'sentry.user') AND true), sumIf(divide(1, sampling_factor), mapContains(attributes_string_31, 'sentry.user') AND true)) AS `__snuba_custom_column__average_sample_rate$count_unique(user)`), (countIf(`sentry.user_TYPE_STRING`, mapContains(attributes_string_31, 'sentry.user') AND true) AS `__snuba_custom_column__count$count_unique(user)`), (sumIfOrNull(multiply(`sentry.duration_ms_TYPE_DOUBLE`, divide(1, sampling_factor)), mapContains(attributes_float_5, 'sentry.duration_ms') AND true) AS `sum(span.duration)`), (multiply(1.96, sqrt(sumIf(multiply(multiply(`sentry.duration_ms_TYPE_DOUBLE`, `sentry.duration_ms_TYPE_DOUBLE`), minus(multiply(divide(1, sampling_factor), divide(1, sampling_factor)), divide(1, sampling_factor))), mapContains(attributes_float_5, 'sentry.duration_ms') AND true))) AS `__snuba_custom_column__confidence_interval$sum(span.duration)$function_type:sum`), (divide(countIf(`sentry.duration_ms_TYPE_DOUBLE`, mapContains(attributes_float_5, 'sentry.duration_ms') AND true), sumIf(divide(1, sampling_factor), mapContains(attributes_float_5, 'sentry.duration_ms') AND true)) AS `__snuba_custom_column__average_sample_rate$sum(span.duration)`), (countIf(`sentry.duration_ms_TYPE_DOUBLE`, mapContains(attributes_float_5, 'sentry.duration_ms') AND true) AS `__snuba_custom_column__count$sum(span.duration)`) FROM eap_items_1_dist WHERE in(project_id, [2]) AND equals(organization_id, 1) AND less(timestamp, toDateTime('2025-12-04 23:28:06')) AND greaterOrEquals(timestamp, toDateTime('2025-12-04 23:27:06')) AND ((notEquals(`sentry.op_TYPE_STRING`, 'pageload') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('pageload'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'navigation') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('navigation'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.render') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.render'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'interaction') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('interaction'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.action.swipe') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.action.swipe'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.action.scroll') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.action.scroll'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.action.click') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.action.click'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.action') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.action'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.load') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.load'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'app.lifecycle') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('app.lifecycle'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.interaction') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.interaction'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.interaction.click') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.interaction.click'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.interaction.hover') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.interaction.hover'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.interaction.drag') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.interaction.drag'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.interaction.press') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.interaction.press'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.webvital.cls') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.webvital.cls'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.webvital.lcp') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.webvital.lcp'))) AND (notEquals(`sentry.op_TYPE_STRING`, 'ui.webvital.fcp') OR xor(isNull(`sentry.op_TYPE_STRING`), isNull('ui.webvital.fcp'))) OR equals(`sentry.op_TYPE_STRING`, 'http.server') OR isNull(`sentry.op_TYPE_STRING`) AND isNull('http.server')) AND (equals((cast(if(mapContains(attributes_float_26, 'sentry.is_segment'), arrayElement(attributes_float_26, 'sentry.is_segment'), NULL), 'Nullable(Boolean)') AS `sentry.is_segment_TYPE_BOOLEAN`), true) OR isNull(`sentry.is_segment_TYPE_BOOLEAN`) AND isNull(true)) AND (mapContains(attributes_float_5, 'sentry.duration_ms') OR mapContains(attributes_float_5, 'sentry.duration_ms') OR mapContains(attributes_string_31, 'sentry.user') OR mapContains(attributes_float_5, 'sentry.duration_ms')) AND equals(item_type, 1) GROUP BY is_starred_transaction, `sentry.request.method_TYPE_STRING`, `sentry.transaction_TYPE_STRING`, `sentry.op_TYPE_STRING`, project ORDER BY is_starred_transaction DESC, `sum(span.duration)` DESC LIMIT 52 OFFSET 0

volokluev avatar Dec 18 '25 21:12 volokluev