cqex
cqex copied to clipboard
Published
20 hours ago •
cqerl
cqerl
Fix: Allow filtering empty page enumerable premature reduce exit
Because of ALLOW FILTERING, it is possible for a Scylla query to return pages of rows which are often empty, followed by non empty pages. This causes CQEx's enumerable to fail to iterate past the empty page and therefore not stream all the results
I populate my table (partitioned by id) where 5% (by random chance) of the events match the WHERE clause and 95% do not
cqlsh:dev> SELECT id, ad_source FROM events_v4 WHERE ad_source = 'facebook' ALLOW FILTERING ; # Then me holding down ENTER
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
8c74bfc7-700f-4ab5-b513-e15c7713ac46 | facebook
---MORE---
id | ad_source
--------------------------------------+-----------
24c80ed1-e73e-4a3b-9687-abe91c4fb7ee | facebook
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
01d19351-80f6-4349-a162-14d3edaee365 | facebook
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
de43a3c9-92af-4cac-af38-ee0fb7857e41 | facebook
202bb937-9da4-4e27-9d9a-a1a3ab13b8ab | facebook
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
56828ab1-6c45-410a-89ed-1f0ec5d2198b | facebook
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
a87b7517-78e5-4103-b7e5-0ad4d3e97099 | facebook
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
7fa8a5f7-3631-474d-80e8-0fba1c9a07ec | facebook
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
432e9c1c-2e14-432c-998b-1ddce6442543 | facebook
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
f6c2c9d8-96bd-4b66-8f0d-7e8b329d3815 | facebook
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
45b119c9-ab53-49c8-bd08-9ffd51920e24 | facebook
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
c9132aac-af54-4550-b382-aba7a3784b99 | facebook
---MORE---
id | ad_source
--------------------------------------+-----------
7c6b0afd-8654-4018-886d-8c0d2648465a | facebook
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
f53ada20-cd2f-4915-b0ba-c046d5ab897b | facebook
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
d0f34c5b-d690-4d80-b345-df10d077982a | facebook
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
053f5200-c862-46d4-aef8-5c2162abaafc | facebook
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
fe5190e2-a546-4035-858b-76544c0f08b9 | facebook
---MORE---
id | ad_source
--------------------------------------+-----------
b6aa50f9-f165-40b7-ab6f-13375a9eecd6 | facebook
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
4d369465-233d-4539-83ce-8ee3e711b0a0 | facebook
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
cc44dfbb-c197-4157-8b5d-ec068176724a | facebook
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
7437131b-8449-47df-b013-c9afde14ae7e | facebook
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
2d3c6594-8505-4f10-95c6-d1579c44a21b | facebook
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
fab43882-f7f3-4123-991d-be7941427e33 | facebook
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
fb0ca097-29dc-4008-bd34-5d291013d42e | facebook
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
892226fe-3559-42b6-b74e-8372c04b7803 | facebook
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
2996bba4-875b-496e-bca5-e179c68d1cbe | facebook
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
068a3b11-6f92-4740-ae82-41fe86f6d547 | facebook
---MORE---
id | ad_source
--------------------------------------+-----------
caa21abe-ed65-484b-8770-4bf9319a0e67 | facebook
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
df2e824b-d42e-49ab-97f9-3996b9a8112f | facebook
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
7f105c85-b5bd-4402-a720-9671117a91ee | facebook
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
43673514-2957-4a62-9127-e73154de53f6 | facebook
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
7dcce63f-4469-44ab-812e-ea5075250fd0 | facebook
---MORE---
id | ad_source
--------------------------------------+-----------
b3e8ed89-5fa4-4817-9b22-ab1c11546e53 | facebook
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
ac9eb32d-9c2a-4a64-853f-13d03d5b8acf | facebook
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
d81bc50f-04eb-4fb6-86d7-e71545acf3f0 | facebook
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
6bd2fdc3-4940-4ff2-9986-5c215a237662 | facebook
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
e349984c-4383-4905-8601-f0625ee7bf12 | facebook
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
id | ad_source
--------------------------------------+-----------
49f00387-57e5-44a9-b42d-61221587a6a3 | facebook
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
(38 rows)
cqlsh:dev>
cqlsh:dev>
As you can see there are many empty pages by there are still several events
I now try this using CQEx and find no events
defmodule T do
def t do
IO.puts("START")
query = %CQEx.Query{
statement: """
SELECT id, ad_source
FROM events_v4
WHERE ad_source = 'facebook'
ALLOW FILTERING
""",
named: "fix_mmmmm_fb_event_query"
}
CQEx.Client.new!()
|> CQEx.Query.call!(query)
|> Enum.each(fn event ->
event[:id] |> IO.inspect(label: "found event ")
end)
IO.puts("END")
end
end
iex(85)> T.t
START
END
:ok
I now run this again but with redbug
iex(87)> :redbug.start('cqerl->return', msgs: 10000, time: 15_000)
{2144, 36}
iex(88)> T.t
START
% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:get_client()
% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:get_client/0 -> {ok,{<0.4459.0>,#Ref<0.2140264969.181403653.147479>}}
% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:run_query({<0.4459.0>,#Ref<0.2140264969.181403653.147479>}, {cql_query,<<"SELECT id, ad_source\nFROM events_v4\nWHERE ad_source = 'facebook'\nALLOW FILTERING\n">>,
#{},undefined,<<"fix_mmmmm_fb_event_query">>,100,undefined,1,
undefined,undefined})
% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:run_query({<0.4459.0>,#Ref<0.2140264969.181403653.147479>}, {cql_query,<<"SELECT id, ad_source\nFROM events_v4\nWHERE ad_source = 'facebook'\nALLOW FILTERING\n">>,
#{},undefined,<<"fix_mmmmm_fb_event_query">>,100,undefined,1,
undefined,undefined}, 2)
% 15:24:31 <0.4459.0>({cqerl_client,init,1})
% cqerl:get_protocol_version()
% 15:24:31 <0.4459.0>({cqerl_client,init,1})
% cqerl:get_protocol_version/0 -> 4
% 15:24:31 <0.3006.3>({cqerl_processor,process,4})
% cqerl:put_protocol_version(4)
% 15:24:31 <0.3006.3>({cqerl_processor,process,4})
% cqerl:put_protocol_version/1 -> undefined
% 15:24:31 <0.3006.3>(dead)
% cqerl:get_protocol_version()
% 15:24:31 <0.3006.3>(dead)
% cqerl:get_protocol_version/0 -> 4
% 15:24:31 <0.4459.0>({cqerl_client,init,1})
% cqerl:get_protocol_version()
% 15:24:31 <0.4459.0>({cqerl_client,init,1})
% cqerl:get_protocol_version/0 -> 4
% 15:24:31 <0.3007.3>(dead)
% cqerl:put_protocol_version(4)
% 15:24:31 <0.3007.3>(dead)
% cqerl:put_protocol_version/1 -> undefined
% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:run_query/3 -> {ok,{cql_result,
[{cqerl_result_column_spec,<<"dev">>,
<<"events_v4">>,id,varchar},
{cqerl_result_column_spec,<<"dev">>,
<<"events_v4">>,ad_source,varchar}],
[],
{cql_query,
<<"SELECT id, ad_source\nFROM events_v4\nWHERE ad_source = 'facebook'\nALLOW FILTERING\n">>,
#{},undefined,
<<"fix_mmmmm_fb_event_query">>,100,
<<0,0,0,0,163,0,0,0,48,0,0,0,1,0,0,0,36,0,0,
0,101,49,50,98,52,48,50,101,45,50,54,48,
56,45,52,102,49,97,45,56,57,48,100,45,99,
53,99,54,54,97,51,101,100,56,54,101,1,20,
0,0,0,1,0,0,0,8,0,0,0,0,0,1,126,110,162,
112,184,255,255,255,255,188,74,251,163,
143,140,21,94,93,101,253,253,167,16,244,
164,1,0,0,0,32,0,0,0,0,1,25,0,0,0,20,0,0,
0,1,0,0,0,8,0,0,0,129,118,159,166,31,252,
91,35,1,0,1,0,0,0,6,78,180,144,10,181,28,
97,213,110,183,148,64,75,31,131,1,0,0,0,0,
0,255,255,255,255,0,0,0,0>>,
1,undefined,undefined},
{<0.4459.0>,
#Ref<0.2140264969.181403653.147479>}}}
% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:run_query/2 -> {ok,{cql_result,
[{cqerl_result_column_spec,<<"dev">>,
<<"events_v4">>,id,varchar},
{cqerl_result_column_spec,<<"dev">>,
<<"events_v4">>,ad_source,varchar}],
[],
{cql_query,
<<"SELECT id, ad_source\nFROM events_v4\nWHERE ad_source = 'facebook'\nALLOW FILTERING\n">>,
#{},undefined,
<<"fix_mmmmm_fb_event_query">>,100,
<<0,0,0,0,163,0,0,0,48,0,0,0,1,0,0,0,36,0,0,
0,101,49,50,98,52,48,50,101,45,50,54,48,
56,45,52,102,49,97,45,56,57,48,100,45,99,
53,99,54,54,97,51,101,100,56,54,101,1,20,
0,0,0,1,0,0,0,8,0,0,0,0,0,1,126,110,162,
112,184,255,255,255,255,188,74,251,163,
143,140,21,94,93,101,253,253,167,16,244,
164,1,0,0,0,32,0,0,0,0,1,25,0,0,0,20,0,0,
0,1,0,0,0,8,0,0,0,129,118,159,166,31,252,
91,35,1,0,1,0,0,0,6,78,180,144,10,181,28,
97,213,110,183,148,64,75,31,131,1,0,0,0,0,
0,255,255,255,255,0,0,0,0>>,
1,undefined,undefined},
{<0.4459.0>,
#Ref<0.2140264969.181403653.147479>}}}
% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:size({cql_result,
[{cqerl_result_column_spec,<<"dev">>,<<"events_v4">>,id,varchar},
{cqerl_result_column_spec,<<"dev">>,<<"events_v4">>,ad_source,varchar}],
[],
{cql_query,
<<"SELECT id, ad_source\nFROM events_v4\nWHERE ad_source = 'facebook'\nALLOW FILTERING\n">>,
#{},undefined,<<"fix_mmmmm_fb_event_query">>,100,
<<0,0,0,0,163,0,0,0,48,0,0,0,1,0,0,0,36,0,0,0,101,49,50,98,52,48,50,
101,45,50,54,48,56,45,52,102,49,97,45,56,57,48,100,45,99,53,99,54,54,
97,51,101,100,56,54,101,1,20,0,0,0,1,0,0,0,8,0,0,0,0,0,1,126,110,162,
112,184,255,255,255,255,188,74,251,163,143,140,21,94,93,101,253,253,
167,16,244,164,1,0,0,0,32,0,0,0,0,1,25,0,0,0,20,0,0,0,1,0,0,0,8,0,0,
0,129,118,159,166,31,252,91,35,1,0,1,0,0,0,6,78,180,144,10,181,28,97,
213,110,183,148,64,75,31,131,1,0,0,0,0,0,255,255,255,255,0,0,0,0>>,
1,undefined,undefined},
{<0.4459.0>,#Ref<0.2140264969.181403653.147479>}})
% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:size/1 -> 0
% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:has_more_pages({cql_result,
[{cqerl_result_column_spec,<<"dev">>,<<"events_v4">>,id,varchar},
{cqerl_result_column_spec,<<"dev">>,<<"events_v4">>,ad_source,varchar}],
[],
{cql_query,
<<"SELECT id, ad_source\nFROM events_v4\nWHERE ad_source = 'facebook'\nALLOW FILTERING\n">>,
#{},undefined,<<"fix_mmmmm_fb_event_query">>,100,
<<0,0,0,0,163,0,0,0,48,0,0,0,1,0,0,0,36,0,0,0,101,49,50,98,52,48,50,
101,45,50,54,48,56,45,52,102,49,97,45,56,57,48,100,45,99,53,99,54,54,
97,51,101,100,56,54,101,1,20,0,0,0,1,0,0,0,8,0,0,0,0,0,1,126,110,162,
112,184,255,255,255,255,188,74,251,163,143,140,21,94,93,101,253,253,
167,16,244,164,1,0,0,0,32,0,0,0,0,1,25,0,0,0,20,0,0,0,1,0,0,0,8,0,0,
0,129,118,159,166,31,252,91,35,1,0,1,0,0,0,6,78,180,144,10,181,28,97,
213,110,183,148,64,75,31,131,1,0,0,0,0,0,255,255,255,255,0,0,0,0>>,
1,undefined,undefined},
{<0.4459.0>,#Ref<0.2140264969.181403653.147479>}})
% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:has_more_pages/1 -> true
% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:fetch_more({cql_result,
[{cqerl_result_column_spec,<<"dev">>,<<"events_v4">>,id,varchar},
{cqerl_result_column_spec,<<"dev">>,<<"events_v4">>,ad_source,varchar}],
[],
{cql_query,
<<"SELECT id, ad_source\nFROM events_v4\nWHERE ad_source = 'facebook'\nALLOW FILTERING\n">>,
#{},undefined,<<"fix_mmmmm_fb_event_query">>,100,
<<0,0,0,0,163,0,0,0,48,0,0,0,1,0,0,0,36,0,0,0,101,49,50,98,52,48,50,
101,45,50,54,48,56,45,52,102,49,97,45,56,57,48,100,45,99,53,99,54,54,
97,51,101,100,56,54,101,1,20,0,0,0,1,0,0,0,8,0,0,0,0,0,1,126,110,162,
112,184,255,255,255,255,188,74,251,163,143,140,21,94,93,101,253,253,
167,16,244,164,1,0,0,0,32,0,0,0,0,1,25,0,0,0,20,0,0,0,1,0,0,0,8,0,0,
0,129,118,159,166,31,252,91,35,1,0,1,0,0,0,6,78,180,144,10,181,28,97,
213,110,183,148,64,75,31,131,1,0,0,0,0,0,255,255,255,255,0,0,0,0>>,
1,undefined,undefined},
{<0.4459.0>,#Ref<0.2140264969.181403653.147479>}})
% 15:24:31 <0.4459.0>({cqerl_client,init,1})
% cqerl:get_protocol_version()
% 15:24:31 <0.4459.0>({cqerl_client,init,1})
% cqerl:get_protocol_version/0 -> 4
% 15:24:31 <0.3008.3>(dead)
% cqerl:put_protocol_version(4)
% 15:24:31 <0.3008.3>(dead)
% cqerl:put_protocol_version/1 -> undefined
% 15:24:31 <0.3008.3>(dead)
% cqerl:get_protocol_version()
% 15:24:31 <0.3008.3>(dead)
% cqerl:get_protocol_version/0 -> 4
% 15:24:31 <0.4459.0>({cqerl_client,init,1})
% cqerl:get_protocol_version()
END
% 15:24:31 <0.4459.0>({cqerl_client,init,1})
% cqerl:get_protocol_version/0 -> 4
% 15:24:31 <0.3009.3>(dead)
% cqerl:put_protocol_version(4)
% 15:24:31 <0.3009.3>(dead)
% cqerl:put_protocol_version/1 -> undefined
:ok
% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:fetch_more/1 -> {ok,{cql_result,
[{cqerl_result_column_spec,<<"dev">>,
<<"events_v4">>,id,varchar},
{cqerl_result_column_spec,<<"dev">>,
<<"events_v4">>,ad_source,varchar}],
[],
{cql_query,
<<"SELECT id, ad_source\nFROM events_v4\nWHERE ad_source = 'facebook'\nALLOW FILTERING\n">>,
#{},undefined,
<<"fix_mmmmm_fb_event_query">>,100,
<<0,0,0,0,9,1,0,0,48,0,0,0,1,0,0,0,36,0,0,
0,48,48,56,48,53,100,55,51,45,48,51,98,
99,45,52,102,48,98,45,98,101,101,50,45,
100,97,48,100,55,98,49,55,55,55,101,54,1,
20,0,0,0,1,0,0,0,8,0,0,0,0,0,1,126,111,
182,192,210,255,255,255,255,188,74,251,
163,143,140,21,94,93,101,253,253,167,16,
244,164,2,0,0,0,57,0,0,0,1,25,0,0,0,20,0,
0,0,1,0,0,0,8,0,0,0,133,10,128,146,212,
212,104,171,0,1,25,0,0,0,20,0,0,0,1,0,0,
0,8,0,0,0,134,20,223,84,240,81,85,118,1,
0,1,0,0,0,6,78,180,144,10,181,28,97,213,
110,183,148,64,75,31,131,57,0,0,0,1,25,0,
0,0,20,0,0,0,1,0,0,0,8,0,0,0,129,118,159,
166,31,252,91,35,0,1,25,0,0,0,20,0,0,0,1,
0,0,0,8,0,0,0,133,10,128,146,212,212,104,
171,1,0,1,0,0,0,6,78,180,144,10,181,28,
97,213,110,183,148,64,75,31,131,1,0,0,0,
0,0,255,255,255,255,0,0,0,0>>,
1,undefined,undefined},
{<0.4459.0>,
#Ref<0.2140264969.181403653.147479>}}}
% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:next({cql_result,
[{cqerl_result_column_spec,<<"dev">>,<<"events_v4">>,id,varchar},
{cqerl_result_column_spec,<<"dev">>,<<"events_v4">>,ad_source,varchar}],
[],
{cql_query,
<<"SELECT id, ad_source\nFROM events_v4\nWHERE ad_source = 'facebook'\nALLOW FILTERING\n">>,
#{},undefined,<<"fix_mmmmm_fb_event_query">>,100,
<<0,0,0,0,9,1,0,0,48,0,0,0,1,0,0,0,36,0,0,0,48,48,56,48,53,100,55,51,
45,48,51,98,99,45,52,102,48,98,45,98,101,101,50,45,100,97,48,100,55,
98,49,55,55,55,101,54,1,20,0,0,0,1,0,0,0,8,0,0,0,0,0,1,126,111,182,
192,210,255,255,255,255,188,74,251,163,143,140,21,94,93,101,253,253,
167,16,244,164,2,0,0,0,57,0,0,0,1,25,0,0,0,20,0,0,0,1,0,0,0,8,0,0,0,
133,10,128,146,212,212,104,171,0,1,25,0,0,0,20,0,0,0,1,0,0,0,8,0,0,0,
134,20,223,84,240,81,85,118,1,0,1,0,0,0,6,78,180,144,10,181,28,97,
213,110,183,148,64,75,31,131,57,0,0,0,1,25,0,0,0,20,0,0,0,1,0,0,0,8,
0,0,0,129,118,159,166,31,252,91,35,0,1,25,0,0,0,20,0,0,0,1,0,0,0,8,0,
0,0,133,10,128,146,212,212,104,171,1,0,1,0,0,0,6,78,180,144,10,181,
28,97,213,110,183,148,64,75,31,131,1,0,0,0,0,0,255,255,255,255,0,0,0,
0>>,
1,undefined,undefined},
{<0.4459.0>,#Ref<0.2140264969.181403653.147479>}})
% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:next/1 -> empty_dataset
what is suspicious to me is the last call: % cqerl:has_more_pages/1 -> true which returns true.
This I guess indicates the Enumerable recursion exits before cqerl:has_more_pages can return false.
This last line % cqerl:next/1 -> empty_dataset indicates that we prematurely exit.
defp maybe_fetch_and_continue(result, acc, fun) do
case R.has_more_pages?(result) do
true ->
next_page = R.fetch_more!(result)
case R.next(next_page) do ######### returns `:empty_dataset`
{h, t} -> reduce(t, fun.(h, acc), fun)
:empty_dataset -> {:done, acc} ######### this prematurely exists on an empty page
end
false ->
{:done, acc}
end
end
Now it works! 🥳
iex(1)> T.t
START
found event : "8c74bfc7-700f-4ab5-b513-e15c7713ac46"
found event : "24c80ed1-e73e-4a3b-9687-abe91c4fb7ee"
found event : "01d19351-80f6-4349-a162-14d3edaee365"
found event : "de43a3c9-92af-4cac-af38-ee0fb7857e41"
found event : "202bb937-9da4-4e27-9d9a-a1a3ab13b8ab"
found event : "56828ab1-6c45-410a-89ed-1f0ec5d2198b"
found event : "a87b7517-78e5-4103-b7e5-0ad4d3e97099"
found event : "7fa8a5f7-3631-474d-80e8-0fba1c9a07ec"
found event : "432e9c1c-2e14-432c-998b-1ddce6442543"
found event : "f6c2c9d8-96bd-4b66-8f0d-7e8b329d3815"
found event : "45b119c9-ab53-49c8-bd08-9ffd51920e24"
found event : "c9132aac-af54-4550-b382-aba7a3784b99"
found event : "7c6b0afd-8654-4018-886d-8c0d2648465a"
found event : "f53ada20-cd2f-4915-b0ba-c046d5ab897b"
found event : "d0f34c5b-d690-4d80-b345-df10d077982a"
found event : "053f5200-c862-46d4-aef8-5c2162abaafc"
found event : "fe5190e2-a546-4035-858b-76544c0f08b9"
found event : "b6aa50f9-f165-40b7-ab6f-13375a9eecd6"
found event : "4d369465-233d-4539-83ce-8ee3e711b0a0"
found event : "cc44dfbb-c197-4157-8b5d-ec068176724a"
found event : "7437131b-8449-47df-b013-c9afde14ae7e"
found event : "2d3c6594-8505-4f10-95c6-d1579c44a21b"
found event : "fab43882-f7f3-4123-991d-be7941427e33"
found event : "fb0ca097-29dc-4008-bd34-5d291013d42e"
found event : "892226fe-3559-42b6-b74e-8372c04b7803"
found event : "2996bba4-875b-496e-bca5-e179c68d1cbe"
found event : "068a3b11-6f92-4740-ae82-41fe86f6d547"
found event : "caa21abe-ed65-484b-8770-4bf9319a0e67"
found event : "df2e824b-d42e-49ab-97f9-3996b9a8112f"
found event : "7f105c85-b5bd-4402-a720-9671117a91ee"
found event : "43673514-2957-4a62-9127-e73154de53f6"
found event : "7dcce63f-4469-44ab-812e-ea5075250fd0"
found event : "b3e8ed89-5fa4-4817-9b22-ab1c11546e53"
found event : "ac9eb32d-9c2a-4a64-853f-13d03d5b8acf"
found event : "d81bc50f-04eb-4fb6-86d7-e71545acf3f0"
found event : "6bd2fdc3-4940-4ff2-9986-5c215a237662"
found event : "e349984c-4383-4905-8601-f0625ee7bf12"
found event : "49f00387-57e5-44a9-b42d-61221587a6a3"
END
:ok
This affects scylla 3.x and 4.x. Have not tried Cassandra
This issue will also affect the defp find function as well
My god, this issue has plagued us for a few years now. I'm so happy it's fixed now 😭 😄
Omg I would love for this to get merged. This has been bugging me for so long. Can't wait for code review
