pinot Support maxNumRowsPerTask in RealtimeToOfflineSegmentsTask

Problem: Currently RealtimeToOfflineSegmentsTask that is used to move real time segments to offline segments, does not have the ability to tune maxNumRowsPerTask. This is the parameter that determines the input to a task. Without this configuration, we end up creating one minion task, which takes in all the input (i.e all segments that meet the criteria to be converted to offline segments) which prevents us from using other minions. There's no parallelism for this task.

Apr 09 '24 16:04 swaminathanmanish

cc @aishikbh - Please take a look

Apr 09 '24 16:04 swaminathanmanish

I'd like to take this up!

Jun 20 '24 07:06 pratikpugalia

@pratikpugalia - Would you be working on this now or taking it up later ? If not can I reassign to someone else. Thanks !

Oct 08 '24 12:10 swaminathanmanish

@swaminathanmanish I would be taking this up by the end of the month if that is okay! sorry for the delay!

Oct 08 '24 23:10 pratikpugalia

@pratikpugalia - We have a slightly urgent ask to pick this up to address scaling issues in this task, thats happening now. Could I or someone from my team pick this up and you could pick something else ?

Oct 20 '24 14:10 swaminathanmanish

Hey @swaminathanmanish No worries, I'll look at other issues I could possibly pickup!

Oct 20 '24 20:10 pratikpugalia

Hey @swaminathanmanish No worries, I'll look at other issues I could possibly pickup!

Thanks @pratikpugalia

Hey @swaminathanmanish No worries, I'll look at other issues I could possibly pickup!

Thanks @pratikpugalia

Oct 21 '24 09:10 swaminathanmanish