velox icon indicating copy to clipboard operation
velox copied to clipboard

add substrait extension yaml files for further function registry

Open chaojun-zhang opened this issue 2 years ago • 5 comments

copy substrait extension yaml files from https://github.com/substrait-io/substrait/tree/main/extensions for further function registry

chaojun-zhang avatar Aug 17 '22 11:08 chaojun-zhang

Hi @chaojun-zhang. For future reference, could you add more context on what these extensions are and why they are needed?

pedroerp avatar Aug 17 '22 16:08 pedroerp

Hi @chaojun-zhang. For future reference, could you add more context on what these extensions are and why they are needed?

When convert a velox plan into substrait plan, we have logic to generate substrait function signature to represent function signature in velox , and these Substrait YAML extension files will be used as input to establish the function signatures instead of a hard-coded function signatures here

chaojun-zhang avatar Aug 17 '22 23:08 chaojun-zhang

@chaojun-zhang ChaoJun, would it be possible to provide the overall design for how the function mappings will be handled in the Substrat-to-Velox plan conversions? I see that functions are currently hard-coded and hope we can move away from doing that. However, it is not clear to me what the new logic will be.

mbasmanova avatar Aug 22 '22 14:08 mbasmanova

@chaojun-zhang ChaoJun, would it be possible to provide the overall design for how the function mappings will be handled in the Substrat-to-Velox plan conversions? I see that functions are currently hard-coded and hope we can move away from doing that. However, it is not clear to me what the new logic will be.

Sure, let's schedule a team meeting for this, how about 5:00-6:00 PM the day on 24/08 or 25/08 based on your time zone ? or you can provide your available time slot?

chaojun-zhang avatar Aug 23 '22 06:08 chaojun-zhang

@chaojun-zhang Let's try to find a time slot on Slack. We would also need a written design document. Do you have something I can read ahead of the meeting?

mbasmanova avatar Aug 23 '22 07:08 mbasmanova