OpenMetadata icon indicating copy to clipboard operation
OpenMetadata copied to clipboard
verified publisher icon open-metadata

Profiler support for Complex Data Types (json, arrays, geo...)

Open sushi30 opened this issue 1 year ago • 3 comments

Is your feature request related to a problem? Please describe. OM currently manages a set of data types for which the profiler does not compute any metrics. This is inconvinent becuase there are metrics which are useful in the context of complex data types (null counts, size for data structures, uniqueness for geo).

Describe the solution you'd like

1. Handle nullCount for all data types.

2. Add 3 more groups of metrics:

  1. Collections (arrays, lists)
  2. Structs (json, etc...)
  3. Complex (Geo)

Handle each of these groups with specific sets of metrics to compute.

Describe alternatives you've considered

None...

Additional context

This is an example of profiler result for a Redshift table with complex data type columns. We can observe no metrics are collected for the SUPER, GEOMETRY and GEOGRAPHY columns.

image

sushi30 avatar Mar 20 '24 11:03 sushi30

hey @sushi30 I want to contribute to this, from where I can start. I am new.

samarth-jain28 avatar Mar 21 '24 11:03 samarth-jain28

@samarth-jain28 please connect on our slack and post a message in #contributor. That will be a more appropriate place to handle the discussion.

sushi30 avatar Mar 21 '24 11:03 sushi30

Hi @sushi30 , I noticed there's been no recent activity on this issue. If you're not working on it, could it be reassigned to me? I'd be happy to help.

Thanks!

BVK21 avatar Aug 15 '24 04:08 BVK21

May I be assigned this issue?

tristanhendry avatar Sep 06 '24 15:09 tristanhendry

@tristanhendry are you planning on contributing to this issue?

harshach avatar Dec 12 '24 19:12 harshach

No, I apologize for the inactivity and thank you for reaching out.

From: Sriharsha Chintalapani @.> Sent: Thursday, December 12, 2024 2:56 PM To: open-metadata/OpenMetadata @.> Cc: Hendry, Tristan R. @.>; Mention @.> Subject: Re: [open-metadata/OpenMetadata] Profiler support for Complex Data Types (json, arrays, geo...) (Issue #15627)

[External Email]

@tristanhendryhttps://github.com/tristanhendry are you planning on contributing to this issue?

— Reply to this email directly, view it on GitHubhttps://github.com/open-metadata/OpenMetadata/issues/15627#issuecomment-2539891672, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BDGSGUE2J7F67FXRBZPS6QD2FHS6VAVCNFSM6AAAAABE7IA4W2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMZZHA4TCNRXGI. You are receiving this because you were mentioned.Message ID: @.***>

tristanhendry avatar Dec 12 '24 20:12 tristanhendry