Some fields cannot be further parsed in avro schema

[xxniuren]

Describe the bug I ingested avro schema into datahub, and found some nest fields cannot be expanded further.

To Reproduce Steps to reproduce the behavior:

1. Ingest avro schema as follows

{
  "type": "record",
  "name": "Payment",
  "namespace": "some.event.namespace",
  "fields": [
    {
        "name":"networks",
        "type":[
            "null",
            {
                "type":"record",
                "name":"Networks",
                "namespace":"some.event.namespace.report",
                "fields":[
                    {
                        "name":"cro",
                        "type":[
                            "null",
                            {
                                "type":"record",
                                "name":"Network",
                                "fields":[
                                    {
                                        "name":"role",
                                        "type":[
                                            "null",
                                            "string"
                                        ]
                                    }
                                ]
                            }
                        ]
                    },
                    {
                        "name":"distributors",
                        "type":[
                            "null",
                            {
                                "type":"array",
                                "items":"Network"
                            }
                        ]
                    },
                    {
                        "name":"entire_chain",
                        "type":[
                            "null",
                            {
                                "type":"array",
                                "items":"Network"
                            }
                        ]
                    }
                ]
            }
        ]
    }
  ]
}

2. The output schema information is as follows: Column 'entire_chain' can be parsed further.

Expected behavior Expect output schema information is as follows:

[xxniuren]

I don't think it is as expect. actual: Field 'entire_chain' can not be expand further. expect: Field 'entire_chain' can be expand further.

[laulpogan]

@hsheth2 is this outside of expected behavior?

[xxniuren]

@laulpogan, Thanks for your reply.

1. This behavior may be not expected.
2. From the user point of view, I want each field can be expanded step by step until it reaches the base type. FYI @hsheth2

[hsheth2]

@xxniuren you're right that this is a known bug.

syu on the DataHub Slack is actually already working on a fix for this

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io

[hsheth2]

@xxniuren are you still working on this?

[xxniuren]

Hello Harshal Sheth, I was on vacation recently, sorry for not reply to your message. I try to fix this bug, but not found any solution, so I am not working on this. If you fix it, please tell me, thanks.

获取 Outlook for iOShttps://aka.ms/o0ukef

---

发件人: Harshal Sheth @.> 发送时间: Saturday, January 14, 2023 4:11:20 AM 收件人: datahub-project/datahub @.> 抄送: xxniuren @.>; Mention @.> 主题: Re: [datahub-project/datahub] Some fields cannot be further parsed in avro schema (Issue #6497)

@xxniurenhttps://github.com/xxniuren are you still working on this?

― Reply to this email directly, view it on GitHubhttps://github.com/datahub-project/datahub/issues/6497#issuecomment-1382339039, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AELN6EHOPRQCPNWDOYORHGLWSGZGRANCNFSM6AAAAAASGFGSCU. You are receiving this because you were mentioned.Message ID: @.***>

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io

[github-actions[bot]]

This issue was closed because it has been inactive for 30 days since being marked as stale.