OM v1.3.0 unable to ingest json with array element

[haonstate]

Affected module I believe it impact the Ingestion Framework thus the UI can only display what can be ingested.

Describe the bug Use the following link, https://docs.open-metadata.org/v1.3.x/connectors/storage/s3/yaml. Created a S3 ingestion for JSON files. The data in the JSON contains array elements. Example of a file we tested has the following structure: { "data": { "users": [ { "id": 1, "name": "John Doe", "email": "[email protected]" }, { "id": 2, "name": "Jane Smith", "email": "[email protected]" } ] } }

After ingestion, only "data" and "users" fields are visible in the UI. The fields in the array will NOT appear in the UI.

To Reproduce

1. Upload a json file with the json data provided earlier to a S3 location
2. Follow the link for S3 external ingestion, also provided earlier.
3. View the schema in the UI or use API to get the schema.

Expected behavior The JSON schema after ingestion should includes: "data" > "users" > "id" "name" "email"

Version:

• OS: [e.g. iOS] AWS EC2 with default linux OS
• Python version: Used 3.9.x
• OpenMetadata version: [e.g. 0.8]. OM v1.3.0
• OpenMetadata Ingestion package version: [e.g. openmetadata-ingestion[docker]==XYZ] 'openmetadata-ingestion[athena]==1.3.0'

Additional context Add any other context about the problem here.

[ayush-shah]

the above is a expected behavior, we do not flatten the array.

[ayush-shah]

We currently are not unflattening the above. we do though ingest Json and array elements for all connectors..not the nested part under array That is an expected behavior