Incomplete or Broken Lineage Among Custom Storage Container entities

[darshanik]

Affected module UI, Backend API

Describe the bug I’m trying to create lineage between 2 custom storage container entities. The lineage created is fine and as expected when the container entities have up to 5 nested containers.

However, if 6th storage container is added, the lineage is not being displayed even after hitting the "entity-side-bar" (the "+" button to expand the remaining downstream or upstream).

To Reproduce

Create a nested Custom Container A with FQN:

Custom_test_storage.Sample_Test_Parent_Dir.Sample_Test_Sub_Dir_1.Sample_Test_Sub_Dir_2.Sample_Test_Sub_Dir_3.Sample_Test_Sub_Dir_4.Sample_Test_Sub_Dir_5."YYYY_MM_DD_Sample_Test_File.txt"

Similarly Create another Custom Storage Container entity B with FQN:

Custom_Staging_Service.Temp_Parent_Dir.Temp_Sub_Dir_1.Temp_Sub_Dir_2.Temp_Sub_Dir_3.Temp_Sub_Dir_4.Temp_Sub_Dir_5."YYYY_MM_DD_Sample_Test_File.txt"

And then Create a sample Airflow Pipeline entity to create lineage edge between above 2 nested container entities.

Sample_Pipeline_FQN: Test_airflow_service.Sample_Airflow_Source_Pipeline

Finally, Create Lineage Edge between Container A and Container B:

From_entity: nested Custom Container A To_Entity: nested Custom Container B LineageDetails: Test_airflow_service.Sample_Airflow_Source_Pipeline

Screenshots or steps to reproduce

Expected behavior

Version:

• OS: Linux/Red Hat
• JDK- version: 17
• Open Metadata version: 1.6.9
• Open Metadata Ingestion package version: 1.6.9.0
• Elastic Search version: 8.17.2
• Backend Database: Enterprise PostgreSQL

Additional Context

Moreover, when I make an API call to "getLineage" from postman, there is a difference in the response def returned for full display of lineage (the expected one in screenshot) to the problematic one.

In the correct lineage, I'm able to see the relationships Def (JSON Payload response from API) among the containers (both upstream/downstream).

However, for the faulty lineage, it is only showing one container in the JSON definition but not the other subsequent entities in the stream.

Based on what I observed, I think the API itself is not returning the full depth of entities involved if the nested container count is 6 or above.

Attaching the JSON Response Definition For Expected Lineage example and Incorrect/faulty Lineage example

Expected_Lineage_Response_Def_For_Nested_Container_Entity.json Faulty_Lineage_Response_Def_For_Nested_Container_entity.json

Initially, I discussed the same in Slack channel and in community discussion hours, I have been suggested to create an Issue.