Manual comments on table fields disappear after reloading metadata

[KAKuznetsov]

Affected module Does it impact the UI, backend or Ingestion Framework?

Describe the bug A clear and concise description of what the bug is.

I load metadata from SQL server. Then I add a description of the columns in the table in the openmetadata UI. After that, when I reload the metadata from SQL server, all column descriptions are deleted. Why is this happening?

And is it possible to make column comments not be deleted the next time the metadata is reloaded?

The same happens when reloading metadata from ClickHouse All manual comments disappear

To Reproduce

https://drive.google.com/file/d/1g2K8KzO4QOVmBSFY0erWVcdau6MVTPbz/view?usp=sharing https://drive.google.com/file/d/1kWIyzywPS3AK3hiTf6KZHiOEh1xJyUep/view?usp=sharing https://drive.google.com/file/d/1UMHyLG14GqlW0RbDv3vaJ9UJHmGi3OKU/view?usp=sharing]

Expected behavior

After updating the metadata from the database server, manual comments for the same columns in the table should not be removed, but should remain the same as before the metadata was reloaded.

Version:

• OS: Windows 10
• Python version:
• OpenMetadata version: 0.12
• OpenMetadata Ingestion package version: [e.g. openmetadata-ingestion[docker]==XYZ]

Additional context Add any other context about the problem here.

[nahuelverdugo]

@harshach this is similar to what is happening here https://github.com/open-metadata/OpenMetadata/issues/6844

[harshach]

In non-secure mode all users are of type user, hence the backend cannot distinguish between a bot vs user. In OpenMetadata a bot user can update the description if it's empty but a user can override the description. This behavior is to do following

1. As a bot user we want to populate the initial descriptions if they are available in source systems such as table descriptions in databases
2. However, we want users to improve the description, tags and improve overall documentation . If a user is edited a description bot cannot override to give preference to user generated content

Ingestion/Bot framework uses put to create or update. Previously we disabled updating the description if it's not empty through put requests as those are solely used by our ingestion framework. But as more and more users are picking up our APIs we made it possible for PUT requests to update the description if even if the description is not empty as long as the user not a bot.

The above condition will not work in non-secure mode as all users are of type "anonymous" we will run into this issue of overriding generated description.

In 0.12.1 we are introducing the username/password login and removing non-secure mode. This will enforce everyone to have a bot account and use that as part of ingestion.

cc @nahuelverdugo @sureshms

[harshach]

@chirag-madlani we need a cypress test to cover this.

1. Run ingestion workflow
2. Update a table description
3. Re-run ingestion workflow
4. Updated description should stay intact