Entities with different numbers of properties by language

[mfacar]

Describe the bug Adding a property to a template creates an empty field in every entity belonging to that template except the ones in the default language. Some custom pages assume that properties are always in the entities (with empty values, which is happening for other languages than the default one).

Depending on the right behavior it could be needed to update the pages to make the access conditional and have a consistent update of the entities when a template is changed.

To Reproduce

1. Create a template on an instance with at least two languages
2. Create an entity with the created template
3. Edit the template adding a property
4. Check the entity in the database and see that the property is present with an empty array in the other languages but not in the default one.

Expected behavior The entity objects in all the languages should have the same structure.

Additional context Identified from Sentry cc. @meegido

[fnocetti]

After debugging these are my findings:

• This issue is only reproducible when the template has relationship properties and when the added property (step 3 of the "To Reproduce" section provided) is a property which value should be equal for an entity across languages. Refer to https://github.com/huridocs/uwazi/blob/9b84bb41aa592405123ac43a6088136ab157c162/app/api/entities/entities.js#L33
• The reason for this preconditions is that when the template has relationships properties it enters the bulkUpdateMetadataFromRelationships flow. Refer to https://github.com/huridocs/uwazi/blob/9b84bb41aa592405123ac43a6088136ab157c162/app/api/entities/entities.js#L675 This flow makes a lot of unnecessary entity loads and saves (reported in #6077). Those saves are performed through entities.updateEntity operations. entities.updateEntity has some logic to keep properties meant to have the same value in sync across languages. That logic is only applied to the entities in the languages other than the one that the request is being executed on (which explains why all but one language are affected). This keep-in-sync logic is assigning an empty array (for the other languages) in every not-yet-valued and meant-to-be-in-sync property of the entity. Refer to https://github.com/huridocs/uwazi/blob/9b84bb41aa592405123ac43a6088136ab157c162/app/api/entities/entities.js#L118

[txau]

@fnocetti is this meant to be fixed? Or do we wait for the new data layer?

[RafaPolit]

I believe this needs to be fixed, this cannot wait for the data layer. As mentioned in the OP, we really do depend on certain properties being there with empty values. This is not only for pages.

[fnocetti]

@fnocetti is this meant to be fixed? Or do we wait for the new data layer?

I don't think we should wait for the new data layer. While the data is semantically correct, it's still an inconsistency. And should be kind of simple to fix.

I believe this needs to be fixed, this cannot wait for the data layer. As mentioned in the OP, we really do depend on certain properties being there with empty values. This is not only for pages.

I don't think the fix is to have "properties being there with empty values". I can't remember if we decided to make it standard to always have empty values (empty arrays for properties), but I couldn't find code in charge of that. Historically we have just let empty properties be either unset or empty arrays and, everywhere used, we handle both cases. As per my research, the reason of the inconsistency is not that we are failing to keep the empty default, but that there's an algorithm that is wrongly/unnecessarily changing properties, not in all languages, from unset to empty array.

[LaszloKecskes]

After the initial debugging at the start of the work, the key point seems to be the entity update action. On entity creation, the structures are still consistent. This issue and #4220 triggers the same thing, just in different ways, so it makes sense to handle them together.