HTML character encoding issue for “ when upgrading

[portedison]

Describe the bug

Similar issue : https://github.com/verbb/vizy/issues/115

When upgrading from Vizy 1.0.23 and Craft CMS 3.9.10, to Vizy 2.1.23 and Craft CMS 4.13.1.1

Character were previously stored in the database as “ any newly created content saves in the database as “ though on an upgrade none of these characters have been migrated. When inspecting the code in the editors html view they appear as “ - though in the database they are still stored as “. A manual update fixes these.

Are there any recommended steps to migrate these?

Steps to reproduce

1. Create a project with Vizy 1.0.23 and Craft CMS 3.9.10
2. Create some content with a “ in a Vizy Field.
3. Migrate to Vizy 2.1.23 and Craft CMS 4.13.1.1

Craft CMS version

3.9.10 -> 4.13.1.1

Plugin version

1.0.23 -> 2.1.23

Multi-site?

No

Additional context

We have written a migration to update each entry in the database, though we've come across an issue where SuperTable content inside Vizy field, is difficult to access/update. (I'll mention that in another ticket and attach).

[engram-design]

Sounds like this'll be encoding changes. Just to be clear, re-saving the entry doesn't change anything, but re-typing the encoded values (swapping & for & for example) works correctly?

I might have to provide a manual console command to run on field content to take care of this.

In addition, there's actually an internal helper function to help with updating content, which handles global, Matrix and Super Table content. We use this in Hyper migrations.

[portedison]

Sounds like this'll be encoding changes. Just to be clear, re-saving the entry doesn't change anything, but re-typing the encoded values (swapping & for & for example) works correctly?

Yes that is correct.

That command could be helpful, the other issue we had was where we have the structure - pageBlocks(Neo) > richText(Neo : Common Block) > blockRichText (Vizy) > vizy2ColumnsTable (Super Table)

But the last one vizy2ColumnsTable 'disappears', though it still appears to be in the database in the old craft format. I can make a ticket explaining more about that if it's useful? There is a comment relating to this already https://github.com/verbb/vizy/issues/292#issuecomment-2268766550

[Kethatril]

This is the change that surfaced the issue: https://github.com/verbb/vizy/commit/819a013d577925488e6f565e3ac1d02edf544dc0

In 2.1.22 the characters showed up correctly in the editor and if you modified the field and saved it the database values would no longer have the escaped characters and that field would show up correctly in 2.1.23 in the editor. But because it only gets fixed by code in the editor resaving all entries using the cli doesn't fix the issue.

[engram-design]

Thanks for pointing that out @Kethatril that's a good start.

[rungta]

@engram-design We’ve noticed a similar issue. After we upgraded Vizy from 2.1.20 to 2.1.23, all HTML entities present in the content have started showing up as text in the control panel. Re-saving the entry (in the CP or via the CLI) does not appear to fix the issue either.

We are currently on Vizy 2.1.25 (Craft 4.14.11.1) but the issue persists. I went back a couple versions and was able to successfully isolate entities becoming visible in the editor to the point where we upgraded to Vizy 2.1.23.

Here is an example:

[{"type":"paragraph","attrs":[],"content":[{"type":"text","text":"We are an international development firm specializing in four pivotal domains that collectively aim to promote prosperity and resilience, specifically in "},{"type":"text","marks":[{"type":"textStyle"}],"text":"underdeveloped, fragile, and conflict-affected contexts"},{"type":"text","text":". Our expertise spans Research and Analysis, Monitoring, Evaluation, and Learning, Institutional Strengthening, and Social and Behavior Change Communication. "}]}]

This is troublesome for two reasons:

1. HTML entities are being rendered as text, leading to authors encountering “corrupted” content that they don't quite understand.
2. Authors are now able to write un-escaped markup inside a Vizy field, including HTML tags and entities.

Any suggestions on how to resolve this?

[engram-design]

If you remove those   texts, do they come back? Was the text pasted from somewhere like Word?

I've got a fairly comprehensive list of entities that when entered into the field are rendering correctly.

While I can see the characters with your content, that would make sense as it's the literal   text present.

You've always been able to write markup in Vizy fields, but we certainly protect against anything malicious making its way in there. This is commonly for when you want to write code in the editor content.

[bartdigitalpulse]

We're seeing this issue as wel. Upgraded from craft 3 to 4 (latest). All the French text from the Globals has faulty encoding:

Original Craft 3: De la fraîcheur à l'intérieur et de l'ombre. Les systèmes de protection solaire, créer une maison agréable.

Craft 4: De la fraîcheur à l'intérieur et de l'ombre. Les systèmes de protection solaire, créer une maison agréable.

Edit: this is a vizy field inside a supertable field

[engram-design]

Fixed for the next release. To get this early, run composer require verbb/vizy:"dev-craft-4 as 2.1.25".

You can also safely apply this on your already Craft 4 upgraded site. In Vizy 1, we stored the encoded value of HTML entities, and in Vizy 2, we don't encode them - causing this issue. The migration that will run upon upgrade will convert these for you, as well as sanitize any potentially malicious content correctly and better than in Vizy 1.

[BenPicard]

Fixed for the next release. To get this early, run composer require verbb/vizy:"dev-craft-4 as 2.1.25".

You can also safely apply this on your already Craft 4 upgraded site. In Vizy 1, we stored the encoded value of HTML entities, and in Vizy 2, we don't encode them - causing this issue. The migration that will run upon upgrade will convert these for you, as well as sanitize any potentially malicious content correctly and better than in Vizy 1.

@engram-design

I still have the bug even with this dev update

It fixed my save error but now all my contents have html tag in them

https://l.picard.ca/erxHV https://l.picard.ca/0iTay

[BenPicard]

@engram-design just to add to that

front-end before update: https://l.picard.ca/ijOXJ back-end before update: https://l.picard.ca/IzdO2

front-end after update: https://l.picard.ca/aUOMo back-end after update: https://l.picard.ca/cY3F6

I restore our website to the old version of vizy, we can't save update on the page but at least the front-end is not broken.

[engram-design]

Right, so you have a CKEditor field inside the Vizy field content, that complicates things considerably.

Just pushed another fix to address this properly. To get this early, run composer require verbb/vizy:"dev-craft-4 as 2.1.25".

[engram-design]

Updated in 2.1.26.