DataTransfer.req Data field quotation mark handling

[andle]

Checklist

• [ x] I checked other issues already, but found no answer/solution
• [x ] I checked the documentation and wiki, but found no answer/solution
• [ x] I am running the latest version and the issue still occurs
• [ x] I am sure that this issue is about SteVe (and not about the charging station software or something unrelated to SteVe)

Specifications

SteVe Version     : 3.4.9
Operating system  : Raspi/Debian
JDK               : openjdk 11.0.9.1 2020-11-04
Database          : mariadb  Ver 15.1 Distrib 10.3.27-MariaDB

Expected Behavior

DataTransfer.req data field can send characters like quotation marks. Sample data field (string-ish): {"txId":"123456","description": “Charging:$2.81 @ $0.12/kWh, ParkingFee:$0.50 @$1/min, ActivationFee:$1.0, TOTAL KWH: 23.4 TIME:03.50 COST:$4.31"}

Actual Behavior

Quotation marks seem to be getting interpreted as html: {"txId":"123456","description": “Charging:$2.81 @ $0.12/kWh, ParkingFee:$0.50 @$1/min, ActivationFee:$1.0, TOTAL KWH: 23.4 TIME:03.50 COST:$4.31"}

Also tried backslash escaping quotation marks: {\"txId\":\"123456\",\"description\": \“Charging:$2.81 @ $0.12/kWh, ParkingFee:$0.50 @$1/min, ActivationFee:$1.0, TOTAL KWH: 23.4 TIME:03.50 COST:$4.31\"}

LOL: The above strings are getting converted in this comment box to normal quotation marks. Below is a screen shot of the strings when editing:

Steps to Reproduce the Problem

1. Operations > OCPP 1.6 > DataTransfer
2. Enter Vendor ID, Message ID, and Data values
3. Click Perform

Additional context

The above "Actual Behavior" strings were taken from Steve logs, so it seems the Data field doesn't handle quotation marks (or backslashes) as raw data characters. The Data field is defined as a private String, so maybe there is a different way to define it for raw data? Is there some way to escape them properly?

This will be important for upcoming CTEP pricing implementations for OCPP 1.6 as the data transfer messages contain quotation marks, both single and double. I am still learning the requirements as they come out.

[minhtrietdiep]

Were you using the 3.4.9 release, or did you build from master at the time of posting?

It looks like commit 630fa18b598d807ebc280fff2320cabe86e58428 escapes html characters, but that messes up quotes in the payload, which previously were escaped with a blackslash (\"), but now become ".

I'm not sure whether the OCPP-client side needs to convert the " back to " (can't find anything about it in the json spec at least), or that the commit introduces a regression (as it worked fine before).

[mgineprini]

Hi! I know that this issue was reported a while ago, but it seems like the problem about the “quotation mark” handling it’s still there. When trying to send the “DataTransfer” message with a custom data payload it always handle wrongly the interpretation of \”.

Any plans on a possible fix for this one? I would like to avoid an OCPP Client-Side fix, since it’s not really a feasible solution.

Thanks!

[juherr]

It is not something nicely fixable from Steve: the specification doesn't say the way the data should be encoded and it will depend on the station itself. Some will accept HTML escaping while some others will accept \ escaping.

As mentioned before, maybe using the default way to escape JSON (\) is a better solution by default.

@goekay Do you remember why you modified the behavior in https://github.com/steve-community/steve/commit/630fa18b598d807ebc280fff2320cabe86e58428 ?

[JensHeilig]

We modified the encoding from forHtml introduced in 630fa18b598d807ebc280fff2320cabe86e58428 to CDATA. This worked well for us, but may break encoding for other use cases.

For anybody interested, this is the patch we made:

ocpp/ws/custom/CustomStringModule.java
index 7b106e12..36ed44fe 100644
--- a/src/main/java/de/rwth/idsg/steve/ocpp/ws/custom/CustomStringModule.java
+++ b/src/main/java/de/rwth/idsg/steve/ocpp/ws/custom/CustomStringModule.java
@@ -89,7 +89,7 @@ public class CustomStringModule extends SimpleModule {
         }

         private static String objectToString(Object value) {
-            return Encode.forHtml((String) value);
+            return Encode.forCDATA((String) value);
         }
     }

@@ -100,7 +100,7 @@ public class CustomStringModule extends SimpleModule {
         @Override
         public String deserialize(JsonParser p, DeserializationContext ctxt) throws IOException {
             String val = super.deserialize(p, ctxt);
-            return Encode.forHtml(val);
+            return Encode.forCDATA(val);
         }
     }
}

[maaikez]

I had this problem too, is there any incentive to change this? It is quite annoying and I went back to an older version now.