smarthome
smarthome copied to clipboard
eclipse-archived
Umlauts are broken after restart
I have some things, which contain umlauts in their name, e.g:
Rollladen EG Büro Straße
I can enter those in PaperUI and they are correctly shown, but after a restart of openhab, they become corrupted:
Rollladen EG B?ro Stra?e
I did some checks. The umlauts are broken on my notebook (MacOS/Chrome) and mobile (iOS/Safari). Also it doesn’t matter if the umlauts where entered on the notebook or mobile. Without restarted they are alway shown correctly on both devices. I did have a look to the JSON-DB (org.eclipse.smarthome.core.thing.Thing.json). The umlauts are broken in the json (saved with ?). I also did some checks with curling the REST-API the umlauts are broken (after restart) / correct (before restore) there in the same way as in paperui. So it is not a problem in the browser or paperui.
UI is in English.
Openhab Version 2.4final
https://community.openhab.org/t/umlauts-broken-after-restart-in-oh-2-4/60167?u=maihacke
I use MapDB instead of JSON DB and never hit that problem. Can you try to switch to MapDB for a moment and check if the problem exists there, too? If it is working using MapDB, we have to fix the (de)serialization on writing the JSON files (e.g. force UTF-8 encoding on read / write).
How is it possible to switch to MapDB, I thought that is no longer an option?
In general you have to remove the jsondb bundle and add the mapdb bundle. For Karaf based distributions I would expect using "feature:uninstall" and "feature:install" with the respective Karaf features should to its job. But for openHAB itself (it is no option or how it can be done correctly) you have to question e.g. the community forum.
I did some tests. Starting the JVM with -Dfile.encoding=UTF8 solves the problem. Looking at the code in https://github.com/eclipse/smarthome/blob/master/bundles/storage/org.eclipse.smarthome.storage.json/src/main/java/org/eclipse/smarthome/storage/json/internal/JsonStorage.java the String conversion to/from bytes is a little bit "dangerous", since the getBytes-Method and instantiation of the FileReader rely on the platform encoding. Maybe it would be better to enforce UTF-8 here.
For openHAB on Windows there is a PR that was already merged. The patch will ensure that on Windows the encoding will default to UTF-8 (see https://github.com/openhab/openhab-distro/pull/833).
Where are you running your openHAB server?
That' interesting. I am using openHAB on a Raspberry and it seemed to me that UTF-8 is the default encoding on this platform. Are you using the Oracle JDK or another JDK? Anyway, I think it would be best to set the encoding explicitly on all platforms to UTF-8 to avoid this kind of problems.
I‘m using zulu-embedded java. The default encoding for java depends on the environment (locale). So yes it would be good idea to enforce UTF-8 e.g. via the startup scripts.
The JSON DB should use UTF-8 encoding regardless of the system locale, environment variables, startup scripts etc.
@maihacke Can you give this changes a try: https://github.com/maggu2810/smarthome/pull/new/jsondb-utf8
Tried to compile that, but it fails with
[ERROR] Failed to execute goal org.eclipse.xtend:xtend-maven-plugin:2.14.0:compile (default) on project org.eclipse.smarthome.model.core: Execution default of goal org.eclipse.xtend:xtend-maven-plugin:2.14.0:compile failed: An API incompatibility was encountered while executing org.eclipse.xtend:xtend-maven-plugin:2.14.0:compile: java.lang.NoSuchMethodError: org.eclipse.jdt.internal.compiler.batch.FileSystem.getClasspath(Ljava/lang/String;Ljava/lang/String;ZLorg/eclipse/jdt/internal/compiler/env/AccessRuleSet;Ljava/lang/String;Ljava/util/Map;)Lorg/eclipse/jdt/internal/compiler/batch/FileSystem$Classpath;
I never compiled the smarthome project before. Is there a special java/maven version required?
I will compile the branch and post a link to a bundle soon. You need a JDK version 1.8 to build ESH.
What whould be the best way to test that jar? Should I replace ./runtime/system/org/eclipse/smarthome/storage/org.eclipse.smarthome.storage.json/0.10.0.oh240/org.eclipse.smarthome.storage.json-0.10.0.oh240.jar with it?
I think the root cause is similar to #6630 and has been fixed with https://github.com/openhab/openhab-distro/pull/833. This issue can be closed.
https://github.com/openhab/openhab-distro/pull/833 changes encoding for windows platform in setenv.bat. Since I do not use that file on linux, this cannot fix this problem here.