[Bug] Broken support for cyrillic characters in openmsx-catapult in mingw32 [sf#476]

[openMSX-import]

Reported by joxy on 2013-07-18 19:45 UTC openmsx catapult 0.9.1:

Cyrillic chars show up as garbage:

Status Info:

Screen saved to /home/egp/Изображения/MSX/ALESTE.png

[openMSX-import]

Commented by joxy on 2013-07-18 19:46 UTC See also: https://sourceforge.net/p/openmsx/bugs/475/ for the related bug in openMSX.

[openMSX-import]

Updated by manuelbi on 2013-07-18 19:57 UTC

• labels: --> wxcatapult
• Group: Next_release --> Postpone_until_after_next_release
• Priority: 5 --> 9

[openMSX-import]

Commented by manuelbi on 2013-07-18 19:57 UTC As the old Catapult is currently unmaintained, the current openMSX team will not fix it. Please provide a patch to get this fixed.

[openMSX-import]

Commented by joxy on 2013-07-24 02:32 UTC In process of debugging -

(11:19:46) egp_: got to the meat (11:23:02) egp_: =) this stuff fails - an intermediate debug log of catapult

in CatapultXMLParser::cb_text .cpp

#define X1 (wxT("UTF-8"))
#define X2 (wxCSConv(X1))
#define X3 ((const wxChar *)X2.cMB2WX((const char *)chars))
#define X4 (wxString(X3,len))

{Joe
x1 'UTF-8'
x3 'Screen saved to /home/egp/&#x418;&#x437;&#x43E;&#x431;&#x440;&#x430;&#x436;&#x435;&#x43D;&#x438;&#x44F;/MSX/tmp1.png</log>
'
x4 'Screen saved to /home/egp/'
len=26
}
{Joe
x1 'UTF-8'
x3 'И'
x4 'И'
len=2
}
{Joe
x1 'UTF-8'
x3 'з'
x4 'з'
len=2
}
{Joe
x1 'UTF-8'
x3 'о'
x4 'о'
len=2
}
{Joe
x1 'UTF-8'
x3 'б'
x4 'б'
len=2
}
{Joe
x1 'UTF-8'
x3 'р'
x4 'р'
len=2
}
{Joe
x1 'UTF-8'
x3 'а'
x4 'а'
len=2
}
{Joe
x1 'UTF-8'
x3 'ж'
x4 'ж'
len=2
}
{Joe
x1 'UTF-8'
x3 'е'
x4 'е'
len=2
}
{Joe
x1 'UTF-8'
x3 'н'
x4 'н'
len=2
}
{Joe
x1 'UTF-8'
x3 'и'
x4 'и'
len=2
}
{Joe
x1 'UTF-8'
x3 'я'
x4 'я'
len=2
}
{Joe
x1 'UTF-8'
x3 '/MSX/tmp1.png</log>
'
x4 '/MSX/tmp1.png'
len=13
}

[openMSX-import]

Commented by joxy on 2013-07-24 03:34 UTC Here's a patch (attached). It fixes this bug completely. It is based on on mth's words that openmsx communicates using UTF-8. Attached file patch.txt:

Index: CatapultXMLParser.cpp
===================================================================
--- wxcatapult/src/CatapultXMLParser.cpp    (revision 13250)
+++ wxcatapult/src/CatapultXMLParser.cpp    (working copy)
@@ -99,17 +99,23 @@
 void CatapultXMLParser::cb_text (CatapultXMLParser * parser, const xmlChar * chars, int len)
 {
    wxString temp;
+   //len means number of bytes (octets)
+   assert(len>=0);
+   char* firstLenBytes=new char[len+1];
+   (void)strncpy(firstLenBytes, (const char*)chars, len);
+   firstLenBytes[len]=(char)0;
    switch (parser->parseResult.parseState)
    {
        case TAG_UPDATE:
        case TAG_LOG:
        case TAG_REPLY:
-           temp =  wxString((const wxChar *)wxCSConv(wxT("ISO8859-1")).cMB2WX((const char *)chars),len);
-           parser->parseResult.contents.Append (temp.Left(len));
+           temp =  wxString((const wxChar *)wxCSConv(wxT("UTF-8")).cMB2WX(firstLenBytes));
+           parser->parseResult.contents.Append (temp);
            break;
        default:
            break;
    }
+   delete[] firstLenBytes;
 }

 void CatapultXMLParser::SendParsedData()

[openMSX-import]

Commented by joxy on 2013-07-24 04:21 UTC More efficient patch2, created with help by mth. Works OK. Attached file patch2.txt:

Index: CatapultXMLParser.cpp
===================================================================
--- wxcatapult/src/CatapultXMLParser.cpp    (revision 13250)
+++ wxcatapult/src/CatapultXMLParser.cpp    (working copy)
@@ -104,8 +104,8 @@
        case TAG_UPDATE:
        case TAG_LOG:
        case TAG_REPLY:
-           temp =  wxString((const wxChar *)wxCSConv(wxT("ISO8859-1")).cMB2WX((const char *)chars),len);
-           parser->parseResult.contents.Append (temp.Left(len));
+           temp =  wxString((const char*)chars, wxConvUTF8, (size_t) len);
+           parser->parseResult.contents.Append (temp);
            break;
        default:
            break;

[openMSX-import]

Commented by joxy on 2013-07-30 03:02 UTC patch is not yet committed to git. Test on Linux shows:

Screen saved to /home/egp/Изображения/tmp1.png

[openMSX-import]

Commented by joxy on 2013-07-30 03:09 UTC committed. tested on linux, all works OK. tests on msvc+mingw are pending.

[openMSX-import]

Commented by joxy on 2013-07-30 06:00 UTC broken under mingw32:

Screen saved to C:/MinGW/msys/1.0/home/egp/Изаобра.png

[openMSX-import]

Updated by manuelbi on 2013-10-05 12:06 UTC

• summary: Broken support for cyrillic characters in openmsx-catapult --> Broken support for cyrillic characters in openmsx-catapult in mingw32
• Priority: 9 --> 5