MySQL encoding issues

[grazor]

When performing async queries using peewee-async some UTF-8 data is fetched as sequence of question marks while peewee's sync queries return this data as is. This issue doesn't occur with PostgreSQL database.

How to reproduce

Creating database

DROP DATABASE IF EXISTS `test`;

CREATE DATABASE IF NOT EXISTS `test`
    DEFAULT CHARACTER SET utf8
    DEFAULT COLLATE utf8_general_ci;

Code

import peewee
import peewee_async
import asyncio


texts = ['[en] The quick brown fox jumps over the lazy dog', 
        '[ru] Быстрая коричневая лиса прыгает через ленивую собаку',
        '[pl] Szybki brązowy lis przeskoczył nad leniwym psem',
        '[de] Der schnelle braune Fuchs springt über den faulen Hund',
        '[ko] 빠른 갈색 여우가 게으른 개 점프']

database = peewee_async.PooledMySQLDatabase(host='localhost', user='root', 
        password='', database='test')

class TestModel(peewee.Model):
    text = peewee.CharField()

    class Meta:
        database = database


TestModel.create_table(True)

for t in texts:
    obj = TestModel.create(text=t)
    print('(%3s) Created:\t %s' % (('ok' if obj.text in texts else 'err'), obj.text))

print()
for obj in TestModel.select():
    print('(%3s) Sync:\t %s' % (('ok' if obj.text in texts else 'err'), obj.text))


database.close()
objects = peewee_async.Manager(database)
database.allow_sync = False

async def handler():
    all_objects = await objects.execute(TestModel.select())
    for obj in all_objects:
        print('(%3s) Async:\t %s' % (('ok' if obj.text in texts else 'err'), obj.text))

print()
loop = asyncio.get_event_loop()
loop.run_until_complete(handler())
loop.close()

with objects.allow_sync():
    TestModel.drop_table(True)

Result

( ok) Created:   [en] The quick brown fox jumps over the lazy dog
( ok) Created:   [ru] Быстрая коричневая лиса прыгает через ленивую собаку
( ok) Created:   [pl] Szybki brązowy lis przeskoczył nad leniwym psem
( ok) Created:   [de] Der schnelle braune Fuchs springt über den faulen Hund
( ok) Created:   [ko] 빠른 갈색 여우가 게으른 개 점프

( ok) Sync:      [en] The quick brown fox jumps over the lazy dog
( ok) Sync:      [ru] Быстрая коричневая лиса прыгает через ленивую собаку
( ok) Sync:      [pl] Szybki brązowy lis przeskoczył nad leniwym psem
( ok) Sync:      [de] Der schnelle braune Fuchs springt über den faulen Hund
( ok) Sync:      [ko] 빠른 갈색 여우가 게으른 개 점프

( ok) Async:     [en] The quick brown fox jumps over the lazy dog
(err) Async:     [ru] ??????? ?????????? ???? ??????? ????? ??????? ??????
(err) Async:     [pl] Szybki br?zowy lis przeskoczy? nad leniwym psem
( ok) Async:     [de] Der schnelle braune Fuchs springt über den faulen Hund
(err) Async:     [ko] ?? ?? ??? ??? ? ??

Additional info

• Mysql server: mariadb 10.1.17-1;
• Table engine: InnoDB;
• Python version: 3.5.2;
• peewee-async: latest from git.

[rudyryk]

@grazor does the same error happen with sync queries? UPD Oh, I see, with sync queries it's OK. Probably some issue with aiomysql or with connection initialization.

[grazor]

@rudyryk no, peewee's sync queries work fine.

[rudyryk]

It seems like this issue is related: https://github.com/aio-libs/aiomysql/issues/62

[grazor]

@rudyryk yes, seems so. I'll have a look, thank you!