Cyrillic is displayed poorly in the web interface if it was written to the database from the console

[lepata]

If errors, for example, from libvirt, are written in Cyrillic, then they are poorly displayed on the web interface:

In the DB, everything is fine:

Select message from messages;
| libvirt error code: 55, message: Недопустимая операция: домен уже работает
| libvirt error code: 27, message: Ошибка XML: Недопустимый адрес PCI 0000:01:06.0: слот должен быть <= 0

In the web interface, only the VM description can be set in Cyrillic. It displays well:

In the database, the description is written with Unicode Decimal Code:

One more example. You cannot create a user with non-Latin characters in the web interface, but in the console you can):

rvd_back --add-user админ
админ password: admin
is admin ? : [y/n] y

In the database:

Select name from users;
+------------+
| name       |
+------------+
| admin      |
| daemon     |
| админ      |
+------------+

On the web interface:

It turns out that data is written to the database differently when it comes from the console and when it comes from the web interface.

[frankiejol]

I am going to need some help getting the libvirt messages in russian. I tried setting the locales to ru_RU and now I get LANG=ru_RU.UTF-8. I also set the default locale to this one. Now the system returns text like this : Чтение списков пакетов… Готово so it looks it is properly configured. I also installed those packages: language-pack-ru and language-pack-ru-base. I made sure I rebooted everything but I keep getting libvirt messages in english: libvirt error code: 67, message: unsupported configuration: unknown CPU feature.

Anyway my guess this may come from the charset configuration in mysql. I am not an expert in this but checking with mysqldump -d ravada users I get this:

/*!40101 SET character_set_client = utf8 */;
....
) ENGINE=InnoDB AUTO_INCREMENT=11556 DEFAULT CHARSET=utf8mb3;

What is your output ?

[frankiejol]

Anyway, I could reproduce adding an user in the CLI and it shows wrong in the web. I am working on it.

[lepata]

Anyway my guess this may come from the charset configuration in mysql. I am not an expert in this but checking with mysqldump -d ravada users I get this:

/*!40101 SET character_set_client = utf8 */;
....
) ENGINE=InnoDB AUTO_INCREMENT=11556 DEFAULT CHARSET=utf8mb3;

What is your output ?

My output:

/*!40101 SET character_set_client = utf8 */;
….
) ENGINE=InnoDB AUTO_INCREMENT=58 DEFAULT CHARSET=utf8;

I also wanted to suggest adding a user)

[frankiejol]

I have managed to render cyrillic text forcing a decode utf8 in some strings. I will work on it and hopefully this will be fixed properly.

[frankiejol]

Well, this has been a pretty interesting request. I tried creating virtual machines with UTF8 names. That is possible in theory, but creates all kind of havok. I am toying with the idea of allowing to create that virtual machine, but giving it an internal proper non-utf8 name. An alias is stored with the user desired name. So far it looks like it is going well.

[frankiejol]

@lepata , Finally it looks like we are headed in the right direction. We are passing the tests but some things are not easy to try. Meanwhile I created a new release with this feature. It is ravada-1.8.0-alpha . Download from our own repo at UPC https://infoteleco.upc.edu/img/debian/ So far we checked a lot of things and it looks it is working fine. Let me know if you spot any problem and we will look at it.

[lepata]

Hi @frankiejol, ravada-1.8.0-alpha:

1. User with non-Latin characters - now it's ok (DB, web, CLI) : rvd_back --add-user пользователь
2. Messages from libvirt: message in the popup window (and DB) - fine , the same message in messages - no:

Thanks a lot for your work )

[frankiejol]

TODO:

• [ ] fix messages
• [ ] check duplicated aliases
• [ ] reservations

[lepata]

Reservation is also not displayed in Cyrillic:

[frankiejol]

I merged all the cyrillic changes on develop branch and I created ravada-2.0-beta5. We need to work on that release for our ow users. I have been testing it. There are still some parts that need translation in the messages but so far we have made great advances thanks to @lepata . This week we will test the beta and release a stable version 2. Only bug fixes will be released for v1.

[frankiejol]

After all the recent changes applied the non-latin text display has been improved, mostly thanks to @lepata

[frankiejol]

If that is okay for you I would like to close this issue and continue with #1824 @lepata

[lepata]

I've problem with websockets and text encoding.

For example, ws.onopen = function(event) { ws.send('list_isos') New iso_images (description-cyrillic): INSERT INTO iso_images (name, description, arch, xml, xml_volume, md5, sha256, device) VALUES ('Slinux','Симпли Линукс', 'x86_64', 'slinux.xml' ,'slinux-volume.xml','d11812a1e7bde5d5cf4ccbb60f9b7f19','mysha256sum','/var/lib/libvirt/images/slinux-10.1-x86_64.iso');

Messages (list_alerts):

At the same time, everything is written to the database in normal encoding, and data is also obtained from the database in Cyrillic. For example, subscribe_list_bookings:

File /usr/share/perl5/Ravada/WebSocket.pm:

sub _send_answer($self, $ws_client, $channel, $key = $ws_client) {
….
my $ret;
    eval { $ret = $exec->($self->ravada, $self->clients->{$key}) };
    warn $@ if $@;

    if ( defined $ret && _different($ret, $old_ret )) {
        **open(FH, '>>', '/tmp/rvd.log') or die $!;
        print FH "=========".Dumper($ret); 
        close(FH);**
        warn localtime(time)." WS: send $channel " if $DEBUG;
        $ws_client->send( {json => $ret} );
        $self->clients->{$key}->{ret} = $ret;
    }
….

In /tmp/rvd.log (OK!):

=========$VAR1 = [
          {
            'title' => 'Еще одно событие',
            'description' => 'Описание еще одного события',
            'background_color' => '#7ab2fa',
            'date_changed' => '2022-09-08 14:29:32',
            'user_allowed' => 1,
            'id_booking' => 2,
            'id' => 2,
            'time_end' => '17:30:00',
            'visibility' => 'public',
            'time_start' => '17:00:00',
            'date_booking' => '2022-09-08'
          }
        ];

File /usr/share/ravada/public/js/ravada.js

var subscribe_list_bookings = function(url) {
                var ws = new WebSocket(url);
                ws.onopen = function(event) { ws.send('list_next_bookings_today') };
               
                ws.onmessage = function(event) {
                    **console.log(event.data);**
                    var data = JSON.parse(event.data);
                    $scope.$apply(function () {
                        $scope.bookings_today = data;
                    });
                }

            }

Log (something broke along the way): [{"background_color":"#7ab2fa","date_booking":"2022-09-08","date_changed":"2022-09-08 14:29:32","description":"�пи�ание е�е одного �об��и�","id":2,"id_booking":2,"time_end":"17:30:00","time_start":"17:00:00","title":"��е одно �об��ие","user_allowed":1,"visibility":"public"}]

I don't know how to deal with this problem((

[frankiejol]

I just release version 2.0-beta6 that should fix problems with messages and bookings. http://infoteleco.upc.edu/img/debian/ I tried with some random text pasted from parts of Russian. Sorry it does not makes sense. It is only to show it works.

[frankiejol]

Also, messages should be correct now

[frankiejol]

But text from the calendar is rendered by the third party app. I still don't know how to handle this.

[lepata]

Yes, problems with messages and bookings fixed in 2.0-beta6.

If lines are added to the Front.pm file, then the problem with displaying Cyrillic in the list of templates (https://github.com/UPC/ravada/issues/1804#issuecomment-1240669705) will also be solved:

sub list_iso_images {
….
    while (my $row = $sth->fetchrow_hashref) {
       $row->{name} = Encode::decode_utf8($row->{name});
       $row->{description} = Encode::decode_utf8($row->{description});
       $row->{options} = decode_json($row->{options})

And on the delete VM page, the VM name is displayed incorrectly:

[frankiejol]

Thank you for your accurate report @lepata . I only could take a quick look but it looks like we almost have it solved. I just fixed #1827 that was triggered because I missed to take care of renaming when there was a non-ascii name.

[frankiejol]

I just released 2.0-beta8 that fixes problems with removing and some other. My guess is there may be some templates to fix. Next week we will add another person to our team and he will start reviewing all the html files.