Error doing a backup when Database file name has extended ASCII chars

[diegostests]

We have a program which makes backups of our Firebird databases. If the database name has only ASCII chars, it works correctly. But if the database name has extended ASCII chars (for example "Ñ" or "Ç"), the backup fails.

We tested it on Firebird-5.0.3.1659-0-f4c0f85-windows-x64.

We use Delphi 12.3, and the component TFDFBNBackup.Backup, which breaks with error "[FireDAC][Phys][FB]I/O error during "CreateFile (open)" operation for file".

If we try to make an nbackup directly, with Windows command "nbackup" it works correctly.

Initially we thought it was a bug in Delphi component, so we contacted Delphi support.

But they say "It seems there is a bug in Firebird isc_action_svc_nbak service (behind TFDFBNBackup). It does not support isc_spb_utf8_filename properly. All the possible combinations of isc_spb_utf8_filename and isc_spb_dbname value encoding fail with I/O or encoding errors. The issue should be reported to Firebird team."

So we asked about this issue in Firebird forums (https://groups.google.com/g/firebird-support/c/fGcByzSazNo). In forums, they said that if it is reproducible with fbsvcmgr we shoud open a ticket in the Firebird tracker providing full fbsvcmgr command line. In Firebird forums they also said there are bug reports about this in the tracker eg https://github.com/FirebirdSQL/firebird/issues/2603.

We have tried the command with fbsvcmgr and it breaks. We have also seen the firebird issue 2603, and we think that our issue could be related to that issue, but our case is specifically about nbackup and we have tested it on a new version of Firebird (Firebird-5.0.3.1659-0-f4c0f85-windows-x64). So we think this could be reported as an independent issue.

---

The database filename we used is "C:\Users\Diego\Desktop\EMPLOYEEÑ.FDB", and the backup filename is "C:\Users\Diego\Desktop\EMPLOYEEÑ.NBK" This database is a copy of standard EMPLOYEE.FDB database. The only change is the database filename, which in our case contains a "Ñ" character.

The commandline we used is: "C:\Program Files\Firebird\Firebird_5_0\fbsvcmgr" service_mgr -user sysdba -password masterkey -action_nbak -nbk_level 0 -dbname C:\Users\Diego\Desktop\EMPLOYEEÑ.FDB -nbk_file C:\Users\Diego\Desktop\EMPLOYEEÑ.NBK

We used the exact filename for the commandline, as the filename on Windows.

The commandline tool breaks with error:

"I/O error during "CreateFile (open)" operation for file "C:\USERS\DIEGO\DESKTOP\EMPLOYEEÃ'.FDB" -Error while trying to open file -El sistema no puede encontrar el archivo especificado."

Thanks in advance

[AlexPeshkoff]

Is it specific for isc_action_svc_nbak? I.e. did you try some other service, like isc_action_svc_backup? I ask cause I use linux, it always works with UTF8, therefore my check will be rather artificial.

[diegostests]

I have just tried to perform a normal backup (not incremental) and it worked. I used the following command:

"C:\Program Files\Firebird\Firebird_5_0\fbsvcmgr" service_mgr -user sysdba -password masterkey -action_backup -dbname C:\Users\Diego\Desktop\EMPLOYEEÑ.FDB -bkp_file C:\Users\Diego\Desktop\EMPLOYEEÑ.FBK

---

I also tried to make a normal restore (not incremental) and it also worked. I used the following command:

"C:\Program Files\Firebird\Firebird_5_0\fbsvcmgr" service_mgr -user sysdba -password masterkey -action_restore -dbname C:\Users\Diego\Desktop\EMPLOYEEÑ.FDB -bkp_file C:\Users\Diego\Desktop\EMPLOYEEÑ.FBK

---

So it seems the issue does not affect all services. But at least, with the option "-action_nbak" it does not work.

And it seems there is also an issue in the command for restoring an incremental backup. I tried to make an incremental backup, with command "nbackup" (not with fbsvcmgr, because it does not work), and then I tried to restore it with fbsvcmgr, using the command:

"C:\Program Files\Firebird\Firebird_5_0\fbsvcmgr" service_mgr -user sysdba -password masterkey -action_nrest -nbk_level 0 -dbname C:\Users\Diego\Desktop\EMPLOYEEÑ.FDB -nbk_file C:\Users\Diego\Desktop\EMPLOYEEÑ.NBK

The restore command did not show any error, but the restored file was created with an incorrect file name (EMPLOYEEÑ.FDB, instead of EMPLOYEEÑ.FDB).

So there are issues at least with the fbsvcmgr options for performing incremental backups and restores. The command for normal (not incremental) backups and restores worked correctly.

Thanks in advance

[AlexPeshkoff]

Please try next (Sunday should be ok) snapshot of FB6 to check did my fix help (should) or not.

[pavel-zotov]

Please try next (Sunday should be ok) snapshot of FB6 to check did my fix help (should) or not.

Please unpack script (.py) and logs from attached zip: gh_8588_-_py_and_logs_for_ruRU_and_enUS.zip

Open cmd.exe and do there:

chcp 65001
python C:\path\to\gh_8588.py C:\FB\60SS\fbclient.dll
// where "C:\FB\60SS\fbclient.dll" = path+name of client library

Python must have full access rights to the c:\temp\ directory. Script tries to dir C:\temp\non-ascii-db.tmp\ and create several databases in it, with names from following tuple: ('rus_сотрудники', 'czh_zaměstnanci', 'fra_employés', 'swe_anställda', 'ptg_funcionários', 'grk_εργαζομένων', 'tur_çalışanlar', 'est_töötajad')

Then, if DB created OK, script obtains its actual file name (i.e. as it is shown by dir command) and make severa actions with this DB using FB utilities: gbak, gfix, nbackup, fbsvcmgr, isql and gstat.

I've tried this on two machines, one with system locale = ru_RU and second with en_US.

Comparison of logs shows that:

0. every DB can be created but its actual name will differ from what we put in DPB

1. value from DPB can be obtained using RDB$GET_CONTEXT('SYSTEM', 'DB_NAME') - WITH EXCEPTION see below "3)"

2. actual file name can be obtained using query to MON$DATABASE.MON$DATABASE_NAME

3. value RDB$GET_CONTEXT('SYSTEM', 'DB_NAME') differs from what we could expect to locale = ru_RU and when we try to create DB file cyrillic (!) and also greek or turkish characters:

4. same for SHOW DATABASE command: its output for locale = ru_RU differs from expected, in several cases:

5. case of non-ascii characters in RDB$GET_CONTEXT() equals to what was specified in DPB, i.e. characters are not uppercased as show in this example: FRA_EMPLOYéS.FDB

6. file names like RUS_сотрудники.FDB (and others) - have weird encoding (i've tried using tema.ru/decoder and dencode.com/ sevrices -- no success)

7. all FB utilities works with such files normally, i.e. one may to change some DB-header attributes etc. Of course, we have to pass argument = actual_file_name to every such utility in order to make it work OK. But fbsvcmgr action_nbak ... will issue smth like: RUS_сорŠÃ‘‚¬Ã‘ƒÐ´Ð½Ð¸ÐºÐ / -The system cannot find the file specified. -- although .nbk file will be created.

[aafemt]

Please unpack script (.py) and logs from attached zip:

This if far different issue duplicating a lot of tickets including, for example, https://github.com/FirebirdSQL/firebird/issues/3546, https://github.com/FirebirdSQL/firebird/issues/2603 and so on.

This ticket is about database name including characters available in ANSI charset and affect nbackup only.

[pavel-zotov]

What exactly means 'ansi charset' ? win1252 / iso8859_1 or smth other ?

[aafemt]

https://learn.microsoft.com/en-us/windows/win32/intl/code-pages

[pavel-zotov]

Changed script for usage only cp1252, only one file for DB was selected ('fra_employés'). gh_8588_cp1252.zip

When system locale is ru_RU then actual DB file name (after its creation) will be 'FRA_EMPLOYЙS.FDB'. Proper (expected) name is returned by RDB$GET_CONTEXT('SYSTEM', 'DB_NAME') but attempt to get DB name from mon$database.mon$database_name fails with "Cannot transliterate character between character sets":

All subsequent actions with such file can not be done:

When system locale = en_US than almost all works fine, except (again) fbscvcmgr action_nbak ... For attempt to make incremental backup level 0 message 'file not found' is raised: -- but eventually all expected files are created:

C:\temp\non-ascii-db.tmp>dir /b
FRA_EMPLOYÉS.FDB
fra_employés.gbak_bk
fra_employés.svc_fbk
fra_employés.svc_nbk0

[pavel-zotov]

PS. It seems that test can not be implemented because of dependency between DB file name and appropriate system locale that must present on testing host.

[aafemt]

Use locale.getencoding() to get ANSI code page. Then for 1251 use russian names, 1252 - french and so on.

Or better yet let this test fail until Firebird get a true Unicode support.

[pavel-zotov]

Use locale.getencoding() to get ANSI code page. Then for 1251 use russian names, 1252 - french and so on.

Total count of codepages is... what ? 50 ...100 ?

better yet let this test fail until Firebird get a true Unicode support.

Yes, this seems more reasonable

[diegostests]

Please try next (Sunday should be ok) snapshot of FB6 to check did my fix help (should) or not.

I have tested it with Firebird 6 snapshot Firebird-6.0.0.800-0-1f226fc-windows-x64.exe.

I tried to make a nbackup with the following command:

"C:\Program Files\Firebird\Firebird_6_0\fbsvcmgr" service_mgr -user sysdba -password masterkey -action_nbak -nbk_level 0 -dbname C:\Users\Administrator\Desktop\EMPLOYEEÑ.FDB -nbk_file C:\Users\Administrator\Desktop\EMPLOYEEÑ.NBK

This command creates a file EMPLOYEEÑ.NBK, which has the expected file name, so the file name is correct. But the created file is empty. And the command breaks with the following error: "Error opening database file: C:\USERS\ADMINISTRATOR\DESKTOP\EMPLOYEEÃ'.FDB -The system cannot find the file specified."

So the issue is not solved yet with the FB6 snapshot.

Thanks in advance

[AlexPeshkoff]

One more try tomorrow please.

[diegostests]

I have tested it with Firebird 6 snapshot TMP_Firebird-6.0.0.835-0-2cf29c1-windows-x64.exe.

I tried to make a nbackup with the following command:

"C:\Program Files\Firebird\Firebird_6_0\fbsvcmgr" service_mgr -user sysdba -password masterkey -action_nbak -nbk_level 0 -dbname C:\Users\Administrator\Desktop\EMPLOYEEÑ.FDB -nbk_file C:\Users\Administrator\Desktop\EMPLOYEEÑ.NBK

Now the command worked correctly. The nbackup was created with correct file name and it is not empty.

But if I try to execute the same command again, it breaks with an error: "Error creating backup file: C:\Users\Administrator\Desktop\EMPLOYEEÃ'.NBK -The file exists."

I understand the command should show an error, because the nbackup file was already created with the previous command. But the error message shows an incorrect file name "EMPLOYEEÃ'.NBK", instead of "EMPLOYEEÑ.NBK"

So there are some issue with error messages.

When the issue is ready in FB 6, will the fix also be applied also to Firebird 5.0.3 ? Thank you in advance

[AlexPeshkoff]

Answer to last question is very simple - yes, certainly bugfix will be backported to stable branches.

What about bad diags - will look next week.

[AlexPeshkoff]

@diegostests Please try tomorrow fb6 snapshot.

[diegostests]

I have tested it with Firebird 6 snapshot Firebird-6.0.0.848-0-c483c09-windows-x64.exe.

Now, if I try to make an incremental backup but the backup file already exists, the error message shows the correct file name. So this works ok.

But I have made another test, which shows an incorrect file name. I tried to make a normal backup (not incremental) with a path that not exists, but the error message shows an incorrect file name. I used the following command:

"C:\Program Files\Firebird\Firebird_6_0\fbsvcmgr" service_mgr -user sysdba -password masterkey -action_backup -dbname C:\Users\Administrator\Desktop\EMPLOYEEÑ.FDB -bkp_file C:\Users\Adminitrator\Desktop2\EMPLOYEEÑ.FBK

It breaks with an error:

cannot open backup file C:\Users\Adminitrator\Desktop2\EMPLOYEEÃ'.FBK

The path is incorrect, because the folder "Desktop2" does not exists, so it is correct that the command shows an error. But the error message shows an incorrect filename "EMPLOYEEÃ'.FBK" instead of "EMPLOYEEÑ.FBK"

So the issues with incremental backup are fixed. But there is an issue with error messages in normal (not incremental) backups.

Thank you in advance

[AlexPeshkoff]

Create new ticket please, this is getting offtopic here.

[AlexPeshkoff]

Also please check fb5 snapshot to ensure nbackup service is OK.

[diegostests]

I have tested the nbackup command with Firebird-5.0.3.1674-0-d366ddf-windows-x64.exe and it also worked correctly.

Thanks