cloudberry icon indicating copy to clipboard operation
cloudberry copied to clipboard
verified publisher icon apache

Movable DataBase Locales for Cloudberry

Open leborchuk opened this issue 3 months ago • 2 comments

We inherited this issue from PostgreSQL.

PostgreSQL uses glibc to sort strings. In version glibc=2.28, collations broke down badly (in general, there are no guarantees when updating glibc). Changing collations breaks indexes. Similarly, a cluster with different collations also behaves unpredictably.

What and when something has changed in glibc can be found on https://github.com/ardentperf/glibc-unicode-sorting Also there is special postgresql-wiki https://wiki.postgresql.org/wiki/Locale_data_changes And you tube video https://www.youtube.com/watch?v=0E6O-V8Jato

In short, the issue can be seen through the use of bash:

( echo "1-1"; echo "11" ) | LC_COLLATE=en_US.UTF-8 sort

gives the different results in ubuntu 18.04 and 22.04.

There is no way to solve the problem other than by not changing the symbol order. We freeze symbol order and use it instead of glibc.

Here the solution https://github.com/postgredients/mdb-locales.

In this PR I have added PostgreSQL patch that replaces all glibc locale-related calls with a calls to an external libary. It activates using new configure parameter --with-mdblocales, which is off by default.

Using custom locales needs libmdblocales1 package and mdb-locales package with symbol table.

Build needs libmdblocales-dev package with headers.

Fixing the symbol order is necessary for OS upgrade. For example Ubuntu 22.04 EOL is April 2027, Rocky 8 Active Support ended May 2024, and Security support ends in 2029.

We use Movable DataBase Locales in Greenplum 6 and all our PostgreSQL installations (starting with PostgreSQL 12). This patch is adopted patch version from our internal PostgreSQL 14 fork.

leborchuk avatar Sep 19 '25 12:09 leborchuk

Thank you very much for bringing up this issue and providing the corresponding patch. Yes, we've run into this problem as well - the same database version behaving inconsistently when deployed on different OS platforms. This is definitely an issue that requires fixing.

Lirong

Leonid @.***> 于2025年9月19日周五 20:26写道：

We inherited this issue from PostgreSQL.

PostgreSQL uses glibc to sort strings. In version glibc=2.28, collations broke down badly (in general, there are no guarantees when updating glibc). Changing collations breaks indexes. Similarly, a cluster with different collations also behaves unpredictably.

What and when something has changed in glibc can be found on https://github.com/ardentperf/glibc-unicode-sorting Also there is special postgresql-wiki https://wiki.postgresql.org/wiki/Locale_data_changes And you tube video https://www.youtube.com/watch?v=0E6O-V8Jato

In short, the issue can be seen through the use of bash:

( echo "1-1"; echo "11" ) | LC_COLLATE=en_US.UTF-8 sort

gives the different results in ubunru 18.04 and 22.04.

There is no way to solve the problem other than by not changing the symbol order. We freeze symbol order and use it instead of glibc.

Here the solution https://github.com/postgredients/mdb-locales.

In this PR I have added PostgreSQL patch that replaces all glibc locale-related calls with a calls to an external libary. It activates using new configure parameter --with-mdblocales, which is off by default.

Using custom locales needs libmdblocales1 package and mdb-locales package with symbol table.

Build needs libmdblocales-dev package with headers.

Fixing the symbol order is necessary for OS upgrade. For example Ubuntu 22.04 EOL is April 2027, Rocky 8 Active Support ended May 2024, and Security support ends in 2029.

We use Movable DataBase Locales in Greenplum 6 and all our PostgreSQL installations (starting with PostgreSQL 12). This patch is adopted patch version from our internal PostgreSQL 14 fork.

You can view, comment on, or merge this pull request online at:

https://github.com/apache/cloudberry/pull/1363 Commit Summary

File Changes

(25 files https://github.com/apache/cloudberry/pull/1363/files)

Patch Links:

  • https://github.com/apache/cloudberry/pull/1363.patch
  • https://github.com/apache/cloudberry/pull/1363.diff

— Reply to this email directly, view it on GitHub https://github.com/apache/cloudberry/pull/1363, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAD6XVHICRZ6I2NQCJL6PZD3TPY5VAVCNFSM6AAAAACG62TJISVHI2DSMVQWIX3LMV43ASLTON2WKOZTGQZTIMJYGAZTCNI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

jianlirong avatar Sep 19 '25 13:09 jianlirong

Fixed all the issues, failed tests still waiting for the working CI

leborchuk avatar Oct 06 '25 12:10 leborchuk