5.38 outputs locale errors with UTF-8 LANG environment variable set

[LordAro]

Any execution of perl 5.38 with a UTF-8 language set results in

~/Downloads/strawberry-perl-5.38.0.1-64bit-portable/perl/bin/perl -v | head -2
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LC_ALL = (unset),
        (possibly more locale environment variables)
        LANG = "en_GB.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the system default locale ("English_United Kingdom.1252").

This is perl 5, version 38, subversion 0 (v5.38.0) built for MSWin32-x64-multi-thread

it's not quite clear what languages are supported and installed on my system, and why this is only an issue now.

There's a few locale related things in 5.38, which it appears to be related to:

https://perldoc.perl.org/perl5380delta#Locale-category-LC_NAME-now-supported-on-participating-platforms https://perldoc.perl.org/perl5380delta#Changes-to-%22thread-safe%22-locales

though I've no idea which.

Codepage is 850, but there's no difference if I set it to 65001 either.

Happens in both powershell and msys, and with either of LANG & LC_NAME - which would definitely suggest that it's perl at fault rather than MSYS / powershell.

There is a workaround, but it's hacky:

$ PERL_BADLANG=0 ~/Downloads/strawberry-perl-5.38.0.1-64bit-portable/perl/bin/perl -v | head -2

This is perl 5, version 38, subversion 0 (v5.38.0) built for MSWin32-x64-multi-thread

No issues with 5.32 or 5.36.

This feels related to, but distinct from, #119 . I have tried 5.38.0.2, it had no effect.

[shawnlaffan]

These are the same symptoms as in #97

The workaround for that (not documented in that issue) was to ensure LC_ALL was set.

See also https://github.com/Perl/perl5/pull/21567

[shawnlaffan]

Also, are you running under docker?

[LordAro]

setting LC_ALL doesn't seem to work for me?

$ LC_ALL=en_GB.UTF-8 LANG= ~/Downloads/strawberry-perl-5.38.0.1-64bit-portable/perl/bin/perl -v | head -2
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LC_ALL = "en_GB.UTF-8",
        (possibly more locale environment variables)
        LANG = (unset)
    are supported and installed on your system.
perl: warning: Falling back to the system default locale ("English_United Kingdom.1252").

This is perl 5, version 38, subversion 0 (v5.38.0) built for MSWin32-x64-multi-thread

$ LC_ALL=en_GB.UTF-8 LANG=en_GB.UTF-8 ~/Downloads/strawberry-perl-5.38.0.1-64bit-portable/perl/bin/perl -v | head -2
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LC_ALL = "en_GB.UTF-8",
        (possibly more locale environment variables)
        LANG = "en_GB.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the system default locale ("English_United Kingdom.1252").

This is perl 5, version 38, subversion 0 (v5.38.0) built for MSWin32-x64-multi-thread

I'm not running in docker (yet..), just a standard MSYS2 UCRT64 terminal

[shawnlaffan]

What happens if you run using a Cmd shell?

Run the portableshell.bat file at the top of the Strawberry Perl distribution to set things up.

[LordAro]

No changes with or without portableshell.bat

Without:

C:\Windows\System32>C:\Users\cpigott\Downloads\strawberry-perl-5.38.0.1-64bit-portable\perl\bin\perl.exe -v

This is perl 5, version 38, subversion 0 (v5.38.0) built for MSWin32-x64-multi-thread
...

C:\Windows\System32>set LANG=en_GB.UTF-8

C:\Windows\System32>C:\Users\cpigott\Downloads\strawberry-perl-5.38.0.1-64bit-portable\perl\bin\perl.exe -v
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LC_ALL = (unset),
        (possibly more locale environment variables)
        LANG = "en_GB.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the system default locale ("English_United Kingdom.1252").

This is perl 5, version 38, subversion 0 (v5.38.0) built for MSWin32-x64-multi-thread
...

C:\Windows\System32>set LC_ALL=en_GB.UTF-8

C:\Windows\System32>C:\Users\cpigott\Downloads\strawberry-perl-5.38.0.1-64bit-portable\perl\bin\perl.exe -v
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LC_ALL = "en_GB.UTF-8",
        (possibly more locale environment variables)
        LANG = "en_GB.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the system default locale ("English_United Kingdom.1252").

This is perl 5, version 38, subversion 0 (v5.38.0) built for MSWin32-x64-multi-thread
...

With:

C:\Windows\System32>C:\Users\cpigott\Downloads\strawberry-perl-5.38.0.1-64bit-portable\portableshell.bat
----------------------------------------------
 Welcome to Strawberry Perl Portable Edition!
 * URL - http://www.strawberryperl.com/
 * see README.TXT for more info
----------------------------------------------
Perl executable: C:\Users\cpigott\Downloads\strawberry-perl-5.38.0.1-64bit-portable\perl\bin\perl.exe
Perl version   : 5.38.0 / MSWin32-x64-multi-thread

C:\Windows\System32>perl -v

This is perl 5, version 38, subversion 0 (v5.38.0) built for MSWin32-x64-multi-thread
...

C:\Windows\System32>set LANG=en_GB.UTF-8

C:\Windows\System32>perl -v
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LC_ALL = (unset),
        (possibly more locale environment variables)
        LANG = "en_GB.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the system default locale ("English_United Kingdom.1252").

This is perl 5, version 38, subversion 0 (v5.38.0) built for MSWin32-x64-multi-thread
...

C:\Windows\System32>set LC_ALL=en_GB.UTF-8

C:\Windows\System32>perl -v
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LC_ALL = "en_GB.UTF-8",
        (possibly more locale environment variables)
        LANG = "en_GB.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the system default locale ("English_United Kingdom.1252").

This is perl 5, version 38, subversion 0 (v5.38.0) built for MSWin32-x64-multi-thread
...

[shawnlaffan]

Thanks for testing. I suspect this is an issue in the perl core and might have been addressed in the 5.39 development series.

Just as a final check, what happens if you change the code page in the Cmd prompt before calling perl?

For example for "plain" UTF-8: chcp 65001

[LordAro]

No change

[sisyphus]

I think this is just part of the general poor support for UTF-8 locales with mingw-w64 msvcrt runtimes. On Strawberry-5.38.0:

C:\>perl -MPOSIX -wle "$loc = POSIX::setlocale( LC_ALL, 'en_GB.UTF-8' ); print $loc;"
Use of uninitialized value $loc in print at -e line 1.

Same system, perl-5.38.0 built by Visual Studio 2022:

C:\>perl -MPOSIX -wle "$loc = POSIX::setlocale( LC_ALL, 'en_GB.UTF-8' ); print $loc;"
en_GB.UTF-8

Same system, perl-5.39.3, built mingw-w64 port of gcc-13.2.0 ( "ucrt" build, not "msvcrt" build):

C:\>perl -MPOSIX -wle "$loc = POSIX::setlocale( LC_ALL, 'en_GB.UTF-8' ); print $loc;"
en_GB.UTF-8

It's tempting to therefore think that this issue has been fixed in perl-5.39.3. But I think it works with 5.39.3 only because that perl was built with a mingw-w64 UCRT compiler.

I haven't tried building perl-5.38.0 with that same UCRT mingw-w64 compiler. I think it requires a patch or two. IIRC, 5.39.2 was the first release that could be built straight out of the box with a mingw-w64 UCRT compiler. If you build perl-5.39.3 using a mingw-w64 MSVCRT compiler, I think you'll still hit the same problem with the en_GB.UTF-8 locale. (But I'm not 100% sure about that.)

Cheers, Rob

[shawnlaffan]

I think another batch of locale changes landed in time for perl 5.39.4, which was released yesterday. Would be worth testing.

[LordAro]

I'm game. ...how do I do that? :)

[shawnlaffan]

For a basic vanilla perl, which is all that is needed here, you can adapt the instructions in https://github.com/StrawberryPerl/Perl-Dist-Strawberry/issues/39#issuecomment-1181826988

Alternately you could run with the strawberry perl infrastructure but delete any of the CPAN entries to save time from the config file. The one for 5.38 is at https://github.com/StrawberryPerl/Perl-Dist-Strawberry/blob/master/share/64bit-5.38.0.1.pp .

Clone this repo, then build the Perl::Dist::Strawberry module using whichever perl you like (edit - whichever strawberry perl). Then adapt the script at https://github.com/StrawberryPerl/Perl-Dist-Strawberry/blob/master/devel.utils/_build-5.38.bat to refer to the correct locations.

[sisyphus]

You'll need a UCRT compiler - eg from https://winlibs.com. Strawberry Perl compilers are MSVCRT, and therefore (apparently) suffer the deficiency.

On a perl-5.39.4 built using gcc version 13.2.0 (MinGW-W64 x86_64-ucrt-mcf-seh, built by Brecht Sanders) I get:

D:\>perl -MPOSIX -wle "print $]; $loc = POSIX::setlocale( LC_ALL, 'en_GB.UTF-8' ); print $loc;"
5.039004
en_GB.UTF-8

But the output of that one-liner reverts to:

5.039004
Use of uninitialized value $loc in print at -e line 1.

when perl-5.39.4 is built using gcc version 13.2.0 (MinGW-W64 x86_64-msvcrt-posix-seh, built by Brecht Sanders).

Cheers, Rob

[shawnlaffan]

Well that answers my suggestion that recent perl changes might fix the issue for MSVCRT compilers.

So the next question is whether the MSVCRT limitations can be worked around?

[sisyphus]

So the next question is whether the MSVCRT limitations can be worked around?

I don't know. I haven't really thought beyond "there's something missing from this runtime".

The latest gcc-13.2.0 compilers have boosted the runtime version from, 11.0.0 to 11.0.1. Perhaps (unlikely ?) they address this issue. (I haven't tested.)

There was a request for submissions re the backporting of stuff into 5.38.1. You could request that the option of building perl-5.38.1 with a UCRT compiler be provided. But there's probably not much point in that if StrawberryPerl is committed to staying with the current gcc toolchain for the 5.38.x cycle. I actually think you would find that if 5.38.1 was buildable using a UCRT compiler, then you could switch to a 13.1.0-UCRT compiler without any need to rebuild any of the 3rd party libraries that are provided, (But this is, of course, unproven and might even be seen as obviously silly to a more knowledgeable person.)

For 5.40 onwards, I think that using a UCRT compiler will be the thing to do.

BTW, I have just checked that 5.38.0 doesn't build with a UCRT compiler - and it doesn't. .... gcc -c -I.\include -I. -I.. -DWIN32 -DWIN64 -DPERLDLL -DPERL_CORE -Os -D__USE_MINGW_ANSI_STDIO -fwrapv -fno-strict-aliasing -DPERL_IS_MINIPERL -omini\fcrypt.o fcrypt.c In file included from ..\perl.h:3783, from ..\perlio.c:43: ..\perlio.c: In function 'PerlIOStdio_invalidate_fileno': ./win32.h:314:33: error: 'FILE' {aka 'struct _iobuf'} has no member named '_file' .... more missing member reports followed

Cheers, Rob

[shawnlaffan]

This discussion looks to also be relevant to #18

[sisyphus]

This discussion looks to also be relevant to https://github.com/StrawberryPerl/Perl-Dist-Strawberry/issues/18

Are you suggesting that modifying perlexe.manifest as @ikegami suggested (https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page) might fix @LordAro's issue ?

I take it that it should be fairly simple to test. Just amend perl-5.38.0 source's win32/perlexe.manifest to its appropriate form and then build perl from that source as normal. Is that it ?

I'm not entirely sure what that "appropriate form" looks like. I envisage it can be attained by simply appending one of the 2 examples (from the learn.microsoft.com page) to the existing perlexe.manifest. Is it that simple ? I don't know which example is the correct one for our case ... but there's only 2 to try out.

I'll give that a try later and see what happens.

Even simpler: Create a manifest file (manifest_cp.txt) containing:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly manifestVersion="1.0" xmlns="urn:schemas-microsoft-com:asm.v1">
  <assemblyIdentity type="win32" name="..." version="6.0.0.0"/>
  <application>
    <windowsSettings>
      <activeCodePage xmlns="http://schemas.microsoft.com/SMI/2019/WindowsSettings">UTF-8</activeCodePage>
    </windowsSettings>
  </application>
</assembly>

Place that file in the directory that contains perl.exe, cd to that directory and run:

>mt.exe -manifest manifest_cp.txt -outputresource:perl.exe;#1

Win32::GetACP() then returns 65001.

What else do we want to test ?

NOTE 1: We should probably also bring perl5.38.0.exe into line with perl.exe:

>mt.exe -manifest manifest_cp.txt -outputresource:perl5.38.0.exe;#1

NOTE 2: It might be a good idea to do all of this in a separate installation of portable SP-5.38.0. (Just in case we end up doing something that later turns out to be difficult to reverse.)

I used mt.exe from my VisualStudio-2022 installation.

Cheers, Rob

[shawnlaffan]

Are you suggesting that modifying perlexe.manifest as @ikegami suggested (https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page) might fix @LordAro's issue ?

I was more thinking the UTF-8 issues raised here would also impact that request, but if there is a solution without needing a new build then that's also welcome.

[sisyphus]

I was more thinking the UTF-8 issues raised here would also impact that request,

Heh ... I see. I think you're right. Having altered the ACP to 65001, the en_GB.UTF-8 locale is still unavailable on SP-5.38.0

@LordAro earlier mentioned the "workaround" of setting the PERL_BADLANG environment variable to a true value. AFAICT that will simply suppress the warning, but it won't alter the functionality in any way.

@LordAro, are you sure that the requested locale was actually being set in perl-5.32 and 5.36 ? I think it's more likely that it was not set, but the failure was simply not reported.

Cheers, Rob

[tonycoz]

BTW, I have just checked that 5.38.0 doesn't build with a UCRT compiler - and it doesn't.

The changes need to build with UCRT are a lot simpler than I expected, See https://github.com/Perl/perl5/commit/26b63da690b28dba2297996624585ef2813f8409

I think it's more likely that it was not set, but the failure was simply not reported.

You can test whether the locale supported with something like:

C:\Users\tony>perl -MPOSIX -E "setlocale(LC_ALL,  'English_United States.1252') or die"

C:\Users\tony>perl -MPOSIX -E "setlocale(LC_ALL,  'English_United States.65001') or die"
Died at -e line 1.

If you have the perl source unpacked, the loc_tools.pl test library can list the libraries it finds on the system:

# MSVCRT build of perl
C:\Users\tony>perl -MPOSIX -E "require 'c:/Users/tony/dev/perl/git/perl/t/loc_tools.pl'; say for find_locales('ALL', 1);"
Afrikaans_South Africa.1252
Albanian_Albania.1250
Arabic_Saudi Arabia.1256
Basque_Spain.1252
Breton_France.1252
Bulgarian_Bulgaria.1251
C
Catalan_Spain.1252
Croatian_Croatia.1250
Czech_Czechia.1250
Danish_Denmark.1252
Dutch_Netherlands.1252
English_United States.1252
English_United States.1252
Estonian_Estonia.1257
...
# UCRT build of perl
C:\Users\tony\dev\perl\git\perl>perl -Ilib -E "require './t/loc_tools.pl'; say for find_locales('ALL', 1)"
Afrikaans_South Africa.1252
Afrikaans_South Africa.utf8
Afrikaans_South Africa.utf8
Afrikaans_South Africa.utf8
Afrikaans_South Africa.utf8
...
sr_yu
sr_yu.UTF-8
sv
sv_fi
sv_fi.UTF-8
sv_se
sv_se.UTF-8
th
th_th
th_th.UTF-8
tr
tr_tr
tr_tr.UTF-8
zh_cn.UTF-8
zh_tw.UTF-8

Perl is limited to the locales available from the CRT whether on Windows or POSIX systems.

[tonycoz]

The workaround for that (not documented in that issue) was to ensure LC_ALL was set.

See also Perl/perl5#21567

Perl/perl5#21567 is unrelated to LC_ALL being set in the environment.

[LordAro]

@LordAro, are you sure that the requested locale was actually being set in perl-5.32 and 5.36 ? I think it's more likely that it was not set, but the failure was simply not reported.

This seems.. possible? (Reverted to en_US to make sure that my output matches yours, also couldn't work out what the "proper" full name of en_GB.UTF-8 was)

C:\Users\LordAro\Downloads>set LANG=

C:\Users\LordAro\Downloads>.\strawberry-perl-5.32.1.1-64bit-portable\perl\bin\perl.exe -MPOSIX -E "setlocale(LC_ALL, 'English_United States.1252') or die"

C:\Users\LordAro\Downloads>.\strawberry-perl-5.32.1.1-64bit-portable\perl\bin\perl.exe -MPOSIX -E "setlocale(LC_ALL, 'English_United States.65001') or die"
Died at -e line 1.

C:\Users\LordAro\Downloads>.\strawberry-perl-5.36.1.1-64bit-portable\perl\bin\perl.exe -MPOSIX -E "setlocale(LC_ALL, 'English_United States.1252') or die"

C:\Users\LordAro\Downloads>.\strawberry-perl-5.36.1.1-64bit-portable\perl\bin\perl.exe -MPOSIX -E "setlocale(LC_ALL, 'English_United States.65001') or die"
Died at -e line 1.

C:\Users\LordAro\Downloads>.\strawberry-perl-5.38.0.1-64bit-portable\perl\bin\perl.exe -MPOSIX -E "setlocale(LC_ALL, 'English_United States.1252') or die"

C:\Users\LordAro\Downloads>.\strawberry-perl-5.38.0.1-64bit-portable\perl\bin\perl.exe -MPOSIX -E "setlocale(LC_ALL, 'English_United States.65001') or die"
Died at -e line 1.

C:\Users\LordAro\Downloads>set LANG=en_US.UTF-8

C:\Users\LordAro\Downloads>.\strawberry-perl-5.32.1.1-64bit-portable\perl\bin\perl.exe -MPOSIX -E "setlocale(LC_ALL, 'English_United States.1252') or die"

C:\Users\LordAro\Downloads>.\strawberry-perl-5.32.1.1-64bit-portable\perl\bin\perl.exe -MPOSIX -E "setlocale(LC_ALL, 'English_United States.65001') or die"
Died at -e line 1.

C:\Users\LordAro\Downloads>.\strawberry-perl-5.36.1.1-64bit-portable\perl\bin\perl.exe -MPOSIX -E "setlocale(LC_ALL, 'English_United States.1252') or die"

C:\Users\LordAro\Downloads>.\strawberry-perl-5.36.1.1-64bit-portable\perl\bin\perl.exe -MPOSIX -E "setlocale(LC_ALL, 'English_United States.65001') or die"
Died at -e line 1.

C:\Users\LordAro\Downloads>.\strawberry-perl-5.38.0.1-64bit-portable\perl\bin\perl.exe -MPOSIX -E "setlocale(LC_ALL, 'English_United States.1252') or die"
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LC_ALL = (unset),
        (possibly more locale environment variables)
        LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the system default locale ("English_United Kingdom.1252").

C:\Users\LordAro\Downloads>.\strawberry-perl-5.38.0.1-64bit-portable\perl\bin\perl.exe -MPOSIX -E "setlocale(LC_ALL, 'English_United States.65001') or die"
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LC_ALL = (unset),
        (possibly more locale environment variables)
        LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the system default locale ("English_United Kingdom.1252").
Died at -e line 1.

(with codepage 850)

[sisyphus]

also couldn't work out what the "proper" full name of en_GB.UTF-8 was

When, on a UCRT build of perl-5.39.4 I run the script that @tonycoz provided, I see the following listings for "English":

English_United States.1252
English_United States.utf8
en
en_au
en_au.UTF-8
en_ca
en_ca.UTF-8
en_gb
en_gb.UTF-8
en_ie
en_ie.UTF-8
en_nz
en_nz.UTF-8
en_uk
en_uk.UTF-8
en_us
en_us.UTF-8
en_zw
en_zw.UTF-8

(That's a bit odd - my locale is apparently English_Australia.1252, and that's not even mentioned.) With Strawberry Perl, the only listing for "English" that I see is the one at the top of that list. I guess those listings reflect "official names", but there seems to be a fair amount of case-insensitive leniency regarding what one must specify.

Oddly, with SP-5.38.0, when I try to set LC_ALL to English_United States.65001, I don't get those warnings that you see :

>perl.exe -MPOSIX -E "setlocale(LC_ALL, 'English_United States.65001') or die"
Died at -e line 1.

I can't explain that difference. However, it looks very much to me that (except for that warning) you're getting the same behaviour on the three Strawberry Perls.

Cheers, Rob

[tonycoz]

The warnings are based on what's in the environment.

As to the Australia locale not being listed, the list on Windows is synthesized from a list at the end of loc_tools.pl, it's possible it doesn't have the right entry to pick it up.

[khwilliamson]

That list inloc_tools.pl was created by someone a long time ago. If anyone knows how to get Windows to output the list of its known locales, I would appreciate getting that.

[shawnlaffan]

If anyone knows how to get Windows to output the list of its known locales, I would appreciate getting that.

Maybe the powershell code in this link is a good place to start? https://ss64.org/viewtopic.php?f=6&t=314

[sachiesaaan]

Did anybody solve this?

[sisyphus]

Did anybody solve this?

If you're asking whether there's a known solution that will fix this problem on current Strawberry Perl builds, then AFAIK the answer is "no".

If you're asking whether there's a known solution that will avoid this problem on future releases of Strawberry Perl then I think the answer is "yes". That solution is, I believe, to build perl with a UCRT (as opposed to MSVCRT) compiler.

Based on the behaviour of my current perl builds that were built using UCRT compilers (both Visual Studio 2022 and mingw-w64 ports of gcc-13.2.0), it seems that will fix many locale-related issues. But there might be other stuff that I'm unaware of going on.

[shawnlaffan]

Did anybody solve this?

UCRT aside, this is something that needs to be handled in Perl itself. See https://github.com/Perl/perl5/issues/21562

[sisyphus]

> UCRT aside, this is something that needs to be handled in Perl itself. See https://github.com/Perl/perl5/issues/21562

Yes, thanks for pointing that out. That issue has been labelled as a "Release Blocker", so there's obviously something there that needs to be addressed by perl itself, in addition to what UCRT takes care of.

UPDATE: Just to make it a little clearer, it was incorrect of me to say that UCRT will help with this particular issue. UCRT will help with another issue (regarding setting of UTF8 locales), but not this one.

[khwilliamson]

I just ran the original test case on blead, and got the same result as reported. Further investigation showed that the message displayed is actually correct. There is no en_GB.UTF-8 locale on my system. If I run

LANG=English perl -v

I get no message. Similarly with LANG=uk I do get a message with LANG=au It appears to me, then, that this change is actually a bug fix, notifying you that you are trying to use a locale that doesn't exist on your system.

The release blocker is more about the default ACP. It comes up not because the locale doesn't exist on the system, but because using the locale could crash perl. These are multi-byte locales that aren't UTF-8, and it would be a lot of work to support them. But just because the machine's default locale is unsupported by perl doesn't mean that things will actually proceed to have perl use that locale. XS code certainly could and run into trouble, and some POSIX:: functions also could, but pure perl requires a 'use locale' to enable any use of said unsupported locale, and the locale may have been changed to a legal one in the meantime. So the message is potentially bogus.

The case in this ticket is that the environment is explicitly set up to use a locale unknown to the system. And perl has long warned about this; it just didn't catch these instances before.