Unit tests fail on macOS (and probably on Linux too)

[0xced]

Here is how to reproduce:

$ cd src/Humanizer.Tests
$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.15.6
BuildVersion:	19G2021
$ dotnet --version
5.0.101
$ dotnet test --framework netcoreapp2.1
Microsoft (R) Build Engine version 16.8.0+126527ff1 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  All projects are up-to-date for restore.
  Humanizer -> ~/Projects/Humanizer/src/Humanizer/bin/Debug/netstandard2.0/Humanizer.dll
  Humanizer.Tests -> ~/Projects/Humanizer/src/Humanizer.Tests/bin/Debug/netcoreapp2.1/Humanizer.Tests.dll
Test run for ~/Projects/Humanizer/src/Humanizer.Tests/bin/Debug/netcoreapp2.1/Humanizer.Tests.dll (.NETCoreApp,Version=v2.1)
Microsoft (R) Test Execution Command Line Tool Version 16.8.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:06.33]     Years(days: 366, expected: "1 年") [FAIL]
  Failed Years(days: 366, expected: "1 年") [4 ms]
  Error Message:
   Assert.Equal() Failure
            ↓ (pos 2)
Expected: 1 年
Actual:   1 year
            ↑ (pos 2)
  Stack Trace:
     at Humanizer.Tests.Localisation.zhCN.TimeSpanHumanizeTests.Years(Int32 days, String expected) in ~/Projects/Humanizer/src/Humanizer.Tests.Shared/Localisation/zh-CN/TimeSpanHumanizeTests.cs:line 18

[...] Many more failed tests [...]

Failed!  - Failed:   102, Passed: 10366, Skipped:     0, Total: 10468, Duration: 38 s - ~/Projects/Humanizer/src/Humanizer.Tests/bin/Debug/netcoreapp2.1/Humanizer.Tests.dll (netcoreapp2.1)

I have noticed that the zh-CN resources are not copied into src/Humanizer.Tests/bin/Debug/netcoreapp2.1. All other localizations are there (af, ar, ... fr, fr-BE, ..., zh-Hans, zh-Hant) but there is no zh-CN directory containing a Humanizer.resources.dll file.

So I ran dotnet build -bl and examined the binary log with MSBuild Structured Log Viewer. It turns out that the AssignCulture task (which is run by the SplitResourcesByCulture target) miscategorizes the zh-CN culture which ends up in the ResxWithNoCulture item group instead of the ResxWithCulture item group where all other localized resources are.

I'm currently investigating why the AssignCulture task does not recognize zh-CN as a valid culture on macOS.

[0xced]

Here's the small program I wrote to diagnose the AssignCulture task.

assignculture.csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net5.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="NSubstitute" Version="4.2.2" />
    <PackageReference Include="ReflectionMagic" Version="4.1.0" />
  </ItemGroup>

  <ItemGroup Condition="'$(ReferenceMsBuildNuGetPackages)' != 'true'">
    <Reference Include="/usr/local/share/dotnet/sdk/5.0.101/Microsoft.Build.Framework.dll" />
    <Reference Include="/usr/local/share/dotnet/sdk/5.0.101/Microsoft.Build.Tasks.Core.dll" />
    <Reference Include="/usr/local/share/dotnet/sdk/5.0.101/Microsoft.Build.Utilities.Core.dll" />
  </ItemGroup>

  <ItemGroup Condition="'$(ReferenceMsBuildNuGetPackages)' == 'true'">
    <PackageReference Include="Microsoft.Build.Tasks.Core" Version="16.8.0" />
  </ItemGroup>
  
</Project>

Program.cs

using System;
using System.Globalization;
using System.Linq;
using System.Runtime.InteropServices;
using Microsoft.Build.Framework;
using Microsoft.Build.Tasks;
using Microsoft.Build.Utilities;
using NSubstitute;
using ReflectionMagic;

try
{
    SimulateMsBuildInitialization();

    PrintAllChineseCultures();

    var task = RunAssignCultureTask("Resources.zh-CN.resx", "Resources.zh-Hans.resx");

    PrintAssignedFiles(task);
}
catch (Exception exception)
{
    Console.Error.WriteLine(exception);
}

static void SimulateMsBuildInitialization()
{
    var assembly = typeof(AssignCulture).Assembly;
    assembly.GetType("Microsoft.Build.Shared.AssemblyUtilities", throwOnError: true).AsDynamicType().Initialize();
    Console.WriteLine($"Running on {RuntimeInformation.FrameworkDescription} using {assembly} at {assembly.Location}");
    Console.WriteLine();
}

static void PrintAllChineseCultures()
{
    Console.WriteLine("*** All Chinese cultures ***");
    foreach (var culture in CultureInfo.GetCultures(CultureTypes.AllCultures).Where(e => e.Name.StartsWith("zh")))
    {
        Console.WriteLine($"[{culture.Name}] {culture.EnglishName}");
    }
    Console.WriteLine();
}

static AssignCulture RunAssignCultureTask(params string[] cultureNames)
{
    var buildEngine = Substitute.For<IBuildEngine>();
    buildEngine.LogMessageEvent(Arg.Do<BuildMessageEventArgs>(e => Console.WriteLine($"[{e.Importance}] {e.Message}")));
    var task = new AssignCulture
    {
        BuildEngine = buildEngine,
        Files = cultureNames.Select(e => new TaskItem(e)).Cast<ITaskItem>().ToArray(),
    };
    task.Execute();
    Console.WriteLine();
    return task;
}

static void PrintAssignedFiles(AssignCulture task)
{
    PrintTaskItems(task.AssignedFilesWithNoCulture, nameof(task.AssignedFilesWithNoCulture));
    PrintTaskItems(task.AssignedFilesWithCulture, nameof(task.AssignedFilesWithCulture));
}

static void PrintTaskItems(ITaskItem[] taskItems, string name)
{
    var itemSpecModifiers = typeof(AssignCulture).Assembly.GetType("Microsoft.Build.Shared.FileUtilities+ItemSpecModifiers", throwOnError: true).AsDynamicType();
    Console.WriteLine($"{name}: {taskItems.Length}");
    foreach (var taskItem in taskItems)
    {
        Console.WriteLine($"  {taskItem.ItemSpec}");
        foreach (var metadataName in taskItem.MetadataNames.Cast<string>().Where(e => !itemSpecModifiers.IsItemSpecModifier(e)))
        {
            var metadata = taskItem.GetMetadata(metadataName);
            if (!string.IsNullOrEmpty(metadata))
            {
                Console.WriteLine($"    {metadataName} = {metadata}");
            }
        }
    }
}

Here's the output when run with dotnet run on macOS:

Running on .NET 5.0.1-servicing.20575.16 using Microsoft.Build.Tasks.Core, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a at assignculture/bin/Debug/net5.0/Microsoft.Build.Tasks.Core.dll

*** All Chinese cultures ***
[zh] Chinese
[zh-Hans] Chinese
[zh-Hans-CN] Chinese, Simplified (China mainland)
[zh-Hans-HK] Chinese, Simplified (Hong Kong)
[zh-Hans-MO] Chinese, Simplified (Macao)
[zh-Hans-SG] Chinese, Simplified (Singapore)
[zh-Hant] Chinese
[zh-Hant-CN] Chinese, Traditional (China mainland)
[zh-Hant-HK] Chinese, Traditional (Hong Kong)
[zh-Hant-MO] Chinese, Traditional (Macao)
[zh-Hant-TW] Chinese, Traditional (Taiwan)

[Low] Culture of "" was assigned to file "Resources.zh-CN.resx".
[Low] Culture of "zh-Hans" was assigned to file "Resources.zh-Hans.resx".

AssignedFilesWithNoCulture: 1
  Resources.zh-CN.resx
    WithCulture = false
    OriginalItemSpec = Resources.zh-CN.resx
AssignedFilesWithCulture: 1
  Resources.zh-Hans.resx
    Culture = zh-Hans
    WithCulture = true
    OriginalItemSpec = Resources.zh-Hans.resx

And here's the result when running on Windows with dotnet build /p:ReferenceMsBuildNuGetPackages=true && dotnet run --no-build:

Running on .NET 5.0.1-servicing.20575.16 using Microsoft.Build.Tasks.Core, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a at c:\Projects\Experiments\assignculture\bin\Debug\net5.0\Microsoft.Build.Tasks.Core.dll

*** All Chinese cultures ***
[zh] Chinese
[zh-Hans] Chinese
[zh-CN] Chinese (China)
[zh-Hans-HK] Chinese (Simplified, Hong Kong SAR)
[zh-Hans-MO] Chinese (Simplified, Macao SAR)
[zh-SG] Chinese (Singapore)
[zh-Hant] Chinese
[zh-HK] Chinese (Hong Kong SAR)
[zh-MO] Chinese (Macao SAR)
[zh-TW] Chinese (Taiwan)

[Low] Culture of "zh-CN" was assigned to file "Resources.zh-CN.resx".
[Low] Culture of "zh-Hans" was assigned to file "Resources.zh-Hans.resx".

AssignedFilesWithNoCulture: 0
AssignedFilesWithCulture: 2
  Resources.zh-CN.resx
    Culture = zh-CN
    WithCulture = true
    OriginalItemSpec = Resources.zh-CN.resx
  Resources.zh-Hans.resx
    Culture = zh-Hans
    WithCulture = true
    OriginalItemSpec = Resources.zh-Hans.resx

The difference, as we can see, is that zh-CN is only known to Windows.

I'm not sure yet what is the best solution to this problem.

[0xced]

Well, it turns out I just re-discovered a known MSBuild issue apparently: https://github.com/dotnet/msbuild/issues/3897