Bad codegen when building dmd with optimizations

[rainers]

Sometimes when building a release build of dmd with optimizations the resulting executable (windows x64) crashes immediately.

This time I have reduced it to building from the compiler directory with

c:\d\dmd-2.101.1\windows\bin\dmd.exe -Jsrc\dmd\res -w -de -m64 -J..\generated\windows\release\64 -Isrc -O -inline -release -c src\dmd\arraytypes.d src\dmd\mars.d

This generates bad code for this short snippet: https://github.com/dlang/dmd/blob/66bd65c569ca430186374599aa50a6d6138643c4/compiler/src/dmd/mars.d#L281-L284

  0000000000000142: 31 C9              xor         ecx,ecx
  0000000000000144: 4D 8B E0           mov         r12,r8
  0000000000000147: 49 FF C0           inc         r8
  000000000000014A: 41 88 14 24        mov         byte ptr [r12],dl
  000000000000014E: 48 83 C4 01        add         rsp,1
  0000000000000152: E9 6F FF FF FF     jmp         00000000000000C6

Note the addition of 1 to RSP! If the file before mars.d is omitted, R11 is used instead of R12 and the add is encoded correctly (though the addition is futile to begin with).

Maybe the spurious failure of test/runnable/exe1.c is related.

[rainers]

Reduced to

module mars;

extern(C) void* malloc(size_t);

struct Strings
{
    size_t length;
	alias T = const(char)*;
private:
    T[] data;

public:
    ref Strings push(T ptr) return
    {
        reserve(1);
        data[length++] = ptr;
        return this;
    }

    void reserve(size_t nentries)
    {
        void enlarge(size_t nentries)
        {
            const allocdim = length + nentries;
            auto p = cast(T*)malloc(allocdim * T.sizeof);
        }

        if (data.length - length < nentries)  // false means hot path
            enlarge(nentries);
    }
}

void getenv_setargv(const(char)* envvalue, Strings* args)
{
    if (!envvalue)
        return;

    char* env = cast(char*)envvalue; // create our own writable copy

    while (1)
    {
        switch (*env)
        {
            case 0:
                return;

            default:
            {
                args.push(env); // append
                auto p = env;
                while (1)
                {
                    auto c = *env++;
                    switch (c)
                    {
                        case 0:
                            *p = 0;
                            break;

                        default:
                            *p++ = c;
                            continue; // increments RSP here!
                    }
                    break;
                }
                break;
            }
        }
    }
}

Compile with

c:\d\dmd-2.111.0\windows\bin\dmd.exe -m64 -O -inline -c -release mars.d

Happens at least back until dmd 2.101.

[WalterBright]

It's awesome you were able to track this down! Well done!