Visual Studio 2022 support - More const and some VS (Windows) build artifacts to ignore

[ross-wheeler]

Adding Visual Studio build artifacts into the ignore file and adding in a few more const so Visual Studio can build these files. 2nd PR will have the Visual Studio (Windows) build changes. No source file changes needed after adding these const keywords in.

[ross-wheeler]

This is the Solution/Project file (IDE) pull-request for Visual Studio 2022 v17.9 Community Edition. This should work with CMake or any other Windows build tool like MSBuild. To run in Visual Studio - open the solution file in llm.c\platform\windows\llm_c.sln.

Requirements/Assumptions:

 •	You have the latest VS Studio and Cuda SDK (this is optional but highly recommended).
 •	You have followed Andrej’s instructions and created the data files

Data files should be located as described below:

 •	llm.c\platform\windows\ data\*
 •	llm.c\platform\windows\gpt2_124M.bin
 •	llm.c\platform\windows\gpt2_124M_debug_state.bin
      

Windows porting goal was no source file changes needed (we did end up needing some const keywords added).

[karpathy]

This looks like a really extended PR... I am certainly interested in the tiniest, most minimal thing that makes this compile on Windows.

[ross-wheeler]

Sorry - the smallest would be - just the const changes and the unistd.h file which should be located in platform/windows (new directory). I don't really care where it's located as long as it's somewhere in the repo. I will create a separate PR for just these changes. (The .gitignore is a "nice to have" not mandatory.)

[azret]

re: /openmp on Windows @ross-wheeler unfortunately Microsoft's compiler is pretty bad at this and we need to do some cosmetic changes.

e.g.

void matmul_forward(float* out,
                    float* inp, float* weight, float* bias,
                    int B, int T, int C, int OC) {
    #pragma omp parallel for collapse(2)
    for (int b = 0; b < B; b++) {
        for (int t = 0; t < T; t++) {
          ..... 
            }
        }
    }
}

with /openmp on Windows

cl.exe /openmp -DOMP /I. /I .\dev\win train_gpt2.c

train_gpt2.c train_gpt2.c(165): warning C4849: OpenMP 'collapse' clause ignored in 'parallel for' directive train_gpt2.c(166): error C3015: initialization in OpenMP 'for' statement has improper form train_gpt2.c(193): warning C4849: OpenMP 'collapse' clause ignored in 'parallel for' directive train_gpt2.c(242): warning C4849: OpenMP 'collapse' clause ignored in 'parallel for' directive train_gpt2.c(405): warning C4849: OpenMP 'collapse' clause ignored in 'parallel for' directive

Microsoft's compiler wants the loop to look like this:

void matmul_forward(float* out, float* inp, float* weight, float* bias, int B, int T, int C, int OC) {
    int b;
    #pragma omp parallel for collapse(2)
    for (b = 0; b < B; b++) {
        for (int t = 0; t < T; t++) {
            ....
            }
        }
    }
}

It's really annoying.

[rosslwheeler]

@azret - please use the Solution/Project files while in this repo https://github.com/ross-wheeler/llm.c/tree/master. OpenMP build has been working since last week. Please use https://github.com/karpathy/llm.c/pull/134 if you want to use the Makefile changes from the command line like you are doing above. You do NEED the const fixes for OpenMP (as far as I can tell). I have an issue outstanding with the VS compiler team on this but I'm fairly sure it's their issue/problem. I have a work-around implemented in both repo's https://github.com/ross-wheeler/llm.c/tree/master and https://github.com/rosslwheeler/llm.c.

Also, the OpenMP speedup I am seeing on Windows is 10x. Highly recommended.

Thanks for giving it a try!

[karpathy]

Idea: We create a doc/windows.md where we document how to build on Windows, but it's just instructions (and maybe some code snippets) in Markdown. Because it looks very involved right now. And then we link to this from somewhere in the main README so people can CTRL+F for it easily.

[rosslwheeler]

Let's see if we can resolve https://github.com/karpathy/llm.c/pull/134? Then if there are issues, we can go this path...

[ross-wheeler]

Portions of this are now in and have newer PR's.