[CLI, c++] lib_lightgbm dll crashes while loading a 2GB model

[wil70]

Description

lib_lightgbm.dll works great for models up to 1.8GB. When loading a model above 1.8GB then lightgbm or a c++ program that uses that dll crashes. I cc a 2GB model.

I'm not familiar enough with c++ to get the intricacies of the bug. It sounds like a 64bits vs 32 bits issues with memory limitation but I compiled everything in 64 bits, or it might be an array of sort that has not enought memory allocated?

Reproducible example

via CLI c:>lightgbm config=predict.conf make sure task=predict and the input

# task type, support train and predict  save_binary
task = predict
force_col_wise=true
device_type=cpu
boosting_type = gbdt
objective = multiclass
metric = multi_logloss,multi_error
num_class = 2
#metric_freq = 1
#is_training_metric = true
header = false
label_column=9603
categorical_feature=0,95089,95090,95091,95092,95093,95094,95095,95096,95097,95098,95099,9600,9601,9602
max_bin = 255
data = a_test_file.csv  Note: Please use the following c++ program directly that call LGBM_BoosterCreateFromModelfile so the testing data is not needed 
num_trees = 10000
learning_rate = 0.1
input_model = "path/to/your/model_138.txt"
two_round = true
snapshot_freq = 1

Via c++ (This isolate to the function that crashes and doesn't require the test sample)

#include <LightGBM/c_api.h>
#include <iostream>

int main() {
    const char* modelFilename = "path/to/your/model_138.txt";  // Replace with your model file path

    int numIterations;
    BoosterHandle booster;

    int result = LGBM_BoosterCreateFromModelfile(modelFilename, &numIterations, &booster);

    if (result == 0) {
        // Model loaded successfully, you can use the booster
        std::cout << "Model loaded with " << numIterations << " iterations." << std::endl;

        // Use the loaded booster for predictions or other tasks

        // Don't forget to free the booster handle when you're done
        LGBM_BoosterFree(booster);
    } else {
        std::cerr << "Failed to load the model." << std::endl;
    }

    return 0;
}

Environment info

LightGBM version or commit hash: Latest as of dec 24th

Command(s) you used to install LightGBM

I use lightgbm with task=predict and I saw it crashing afor model above 2GB. I use a c# program that calls the lib_lightgbm.dll and it works great for model under 2GB. Models above 2GB crashes as saw and expected from ightgbm.exe with task=predict. After further investigation I saw the crash happen in the lib_lightgbm.dll. The windows event viewer confirmed this (see trace lower).

I use Win 10 PRO , vs 2022, I compile lightgbm with cpu mode and in 64 bits

Additional Comments

As the program (lightgbm.exe or the c++ sample) loads the model to the memory, the memory grows to 5GB and then the program exit and displays (process 32244) exited with code -1073741819. When I look at the windows event viewer I can see the follpowing error with a smaple code and this is the same error if I use lightgbm.exe

Faulting application name: lightgbm_sample_code.exe, version: 1.0.0.0, time stamp: 0x64e841c5
Faulting module name: lib_lightgbm.DLL, version: 0.0.0.0, time stamp: 0x658787fe
Exception code: 0xc0000005
Fault offset: 0x00000000000b26e7
Faulting process id: 0x7df4
Faulting application start time: 0x01da3681d6a4c469
Faulting application path: C:\Users\wilhe\source\lightgbm_sample\lightgbm_sample.exe
Faulting module path: C:\Users\wilhe\source\lightgbm\x64\lib_lightgbm.DLL
Report Id: faf336f4-a010-4b4e-ad5a-3b2796b8f351
Faulting package full name: 
Faulting package-relative application ID: 

Model File is attached: model_138.zip

[wil70]

Unhandled exception thrown: read access violation. cur_p was 0x219CB55DBE5. lightgbm.exe!LightGBM::GBDT::LoadModelFromString$omp$1() Line 559 C++