LGBMRegressor init_score cause difference

[mzl9039]

I'm trying to write customised objective functions and metric functions, first I try to reproduce the exact result with simplest customised functions "MSE", and I noticed I need to set init_score for both train dataset and test dataset. However, when I use the default objective function without init_score in dataset, the result is different from using the default objective function with init_score in dataset.

LDB version: 3.2.1

params keep same:

parameters = {
        'num_leaves': 90,
        'learning_rate': 0.05, 
        'reg_alpha': 0,
        'min_child_samples': 20,
        'max_depth': 20, 
        'n_estimators': 20000,
        }

1. default objective function without init_score in dataset:

gbm = lgb.LGBMRegressor(**parameters)

gbm.fit(
    train[feature],
    train[label],
    eval_metric=["l2"],
    eval_set=[(validation[feature], validation[label])],
    early_stopping_rounds=50,
    verbose=10,
)

and the log is:

Starting training...
Training until validation scores don't improve for 50 rounds
[10]	valid_0's l2: 0.368826
[20]	valid_0's l2: 0.286741
[30]	valid_0's l2: 0.254781
[40]	valid_0's l2: 0.243023
[50]	valid_0's l2: 0.236962
[60]	valid_0's l2: 0.234841
[70]	valid_0's l2: 0.233332
[80]	valid_0's l2: 0.2336
[90]	valid_0's l2: 0.233803
[100]	valid_0's l2: 0.23413
[110]	valid_0's l2: 0.234808
[120]	valid_0's l2: 0.23487
Early stopping, best iteration is:
[74]	valid_0's l2: 0.23274

2.default objective function with init_score in dataset

noticed that the init_score in regression should be the mean of the label, from the function BoostFromScore in class RegressionL2loss in file regression_objective.hpp

gbm = lgb.LGBMRegressor(**parameters)

gbm.fit(
    train[feature],
    train[label],
    eval_set=[(validation[feature], validation[label])],
    eval_metric=["l2"],
    early_stopping_rounds=50,
    init_score=np.full_like(train[label], train[label].mean(), dtype=np.float),
    eval_init_score=[np.full_like(validation[label], train[label].mean(), dtype=np.float)],
    verbose=10,
)

and the log is:

Starting training...
Training until validation scores don't improve for 50 rounds
[10]	valid_0's l2: 0.368826
[20]	valid_0's l2: 0.286741
[30]	valid_0's l2: 0.254781
[40]	valid_0's l2: 0.243023
[50]	valid_0's l2: 0.236998
[60]	valid_0's l2: 0.234876
[70]	valid_0's l2: 0.233381
[80]	valid_0's l2: 0.233651
[90]	valid_0's l2: 0.233854
[100]	valid_0's l2: 0.23418
[110]	valid_0's l2: 0.234859
[120]	valid_0's l2: 0.234922
Early stopping, best iteration is:
[74]	valid_0's l2: 0.232791

Though the difference is slightly, I still wonder what cause this difference?

[guolinke]

For floating-point arithmetic operations, the results are not deterministic. Different hardwares, compilers, algorithms (like the order of performing sum) usually result in different results. So it is normal to get slightly different results.

[mzl9039]

For floating-point arithmetic operations, the results are not deterministic. Different hardwares, compilers, algorithms (like the order of performing sum) usually result in different results. So it is normal to get slightly different results.

Really thx for explanation! In fact, these 2 results are generated on a linux server with multiple processors, maybe this is the reason for the difference.