[BUG] Sporadic MSE C++ test fail
Describe the bug Sporadic MSE test failure as seen in https://github.com/rapidsai/cuml/pull/4585 in log https://gpuci.gpuopenanalytics.com/job/rapidsai/job/gpuci/job/cuml/job/prb/job/cuml-gpu-test/CUDA=11.0,GPU_LABEL=driver-450,LINUX_VER=centos7,PYTHON=3.8/2184/display/redirect
Failure:
[ RUN ] RfTests/MSEObjectiveTestD.MSEObjectiveTest/0
/workspace/.conda-bld/work/cpp/test/sg/rf_test.cu:1182: Failure
The difference between ground_truth_gain and hypothesis_gain is nan, which exceeds params.tolerance, where
ground_truth_gain evaluates to -nan,
hypothesis_gain evaluates to -nan, and
params.tolerance evaluates to 1.0000000000000001e-05.
[ FAILED ] RfTests/MSEObjectiveTestD.MSEObjectiveTest/0, where GetParam() = 32-byte object <DF-BF C0-70 97-8F F2-83 00-08 00-00 40-00 00-00 01-00 00-00
This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.
This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.