Hyperactive icon indicating copy to clipboard operation
Hyperactive copied to clipboard

[BUG] sporadic failure of `test_trafo_1` at timings check

Open fkiraly opened this issue 4 months ago • 1 comments

test_trafo_1 fails sporadically with a timings check. Example traceback below.

If this is not a strict requirement, we should perhaps remove this or make it more lenient?

================================== FAILURES ===================================
______________ test_trafo_1[search_space2-memory_warm_start2-4] _______________

random_state = 4
search_space = {'min_samples_split': [np.int64(22), np.int64(23), np.int64(24), np.int64(25), np.int64(26), np.int64(27), ...]}
memory_warm_start =    min_samples_split  score
0                 22     22
1                 23     23
2                 24     24
3     ...     27
6                 28     28
7                 29     29
8                 30     30
9                 31     31

    @pytest.mark.parametrize("random_state", random_state_para_0)
    @pytest.mark.parametrize("search_space, memory_warm_start", search_space_para_1)
    def test_trafo_1(random_state, search_space, memory_warm_start):
        search_space = search_space
        memory_warm_start = memory_warm_start
    
        c_time_0 = time.perf_counter()
        hyper = Hyperactive()
        hyper.add_search(
            objective_function_1,
            search_space,
            n_iter=10,
            random_state=random_state,
            initialize={"random": 1},
        )
        hyper.run()
        d_time_0 = time.perf_counter() - c_time_0
    
        c_time_1 = time.perf_counter()
        hyper = Hyperactive()
        hyper.add_search(
            objective_function_1,
            search_space,
            n_iter=10,
            random_state=random_state,
            initialize={"random": 1},
            memory_warm_start=memory_warm_start,
        )
        hyper.run()
        d_time_1 = time.perf_counter() - c_time_1
    
>       assert d_time_1 < d_time_0 * 0.5
E       assert 1.027204499999982 < (1.9211578000000031 * 0.5)

fkiraly avatar Jul 27 '25 18:07 fkiraly

We should remove this test. It is one of several tests, that work with measuring and comparing optimization time. This can be unreliable especially in the CI.

SimonBlanke avatar Jul 31 '25 01:07 SimonBlanke