Error changing number of threads in HiGHS

[mtanneau]

I find myself unable to set/change/restrict the number of threads used by HiGHS.

It feels like HiGHS internally initializes how many threads it's going to use, an that modifying it later through MOI.NumberOfThreads creates some sort of internal conflict.

Code to reproduce:

using JuMP
using HiGHS

model = Model(HiGHS.Optimizer)
@variable(model, x >= 0)
@objective(model, Min, x)
optimize!(model)
termination_status(model)  # so far so good

model = Model(HiGHS.Optimizer)
set_optimizer_attribute(model, MOI.NumberOfThreads(), 1)
@variable(model, x >= 0)
@objective(model, Min, x)
optimize!(model)  # will print an error message from HiGHS
termination_status(model)  # will return OTHER_ERROR

will output:

ERROR:   Option 'threads' is set to 1 but global scheduler has already been initialized to use 8 threads. The previous scheduler instance can be destroyed by calling HighsTaskExecutor::shutdown().
OTHER_ERROR::TerminationStatusCode = 24

I ran this on an 8-core machine, hence (I suppose?) the "initialized to use 8 threads" part of HiGHS' error message.

[mtanneau]

I tried to dig into this, but this is beyond my C++ understanding. What I could gather is the following stack of calls:

1. MOI.set(::HiGHS.Optimizer, ::MOI.NumberOfThreads(), ::Int) (julia)
2. MOI.set(::HiGHS.Optimizer, ::MOI.RawAttribute("threads"), ::Int) (julia)
3. HiGHS._set_option(::HiGHS.Optimizer, ::String, ::Integer) (julia)
4. Highs_setIntOptionValue(highs, option, value) (C <-- julia)
5. As far as I can tell, Highs_setIntOptionValue calls this function, at which point I got lost.

The error message itself suggests that, to change the number of threads, one should reset HiGHS' internal scheduler. ~I have no idea whether one could nor how one would do it from Julia / the C interface.~ We would probably need someone from upstream to advise.

Edit: it looks like resetting the scheduler is exposed in the C API here, but I could not find it in src/gen/libhighs.jl. For reference, the is the C API code:

/**
 * Releases all resources held by the global scheduler instance. It is
 * not thread-safe to call this function while calling Highs_run()/Highs_*call()
 * on any other Highs instance in any thread. After this function has terminated
 * it is guaranteed that eventually all previously created scheduler threads
 * will terminate and allocated memory will be released. After this function
 * has returned the option value for the number of threads may be altered to a
 * new value before the next call to Highs_run()/Highs_*call(). If the given
 * parameter has a nonzero value, then the function will not return until all
 * memory is freed, which might be desirable when debugging heap memory but
 * requires the calling thread to wait for all scheduler threads to wake-up
 * which is usually not necessary.
 *
 * @returns No status is returned since the function call cannot fail. Calling
 * this function while any Highs instance is in use on any thread is
 * undefined behavior and may cause crashes, but cannot be detected and hence
 * is fully in the callers responsibility.
 */
void Highs_resetGlobalScheduler(const HighsInt blocking);

That comment is scary 😱 I understand it as "don't ever do this unless you're absolutely sure no other HiGHS instance is running on any thread"

[odow]

Highs_resetGlobalScheduler is now in the C API as part of HiGHS v1.3.0.

[odow]

This works:

julia> using HiGHS

julia> Highs_resetGlobalScheduler(1)

julia> model = HiGHS.Optimizer()
A HiGHS model with 0 columns and 0 rows.

julia> x = MOI.add_variable(model)
Running HiGHS 1.3.0 [date: 1970-01-01, git hash: e5004072b-dirty]
Copyright (c) 2022 ERGO-Code under MIT licence terms
MathOptInterface.VariableIndex(1)

julia> MOI.set(model, MOI.NumberOfThreads(), 1)

julia> MOI.optimize!(model)
Presolving model
0 rows, 0 cols, 0 nonzeros
0 rows, 0 cols, 0 nonzeros
Presolve : Reductions: rows 0(-0); columns 0(-1); elements 0(-0) - Reduced to empty
Solving the original LP from the solution after postsolve
Solving an unconstrained LP with 1 columns
Model   status      : Optimal
Objective value     :  0.0000000000e+00
HiGHS run time      :          0.00

julia> Highs_resetGlobalScheduler(1)

julia> MOI.set(model, MOI.NumberOfThreads(), 2)

julia> MOI.optimize!(model)
Solving LP without presolve or with basis
Solving an unconstrained LP with 1 columns
Model   status      : Optimal
Objective value     :  0.0000000000e+00
HiGHS run time      :          0.00

I don't know if we want to be mucking with Highs_resetGlobalScheduler inside the library. You can call it from user-land if you want to change the number of threads.

[odow]

Closing this as won't fix. If it comes up again, we can revisit. It's a shame HiGHS has a global scheduler, but it isn't safe for us to muck around with that in Julia-land.