add cuopt direct solver

[Iroy30]

Fixes #3626

Summary/Motivation:

Add cuOpt math optimization (includes LP and MILP) solver backend to Pyomo so users can solve pyomo models with cuOpt

Changes proposed in this PR:

• Add cuopt_direct solver plugin

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

1. I agree my contributions are submitted under the BSD license.
2. I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

[codecov[bot]]

Codecov Report

:x: Patch coverage is 77.27273% with 60 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 89.38%. Comparing base (13facca) to head (767d96a).

Files with missing lines	Patch %	Lines
pyomo/solvers/plugins/solvers/cuopt_direct.py	77.09%	60 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3620      +/-   ##
==========================================
- Coverage   89.40%   89.38%   -0.03%     
==========================================
  Files         909      910       +1     
  Lines      105541   105805     +264     
==========================================
+ Hits        94364    94573     +209     
- Misses      11177    11232      +55     

Flag	Coverage Δ
builders	29.08% <20.07%> (-0.03%)	:arrow_down:
default	86.00% <77.27%> (?)
expensive	35.72% <20.45%> (?)
linux	86.55% <19.31%> (-2.63%)	:arrow_down:
linux_other	86.55% <19.31%> (-0.18%)	:arrow_down:
osx	82.71% <19.31%> (-0.17%)	:arrow_down:
win	84.81% <19.31%> (-0.16%)	:arrow_down:
win_other	84.81% <19.31%> (-0.16%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mrmundt]

@Iroy30 - We've been able to make cuopt available on our internal testing machines. Can you please add tests to this PR?

[Iroy30]

@mrmundt Thanks! We have added tests by enabling testing cuopt with LP and MILP capabilities in tests/solvers.py. Let us know if:

1. There is any other testing that needs to be enabled.
2. For testing cuOpt backend in CI, it needs a GPU, is that something pyomo team intends to add or will it be disabled in github CI?

The following is the testing output I get relevant to cuOpt

Testing scenario generation
------------------------------
LP_block, cuopt, python
   ok: 
LP_block_kernel, cuopt, python
   ok: 
LP_compiled, cuopt, python
   ok: 
LP_compiled_dense_kernel, cuopt, python
   ok: 
LP_compiled_sparse_kernel, cuopt, python
   ok: 
LP_constant_objective1, cuopt, python
   ok: 
LP_constant_objective1_kernel, cuopt, python
   ok: 
LP_constant_objective2, cuopt, python
   ok: 
LP_constant_objective2_kernel, cuopt, python
   ok: 
LP_duals_maximize, cuopt, python
   ok: 
LP_duals_maximize_kernel, cuopt, python
   ok: 
LP_duals_minimize, cuopt, python
   ok: 
LP_duals_minimize_kernel, cuopt, python
   ok: 
LP_inactive_index, cuopt, python
   ok: 
LP_inactive_index_kernel, cuopt, python
   ok: 
LP_infeasible1, cuopt, python
   ok: 
LP_infeasible1_kernel, cuopt, python
   ok: 
LP_infeasible2, cuopt, python
   ok: 
LP_infeasible2_kernel, cuopt, python
   ok: 
LP_piecewise, cuopt, python
   ok: 
LP_piecewise_nosuffixes, cuopt, python
   ok: 
LP_simple, cuopt, python
   ok: 
LP_simple_kernel, cuopt, python
   ok: 
LP_trivial_constraints, cuopt, python
   ok: 
LP_trivial_constraints_kernel, cuopt, python
   ok: 
LP_unbounded, cuopt, python
   ok: 
LP_unbounded_kernel, cuopt, python
   ok: 
LP_unused_vars, cuopt, python
   ok: 
LP_unused_vars_kernel, cuopt, python
   ok: 
MILP_discrete_var_bounds, cuopt, python
   ok: 
MILP_discrete_var_bounds_kernel, cuopt, python
   ok: 
MILP_infeasible1, cuopt, python
   ok: 
MILP_infeasible1_kernel, cuopt, python
   ok: 
MILP_simple, cuopt, python
   ok: 
MILP_simple_kernel, cuopt, python
   ok: 
MILP_unbounded, cuopt, python
   ok: 
MILP_unbounded_kernel, cuopt, python
   ok: 
MILP_unused_vars, cuopt, python
   ok: 
MILP_unused_vars_kernel, cuopt, python
   ok: 

[whart222]

@mrmundt What do you think about including this solver interface in pyomo.contrib.solvers? Would it make sense to pull-in new solver interfaces there, since that's where the new solver API is evolving?

[mrmundt]

@mrmundt What do you think about including this solver interface in pyomo.contrib.solvers? Would it make sense to pull-in new solver interfaces there, since that's where the new solver API is evolving?

@whart222 - I am evenly split. Because we are still messing with what the new solver interfaces are going to actually do / how they will handle input and present output, I don't know if we want to put "new" solvers there or just "well-established" ones that we can robustly test / really know what they are supposed to do and return.

[mrmundt]

@Iroy30 - I forgot to post this last week, but all of the failures are of the variety:

self = <pyomo.solvers.tests.checks.test_no_solution_behavior.Test_LP_unbounded_kernel testMethod=test_cuopt_python>

    def return_test(self):
>       return failed_solve_test(self)

/tester/python/python39/pyomo/pyomo/solvers/tests/checks/test_no_solution_behavior.py:92: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/tester/python/python39/pyomo/pyomo/solvers/tests/checks/test_no_solution_behavior.py:61: in failed_solve_test
    opt, results = model_class.solve(
/tester/python/python39/pyomo/pyomo/solvers/tests/models/base.py:123: in solve
    results = opt.solve(
/tester/python/python39/pyomo/pyomo/opt/base/solvers.py:99: in solve
    self._solver_error('solve')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pyomo.opt.base.solvers.UnknownSolver object at 0x7f86ad9ab040>
method_name = 'solve'

        def _solver_error(self, method_name):
>           raise RuntimeError(
                """Attempting to use an unavailable solver.
    
    The SolverFactory was unable to create the solver "%s"
    and returned an UnknownSolver object.  This error is raised at the point
    where the UnknownSolver object was used as if it were valid (by calling
    method "%s").
    
    The original solver was created with the following parameters:
    \t"""
                % (self.type, method_name)
                + "\n\t".join("%s: %s" % i for i in sorted(self._kwds.items()))
                + "\n\t_args: %s" % (self._args,)
                + "\n\toptions: %s" % (self.options,)
            )
E           RuntimeError: Attempting to use an unavailable solver.
E           
E           The SolverFactory was unable to create the solver "cuopt"
E           and returned an UnknownSolver object.  This error is raised at the point
E           where the UnknownSolver object was used as if it were valid (by calling
E           method "solve").
E           
E           The original solver was created with the following parameters:
E           	executable: cuopt
E           	solver_io: python
E           	type: cuopt
E           	_args: ()
E           	options: {}

/tester/python/python39/pyomo/pyomo/opt/base/solvers.py:116: RuntimeError

[mrmundt]

@Iroy30 - Two more things:

1. Please run black -S -C on your files
2. Something in our testing environment isn't loading correctly. I'm looking into it.

[mrmundt]

Hi @Iroy30 - I got the tests running - the name in the solvers.py file was the culprit. That being said, there are quite a few errors (mentioned in the comment above).

It would be a good idea for you to create a test_cuopt.py file in pyomo/solvers/tests/checks and incorporate unit tests, small tests for basic functionality, little models that you know the answer for, etc.

[Iroy30]

I am looking at the "Attempting to use an unavailable solver" error currently. The rest should be fixed.

Something in our testing environment isn't loading correctly. I'm looking into it.

Is this resolved?

[mrmundt]

I am looking at the "Attempting to use an unavailable solver" error currently. The rest should be fixed.

Something in our testing environment isn't loading correctly. I'm looking into it.

Is this resolved?

Yes, I fixed that. Please look at the other errors I posted in different comments above.

[Iroy30]

we are investigating certain Termination condition mismatches on our end

[Iroy30]

@mrmundt Is there a way to disable certain tests temporarily for cuOpt solver as we improve on it? We are facing two CI failures:

1. RC computation for a maximization problem failures
2. cuOpt does not differentiate between infeasible vs unbounded at this point

[mrmundt]

@mrmundt Is there a way to disable certain tests temporarily for cuOpt solver as we improve on it? We are facing two CI failures:

1. RC computation for a maximization problem failures

2. cuOpt does not differentiate between infeasible vs unbounded at this point

Yeah, you can use @unittest.skip with an appropriate message. Are these real failures that need to be addressed, though?

[Iroy30]

There are active improvements and fixes in cuOpt that are being worked on for the next release. I have skipped two above mentioned tests temporarily. The CI tests are passing for me locally. Hopefully all runs successfully in the pipeline as well.

[Iroy30]

@mrmundt How do I trigger the CI ?

[mrmundt]

@mrmundt How do I trigger the CI ?

I needed to click "Approve" on it, which I have now done!

[Iroy30]

Looks like it is failing as cuOpt is not available in CI? I can reproduce the CI error when cuOpt package is not available in the environment.

[jsiirola]

cuOpt is not available on GHA because the GHA runners (to our knowledge) do not have GPUs. The Jenkins job will run on machines with GPUs and cuOpt, and should exercise this interface. Regardless, solver interfaces must be able to be imported and instantiated (to check for availability) without a hard dependency on any "external" packages.

A quick look at you implementation:

• you will need to guard the numpy import in cuopt_direct.py (the easy way it to import it from pyomo.common.sdependencies
• you should defer resolving the _version and checking if the python api exists until after (outside) the CUOPTDirect.__init__().

[Iroy30]

@jsiirola Thanks John, aded the guards and updated version, hopefully this should fix the fails when cuopt isn't present. Should we re-run CI?

[Iroy30]

@mrmundt @jsiirola How can I go ahead and re-run the CI?

[Iroy30]

@mrmundt I don't think the fail we see is cuOpt related. Can you confirm?

[Iroy30]

We are gearing up for launch of cuopt 25.10, it would be great if we could review this PR before the new release so that pyomo & cuopt are in sync

[michaelbynum]

This is very close to the top of my list. When is the cuopt release?

[Iroy30]

Thanks @michaelbynum ! It is October 10th

[mrmundt]

The errors we are seeing on Jenkins are segfaults:

12:44:56 pyomo/pyomo/solvers/tests/checks/test_cuopt_direct.py::CUOPTTests::test_values_and_rc Fatal Python error: Segmentation fault
12:44:56 
12:44:56 Thread 0x00007f706aced640 (most recent call first):
12:44:56   File "/python/3.11.13/lib/python3.11/threading.py", line 331 in wait
12:44:56   File "/python/3.11.13/lib/python3.11/threading.py", line 629 in wait
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/jupyter_client/channels.py", line 130 in _async_run
12:44:56   File "/python/3.11.13/lib/python3.11/asyncio/events.py", line 84 in _run
12:44:56   File "/python/3.11.13/lib/python3.11/asyncio/base_events.py", line 1936 in _run_once
12:44:56   File "/python/3.11.13/lib/python3.11/asyncio/base_events.py", line 608 in run_forever
12:44:56   File "/python/3.11.13/lib/python3.11/asyncio/base_events.py", line 641 in run_until_complete
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/jupyter_client/channels.py", line 149 in run
12:44:56   File "/python/3.11.13/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
12:44:56   File "/python/3.11.13/lib/python3.11/threading.py", line 1002 in _bootstrap
12:44:56 
12:44:56 Thread 0x00007f70684e8640 (most recent call first):
12:44:56   File "/python/3.11.13/lib/python3.11/selectors.py", line 468 in select
12:44:56   File "/python/3.11.13/lib/python3.11/asyncio/base_events.py", line 1898 in _run_once
12:44:56   File "/python/3.11.13/lib/python3.11/asyncio/base_events.py", line 608 in run_forever
12:44:56   File "/python/3.11.13/lib/python3.11/asyncio/base_events.py", line 641 in run_until_complete
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/jupyter_client/threaded.py", line 266 in run
12:44:56   File "/python/3.11.13/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
12:44:56   File "/python/3.11.13/lib/python3.11/threading.py", line 1002 in _bootstrap
12:44:56 
12:44:56 Thread 0x00007f7281fea640 (most recent call first):
12:44:56   File "/host/tester/python/python311/pyomo/pyomo/common/tee.py", line 940 in _mergedReader
12:44:56   File "/python/3.11.13/lib/python3.11/threading.py", line 982 in run
12:44:56   File "/python/3.11.13/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
12:44:56   File "/python/3.11.13/lib/python3.11/threading.py", line 1002 in _bootstrap
12:44:56 
12:44:56 Thread 0x00007f7285dfb640 (most recent call first):
12:44:56   File "/host/tester/python/python311/pyomo/pyomo/common/tee.py", line 940 in _mergedReader
12:44:56   File "/python/3.11.13/lib/python3.11/threading.py", line 982 in run
12:44:56   File "/python/3.11.13/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
12:44:56   File "/python/3.11.13/lib/python3.11/threading.py", line 1002 in _bootstrap
12:44:56 
12:44:56 Current thread 0x00007f72cee64740 (most recent call first):
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/numba/cuda/cudadrv/driver.py", line 326 in safe_cuda_api_call
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/numba/cuda/cudadrv/driver.py", line 505 in __enter__
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/numba/cuda/cudadrv/devices.py", line 121 in ensure_context
12:44:56   File "/python/3.11.13/lib/python3.11/contextlib.py", line 137 in __enter__
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/numba/cuda/cudadrv/devices.py", line 231 in _require_cuda_context
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/numba/cuda/api.py", line 76 in as_cuda_array
12:44:56   File "/snl/python_cuda/25.08.14/lib/python3.11/site-packages/cudf/core/column/column.py", line 705 in data_array_view
12:44:56   File "/snl/python_cuda/25.08.14/lib/python3.11/site-packages/cudf/core/column/column.py", line 819 in values_host
12:44:56   File "/snl/python_cuda/25.08.14/lib/python3.11/site-packages/cudf/core/frame.py", line 762 in <lambda>
12:44:56   File "/snl/python_cuda/25.08.14/lib/python3.11/site-packages/cudf/core/frame.py", line 564 in to_array
12:44:56   File "/snl/python_cuda/25.08.14/lib/python3.11/site-packages/cudf/core/frame.py", line 594 in _to_array
12:44:56   File "/snl/python_cuda/25.08.14/lib/python3.11/site-packages/cudf/utils/performance_tracking.py", line 51 in wrapper
12:44:56   File "/snl/python_cuda/25.08.14/lib/python3.11/site-packages/cudf/core/frame.py", line 761 in to_numpy
12:44:56   File "/snl/python_cuda/25.08.14/lib/python3.11/site-packages/cudf/utils/performance_tracking.py", line 51 in wrapper
12:44:56   File "/snl/python_cuda/25.08.14/lib/python3.11/site-packages/cuopt/linear_programming/solver/solver.py", line 98 in Solve
12:44:56   File "/snl/python_cuda/25.08.14/lib/python3.11/site-packages/cuopt/utilities/exception_handler.py", line 36 in func
12:44:56   File "/host/tester/python/python311/pyomo/pyomo/solvers/plugins/solvers/cuopt_direct.py", line 71 in _apply_solver
12:44:56   File "/host/tester/python/python311/pyomo/pyomo/solvers/plugins/solvers/direct_solver.py", line 142 in solve
12:44:56   File "/host/tester/python/python311/pyomo/pyomo/solvers/tests/checks/test_cuopt_direct.py", line 67 in test_values_and_rc
12:44:56   File "/python/3.11.13/lib/python3.11/unittest/case.py", line 579 in _callTestMethod
12:44:56   File "/python/3.11.13/lib/python3.11/unittest/case.py", line 623 in run
12:44:56   File "/python/3.11.13/lib/python3.11/unittest/case.py", line 678 in __call__
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/_pytest/unittest.py", line 351 in runtest
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/_pytest/runner.py", line 174 in pytest_runtest_call
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/_pytest/runner.py", line 242 in <lambda>
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/_pytest/runner.py", line 341 in from_call
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/_pytest/runner.py", line 241 in call_and_report
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/_pytest/runner.py", line 132 in runtestprotocol
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/_pytest/runner.py", line 113 in pytest_runtest_protocol
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/_pytest/main.py", line 362 in pytest_runtestloop
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/_pytest/main.py", line 337 in _main
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/_pytest/main.py", line 283 in wrap_session
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/_pytest/main.py", line 330 in pytest_cmdline_main
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/_pytest/config/__init__.py", line 175 in main
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/_pytest/config/__init__.py", line 201 in console_main
12:44:56   File "/snl/python_packages/25.08.14/lib/python3.11/site-packages/pytest/__main__.py", line 9 in <module>
12:44:56   File "<frozen runpy>", line 88 in _run_code
12:44:56   File "<frozen runpy>", line 198 in _run_module_as_main
12:44:56 
12:44:56 Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, shiboken6.Shiboken, PySide6.QtCore, PySide6.QtGui, PySide6.QtWidgets, PySide6.QtTest, PIL._imaging, kiwisolver._cext, charset_normalizer.md, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, yaml._yaml, gurobipy._util, gurobipy._exception, gurobipy._attrutil, gurobipy._core, gurobipy._batch, gurobipy._matrixapi, gurobipy._helpers, gurobipy._modelutil, gurobipy._model, cython.cimports.libc.math, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._cython_nnls, scipy._lib._uarray._uarray, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.interpolate._dierckx, scipy.interpolate._ppoly, scipy.interpolate._interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.interpolate._bspl, scipy.special.cython_special, scipy.stats._stats, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._biasedurn, scipy.stats._stats_pythran, scipy.stats._levy_stable.levyst, scipy.stats._ansari_swilk_statistics, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.ndimage._nd_image, scipy.ndimage._rank_filter_1d, _ni_label, scipy.ndimage._ni_label, pyarrow.lib, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, numexpr.interpreter, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, ipopt_wrapper, _xpress, gmpy2.gmpy2, psutil._psutil_linux, psutil._psutil_posix, markupsafe._speedups, cftime._cftime, cupy_backends.cuda._softlink, cupy_backends.cuda.api._runtime_enum, cupy_backends.cuda.api.runtime, cupy._util, cupy.cuda.device, fastrlock.rlock, cupy.cuda.memory_hook, cupy_backends.cuda.stream, cupy.cuda.graph, cupy.cuda.stream, cupy_backends.cuda.api._driver_enum, cupy_backends.cuda.api.driver, cupy.cuda.memory, cupy._core.internal, cupy._core._carray, cupy.cuda.texture, cupy.cuda.function, cupy_backends.cuda.libs.nvrtc, cupy.cuda.pinned_memory, cupy.cuda.common, cupy.cuda.cub, cupy_backends.cuda.libs.nvtx, cupy.cuda.thrust, cupy._core._dtype, cupy._core._scalar, cupy._core._accelerator, cupy._core._memory_range, cupy._core._fusion_thread_local, cupy._core._kernel, cupy._core._routines_manipulation, cupy._core._routines_binary, cupy._core._optimize_config, cupy._core._cub_reduction, cupy._core._reduction, cupy._core._routines_math, cupy._core._routines_indexing, cupy._core._routines_linalg, cupy._core._routines_logic, cupy._core._routines_sorting, cupy._core._routines_statistics, cupy._core.dlpack, cupy._core.flags, cupy._core.core, cupy._core._fusion_variable, cupy._core._fusion_trace, cupy._core._fusion_kernel, cupy._core.new_fusion, cupy._core.fusion, cupy._core.raw, cupy.fft._cache, cupy.fft._callback, cupy.random._bit_generator, cupy.lib._polynomial, scipy.fftpack.convolve, scipy.cluster._vq, scipy.cluster._hierarchy, scipy.cluster._optimal_leaf_ordering, sklearn.__check_build._check_build, sklearn.utils._isfinite, sklearn.utils.sparsefuncs_fast, sklearn.utils.murmurhash, sklearn.utils._openmp_helpers, sklearn.utils._random, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.preprocessing._csr_polynomial_expansion, sklearn.preprocessing._target_encoder_fast, sklearn.metrics._dist_metrics, sklearn.metrics._pairwise_distances_reduction._datasets_pair, sklearn.utils._cython_blas, sklearn.metrics._pairwise_distances_reduction._base, sklearn.metrics._pairwise_distances_reduction._middle_term_computer, sklearn.utils._heap, sklearn.utils._sorting, sklearn.metrics._pairwise_distances_reduction._argkmin, sklearn.metrics._pairwise_distances_reduction._argkmin_classmode, sklearn.utils._vector_sentinel, sklearn.metrics._pairwise_distances_reduction._radius_neighbors, sklearn.metrics._pairwise_distances_reduction._radius_neighbors_classmode, sklearn.metrics._pairwise_fast, sklearn.neighbors._partition_nodes, sklearn.neighbors._ball_tree, sklearn.neighbors._kd_tree, sklearn.utils.arrayfuncs, sklearn.utils._seq_dataset, sklearn.linear_model._cd_fast, _loss, sklearn._loss._loss, sklearn.svm._liblinear, sklearn.svm._libsvm, sklearn.svm._libsvm_sparse, sklearn.utils._weight_vector, sklearn.linear_model._sgd_fast, sklearn.linear_model._sag_fast, sklearn.decomposition._online_lda_fast, sklearn.decomposition._cdnmf_fast, sklearn.tree._utils, sklearn.neighbors._quad_tree, sklearn.tree._tree, sklearn.tree._splitter, sklearn.tree._criterion, sklearn.ensemble._gradient_boosting, sklearn.ensemble._hist_gradient_boosting.common, sklearn.ensemble._hist_gradient_boosting._gradient_boosting, sklearn.ensemble._hist_gradient_boosting._binning, sklearn.ensemble._hist_gradient_boosting._bitset, sklearn.ensemble._hist_gradient_boosting.histogram, sklearn.ensemble._hist_gradient_boosting._predictor, sklearn.ensemble._hist_gradient_boosting.splitting, PySide6.QtUiTools, PySide6.QtDataVisualization, PySide6.QtOpenGL, PySide6.QtOpenGLWidgets, _testcapi, PySide6.QtPrintSupport, PySide6.QtSvg, zmq.backend.cython._zmq, tornado.speedups, lxml._elementpath, lxml.etree, casadi._casadi, gams.core.gdx._gdxcc, gams.core.gmd._gmdcc, gams.core.opt._optcc, gams.core.gmo._gmomcc, gams.core.cfg._cfgmcc, gams.core.gev._gevmcc, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.typing.builtins.itertools, numba.cpython.builtins.math, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, numba.mviewbuf, numba.core.typing.cmathdecl.cmath, cuopt.linear_programming.internals.internals, cuopt.linear_programming.data_model.data_model_wrapper, cuda.bindings._bindings.cydriver, cuda.bindings.cydriver, cuda.bindings.driver, cuda.bindings._lib.utils, cuda.bindings._bindings.cyruntime_ptds, cuda.bindings._bindings.cyruntime, cuda.bindings._lib.cyruntime.utils, cuda.bindings._lib.cyruntime.cyruntime, cuda.bindings.cyruntime, cuda.bindings.runtime, cuda.bindings.utils._get_handle, pylibraft.common.cuda, rmm.pylibrmm.cuda_stream, rmm.pylibrmm.stream, rmm.pylibrmm.helper, rmm.pylibrmm.memory_resource, rmm.pylibrmm.device_buffer, rmm.librmm._logger, rmm.pylibrmm.logger, pylibraft.common.handle, nvtx._lib.lib, nvtx._lib.profiler, pylibcudf.libcudf.types, pylibcudf.types, pylibcudf.libcudf.aggregation, pylibcudf.aggregation, pylibcudf.gpumemoryview, pylibcudf.utils, pylibcudf._interop_helpers, pylibcudf.table, pylibcudf.filling, pylibcudf.traits, pylibcudf.column, pylibcudf.scalar, pylibcudf.libcudf.binaryop, pylibcudf.binaryop, pylibcudf.column_factories, pylibcudf.concatenate, pylibcudf.contiguous_split, pylibcudf.libcudf.copying, pylibcudf.copying, pylibcudf.libcudf.datetime, pylibcudf.datetime, pylibcudf.experimental, pylibcudf.libcudf.expressions, pylibcudf.expressions, pylibcudf.groupby, pylibcudf.hashing, pylibcudf.interop, pylibcudf.io.datasource, pylibcudf.libcudf.io.json, pylibcudf.libcudf.io.types, pylibcudf.io.types, pylibcudf.io.avro, pylibcudf.io.csv, pylibcudf.io.json, pylibcudf.io.orc, pylibcudf.io.parquet, pylibcudf.io.parquet_metadata, pylibcudf.io.text, pylibcudf.io.timezone, pylibcudf.join, pylibcudf.json, pylibcudf.libcudf.labeling, pylibcudf.labeling, pylibcudf.libcudf.lists.combine, pylibcudf.libcudf.lists.contains, pylibcudf.lists, pylibcudf.merge, pylibcudf.null_mask, pylibcudf.nvtext.byte_pair_encode, pylibcudf.nvtext.deduplicate, pylibcudf.nvtext.edit_distance, pylibcudf.nvtext.generate_ngrams, pylibcudf.nvtext.jaccard, pylibcudf.nvtext.minhash, pylibcudf.nvtext.ngrams_tokenize, pylibcudf.nvtext.normalize, pylibcudf.nvtext.replace, pylibcudf.nvtext.stemmer, pylibcudf.nvtext.subword_tokenize, pylibcudf.nvtext.tokenize, pylibcudf.nvtext.wordpiece_tokenize, pylibcudf.partitioning, pylibcudf.quantiles, pylibcudf.libcudf.reduce, pylibcudf.reduce, pylibcudf.libcudf.replace, pylibcudf.replace, pylibcudf.reshape, pylibcudf.rolling, pylibcudf.libcudf.round, pylibcudf.round, pylibcudf.search, pylibcudf.sorting, pylibcudf.libcudf.stream_compaction, pylibcudf.stream_compaction, pylibcudf.strings.attributes, pylibcudf.libcudf.strings.char_types, pylibcudf.strings.capitalize, pylibcudf.strings.case, pylibcudf.strings.char_types, pylibcudf.libcudf.strings.combine, pylibcudf.strings.combine, pylibcudf.libcudf.strings.regex_flags, pylibcudf.strings.regex_flags, pylibcudf.strings.regex_program, pylibcudf.strings.contains, pylibcudf.strings.convert.convert_booleans, pylibcudf.strings.convert.convert_datetime, pylibcudf.strings.convert.convert_durations, pylibcudf.strings.convert.convert_fixed_point, pylibcudf.strings.convert.convert_floats, pylibcudf.strings.convert.convert_integers, pylibcudf.strings.convert.convert_ipv4, pylibcudf.strings.convert.convert_lists, pylibcudf.strings.convert.convert_urls, pylibcudf.strings.extract, pylibcudf.strings.find, pylibcudf.strings.find_multiple, pylibcudf.strings.findall, pylibcudf.strings.padding, pylibcudf.strings.repeat, pylibcudf.strings.replace, pylibcudf.strings.replace_re, pylibcudf.strings.reverse, pylibcudf.libcudf.strings.side_type, pylibcudf.strings.side_type, pylibcudf.strings.slice, pylibcudf.strings.split.partition, pylibcudf.strings.split.split, pylibcudf.strings.strip, pylibcudf.libcudf.strings.translate, pylibcudf.strings.translate, pylibcudf.strings.wrap, pylibcudf.transform, pylibcudf.transpose, pylibcudf.libcudf.unary, pylibcudf.unary, pylibcudf.utilities, numba.cpython.mathimpl.math, numba.cpython.mathimpl.sys, numba.types.itertools, cudf._lib.strings_udf, pyarrow._feather, ujson, cuopt.linear_programming.solver.solver_parameters, cuopt.linear_programming.solver.solver_wrapper, cuopt.routing.utils_wrapper, msgpack._cmsgpack, pyarrow._fs, pyarrow._azurefs, pyarrow._hdfs, pyarrow._gcsfs, pyarrow._s3fs, pyarrow._acero, pyarrow._csv, pyarrow._json, pyarrow._dataset, pyarrow._dataset_orc, pyarrow._parquet, pyarrow._parquet_encryption, pyarrow._dataset_parquet_encryption, pyarrow._dataset_parquet, raft_dask.common.comms_utils, raft_dask.common.nccl, cuopt.routing.vehicle_routing_wrapper, PIL._imagingmath (total: 470)
12:45:00 pyomo/.jenkins.sh: line 266: 790791 Segmentation fault      (core dumped) python -m pytest -v -W ignore::Warning --junitxml="TEST-pyomo.xml" -m "$CATEGORY" $TEST_SUITES $PYTEST_EXTRA_ARGS

[Iroy30]

@mrmundt Would it be possible to update CI to pull cuopt 25.10

[jsiirola]

@Iroy30 Yes (working on that now - it will take a couple hours to update everything and propagate it to the build machines)- but does that mean that this interface will enforce a minimum cuopt version?

[Iroy30]

@jsiirola The segfaults you see in CI with 25.08 is concerning. Was the CI running the 25.08 release or the daily build of cuopt(this could cause the segfault). If 25.08 release was causing segfault then we might have to enforce minimum cuopt 25.10 and meanwhile I will debug 25.08 on my end because we were not seeing this previously

[jsiirola]

Was the CI running the 25.08 release or the daily build of cuopt (this could cause the segfault). If 25.08 release was causing segfault then we might have to enforce minimum cuopt 25.10 and meanwhile I will debug 25.08 on my end because we were not seeing this previously

The CI is running cuopt-cu12/25.8.0 from PyPI. I have added 25.10.0 (also from PyPI), but that won't make it to the build machines for testing until later tonight (it is waiting on finalizing the Pyomo 6.9.5 release)

[jsiirola]

We attempted to upgrade cuopt-cu12 to 25.10.0 (on a machine running RHEL9.6). The packages installed, but importing cuopt fails:

>>> import cuopt
.local/lib/python3.11/site-packages/libcuopt/load.py:95: RuntimeWarning: Failed to load libcuopt library: libcuopt.so. Error: .local/lib/python3.11/site-packages/libcuopt/lib64/libcuopt.so: undefined symbol: _ZSt21__glibcxx_assert_failPKciS0_S0_. Falling back to relying on system loader. cuOpt functionality may be unavailable. This might lead to a generic error such as 'libcuopt.so missing' if the library cannot be found.
  warnings.warn(
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".local/lib/python3.11/site-packages/cuopt/__init__.py", line 24, in <module>
    from cuopt import linear_programming, routing
  File ".local/lib/python3.11/site-packages/cuopt/linear_programming/__init__.py", line 16, in <module>
    from cuopt.linear_programming import internals
  File ".local/lib/python3.11/site-packages/cuopt/linear_programming/internals/__init__.py", line 30, in <module>
    from cuopt.linear_programming.internals.internals import (
ImportError: libcuopt.so: cannot open shared object file: No such file or directory