Python - Pyomo - R - Reticulate - Failed to create solver [WinError 6]

[AlexanderMeisinger]

Hey community,

I tried to run the following code via R with reticulate and pyomo based on the solver "cbc". The Python-code is working if I execute the code independently of R. Executing the code in R with the package "reticulate" doesn't work.

Information about the Python-code: A Simple Concrete Pyomo Model - https://pyomo.readthedocs.io/en/stable/pyomo_overview/simple_examples.html

Thanks in advance for your help!

Br, Alex

Code in Python:

from pyomo.environ import *

import pyutilib.subprocess.GlobalData
pyutilib.subprocess.GlobalData.DEFINE_SIGNAL_HANDLERS_DEFAULT = False

model = ConcreteModel()

model.x = Var([1,2], domain=NonNegativeReals)

model.OBJ = Objective(expr = 2*model.x[1] + 3*model.x[2])

model.Constraint1 = Constraint(expr = 3*model.x[1] + 4*model.x[2] >= 1)

opt = SolverFactory('cbc')
opt.solve(model)

Code in R:

library("reticulate")

use_python("C:/Anaconda")
setwd("C:/Users/SimpleConcretePyomoModel")
source_python("C:/Users/SimpleConcretePyomoModel/SimpleConcretePyomoModel.py")

Error in R

Error in py_run_file_impl(file, local, convert) : 
  RuntimeError: Attempting to use an unavailable solver.

The SolverFactory was unable to create the solver "cbc"
and returned an UnknownSolver object.  This error is raised at the point
where the UnknownSolver object was used as if it were valid (by calling
method "solve").

The original solver was created with the following parameters:
	type: cbc
	_args: ()
	options: {}
WARNING: Failed to create solver with name 'cbc': Could not execute the
    command: 'C:\Program Files\cbc-win64\cbc.exe -stop'
    	Error message: [WinError 6] Das Handle ist ungültig

Additional information:

python:             C:/Anaconda/python.exe
libpython:          C:/Anaconda/python38.dll
pythonhome:         C:/Anaconda
version:            3.8.8 (default, Apr 13 2021, 15:08:03) [MSC v.1916 64 bit (AMD64)]
Architecture:       64bit
numpy:              C:/Anaconda/Lib/site-packages/numpy
numpy_version:      1.20.3
pyomo_version:      5.7.2
reticulate_version: 1.20
cbc_version:        2.10.3
RStudio_version:    1.4.1717
Spyder_version:     5.0.5

[jsiirola]

This appears to be an issue with Python's subprocess within reticulate. A little googling (see here, which references here). Unfortunately, none of the development team works with R / reticulate, and this platform is not part of the standard testing library.

You might try the following patch to pyomo/colvers/plugins/solvers/CBCplugin.py:

--- a/pyomo/solvers/plugins/solvers/CBCplugin.py
+++ b/pyomo/solvers/plugins/solvers/CBCplugin.py
@@ -52,12 +52,15 @@ def configure_cbc():
         return
     cbc_exec = executable.path()
     results = subprocess.run([cbc_exec,"-stop"], timeout=1,
-                             stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+                             stdout=subprocess.PIPE,
+                             stderr=subprocess.STDOUT,
+                             stdin=subprocess.DEVNULL,
                              universal_newlines=True)
     _cbc_version = _extract_version(results.stdout)
     results = subprocess.run([cbc_exec, "dummy", "-AMPL", "-stop"], timeout=1,
                              stdout=subprocess.PIPE,
                              stderr=subprocess.STDOUT,
+                             stdin=subprocess.DEVNULL,
                              universal_newlines=True)
     _cbc_compiled_with_asl = not ('No match for AMPL' in results.stdout)
     if _cbc_version is not None:

If that works for you, we can look into promulgating that through Pyomo.

[AlexanderMeisinger]

Hey John, thanks a lot for answering! I will try your suggestion and come back to you. Another thing what I noticed is, that I also get the same error when using different solvers, for example glpk or gurobi. In addition to that, I found out that I can execute the code in the console (R.exe) without getting an error. However, executing the code in RStudio still doesn't work. Br, Alex

[notesofdabbler]

Not sure if this will be helpful but I had written a blogpost about using pyomo from R with reticulate a while back. I remember having to put the following 2 commands within the R session to make things work but not sure if was related to the issue you are having.

pyulib = import("pyutilib.subprocess.GlobalData")
pyulib$DEFINE_SIGNAL_HANDLERS_DEFAULT = FALSE

[jsiirola]

@notesofdabbler: Thanks for reminding me of that! For the record, those lines are only necessary for Pyomo < 6.0. Starting in 6.0, Pyomo no longer has a dependence on PyUtilib (and no longer defines signal handlers for subprocesses). That said, we now set the timeout option through Python's subprocess, and I wonder if subprocess implements some form of signal handling to pull off the timeout?

[AlexanderMeisinger]

@jsiirola Hey John, I added your suggested patch in pyomo/colvers/plugins/solvers/CBCplugin.py. Now, executing SimpleConcretePyomoModel.py raises the error: Executing line 23: opt = SolverFactory('cbc')

WARNING: Failed to create solver with name 'cbc': module 'pyutilib.subprocess'
    has no attribute 'DEVNULL'

Executing line 24: opt.solve(model)

RuntimeError: Attempting to use an unavailable solver.

The SolverFactory was unable to create the solver "cbc"
and returned an UnknownSolver object.  This error is raised at the point
where the UnknownSolver object was used as if it were valid (by calling
method "solve").

The original solver was created with the following parameters:
	type: cbc
	_args: ()
	options: {}
```
Thanks a lot for your help!
Br, Alex

[AlexanderMeisinger]

@notesofdabbler Thanks for your answer! I already tried your examples on the blog, but unfortunately none of them are working for me. At the moment I am using the pyomo version 5.7.2. However, adding your suggestion still doesn't work. Do you have any other ideas? Br, Alex

[jsiirola]

@AlexanderMeisinger: sorry, I should have mentioned that that patch is against the main Pyomo development branch. It should work for any version of Pyomo > 6.0. If you are still running 5.x, you will need a different patch. Assuming you are running Python 3.6 or later, adding something like this at the very beginning of that module:

from subprocess import DEVNULL

Then just use stdin=DEVNULL for the subprocess call might work.

Also note that this will not solve your issue completely: there are subprocess calls all over Pyomo and you will just end up hitting an error at a later call. If this helps, then we will have to propagate this change to every subprocess call.

[AlexanderMeisinger]

@jsiirola Thank you for your advice. If it is possible, then would like to use Pyomo 5.x. I get a new error message in Python when I add the following code at the very beginning of the module:

from pyutilib.subprocess import run
from subprocess import PIPE, STDOUT, DEVNULL

Executing line 23: opt = SolverFactory('cbc')

WARNING: Failed to create solver with name 'cbc': run_command() got an
    unexpected keyword argument 'timeout'

Executing line 24: opt.solve(model)

RuntimeError: Attempting to use an unavailable solver.

The SolverFactory was unable to create the solver "cbc"
and returned an UnknownSolver object.  This error is raised at the point
where the UnknownSolver object was used as if it were valid (by calling
method "solve").

The original solver was created with the following parameters:
	type: cbc
	_args: ()
	options: {}

[blnicho]

@AlexanderMeisinger I'm cleaning up old issues, can this be closed?