Bug: OpcDaClient problems with Multithreading

[jamesbraza]

I have discovered that if you try to connect the OpcDaClient on a thread, it throws an Exception. Please see the below code snippet (done with MatrikonOPC Simulation Server):

from threading import Thread

from openopc2.config import OpenOpcConfig
from openopc2.da_client import OpcDaClient

MATRIKON_SIMULATION_SERVER = "Matrikon.OPC.Simulation"

open_opc_config = OpenOpcConfig()
open_opc_config.OPC_SERVER = MATRIKON_SIMULATION_SERVER

# 1. Instantiate client in main thread
opc = OpcDaClient(open_opc_config=open_opc_config)
# 2. Connect client in another thread (doesn't work)
thread = Thread(
    target=opc.connect, kwargs={"opc_server": MATRIKON_SIMULATION_SERVER}
)
thread.start()
thread.join()
# 2. Connect client in the main thread (works)
opc.connect(opc_server=MATRIKON_SIMULATION_SERVER)
# 3. Try reading tag
result = opc.read("Random.Boolean")
_ = 0  # Debug here

If you use the main thread the whole time, this works.

If you call OpcDaClient.connect on a thread, you get this Exception when OpcCom.connect calls self.opc_client.Connect:

pywintypes.com_error(-2147352567,
                     'Exception occurred.',
                     (0, None, None, None, 0, -2147467259),
                     None)

Notable, OpenOPC (not OpenOPC 2) doesn't have this error:

from threading import Thread

import OpenOPC

MATRIKON_SIMULATION_SERVER = "Matrikon.OPC.Simulation"

# 1. Instantiate client in main thread
opc = OpenOPC.open_client("localhost")  # Open mode
# 2. Connect client in another thread (works)
thread = Thread(
    target=opc.connect, kwargs={"opc_server": MATRIKON_SIMULATION_SERVER}
)
thread.start()
thread.join()
# 3. Try reading tag
result = opc.read("Random.Boolean")
_ = 0  # Debug here

This is a bummer for me, because I use a software framework where all device connections are all made asynchronously (on a thread). I would like to use OpenOPC2, but this is an issue.

[renzop]

Multithreading is supported via the gateway. Just make sure you create the proxy in the thread where you use it. Or pass the ownership (see pyro docs)

What happens in your case when you create the connection in the same thread as you use it?

[jamesbraza]

Thanks for the speedy response!

Multithreading is supported via the gateway.

This is good, it focuses us on the OpcDaClient then.

Just make sure you create the proxy in the thread where you use it.

Thanks for pointing this out, yeah I realize that, and it's at the core of this issue. For some reason, OpenOPC v1 didn't have this issue, perhaps something changed related to security between Pyro4 and Pyro5.

Unfortunately, due to the framework I'm using, I can't control which thread creates the OpcDaClient.

Or pass the ownership (see pyro docs)

Okay, reading here: https://pyro5.readthedocs.io/en/latest/clientcode.html?highlight=thread#proxy-sharing-between-threads

proxy is ‘owned’ by a thread. You cannot use it from another thread.

I work in the biotech space where we have lots of devices being concurrently controlled from one Python process. We make heavy use of multithreading. The fact that Pyro5 doesn't allow for this is imo an issue.

That being said, based on Pyro5's examples/threadproxysharing/client.py, I am trying to use _pyroClaimOwnership():

>>> opc_client._pyroClaimOwnership()
Traceback (most recent call last):
  File "C:\path\to\venv\lib\site-packages\IPython\core\interactiveshell.py", line 3442, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-493afd7395b1>", line 1, in <module>
    self._opc_client._pyroClaimOwnership()
AttributeError: 'OpcDaClient' object has no attribute '_pyroClaimOwnership'
>>> opc_client._opc._pyroClaimOwnership()
Traceback (most recent call last):
  File "C:\path\to\venv\lib\site-packages\IPython\core\interactiveshell.py", line 3442, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-2b72a8ebc0c5>", line 1, in <module>
    self._opc_client._opc._pyroClaimOwnership()
AttributeError: 'OpcCom' object has no attribute '_pyroClaimOwnership'

I am getting attribute errors on both OpcDaClient and OpcCom objects, can you point out to me how to claim ownership? If I can figure this thread handoff out, I can get this to work. Looking forward to hearing back, and thanks again!

[jamesbraza]

@eliabieri a nit on the label applied is I think this a bug, not a question. I don't think it's possible to use OpcDaClient with multiple threads, try running the code snippet in the OP (summarized here) with Matrikon OPC Simulation Server.

opc = OpcDaClient(open_opc_config=open_opc_config)
thread = Thread(
    target=opc.connect, kwargs={"opc_server": MATRIKON_SIMULATION_SERVER}
)
thread.start()
thread.join()
result = opc.read("Random.Boolean")

If you can help me figure this out, I can become an adopter of OpenOPC 2.

[renzop]

You can use it in multithreading. We do so with celery for example. Just make sure you call this function in the thread before you want to call any method on the proxy object. Add this to the start function of your thread.

def start(self): # important for multithreading, Pyro proxy needs ownership in of proxy in the current thread self.open_opc_client._pyroClaimOwnership()

[jamesbraza]

Hi @renzop thank you for responding!

_pyroClaimOwnership is not a method in OpcDaClient, so I don't think your suggestion is valid. See the below:

from openopc2.config import OpenOpcConfig
from openopc2.da_client import OpcDaClient

MATRIKON_SIMULATION_SERVER = "Matrikon.OPC.Simulation"

open_opc_config = OpenOpcConfig()
open_opc_config.OPC_SERVER = MATRIKON_SIMULATION_SERVER

opc = OpcDaClient(open_opc_config=open_opc_config)
opc._pyroClaimOwnership()

This leads to an AttributeError with Pyro5==5.14 and openopc2==0.1.11:

Traceback (most recent call last):
  File "C:\path\to\file.py", line 10, in <module>
    opc._pyroClaimOwnership()
AttributeError: 'OpcDaClient' object has no attribute '_pyroClaimOwnership'

Can you link me to your celery worker code? I don't know how your celery worker runs, given this AttributeError.