gRPC error message during subscription

[ianbarrere]

Hello,

I am setting up a subscription poller using pygnmi in the following setup:

def subscribe(hostname: str, paths: List[str]):
    subscription_def = {
        "subscription": [
            {
                "path": path,
                "mode": "sample",
            }
            for path in paths
        ],
        "mode": "poll",
        "encoding": "json_ietf",
    }
    with get_gnmi_session(hostname) as session:
        subscription = session.subscribe2(subscribe=subscription_def)
        yield subscription

I noticed that every time I poll a device, if I advance to the end of the iteration (by either wrapping the generator in next() or by looping over it) it throws a rather generic looking gRPC error:

Exception in thread Thread-4 (enqueue_updates):
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/[email protected]/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/opt/homebrew/Cellar/[email protected]/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/i/venv/lib/python3.11/site-packages/pygnmi/client.py", line 997, in enqueue_updates
    for update in subscription:
  File "/Users/i/venv/lib/python3.11/site-packages/grpc/_channel.py", line 540, in __next__
    return self._next()
           ^^^^^^^^^^^^
  File "/Users/i/venv/lib/python3.11/site-packages/grpc/_channel.py", line 966, in _next
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.CANCELLED
        details = "Channel closed!"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-09-13T14:46:04.927971+02:00", grpc_status:1, grpc_message:"Channel closed!"}"

This error doesn't seem to disrupt the behavior of the poll, as I'm still able to get the data I'm looking for, but it seems to be hit when the session is closed.

I'm not sure if I'm using the tool wrong or if this is a known issue or something, but I just wanted to see if anybody else had run into it.

[jjochum]

We encountered the same issue when the session is closed. I modified client.py:1079 to work around that. Most likely, the issue is with the grpc usage, but I didn't investigate further.

        except Exception as error:
              self.error = error
              # The connection was terminated by the server. This is generally okay and shouldn't raise an exception.
              if isinstance(error, grpc._channel._MultiThreadedRendezvous) and error.code() == grpc.StatusCode.CANCELLED:
                  return
              raise error

[akarneliuk]

Hey @ianbarrere , Yes, GRPC library internally creates a thread and doesn't seem it is joining properly. The proposal by @jjochum seems as interesting workaround indeed. I think I can add it to the code.

[akarneliuk]

Fixed in #161 , thanks @jjochum

[ianbarrere]

Hi @akarneliuk, thanks for the prompt fix, and apologies for the delay in response. Do you have new release scheduled at some point? I see the latest one is from before the fix.