gnmic icon indicating copy to clipboard operation
gnmic copied to clipboard
verified publisher icon openconfig

gnmi-server not sending sync_response correctly in Subscribe mode = once

Open pboers1988 opened this issue 1 year ago • 1 comments

When using the gnmi-server I'm attempting to create a rudimentary rest endpoint to retrieve interface information from the network through the gnmi-server. The FastAPI endpoint makes use of a pyGNMi client and looks like so.

Endpoint

@router.get("/{target}/interfaces")
def get_capabilities(target: str, gnmi_client: gNMIclient = Depends(yield_gnmi_client)) -> list[dict[Any, Any]]:
    subscription = {
        "subscription" : [{"path" : "/interfaces"}],
        "encoding": "json",
        "mode": "once"
    }
    subscribe_handle = gnmi_client.subscribe_once(subscription, target=target)
    res = []
    while True:
        try:
            res.append(next(subscribe_handle))
        except (StopIteration):
            break
    return res

Client

def yield_gnmi_client() -> Generator:
    with gNMIclient(("gnmi-server", 57400), insecure=True) as client:
        yield client
        client.close()

When calling the endpoint I receive the following strack trace

2024-08-07 19:37:38 [info     ] Application startup complete.  [uvicorn.error] 
Collecting Capabilities...
2024-08-07 19:37:43 [info     ] Collecting Capabilities...     [pygnmi.client] 
Collection of Capabilities is successfull
2024-08-07 19:37:43 [info     ] Collection of Capabilities is successfull [pygnmi.client] 
Unable to detect supported encodings, defaulting to 'json'
2024-08-07 19:37:43 [warning  ] Unable to detect supported encodings, defaulting to 'json' [pygnmi.client] 
Parsing of telemetry information is failed.
Traceback (most recent call last):
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/pygnmi/client.py", line 1351, in telemetryParser
    element_str += element
TypeError: can only concatenate str (not "TypedValue") to str
2024-08-07 19:37:43 [error    ] Parsing of telemetry information is failed. [pygnmi.client] 
Traceback (most recent call last):
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/pygnmi/client.py", line 1351, in telemetryParser
    element_str += element
TypeError: can only concatenate str (not "TypedValue") to str
Exception in thread Thread-5 (enqueue_updates):
Traceback (most recent call last):
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/pygnmi/client.py", line 1080, in enqueue_updates
    raise error
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/pygnmi/client.py", line 1076, in enqueue_updates
    for update in subscription:
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/grpc/_channel.py", line 543, in __next__
    return self._next()
           ^^^^^^^^^^^^
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/grpc/_channel.py", line 969, in _next
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.CANCELLED
        details = "Channel closed!"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Channel closed!", grpc_status:1, created_time:"2024-08-07T19:37:43.325609+02:00"}"
>
2024-08-07 19:37:43 [info     ] 127.0.0.1:65154 - "GET /******/capabilities HTTP/1.1" 500 [uvicorn.access] 
2024-08-07 19:37:43 [error    ] Exception in ASGI application
 [uvicorn.error] 
Traceback (most recent call last):
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 398, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/nwastdlib/logging.py", line 121, in __call__
    await self.app(scope, receive, send)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/fastapi/routing.py", line 193, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/concurrency.py", line 42, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 859, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/boers001/Documents/SURF/projects/gnmic-proxy/server/api/endpoints/interface.py", line 34, in get_capabilities
    res.append(next(subscribe_handle))
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/pygnmi/client.py", line 1146, in __next__
    if self._once and  "sync_response" in result:
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable
WARNING:  StatReload detected changes in 'server/api/endpoints/interface.py'. Reloading...

This is where the stacktrace is generated in pyGNMI https://github.com/akarneliuk/pygnmi/blob/master/pygnmi/client.py#L1178 The last message in the stream is a None that does not contain the sync_response

It would seem to me as the gnmi-server is not sending the sync_response that the protocol and client expects. It just closes the socket. Would you agree this might be an implementation error in the gnmi-server code?

pboers1988 avatar Aug 07 '24 17:08 pboers1988

The sync_response: true is being sent: https://github.com/openconfig/gnmic/blob/main/pkg/app/gnmi_server.go#L372 I also took a pcap while running a test, I could see a sync_response msg sent.

karimra avatar Aug 07 '24 19:08 karimra

Thanks! Ill look into it further and try and figure out why the pyGNMi client lib is not parsing the response correctly then.

pboers1988 avatar Aug 08 '24 06:08 pboers1988

I'm using Subscribe mode "stream" and ran into this same behavior.

It turns out I had to set the updates-only parameter to true, then it would send SyncResponse messages.

Not sure if this applies to mode "once" as well, just leaving this here in case it helps someone.

Mark90 avatar Jan 31 '25 10:01 Mark90