grpc-labview icon indicating copy to clipboard operation
grpc-labview copied to clipboard

Parallel Stream Writes crash LabVIEW

Open NIbbuchanan opened this issue 2 years ago • 2 comments

When multiple stream writes happen, it can crash LabVIEW with the following dialog: image

Here is sample code to illustrate the crash ParallelStreamWrite.zip

After a little debugging, there is already code in event_data.cc to use a _writeSemaphore to ensure multiple writes to a stream don't happen at the same time (see CallData::Write where the _writeSemaphore.wait() is called). I was able to make a simple fix by just making the stream write VI non re-entrant and this fixed the crash. Not sure if you want to update the generated code so this VI is non-reentrant or improve the locking code in C++ to avoid this problem.

One downside to making the VI non-reentrant is that if one stream gets behind (i.e. client isn't reading it), it could result in other streams being blocked. I was able to write about 5000 server stream items without ever reading the client before the server stream write was blocked because the client queue couldn't get the new data added. So after ~5000 writes the server stream would be blocked and no other server streams could be written if the VI was non-reentrant. Not sure how big a use case this is, but for the short term, I'll just make my VI non reentrant to work around this issue.

NIbbuchanan avatar Aug 30 '22 15:08 NIbbuchanan

looks like the abort is called from an assert on line 980 of this file: "..\grpc-labview\third_party\grpc\include\grpcpp\impl\codegen\call_op_set.h", with the error message: GRPC_CALL_ERROR_TOO_MANY_OPERATIONS.

This seems to be expected behavior when you write in parallel since gRPC's comment on that line says this will happen when "doing a Write while another Write is already pending on the same RPC or invoking WritesDone multiple times".

Other people online have seen similar issues like this one, and the response is basically that they need to use the API correctly and not call write multiple times in parallel on the same gRPC.

We can make changes in our use of gRPC to avoid calling write in parallel on the same RPC; however, it would be nice for the LabVIEW API to not allow gRPC to call abort and to instead return an error or warn the user that they can't write while another write is pending.

NIthan314 avatar Sep 01 '22 17:09 NIthan314

I would say instead of LV returning an error or warning the user, the write should just wait internally in the DLL until the other thread is done writing and then it can write...(need to make sure only one writes at a time). This way, user doesn't have to worry about how they write their code and if parallel writes happen, the second will just block until it's valid to write and then it writes normally so no need for error to be returned.

NIbbuchanan avatar Sep 01 '22 21:09 NIbbuchanan