server script gives me an error SO_REUSEPORT using GRPC

[gus6202]

Description of the problem

Hello, I'm trying to run my server program (GRPC) using gramine-sgx but it gives me this error:

~/Desktop/gramine/CI-Examples/python$ gramine-sgx python scripts/_sgx/server.py 
Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - sgx.debug = true                           (this is a debug enclave)
  - loader.insecure__use_cmdline_argv = true   (forwarding command-line args from untrusted host to the app)
  - sgx.allowed_files = [ ... ]                (some files are passed through from untrusted host without verification)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

E0717 22:00:46.201856000       1 socket_utils_common_posix.cc:223] check for SO_REUSEPORT: {"created":"@1658095246.201781000",
/"description":"Protocol not available","errno":92,"file":"src/core/lib/iomgr
/socket_utils_common_posix.cc","file_line":202,"os_error":"Protocol not available","syscall":"getsockopt(SO_REUSEPORT)"}

Still, the server goes up and can communicate with client sending messages to the client and vice versa. The only problem here is that it doesn't show what the client is sending to the server (doesn't print any messages sent from the client). PS: Using gramine-direct shows the same issue.

[kailun-qin]

SO_REUSEPORT is not supported by Gramine at this moment (and no plan to implement it based on a previous discussion as far as I recall). We would expect the workload to fallback to other alternatives when it finds out the flag is not supported.

BTW, would you pls kindly also:

1. provide the commit hash of Gramine that you're working on?
2. clarify/elaborate on whether the C<->S communication (sending/receiving) is working or it's just a no show of print?

Thanks!

[dimakuv]

@gus6202 Could you check this discussion: https://github.com/gramineproject/gramine/issues/361 ? It provides some explanations on what could be done with gRPC. Hopefully it helps.

[gus6202]

@kailun-qin

provide the commit hash of Gramine that you're working on?

fd2e302

clarify/elaborate on whether the C<->S communication (sending/receiving) is working or it's just a no show of print?

With Gramine

Run server with gramine-sgx:

~/Desktop/gramine/CI-Examples/python$ gramine-sgx python scripts/_sgx/server.py 
Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - sgx.debug = true                           (this is a debug enclave)
  - loader.insecure__use_cmdline_argv = true   (forwarding command-line args from untrusted host to the app)
  - sgx.allowed_files = [ ... ]                (some files are passed through from untrusted host without verification)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

Here the server goes up, but it doesn't print ------------------start Python GRPC server in the beginning and when I run client.py it doesn't show any messages from the client (the error message doesn't show anymore because I put on my manifest loader.env.GRPC_VERBOSITY="NONE" like @dimakuv told). .

Run client (I run the client without gramine):

~/Desktop/grpc/examples/python/data_transmission$ python3 client.py 
--------------Call SimpleMethod Begin--------------
resp from server(1), the message=Python server SimpleMethod Ok!!!!
--------------Call SimpleMethod Over---------------
--------------Call ClientStreamingMethod Begin--------------
resp from server(1), the message=Python server ClientStreamingMethod ok
--------------Call ClientStreamingMethod Over---------------
--------------Call ServerStreamingMethod Begin--------------
recv from server(1), message=send by Python server, message=0
recv from server(1), message=send by Python server, message=1
recv from server(1), message=send by Python server, message=2
recv from server(1), message=send by Python server, message=3
recv from server(1), message=send by Python server, message=4
--------------Call ServerStreamingMethod Over---------------
--------------Call BidirectionalStreamingMethod Begin---------------
recv from server(1), message=send by Python server, message= 0
recv from server(1), message=send by Python server, message= 1
recv from server(1), message=send by Python server, message= 2
recv from server(1), message=send by Python server, message= 3
recv from server(1), message=send by Python server, message= 4
--------------Call BidirectionalStreamingMethod Over---------------

As you can see, client and server are communicating, it shows the messages sent from server.

Without Gramine

Run server.py:

~/Desktop/grpc/examples/python/data_transmission$ python3 server.py 
------------------start Python GRPC server

Run client.py:

~/Desktop/grpc/examples/python/data_transmission$ python3 client.py 
--------------Call SimpleMethod Begin--------------
resp from server(1), the message=Python server SimpleMethod Ok!!!!
--------------Call SimpleMethod Over---------------
--------------Call ClientStreamingMethod Begin--------------
resp from server(1), the message=Python server ClientStreamingMethod ok
--------------Call ClientStreamingMethod Over---------------
--------------Call ServerStreamingMethod Begin--------------
recv from server(1), message=send by Python server, message=0
recv from server(1), message=send by Python server, message=1
recv from server(1), message=send by Python server, message=2
recv from server(1), message=send by Python server, message=3
recv from server(1), message=send by Python server, message=4
--------------Call ServerStreamingMethod Over---------------
--------------Call BidirectionalStreamingMethod Begin---------------
recv from server(1), message=send by Python server, message= 0
recv from server(1), message=send by Python server, message= 1
recv from server(1), message=send by Python server, message= 2
recv from server(1), message=send by Python server, message= 3
recv from server(1), message=send by Python server, message= 4
--------------Call BidirectionalStreamingMethod Over---------------

After I run client, server.py:

~/Desktop/grpc/examples/python/data_transmission$ python3 server.py 
------------------start Python GRPC server
SimpleMethod called by client(1) the message: called by Python client
ClientStreamingMethod called by client...
recv from client(1), message= called by Python client, message:0
recv from client(1), message= called by Python client, message:1
recv from client(1), message= called by Python client, message:2
recv from client(1), message= called by Python client, message:3
recv from client(1), message= called by Python client, message:4
ServerStreamingMethod called by client(1), message= called by Python client
BidirectionalStreamingMethod called by client...
recv from client(1), message= called by Python client, message: 0
recv from client(1), message= called by Python client, message: 1
recv from client(1), message= called by Python client, message: 2
recv from client(1), message= called by Python client, message: 3
recv from client(1), message= called by Python client, message: 4

The example that use: data_transmission.zip

[gus6202]

@dimakuv

Hey, I only managed to hide the error message (I added loader.env.GRPC_VERBOSITY="NONE" in my python.manifest.template file), but the main problem (print something in server.py or print messages received from the client) I couldn't do it.

[lejunzhu]

The server.py script actually works correctly, but its output is buffered by Gramine. Add 'flush=True' to the print() calls in server.py, and you will see all the output.

[mkow]

Should be fixed by https://github.com/gramineproject/gramine/pull/1031, which was just merged. If not, feel free to reopen.