Write a doc and sample on how to use gRPC with nginx

[zinuga]

[hsaliak]

@fengli-google would you be able to help out with this?

[hsaliak]

@fengli-google have you had a chance to look into this? @zinuga is P2 the right priority for this content?

[zinuga]

yes, we can drop this from GA milestone.

[shulegaa]

Please! I could use this documentation right away. Bummer that gRPC isn't further along - esp. with respect to NGINX

[clehene]

This may be relevant https://trac.nginx.org/nginx/ticket/923

[zinuga]

@shulegaa we are in the process of submitting a 3rd party module in nginx for grpc support. we cant control the nginx core code or timeline for http/2 support. for now, if its critical for your needs, we can point you to a custom nginx binary where we have built support for grpc.(usable for testing/prototyping).

[shulegaa]

Do tell! Please. I'd be very interested in seeing what you've done. Depending on timing, we may go down a similar path. Long story.

[wangming1993]

I use nginx1.10 which support http2 as grpc server proxy, then go grpc client send a request to server, from nginx debug log, I find such message:

2016/09/12 14:48:57 [debug] 14677#14677: *1 SSL certificate status callback
2016/09/12 14:48:57 [debug] 14677#14677: *1 SSL NPN advertised

2016/09/12 14:48:57 [debug] 14677#14677: *1 http2 preface verified
2016/09/12 14:48:57 [debug] 14677#14677: *1 process http2 frame type:4 f:0 l:0 sid:0
2016/09/12 14:48:57 [debug] 14677#14677: *1 http2 send SETTINGS frame ack:1
2016/09/12 14:48:57 [debug] 14677#14677: *1 http2 frame complete pos:00007F5DA7C0A5F1 end:00007F5DA7C0A5FE

2016/09/12 14:48:57 [info] 14677#14677: *1 client sent stream with data before settings were acknowledged while processing HTTP/2 connection, client: 127.0.0.1, server: 0.0.0.0:443

nginx can receive the connection preface starts with the string PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n), but it send a RST_STREAM frame, and refuse to accpet any request.

Nginx http2 supports required a ack before client send data frame, I know that.

When I use go http2 to send a http request, the nginx can response succeed. So what's difference and reason for that?

Anyone helps

@fengli79 @zinuga @hsaliak @shulegaa @clehene @fengli-google

[danaschwanden]

@zinuga could you please also point me to your 3rd party nginx module for grpc support? thanks

[fengli79]

Nginx today doesn't support HTTP2 as upstream, so you cannot use it as a L7 proxy for GRPC. Instead you can use it as a L4 proxy.

For the RST_STREAM issue, it's a known regression in nginx. https://trac.nginx.org/nginx/ticket/959

[ynezz]

@zinuga Hi, I'm also considering grpc and nginx for my next project. Can you please kindly share the source code of that nginx module? I'm not interested in binary blobs :-) Thanks!

@fengli79 BTW that NGINX RST_STREAM regression seems to be fixed.

[hoswey]

@zinuga could you please also point me to your 3rd party nginx module for grpc support? thanks

[amitripshtos]

Much needed feature indeed, I saw some posts on the internet talking about ngx_stream_core_module

[xiaoyunwu]

@zinuga same here

[amitripshtos]

I went a bit of exploring and understood it better, and created a load balancer that balance automatic using docker + consul: https://github.com/amitripshtos/grpc-docker-lb

i'm using nghttp2 for this one, by the way

[ggaaooppeenngg]

@zinuga same here, it seems nginx supports HTTP/2 but not full. Any updates now?

[idevz]

@zinuga same here, thanks.:)

[siacomuzzi]

@zinuga same here (https://github.com/grpc/grpc/issues/11427)

thanks!

[zinuga]

based on this https://github.com/grpc/grpc/issues/11427, it is matter of a few weeks now before its upstreamed. credit @PiotrSikora. once done, @LisaFC can help with doc on how to use.

[siacomuzzi]

There are some fixes provided by @PiotrSikora in the latest versions of nginx (http://nginx.org/en/CHANGES):

Changes with nginx 1.13.2                                        27 Jun 2017

    *) Feature: the "add_trailer" directive.
       Thanks to Piotr Sikora.

    *) Bugfix: in the ngx_http_v2_module.
       Thanks to Piotr Sikora.

Do you know if we can use gRPC with nginx now? Thanks!

[PiotrSikora]

@siacomuzzi unfortunately, those are only prerequisites and bugfixes for existing HTTP/2 server-side code. HTTP/2 to backends is still pending review / merge.

[dyadaval]

Hello wangming1993, PiotrSikora

Could you please share the working nginx.conf. We are seeing that our nginx h2 proxy(taken from PiotrSikora github) is not forwarding to gRPCServer. our conf file part looks below: server { listen 8443 ssl http2; listen [::]:8443 ssl http2; ssl_certificate /f0/base/nginx/ssl/certs/server.crt; ssl_certificate_key /f0/base/nginx/ssl/private/server.key; # ...

location / {
    proxy_pass http://10.220.73.160:50051;
    proxy_set_header X-Real-IP $remote_addr;
    allow all;
}

10.220.73.160 is the Network element, that has gRPCServer running.

nginx -v

nginx version: nginx/1.13.2

Thanks for help.

Best Regards, Div

[asinha08]

what is the update on using nginx with grpc?

[dyadaval]

No, was not lucky with the patch to get it working. You can use haproxy with grpc.

On Sun, Oct 15, 2017 at 2:51 AM, Ashish Sinha [email protected] wrote:

what is the update on using nginx with grpc?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/grpc/grpc.github.io/issues/230#issuecomment-336668326, or mute the thread https://github.com/notifications/unsubscribe-auth/AS4RIa7n059Ft806GmCaO4qJIz0RaX9dks5ssSXmgaJpZM4JHi3Q .

[snowzach]

So for anyone else here that want's to know the config to make it work I managed to get it going using @PiotrSikora patch. You need to disable buffering in order to make grpc streams work. @PiotrSikora, any idea if they plan to pull this? Pretty happy it works and it would help me out a lot.

location /grpcpackagename.grpcservicename {
        proxy_pass https://127.0.0.1:1234;
        proxy_http_version 2.0;
        proxy_pass_trailers on;
        proxy_request_buffering off;
        proxy_buffering off;
}

[pathbox]

waiting for good news

[mishika186]

@snowzach I am trying GRPC with nginx, can you please let me know detail of config "location /grpcpackagename.grpcservicename"

[snowzach]

@mritunjayblr it's basically just that.. GRPC is listening at a url of whatever the GRP package name is followed by the service name...

So if in your proto definition you have

package bob

service barker { rpc blahblahbl;ah }

your GRPC is technically listening at /bob.barker on your HTTP2 server.

[mhausenblas]

If folks want to, happy to contribute the example based on what I did in https://github.com/mhausenblas/kruiser/blob/master/how-does-it-work.md … however, I'm still struggling with serving from non-root URL. So works fine with location / but not /yages.Echo.

[mdkma]

Good news finally comes! https://www.nginx.com/blog/nginx-1-13-10-grpc/