connectrpc
Add RFC 004-Python Implementation
The latest updates on your projects. Learn more about Vercel for Git ↗︎
| Name | Status | Preview | Comments | Updated (UTC) |
|---|---|---|---|---|
| connect | ✅ Ready (Inspect) | Visit Preview | 💬 Add feedback | Apr 18, 2024 8:47pm |
![vercel[bot] avatar](https://avatars.githubusercontent.com/in/8329?v=4 "Published by a pub.dev verified publisher"){kind=small}
❤️ Many, many thanks for writing this up, Matt! We've been a little swamped, but this hasn't fallen off our radar.
@mattrobenolt is attempting to deploy a commit to the connectrpc Team on Vercel.
A member of the Team first needs to authorize it.
@mattrobenolt Sorry to suddenly jump into this conversation, but I have something to share that may be relevant to your PoC and this PR. At the end of last year, I clearly realized I might need a Python server implementation for my job. To get ahead of this, I spent a few hours over the New Year holiday implementing my own version, independent of your PoC implementation or design. You can find it here: Connecpy. Currently, this implementation supports only Unary RPC of Connect Protocol, but the service handler interface's type signature is compatible with Python's gRPC framework interface, which seems to be sufficient for my current use cases.
@akshayjshah @mattrobenolt Also, I've been using buf CLI and connect-go at work, and they've been incredibly helpful. Big thanks to all the creators and contributors for your fantastic work.
Thank you for creating the POC. The Connect over HTTP/1.1 is exactly what I was looking for. I might be checking the async implementation if other things related to Connect work well.
👋 So to clarify, @mattrobenolt and @i2y if someone wants to contribute code for this effort, would it be best at this point to do so at https://github.com/mattrobenolt/connect-python or at https://github.com/i2y/connecpy and how hard would it be to merge these into one effort? Even more specifically, if I dumped any missing code from one into the other, from which repo to which repo should I go?
I mean, it would also be great if someone could just get paid to work on this at their day job, but I know how hard it is to justify to management investing work on critical behind-the-scenes support when it is indirect to more immediate results. Thanks for all the volunteer work so far! ❤️
Hi folks, I wanted to share a project that I've been working on: https://github.com/dsludwig/sonora-connect
This is a fork of the existing sonora which implemented GRPC-Web in both ASGI and WSGI handlers, without adding additional code-generators beyond the standard protobuf pb2 and pb2_grpc. It also conveniently is written as an ASGI/WSGI middleware, so that GRPC-Web handlers can easily be added to an existing application. I have added support for Connect and GRPC to my fork. (GRPC works only with an ASGI server that implements the trailers extension, which seems to just be https://github.com/nonebot/nonecorn).
I have also added conformance tests for the clients based on @mattrobenolt's conformance code, and written server conformance tests. All supported server test cases are currently passing, and there are only a few remaining client test cases that are failing due to unimplemented features.
@dsludwig Oh, that seems to be a great approach. Also, I have not noticed "sonora." I might contribute to your repository as well. However, as you can see in one of the diagrams I included in my Japanese blog post (https://tech-blog.monotaro.com/entry/2024/06/25/090000), we have already started using "connecpy" as one of the foundational libraries for our internal libraries. For that reason, it might not be possible to switch to your solution or library immediately, although it could become a possibility in the near future. At the same time, I think having alternative approaches or solutions available isn't necessarily a bad thing in open-source initiatives, so I may use your code and @mattrobenolt's code as references to improve "connecpy." In any case, I think your approach is very clever and impressive. Thanks for the sharing👍
bumping up this thread, is there any roadmap items for supporting connectrpc on python?
Hi folks, I wanted to share a project that I've been working on: https://github.com/dsludwig/sonora-connect
So no buf plugin as for now, yes?
Hey all 👋
So just a status update since I've been kinda MIA on this, when I expected to personally spearhead developing this.
For a bit of history, most of my career has been professionally developing Python, starting back in Python 2.3 and I was quite involved in the Python community for a large period of that time.
I mostly ended writing Python when Sentry started moving to Python 3, which was around the time Python 3.6 became popular.
Prior to this, I spent most of that time writing Python 2.6 and 2.7, with Python 3 being for small scripts and small greenfield projects.
I have since stopped writing Python entirely for anything professional. I keep writing Python for small scripts as needed, but basically just using Python 2.7 syntax in modern Python.
As a part of this project, I got really burned out with the modern Python landscape trying to catch up. Very specifically modern Python types and ASGI/asyncio are... extremely not something I like. Publishing a modern Python library would come with the expectation of strong type safety and supporting asyncio, which is something I personally strongly disagree with in Python and that alone is a strong demotivator to write modern Python and publish a public library.
On top of that, I have no personal or professional use for Connect in Python other than just a personal interest in developing it, but that personal interest kinda fizzed out by getting caught up in modern Python semantics. I have learned through this that I do not like writing modern Python, and publishing something how I'd want to do it would not be received very well, so I kinda lost a lot of interest in spending my personal time on something I find very frustrating and annoying. Connect is awesome, I just really hate modern Python.
For those interested, I had a lot of progress on this branch: https://github.com/mattrobenolt/connect-python/tree/ref that was working on passing conformance tests.
It's sad that Matt dropped the effort on it.
But, @i2y, @dsludwig, had you considered transforming your efforts into buf plugin?
But, @i2y, @dsludwig, had you considered transforming your efforts into
bufplugin?
The library I am working on does not require a buf plugin - it uses the output from the existing Python plugins (grpc/python and protocolbuffers/python). This means it cannot support the GET functionality, since that information is not present in the generated output. I am also interested in adding support for betterproto, if it is not too disruptive.
I have a few more things to do before releasing a first version, I will drop a note here when I get to that point.
But, @i2y, @dsludwig, had you considered transforming your efforts into buf plugin?
I’ve considered it a little before. I wish that I or someone else could do that in near future.
I'd like to take this on. My company (Firetiger) and I can support this long-term. I have a good first crack at an implementation at https://github.com/firetiger-oss/connect-python/. It currently has:
- sync clients (using
requests) - async clients (using
aiohttp) - sync servers (as WSGI apps, no external dependencies)
It passes conformance tests, and is fully type hinted. It only does Connect protocol, no multi-protocol support yet. It supports setting headers and trailers in a persnickety API, but also has simpler APIs for the common case where you just want to pass messages back and forth.
It has its own code generator which produces the client and a server typing.Protocol, so it's something you would use as a buf plugin.
I'm talking with @mattrobenolt in the CNCF slack about getting it pushed to PyPI under the 'connect-python' package name right now, and plan to publish a v0.4.1 release which has these early features and is good enough to test in low-stakes settings. I still have to write a bunch of docs on everything.
I plan to add an ASGI implementation for people who feel the need for speed. Once that passes conformance, I'd like to try to get that adopted as an "official" implementation.
I have generated docs at https://connect-python.readthedocs.io/ and I have pushed a v0.4.1 to https://pypi.org/project/connect-python/. Please take it for a spin!
For those following along, https://github.com/connectrpc/connectrpc.com/pull/285 is the successor RFC to this RFC. If approved, the Connect project hopes to have an official Python implementation imminently. Thanks to @spenczar for spearheading this initiative!
@bufdev @spenczar This is wonderful news! 🎉
BTW, I've been maintaining and improving https://github.com/i2y/connecpy for quite some time now. I originally created it to help promote the adoption of protobuf, schema-driven development, gRPC, and Connect RPC at my workplace, where we've been using it alongside connect-go in production.
I would have loved to see Connecpy become an official implementation and would be happy to transfer it to an appropriate organization if that would help. Is there a possibility that Connecpy could eventually be recognized as one of the official Python implementations in the future if it meets certain criteria?
I'm excited about the growth of the Connect ecosystem and would love to contribute to making Python support even stronger!
Quick update - at my workplace, Connect-go is used by many teams and Connecpy by several. Also happy to see @anuraaga recently contributing to Connecpy from outside our company!
Seems like there's growing interest in Python support from multiple places 😊
Closing as https://github.com/connectrpc/connectrpc.com/pull/285 has been merged! We're excited to move connect-python forward!
Just wanted to share that we've reached a "feature-complete" version of Connecpy, with support for all stream types including bidi, sync/async for client/server, compression, interceptors, with no significant TODOs known of. There is an explicit design decision to make sure there aren't any tedious .message type of accesses with access to metadata completely optional via a context manager. The testing infrastructure is maybe particularly innovative, with code coverage and IDE debugging working fairly well (if the conformance CLI supported disabling test timeout by a flag, debugging would be perfect). Coverage is above 95% with a general aim on quality. I'm happy to start migrating my Python projects to Connecpy on the server after having done it on the client before.
I suspect there are hints in the codebase that can help move the upstream project forward. While I don't have high hopes of this comment actually getting read, or there being enough interest to take such a deep dive, but adding it just in case it can connect to any improvements.
Looks good. Does anyone ran or plan to run a benchmark? Comparing to fastapi, django etc, would be interesting to see the diff and maybe create some traction.
I suspect there are hints in the codebase that can help move the upstream project forward. While I don't have high hopes of this comment actually getting read, or there being enough interest to take such a deep dive, but adding it just in case it can connect to any improvements.
@spenczar and I would love to talk about adopting much of connectpy into this effort - feel free to message us on the CNCF slack in the #connectrpc room!