prisma
Add support for AWS Data API (AWS Aurora Serverless)
Problem
Prisma currently doesn't seem to support the Data API for AWS Aurora Serverless.
Solution
It would be great to add support for it which is especially useful for applications running on serverless environments like AWS Lambda.
I absolutely love this and wanted it for some time, however be mindful that there are some limitations right now, especially for those of us in the EU:
-
Aurora Serverless for MySQL only supports MySQL 5.6 right now, which I am not sure is still supported by Prisma 2 (the supported-databases.md only mentions MySQL 5.7). Aurora Serverless for MySQL 5.7 will likely launch this year, but it hasn't so far
-
~~Aurora Serverless for PostgreSQL only supports PostgreSQL 10.7 right now, and is also only available in Ireland, not Frankfurt (Germany) in the EU so far~~ No longer the case, see my comment below
-
~~Data API for Aurora Serverless is only available in some regions so far, e.g. not in Frankfurt (Germany) (EU-Central-1), but in Ireland (EU-West-1)~~ No longer the case, see my comment below
Linking my comment for relevance, as suggested: https://github.com/prisma/prisma/issues/2133#issuecomment-635062736
Will love to do any testing when needed.
A few updates since my comment above.
-
Data API for Aurora Serverless is now available in more regions, including eu-central-1 (EU/Frankfurt), see https://aws.amazon.com/about-aws/whats-new/2020/05/amazon-rds-data-api-and-query-editor-available-additional-regions/
-
Aurora Serverless for PostgreSQL (10.7) is now available in more regions, including eu-central-1 (EU/Frankfurt). It includes support for Data API for Aurora Serverless
May I have a bold idea here:
In terms of supporting the Data API, can we compile the whole GraphQL query into one SQL and execute it over Data API?
What a performance...
I'm using https://redwoodjs.com/ which deploys a graphql API as a serverless function. It would be great to have Data API support so that AWS Aurora Serverless can be used with such serverless applications.
More use cases described in this issue: https://github.com/prisma/prisma/issues/2133
I'm incredibly interested in the possibility of making this happen. Has anyone looked deeply at what it would take yet? I use Prisma all the time but have yet to actually go through the source code. If this is a manageable project it may be one I want to take on, but I'm wondering if anyone has looked into scope, or if Prisma has plans already in place for adding this?
There is Data API client which is also used by TypeORM https://github.com/ArsenyYankovsky/typeorm-aurora-data-api-driver
The Data API seems to be plagued with some issues related to casting types like json, dates etc which is handled by the above client https://github.com/jeremydaly/data-api-client/pull/57
It seems like to be able to use this Prisma would have to explicitly provide the types to the Data API client.
@medv mentioned that they are using this with TypeORM and can probably share their experience.
We opted to use Jeremy Daly's data-api-client instead of the typeorm implementation (which I would not recommend, but this is my opinion). This approach has worked surprisingly well for our serverless use case. However, the dev ux has suffered a fair bit in comparison to what otherwise could have been with prisma.
@medv , would love a bit of expansion on why you ditched typeorm/typeorm-aurora-data-api-driver in favor of data-api-client.
TypeORM itself did not fit our team, so our decision was less about the data api implementation. data-api-client allowed us to retain our temporary ORM solution in the back-end while providing low latency queries and small bundle size through data-api-client for our serverless front-end served from lambda@edge (temporary because we are eagerly awaiting prisma with migrate and data-api to be well defined and widely used).
So, I asked a prisma developer this directly, and he took the question to the product team. I specified that I had a project launching in July, and if the integration would be done by summer, we would build with prisma. They said that currently they have no plans to prioritize this, and it would not likely be done this year. Obviously the more interest the more likely they are to prioritize, but right now I don’t think it is really on the radar.
So, I asked a prisma developer this directly, and he took the question to the product team. I specified that I had a project launching in July, and if the integration would be done by summer, we would build with prisma. They said that currently they have no plans to prioritize this, and it would not likely be done this year. Obviously the more interest the more likely they are to prioritize, but right now I don’t think it is really on the radar.
That's extremely disappointing. @justinhandley what are you planning on doing?
can i use aurora serverless as prisma db or not? (or is this thread only about using data api for internal processing?) https://www.prisma.io/docs/reference/database-reference/supported-databases
'cause aurora serverless is listed as supported db
@softmarshmallow, you can use aurora serverless. This conversation is about the Data API. In the fantasy of moving to a completely scalable and serverless environment, as well as having your database scalable, your endpoints would ideally also be scalable and served from edge servers. Unfortunately, with aurora serverless, you can’t access the database from outside of the region the database is in, so you can’t have an api served on edge lambdas connect. You could theoretically achieve this with the aurora data api. But, you are good to go.
@schickling himself opened this issue so I'm sure he sees the huge opportunity this would mean for developers using Prisma. In this whole JS world that is trending more and more towards Serverless, scalable database access is one of the largest bottlenecks. I believe Prisma would benefit hugely from this move and I wonder what the reasoning is to not offer support in the short and medium term.
Disclaimer: I do have to say though that I do not know how resource-intensive it would be to make these changes (time-wise). From an outsider perspective, wouldn't it "just" mean sending the SQL strings (that the engine already generates) via HTTP rather than a native connection? Looking forward to a quick response, if possible at all :)
On a separate topic but kind of related, I think it will be in the best interest of developers using Prisma to mention the current status of Data API and RDS Proxy compatibility in the docs and wherever else applicable. Right now this information is painfully hard to weed out. I have opened a PR for the same https://github.com/prisma/docs/pull/1409
@medv - I'm wondering if you could share your experience using the Aurora Serverless Data API so far? I have talked to a number of people who had trouble using it at scale reporting issues such as large increase in response time, random failures and sometimes receiving wrong data. Have you experienced something like that or has it been smooth sailing?
@medv - I'm wondering if you could share your experience using the Aurora Serverless Data API so far? I have talked to a number of people who had trouble using it at scale reporting issues such as large increase in response time, random failures and sometimes receiving wrong data. Have you experienced something like that or has it been smooth sailing?
I can definitely confirm all that you said. I wonder how Aurora Serverless 2 would fare and looking at fauna in the meantime, but we have opted for a non-serverless Aurora with 1000 connection limit as a stopgap. Slight gritting teeth at the tape holding this all together. The idea is incredible from the getgo and in theory this is what we wish to do, but we have to bide our time with these interim limbo dances in the face of reality. Our usecase is not massive, but it's still about ~5mill users a month so plenty for us to invalidate a lot of approaches which we have so far.
- Prisma on top of Data API would solve massive dev ux overhead, but may not entirely solve Aurora Serverless issues (may address some client layer issues over time due to sheer number of users and amazing prisma dev team having some ownership). This would already be the front-running solution in our eyes. Even with remaining issues, we would already bite the bullet and use this with workarounds on our end, whether it's cache fallbacks for slow or unavailable data, or retries etc. Perfectly happy to sustain the losses for the productivity boost and beautiful code approach.
- Fauna with a prisma binding would be pretty ace, but probably has unforseen issues in of itself, I will be happy to report back after our experiments which are on the cards but have no real date at of now.
- Ultimately, Prisma + whatever database driver that addresses connection pools would win the internet in our eyes after having played with a lot of arrangements.
Lambda@Edge deployments are absolutely incredible for whatever scale, and certainly help us enter a project with no growth expectation with our existing turnkey deployment while knowing full-well that it would be completely fine with sudden bursts of millions of people in the event of a business success. An hour to bootstrap a whole stack with scaling insurance puts us at absolute rest and allows us to extend massive cost savings to our partners or clients. The distributed nature is incredible and allows us to optimise on a per-region basis by replicating S3 and RDS resources with routing at edge, so it can grow into a super low latency solution when the need arises (getting there). This last hurdle is what keeps this from being truly great, and we're betting on prisma as we absolutely love the approach and the consistency of the prisma team's efforts.
Thank you @medv - that is extremely helpful!
One of our product teams is tasked with solving the problems you are describing. I would love to get on a call together with our pm to make sure we capture as much as possible. Would you be open to a call sometime in the coming weeks? You can contact me directly at [email protected]
I was using https://github.com/ArsenyYankovsky/typeorm-aurora-data-api-driver with TypeORM and it works great. Data API is amazing because:
- you don't need to save a password anywhere
- you can allow only certain functions to talk to the DB with IAM
- no VPC needed - MUCH simpler environment setup - lambdas don't need to be in VPC and don't need NATGWs
- no need for libpq-based native client - can do queries with pure JS/TS
I'm trying to migrate from TypeORM to Prisma and this is the only area where Prisma doesn't meet my needs.
I'm building CDK apps with https://github.com/jetbridge/typescript-cdk-template
How about opening up the connector interface so that others could contribute? Though I wouldn't turn down direct support either. :)
@revmischa - you might be interested in talking to the team working on the new Data Proxy feature that is aiming to solve this problem. If you talk to the PM, we can make sure to design it in a way that meets your needs: https://prisma103696.typeform.com/to/IG9ZKSKk
Yes, I'd love to talk to you about your use case @revmischa. You can schedule time with me using this: https://calendly.com/alberto-prisma/45-min-zoom-call
@revmischa I'm in the same boat. The CDK is absolutely amazing, all it needs is a good ORM to integrate with Aurora Serverless and we're bordering on cloud perfection. TypeORM and the driver you linked work, but they're both full of nasty surprises and workarounds. I asked for an update on this in the Prisma Slack less than 2 weeks ago and was told (and I believe quite authoritatively) that the Data API is orthogonal to how Prisma is designed and so it ain't on the roadmap, at least not for the foreseeable future.
That said, I think in a matter of months the CDK's popularity will have reached a level that the Prisma team will want to get onboard. Hopefully we have to stick out TypeORM for only a little longer...
@revmischa I'm in the same boat. The CDK is absolutely amazing, all it needs is a good ORM to integrate with Aurora Serverless and we're bordering on cloud perfection.
~I've been building out some reusable constructs for Prisma in CDK: https://github.com/jetbridge/jetkit-cdk/pull/13~
Edit: now just using SST
There is no update on support for AWS Data API, but we've been busy working on the Prisma Data Proxy. For anyone interested, it's available in Early Acess and we have documentation available.
It will work with any database supported by Prisma that is accessible via internet (PostgreSQL, MySQL, SQLServer, MongoDB). Currently, there is a requirement for the database to be accessible from public internet but planning to work soon on removing this limitation.