opensource-observer
Epic: Paid tiers for API access
We currently use Hasura for a GraphQL API. If we expect others to use this, then we probably want to at least cover the cost of the serving infrastructure
Going to deprioritize this for now. I think we'll track this manually for now via the data_collective Supabase table rather than do it programmatically
@ccerv1 Notes from today:
LLMs:
- Access Y/N
- Quality of models
pyoso:
- Data reads (BigQuery vs GCS/Iceberg)
- Data model utility/value
- Network egress
- Wall time
GraphQL API:
- Data model utility/value
- Network egress
With some thoughts on the credit system UX
@Jabolol Would love your help designing the metering system for this. Feel free to file more sub-issues
@Jabolol Would love your help designing the metering system for this. Feel free to file more sub-issues
Hello! For the first iteration of our billing strategy I was thinking of something like:
LLM
For our LLM, we'll structure it with four distinct tiers: a Free tier allowing 1K calls monthly, a Pro tier at $X/month with 10K calls, a Team tier priced at $Y/month supporting 50K calls, and a custom Enterprise tier. Pricing will factor in both model quality (categorized as Basic, Advanced, and Expert) and overall call volume.
We'll implement rate-limiting by tier using a credit-based system where 1 credit equals 1 Basic call, 5 credits for Advanced, and 10 credits for Expert-level calls. Resource allocation will place Free and Pro users on shared public servers, while Team and Enterprise customers will benefit from priority queue access. Enterprise clients will additionally receive dedicated compute resources and guaranteed throughput SLAs to ensure optimal performance.
pyoso
Our pyoso library will be priced based on data processed, measured in gigabytes, with BigQuery sources charged at twice the standard rate. We'll establish baseline rates of $0.10/GB for standard sources and $0.20/GB for BigQuery sources, with a 15% volume discount automatically applied for usage exceeding 1TB monthly. The Team feature will enable shared data processing quotas distributed across team members. We'll implement a Priority Queue system that places Team and Enterprise jobs ahead of Free and Pro jobs during peak load times, ensuring better performance for higher-tier customers.
Infra
I'm proposing we implement a Dedicated Infrastructure approach where we maintain lower-cost machines for our Free and Pro tiers, with more powerful dedicated hardware for Team and Enterprise customers. The Enterprise tier could even have fully isolated environments with guaranteed resource allocation, allowing us to offer stronger SLAs and higher throughput guarantees than would be possible on shared infrastructure.
GraphQL API
The GraphQL API will use core metrics of query depth (categorized in levels 1-3) and data volume transferred in megabytes. Standard pricing will be set at $0.05 per Level 1 query, $0.10 per Level 2, and $0.20 per Level 3, with data transfer costs structured as first 10GB free, then $0.15/GB thereafter. We'll maintain separate resource tiers with public API servers handling Free and Pro traffic, while enhanced infrastructure will support Team and Enterprise customers for improved reliability and performance.
Team Management v1
Our team management functionality will include a Team Admin Console where administrators can invite and manage team members and assign appropriate roles. The Credit Allocation feature will allow admins to distribute credits across team members or set individual usage limits. We'll provide robust Usage Tracking at both team-level and individual member levels through comprehensive reporting. Billing will be simplified with a single consolidated bill covering the entire team's usage across all products.
Credit System v1
The initial credit system will maintain either team or individual-level credit balances shared across all products. Users will purchase credits manually through the dashboard interface. The system will send email alerts when credit usage reaches 80% of available balance. For this first version, we won't implement auto-replenishment capabilities, keeping the system straightforward while we gather usage patterns.
Requirements
To support this billing strategy, we'll need several key technical components: a simple API call counter with throttling service, Stripe integration for processing credit purchases, and a basic usage dashboard showing daily and weekly totals. We'll also require a team management interface, priority queue implementation for compute resources, and separate server pools dedicated to Free/Pro versus Team/Enterprise customers.
Next Steps
- implementing metering systems for each product
- setting up the Stripe payment flow
- building a minimal usage dashboard with team views
- creating basic team management functionality
- implementing the priority queue system; and defining exact tier limits and pricing
Here's an idea of how we could implement the server-side billing checks:
Had a quick chat with Carl, and here's some ideas we had:
- The user should be able to buy credits beforehand
- The user should get warned when they're about to reach their max credit allocation and downgraded to a cheaper LLM model so their work is not interrupted
- There should be a notion of "enterprise credit pool" which is a credit pool (bought beforehand) to which all users of an organization have access to and can consume
- The user should be able to buy credits themselves as the main way of getting credits
- Credits must not be a 1:1 with USD. When real currency is used as the metric, people will get more worried about their spending. E.g. with LLMs like Claude: "this code session was 4.22 USD". I think we should be a little more opaque
- We should start setting up measurements and have Carl as our power user in his day to day life using our services, and see which features he uses the most/does not use at all and focus our pilot pricing strategy on those areas
- There should be a way for users to "preview" their queries without racking up in costs, issue #3771 tracks this.
- We should explore the idea of having dedicated infra for enterprise clients, and have a cheaper cluster running for the free plans. This would allow us to save up on costs while we are refining strategies and our client base is still small.
- There should be an opt-in way to asses how efficient/inefficient a query is. If the user does a select of a huge table, they should be warned beforehand. We can leverage a mix of
sqlglot+trinotools (like stage count) and maybe LLMs. I think we should not use LLMs as the default option since pricing should be deterministic and predictable, and we cannot risk hallucinations.
This is an awesome wishlist, but again, my feedback is to scope down, down, down. Feel free to break this epic into paid tiers v1 and paid tiers v2 and many other epics.
Some things that stood out:
For v1, I'd focus on API keys by organization (and purchasing credits by organization), but not give users the ability to buy their own credits. Supporting both is significantly more complicated and not worth doing now.
For v1, no dedicated infrastructure. Multi-tenant only. Supporting dedicated infrastructure is a typical enterprise feature, but that is a separate epic. This is a huge ask and needs to be scoped separately.
Cost estimation tooling is a cool idea. That is a separate epic, not in the MVP of getting paid tiers working at all.
Can we take a stab at defining the bare minimum to get something working end to end? Let's get on the same page about that ASAP.
Haha yeah fair points! I got a bit carried away. So many exciting things to build!
For v1 MVP, lets focus on:
- Basic API key auth by organization
- Simple credit purchase at org level
- Basic metering to track usage
- Minimal dashboard to see credits/usage
We can park all the fancy stuff (dedicated infra, user-level credits, cost estimation) for v2.
Let's discuss on Monday what specific metrics we should track first to get this rolling! :)