Rate limit

[metallicusdev]

Running into rate limit

Rate limit reached for default-global-with-image-limits in organization XXX on requests per min. Limit: 60.000000 / min. Current: 70.000000 / min. Contact [email protected] if you continue to have issues.

[cfortuner]

What are you using in promptable? Can you share your usage?

[metallicusdev]

import { OpenAI, FileLoader, CharacterTextSplitter, Embeddings } from "promptable";

// Create a model provider!
const provider = new OpenAI(process.env.API_KEY);

// Create documents
// Load documents
const loader = new FileLoader('../output.md');
let docs = await loader.load();

// Split documents into chunks
const splitter = new CharacterTextSplitter("\n");
docs = splitter.splitDocuments(docs, {
  chunk: true,
  chunkSize: 1000, // tokens :)
});

// Create embeddings
const embeddings = new Embeddings(
    "proton-ts-embeddings",
    provider,
    docs,
    { cacheDir: "." }
);
await embeddings.index();

It should be able to handle API ratelimits and retry

[cfortuner]

What are your thoughts on using something like this??

https://github.com/cfortuner/promptable/pull/12

[ymansurozer]

I've just discovered this repo and it is very promising. It would sooo good to have an NPM alternative of GPT Index. Thanks for the great work!

My humble view: #12 looks great. But in embedMany, is there a reason why you're not sending the documents to the embeddings endpoint as an array input (string[]) and instead sending them in separate requests as single input (string)?

[cfortuner]

Whoops! Yeah i'll update that @ymansurozer

Thanks for pointing that out.

Also, I'd love your input on the api you think we should make for Indexing. Want to DM me in Discord? https://discord.gg/PGVadrjm

[ymansurozer]

Happy to help! Now we need to somehow handle embedding requests that exceed 250,000 tokens in total to overcome rate limits. :) I'll DM on Discord now. @cfortuner

[hanrelan]

@cfortuner if openai returns a retry-after header in their 429 status code we should use that for the rate limiting

[cfortuner]

Hey just a quick update.

we're still working on the right implementation here, but coming soon!