gpt-crawler icon indicating copy to clipboard operation
gpt-crawler copied to clipboard

Published 20 hours ago • verified publisher icon BuilderIO

Size

Open albrox opened this issue 1 year ago • 3 comments

I have successfully crawled a whole website and have the output file as JSON.

The problem is that the file size is 93MB and after uploading it to ChatGPT I get an error message stating that the file is too large. Is there a known limit in size that can be uploaded and can this be chunked into different parts?

albrox avatar Nov 27 '23 12:11 albrox

hello @albrox please look at my PR it just about that same problem

guillermoscript avatar Nov 28 '23 14:11 guillermoscript

@guillermoscript Thank you very much for your PR. I would like to ask you what is the appropriate size for a single file. I have not found the official instructions of openai.

learner1999 avatar Dec 03 '23 12:12 learner1999

hello @learner1999 sorry but I also don't know a recommended size per file, that's going to be trial an error for you use case and with the PR you can change the size to fit your needs

guillermoscript avatar Dec 03 '23 18:12 guillermoscript