llm-scraper icon indicating copy to clipboard operation
llm-scraper copied to clipboard

break page into chunks html mode

Open EcomGraduates opened this issue 1 year ago • 2 comments

long pages tend to cause a token error, It would be useful if it could calculate the tokens of a page and break it up into chunks or maybe strip some of the html out that we don't technically need? script tags ect

EcomGraduates avatar Apr 25 '24 18:04 EcomGraduates

Really good idea 👍

mishushakov avatar Apr 25 '24 18:04 mishushakov

Feel free to open a pull request, if you have even a rough idea how the code for this could look like

mishushakov avatar Apr 25 '24 20:04 mishushakov