Context length exceded

[jamesmurdza]

I get this error when running any command on Amazon.co.uk:

"This model's maximum context length is 8192 tokens. However, your messages resulted in 8661 tokens. Please reduce the length of the messages."

I guess this is done while trying to maximize the number of tokens within the GPT-4 maximum context size?

[paulVu]

I think problem is scrawl all html, so always get over tokens. I think should not only use GPT, I think should combine with https://webscraper.io and action base on that.

[CryptoMitch]

You're right about using Webscraper, it's a good solution. What if we implement a method to preprocess the HTML content before sending it to GPT-4, reducing the token count while preserving relevant information?

[nithinreddyyyyyy]

How to overcome this issue then?