Feature request: Log intermediate step to file on chain_type "refine" to help with the debugging process.

[ilhamfp]

Feature request

Log intermediate step to file on chain_type "refine" to help with the debugging process.

Have the library dump the intermediate step to a file on every step, even before the final output is calculated. This would the debugging process in the case of an error like this:

InvalidRequestError: This model's maximum context length is 4097 tokens. However, your messages resulted in 4413 tokens. Please reduce the length of the messages.

Motivation

As can be seen from this feature request, I have a use case where I keep hitting the "model's maximum context length" limit on chain_type refine.

Being able to see the intermediate results that lead to this limit being reached would be really helpful.

Your contribution

I would love to contribute to making this feature a reality! Please guide me on where I should look into.