Text encoding and search for handling large contexts

[ATheorell]

Feature description

Currently, improve mode is (very) restricted in the size of the codebases it can read, due to the token limit of the LLM. The state-of-the-art (?) solution is to chunk the read code, embed it in a vector space, search for the vectors most closely related to the prompt and decode these vectors back to text. How to do this is described in:

https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb

Since gpt-engineer is using the langchain to interface LLMs, instead of direct calls to OpenAI like in the linked notebook, the encode calls should be done through langchain:

https://python.langchain.com/docs/integrations/text_embedding

Support for encoding and decoding should be added to the AI class (core/ai.py) and the encode/search/decode should go into a new core class.

Note that, we are only discussing encoding and decoding on an in-memory basis. Extending this to using a vector db will go into a separate feature request if deemed necessary.

All the above is likely to already be implemented somewhere, and finding and evaluating a suitable lightweight module and integrating it as a dependency in gpt-engineer may be the best way to do the above.

Motivation/Application

It will allow gpt-engineer to work with larger code bases and potentially also more complex promts, such as prompts including up-to-date interface documentation.