Consider using new OpenAI features like uploading files, RAG, code interpretter

[paul-gauthier]

It would be great if Aider code could leverage new features of the model and API, such as uploading project files, RAG, and code-interpreter. Aider code would drastically shrink in size, yet increase in ability (by knowing more about your project), and performance (less http i/o). The /add command might even be optional to use; the AI could figure out which files are in scope.

That's a significant change tho, and would result in a much smaller project. Likely some new leaner project should be written that does it. It's probably too early to even make a ticket(s) about this change. Everyone is just learning what's possible.

Nothing to see here. Move along.

Originally posted by @mikeslattery in https://github.com/paul-gauthier/aider/issues/332#issuecomment-1802252271

[man0l]

This is a great idea. A simple embedding would free up the context, but I am not sure if this would improve the consistency of the generated code. It is a worth trying. Even a simple vector index adapter for FAISS, Pinecone or ChromaDB might do the work (preferably running in a docker container).

[mikeslattery]

This is a great idea. A simple embedding would free up the context, but I am not sure if this would improve the consistency of the generated code. It is a worth trying. Even a simple vector index adapter for FAISS, Pinecone or ChromaDB might do the work (preferably running in a docker container).

That's not quite what I was suggesting, but close. I was suggesting the new RAG feature of OpenAI be used, called Knowledge Retrevial. I wasn't suggesting increasing the complexity of this project, but rather decreasing complexity by off-loading some responsibilities to the new Assistant API features (knowledge retrieval [RAG], code interpreter, threads [persistent history]).

[man0l]

Thanks for clarifying. I am looking forward to the implementation. I even began researching how can I implement these features although i am not a python dev.

[mikeslattery]

~~I'm still excited about this ticket, but perhaps it should all wait until the dust clears to see what happens to openai as a company, and its assistant api. Azure AI Services doesn't have the assistant api, and it's possible the api could be discontinued if things continue on the current track. We shall see.~~ (edit: crisis over)

Also, some code file types aren't yet supported by its RAG, such as .js .ts .go.

[joenas]

Looks like js/ts are now available!

[lukestanley]

If Aider was dependent on these OpenAI specific features, it would tightly couple to OpenAI more and ultimately offer less control to end users. As model competition heats up (such as local models, or Claude 3), this seems like a poor trade-off, especially when there are other ways to offload code complexity, such as external libraries that have RAG and code interpreter features.

[mikeslattery]

As model competition heats up (such as local models, or Claude 3), this seems like a poor trade-off, especially when there are other ways to offload code complexity, such as external libraries that have RAG and code interpreter features.

@lukestanley, As the author of this idea, I generally agree. As I originally said, it might be better to do something like it as another project, or abstract it away into an agent base layer.

I'm excited about Claude 3 Opus and Groq, so being able to switch models has become important to me.

[jluyckx]

Respectfully, in my estimation, I think this is a very bad idea:

• Code interpreter: aider is intended to be a tool that can support bigger projects, they don't run automatically in a code interpreter environment from OpenAI, that's just small test scripts etc. Also, there's other languages than python etc.
• Furthermore, a beautiful and impressive part of aider is the way it uses treesitter to look at your code as a graph, use that as the foundation to build repo-maps etc. There are tons of possilibites like understanding method relationships and using that in terms of code etc.
• Of course, you don't want to upload your entire codebase to OpenAI...
• Agreeing with others: I think it's better for a tool like aider to be vendor-neutral as well.

To be honest, I think you have a poor understanding of aider and the RAG capabilities overall, to think that will be better. Aider's way more sophisticated, can do RAG if needed. It's 15K lines of code or so, aider complexity seems not to be the issue.

I like the philosophy of every tool being good at its job, and as good as possible at that. In such a way, an LLM should be great in.. being an LLM. Aider, on the other hand, should be great in using LLMs to create a "virtual pair programmer". I could write several pages of opportunities I think aider could have with going into further depth with its current model, using treesitter, using deeper analytics locally, working semi-formal code creation/validation using pre/postconditions and e.g. a theorem verifier like z3, using deeper code graph analytics to bring aider a better context not necessarily defined by files, ...

Again, all of this respectfully - my opinions, but I think the future of aider is bright, and could become a very sophisticated tool. Where, the LLM part, is just that, a part, a lot of the coding, techniques and approaches I would expect to become just traditional smart diligent and well researched classical algorithms that drive the LLM, which is quite counter to rely on a bag of tricks that OpenAI adds to their models to make them a bit more broadly applicable.

[drhouse]

@jluyckx will you be creating an Aider fork with your code ideas in the upcoming future?

[jluyckx]

I would prefer submitting it as a PR and help making aider better rather than forking.

I might try out some things, and if any presents a proper improvement I’ll come back with it. I’m sure there’s a lot of things that @paul-gauthier tried that did not work out either.

But I’d like to start with something simple. One of the things I see for example is that, because there’s so much going on in the context window, sometimes the llm (even sonnet 3.5) loses track of part of the task, or a piece of what was asked. An (extreme) way to trigger this is to ask it to make a change to all files in a folder or something. I found that with specific prompting where I ask to keep track of things in a list (that is marked as non summarizable) it can do this, so it is possible .

I’ll be experimenting a bit more and if I get a clear view on it make a feature proposal (which I will then also try to implement , proposing is easy, making it work is the real challenge )

[mikeslattery]

@jluyckx Yes, I agree, and have already said so. I am the originator of the idea, but not the creator of the ticket, and have agreed for quite some time, and wasn't even sure it was a good idea from the start. It's all documented above, if you take the time to read.

To be honest, I think you have a poor understanding of aider and the RAG capabilities overall, to think that will be better.

It would be quite surprising if the creator this ticket, whom is Aider's author, has a poor understanding of Aider.

[jluyckx]

Hahahah would be hilarious indeed 😂. The comment says “originally posted by @mikeslattery” and the content is indeed copied mostly from what you wrote, so I consider ticket admin to move it out of an unrelated ticket not the same as standing behind the opinion, but yes still funny.

In any case it seemed an old ticket, my reaction was mostly originating from my hope that we could avoid having aider move in that direction as I truly believe it’s the wrong one.

But seems we’re on the same page 👍.

Going back to experimenting, so many cool things to try, aider is one of the coolest pieces of software I’ve seen and it’s fun to play with and try out various new approaches of how to improve it.