gpt-engineer
gpt-engineer copied to clipboard
gpt-engineer-org
Can GPT engineer be used to modify or extend existing codebase instead?
Today I have come across this open source code completion tool - https://tzap.io/ which uses GPT API (3.5, 4)
They took a different approach of using embeddings and vector databases to scan existing codebase and send it along (or instead) with main prompt. Advantage of this approach as opposite to text-only prompts is that bigger amount of contextual information can be sent to OpenAI model. I was wondering if similar functionality can be implemented in gpt-engineer as well to extend its functionality from creating new projects to analyzing and modifying existing source code repositories.
@morganavr https://github.com/AntonOsika/gpt-engineer/issues/79 https://github.com/AntonOsika/gpt-engineer/issues/119 https://github.com/AntonOsika/gpt-engineer/issues/95
There is an extension for VSCode : Code GPT which has this ability in part... you have to highlight a section of code for it to edit and/or refactor.
There is a way to get gpt-engineer to work on already established code base, but it takes some manual fiddling to do. The way they were re-running this issue's code: https://github.com/AntonOsika/gpt-engineer/issues/101#issuecomment-1595792433 would be similar to how you would run gpt-engineer over an existing code base. But there isn't an automatic process to do this yet. But it is doable.
@patillacode @noxiouscardiumdimidium
There is an extension for VSCode : Code GPT which has this ability in part... you have to highlight a section of code for it to edit and/or refactor.
The thing is - very often I need to write new functionality that is dependent on multiple other classes, structures, function calls and I can not simply highlight sections of the code, instead smart AI tool ideally should understand my requirements for new functionality and then analyze existing code base to understand how it can be implemented and integrated into existing code base.
@jebarpg
There's no AI in existence with enough context to understand any substantial codebase. If you set it free on an entire codebase it'd just make a horrific mess of it as huge swaths of it shimmered in and out of its short term memory.
There's no AI in existence with enough context to understand any substantial codebase. If you set it free on an entire codebase it'd just make a horrific mess of it as huge swaths of it shimmered in and out of its short term memory.
I could see a possible way of it going through the existing code and creating basically bullet points to understand the codebase, and being more effective in creation things you would be adding though.
There's no AI in existence with enough context to understand any substantial codebase. If you set it free on an entire codebase it'd just make a horrific mess of it as huge swaths of it shimmered in and out of its short term memory.
True only very small code bases might fit into the context window. I think 32k is the largest window. Which is only 24000 words, or 48 pages of text. so if you have a code base which is less than this then it can work currently, but like what @different55 was hinting at, that is a fairly small window for typical code bases out there. But obviously there are smaller projects which currently fit into this window. Later on when we get GPT-5 next year we'll see larger windows for more tokens and be able to fit larger code bases into the model's working memory.
My suggestion is that we develop with this in mind and suggest to users in the README.md the limitations of how much context can be read in by each model chosen before running gpt-engineer. This will give users of gpt-engineer the correct information going forward to help make their decision on where gpt-engineer fits one of their requirements, that requirement being a question of 'is this context window large enough for my project I have in mind' i.e. reviewing my code base. Then the user could use a quick tool to glob their entire project's code base text and pump it through a tokenizer and check against whether it fits into the selected model's token window limits.
Actually their could be a way to reduce how much of your code base needs to fit into the context window... you could search through your code base and find only dependent modules to scan (since some code bases have sections of modules which are not dependent on the entire code bases module base) and then only provide the needed models to be scanned for the particular section of code you are asking to edit or enhance. This would actually probably be ideal in all cases when possible. This would reduce cost and increase computational time, of reading extra text for a request to change a small section of the code base. Essentially a package/module manager pruner of sorts. I could see in the future as well that AI could also optimize your module dependencies to optimize for making chunks of your code base more modular and less dependent on large chunks of unused code for particular sections, so that when you are in a file and make a request to improve something the context window needed will be optimally small and exacting for what needs to be taken into consideration for gpt-engineer/gpt-# to evaluate requested improvement/feature/optimization/change(s)/etc...
@AntonOsika @patillacode
What about building a LoRA for a local coding LLM like WizardCoder?
Could OpenAI build a QA dataset for training, then GPT Engineer have a GPT4 (smart agent) the trained LLM that for actual coding?
I have two considerations on this topic.
1 - Do we need to send all codes simultaneously to the GPT prompt? Can't we send each smaller code or function separately in a way that our agent learns as more code we send to it? I imagine that developing with GPT would require a strategy of segregating the files, functions, and classes more than we usually do. However, it is important for the AI to understand the interrelation between each function and be able to trace all the necessary information to correctly develop the new function.
2 - The AI agent could read only the documentation (ie. function's IO and description) of the adjacent files (not the one we want to modify) like we (humans) usually do when using a python library such as pandas or numpy. We don't try to understand all code every time. The generated code should be well documented so the AI can read it in the future. The downside of this strategy is the necessity of documentation of code in a separate file, which can limit who would use it.
We could do a proof of concept using smaller codes to see the difficulty involved in implementing this functionality. I see it working like this:
Suppose you have a project of three small files, all of them could be used with GPT prompt. In one of them, you want to create a new functionality, the other two are adjacent files but the AI needs to understand to make the implementation. We could follow these steps:
- Start the agent
- Send as input adjacent code 1
- Send as input adjacent code 2
- Send as input the main file (the one you wish to add the functionality)
- Ask to do the modification you need to do.
- Recieve the modified file and replace it on the computer.
In all steps above a complementary message should be sent with the code so the agent can understand what to do with it.
As for the user, we only need to specify which and where the code files are located on the computer. The gpt-engineer should be able to take the code string and automatically do steps 1, 2, 3, 4, and 5. The user needs to specify what and where the modification should be.
Yes, probably will have a lot of flaws, but it is a start. What do you guys think?
Looks like we will be able to do whole code bases soon enough with LongNet: https://github.com/kyegomez/LongNet We'll have a billion token context length to put our code bases into it.
Yup available now on main (not yet released so pip install gpt-engineer is not enough)
@AntonOsika Is there a guide on using Longnet with gpt-engineer then? Any guidance is appreciated, thanks.
