sdk
sdk copied to clipboard
meltano
feat: generate taps and targets programatically
Feature scope
Taps (catalog, state, tests, etc.)
Description
the ability to call meltano from python to make a skeleton tap or target would be very useful for generating taps and targets programmatically using an llm in an automated fashion.
Hi @peterclemenko!
This is something I would definitely like to explore!
Do you have an example of a project that does something like this? Maybe a web app generator or similar?
Or perhaps you have an idea of what's needed, or what it would look like in practice?
Not at the moment, I'm cooking something up it would be useful for in private, plan is to open it when ready. Currently in the process of making it work. The theory though is to take an output from an api or a command line app, run it through an llm, create a tap, test it, and if it works, use it instead of the llm for input processing. essentially this cuts the processing time per command execution on the llm, and frees compute for other workloads.
@peterclemenko Cool. Let me know if you have any questions or suggestions for improving generation from this library, I'm more than happy to help.
At the moment the problem is execution of the command from a Python script tied to llm actions
Essentially the current plan is to take nats or Kafka and use it as a message queue to take command output, run scripts, and execute other commands, in order to make data pipelines painless, being able to convert an output from a command or api to a meltano tap using an llm would be effective
After that unit test the tap and use it on further scripts and upload to git
Essentially creating a pipeline to reduce the llm load where possible
The problem is I don't see a way to programmatically do this from Python
Just a thought, what if this was called via an mcp server? Maybe implement the scaffold being able to be called via MCP, run as a service and MCP can be used to trigger new projects being scaffolded.