breadboard icon indicating copy to clipboard operation
breadboard copied to clipboard

Implement a simple ELT provider

Open dglazkov opened this issue 1 year ago • 6 comments

As mentioned in #1037, Breadboard wants to have an extract/load/transform (ELT) provider that allows managing things other than text: PDFs, docs, images, videos, audio.

Here's a couple of scenarios:

  • I want to upload a few PDF files as part of the "persona" prompt in the node. I expect that Breadboard will keep it there.
  • I uploaded an audio file when running the board. I want to run again. Can I just reuse the file? Do I have to upload it again?

Let's figure out how to build the simplest-possible ELT provider that relies on IDB. Here's what we need:

  • The ability to upload, store, and choose various files as part of the llm-content editor
  • A way to store multimodal content in configuration
  • Being able to pass this data efficiently as part of the llm-content data that's passed across the wire
  • (later?) The ability to manage these files in one place, so that I could zap some files

dglazkov avatar Apr 20 '24 17:04 dglazkov

Let me ponder the multimodal content in config first

dglazkov avatar Apr 20 '24 18:04 dglazkov

There's something here about saving an llm-content as one item. For instance, imagine I have a board that represents a single turn of a chat. The board takes llm-content as input and llm-content as output (aka "worker shape"). It would be exceedingly cool if I could save the output and then, when running the board again, pick the saved llm-content from some store of saved things. This would allow me to debug worker-shaped boards very easily and even simulate multiple turns

dglazkov avatar Apr 20 '24 19:04 dglazkov

omg, I just realized something. With FSAPI, we can save a .bb directory on disk and store settings and ELT bits there.

dglazkov avatar Apr 20 '24 21:04 dglazkov

For now:

  • let's just stuff base64 bits directly into configuration values in BGL, and see where it goes. No change needed here.
  • add a simplest possible "Run Again" capability: remember the inputs entered so that they could be used again in the next run (instead of entering them again from scratch)

dglazkov avatar Apr 21 '24 17:04 dglazkov

wait... I can just reuse saved runs

dglazkov avatar Apr 21 '24 17:04 dglazkov

Okay, with InspectableRun.inputs(), I think we have what we need. This is still an important issue, but we can decouple it from the Easy Mode MVP.

dglazkov avatar Apr 21 '24 21:04 dglazkov