Allow CustomTool to work with files

[Tomas2D]

Is your feature request related to a problem? Please describe. Python Code Interpreter (accessible through PythonTool in the framework) can work with files, whereas Custom Python Function (CustomTool) cannot.

Describe the solution you'd like Add the ability to a CustomTool to work files.

[JanPokorny]

How it could work:

• in the Python definition of a custom tool, we introduce a new possible type File which is just an alias to str
• function can specify an argument to be of type File (or list[File] etc.)
• when converting File to JSON Schema, we place a placeholder item at that location
• once constructing the custom tool, we replace the placeholder with the list of available files as an enum
• for fields that are of type File, we upload the file to the working folder in addition to passing the string to the function

So the custom tool could look like:

import pathlib

def my_function(my_file: File):
   pathlib.Path(my_file).read(...) # my_file would be something like "file.txt", and it would be present in the workdir

@Tomas2D Does this sound reasonable? Should we also try to implement file output, somehow?

[Tomas2D]

Perhaps I missed something, but what about reusing the storage concept we use for PythonTool?

[JanPokorny]

@Tomas2D Yes, the files will work the same, but will additionally need some extra semantics due to how the users and the LLM expect them to work with regards to input and output of the custom tool. Plus there will be actually two types of files -- "dynamic" (passed in by the caller) and "static" (pre-determined by the tool creator -- many users requested this use-case).

[JanPokorny]

Also, I realized that instead of introducing our custom File, we could just piggyback the semantics on top of pathlib.Path.

[geneknit]

@Tomas2D can you enumerate some examples/use cases that this feature unlocks? cc: @jenna-winkler @sarahmilleribm

[Tomas2D]

Having file support in a custom tool unlocks a lot of features. For instance, you can build a tool that converts files from one format to another (PDF to DOCX) and so on.

[JanPokorny]

@geneknit I'd say that there are three main groups of use cases, split by essentially three different features all falling under the "work with files" umbrella:

• Static files (supplied by creator, used internally by tool code)
  • SQLite database to query
  • Machine learning model to use
  • ...
• Input files (supplied by the Bee agent as tool input)
  • File processing / conversion
  • Analysis (summarization etc.)
  • ...
• Output files (files created by the custom tool, returned to the agent)
  • File processing / conversion
  • Obtaining external files
  • ...

[Tomas2D]

What is the current status?

[JanPokorny]

@Tomas2D It's going to be a easier to implement with the recent JSONSchema-related improvements in bee-code-interpreter, but no direct progress was made

[Tomas2D]

The code interpreter package is not actively maintained; therefore, closing this. cc @jenna-winkler