griptape icon indicating copy to clipboard operation
griptape copied to clipboard

Add a Tool to display/play media

Open shhlife opened this issue 1 year ago • 0 comments

Is your feature request related to a problem? Please describe. When working with Image Generation Drivers (and eventually video, audio, and any other media), there's no built in way to display that media.

Describe the solution you'd like I would like a tool that I can add to a Griptape Structure that will display the media. For example, I could have an agent generate an image, and then ask it to display the image and it would open it in an appropriate application on my computer. If it's an audio file, then it could play the audio file for me.

Describe alternatives you've considered Because we're using Python, I have created my own tool to do this for displaying images, but it'd be better if it were a part of Griptape so others could do it, and if it were extended to handle all media types.

Additional context An example of a ViewImage tool

from schema import Schema, Literal
from attr import define
import os, subprocess, sys

from griptape.artifacts import TextArtifact, ErrorArtifact
from griptape.utils.decorators import activity
from griptape.tools import BaseTool

def open_image(image_path):
    if sys.platform == "win32":
        os.startfile(image_path)
    elif sys.platform == "darwin":  # macOS
        subprocess.run(["open", image_path])
    else:  # linux variants
        subprocess.run(["xdg-open", image_path])


@define
class ViewImage(BaseTool):
    @activity(
        config={
            "description": "View the image.",
            "schema": Schema(
                {
                    Literal(
                        "filename", description="The filename of the image to view."
                    ): str,
                }
            ),
        }
    )
    def view_image(self, params: dict) -> TextArtifact | ErrorArtifact:
        try:
            filename = params["values"]["filename"]
            agent_response = ""

            # Make sure the path exists
            agent_response = os.path.exists(filename)
            if agent_response:
                open_image(filename)

            return TextArtifact(agent_response)

        except Exception as e:
            return ErrorArtifact(f"Error retrieving projects: {e}")

shhlife avatar Dec 15 '23 15:12 shhlife