Support for Images

[JackHopkins]

Currently, agents only observe token streams from the output of their programs.

We could create a tool render(*entities) that converts the entity objects into an image that can be passed to a visual model.

Requirements:

1. Implement an entities -> image renderer.
2. Implement a render tool that invokes the renderer, with the manual etc.
3. Add a post-tool hook to the render tool in the agent definition that passes the image into the agent
4. Send the image and the program response to the LLM

[A-Vaillant]

A thought here: If you implement integration with https://github.com/redruin1/factorio-draftsman, you can convert the entity objects into a blueprint string and then take advantage of already existing renderers for blueprints. Wouldn't capture the environment nearby though.

[vessenes]

I was reading the draftsman documentation and it mentions that (of course) it uses blueprint strings. As does everything in the wider ecosystem. I propose a better first step is entity object -> blueprint strings. That opens up a lot of possibilities.

[JackHopkins]

We decided against using blueprint strings, because AFAIK they don't contain any stateful information (such as inventories) or set recipes, and therefore don't capture enough information about 'live' factories.

However, I think blueprints would be ideal for specifying lab scenarios (e.g 'fix this broken factory') - which we should definitely aim to support.