factorio-learning-environment Reduce the token count

Currently we are sending the full manual (API description, entities, manual for every tool, few-shot examples of tool usage) as the system prompt to the agents. This ends up blowing up the token count for the system prompt (20k+ tokens) as the manual needs to cover almost everything from early to late game factorio and contains alot of information that the agent does not directly require for the next step.

We should decrease the manual to make it more compact and give the agent the ability to query for specific pieces that are relevant for whatever task the agent is doing currently (almost resembling how humans would look up the factorio wiki). This would significantly reduce the costs of running evals and might improve performance as there is less un-related information regarding the current action agent aims to take

Mar 12 '25 21:03 MartBakler

just a thought but for tool enabled llms, you could give them lookup functions, which would make things very efficient.

Mar 12 '25 21:03 vessenes

I'm thinking something similar, i.e first try a simple lookup function and not over-complicate with RAG

Mar 12 '25 22:03 MartBakler

Hey @MartBakler , I was wondering if we can check something like the factorio calculator, Get only the relevant pages and then go ahead. So hypothetically for red-circuits,

We only send the relevant required stuff. In this case how to make plastic and green-circuits

Mar 17 '25 23:03 saisurbehera

I like this idea, I was also thinking along these lines, we can recursively get all relevant pages along a resource requirement tree.

I'm trying some local modifications out right now and bigger obstacle seems to be getting the agents to use the "query_information" wiki tool overall. If I remove much of the few-shot examples from context and give a new tool that they can use to get relevant info, the agent seems to rather just try to accomplish the task with the limited knowledge it currently has rather than check the wiki for info and then try to solve the task

Mar 18 '25 20:03 MartBakler

Can you give me an example of "the agent seems to rather just try to accomplish the task with the limited knowledge it currently has rather than check the wiki for info and then try to solve the task"?

Mar 18 '25 23:03 saisurbehera