Jakub Matejczyk

So my main questions: - Should there be more categories? - What categories should have more tasks than others? IMO: basic - so getting something from a right topic are...

As disscussed in person with @maciejmajek i will implement 3 ways of grading the Task: - First the difficulty of the task itseld ( already present ) , currently it...

- [x] define complexities for tasks - [x] define N_shot for tasks - [x] define prompt_detail for tasks - [x] adjust the benchmark configs and examples - [x] adjust saving...

define new sets of tasks for: - [x] basic - may require defining more mock topics, also include calling services here, as now its only topics - [x] custom interfaces...

new type of task -> analysis? it would group tasks that require deducing something or answering question based on gathered data from topics, so something like IsSystemHealthyTask, that would require...

add new type of validator -> `optional`, which will pass when any of given subtasks passed. Usefull when there is couple way of doing the same thing

added much more tasks in series of PRs. https://github.com/RobotecAI/rai/pull/620 https://github.com/RobotecAI/rai/pull/656, github.com/RobotecAI/rai/pull/644, https://github.com/RobotecAI/rai/pull/638, https://github.com/RobotecAI/rai/pull/637 https://github.com/RobotecAI/rai/pull/636 spatial was them removed as VLM benchmark was introduced, as well as naivgation which didn't suit...

i've tested it further - it seems like openai models (4o, 4o-mini) do not recognize arguments of type dict. Other types like int, str, list do not cause this problem....

It seems like some ros2 packages have just issue of not closing even after sending shutdown signal, for example moveit stays alive even after sending SIGINT signal or shutdown ros...

tried to apply approach from https://github.com/RobotecAI/rai/pull/445 to monitor processes, shutting down everything when any process is down, then starting all of them once again - That would be great for...