Task.get_tasks does not return all the experiments

[davidenitti]

I'm using the following code tasks = Task.get_tasks(tags=tags) to get the tasks with given tags, but now that I have a lot of experiments, it does not return all the experiments I can see on the clearml web interface. For example, if I have experiments imagenet and experiments coco, tasks = Task.get_tasks(tags=tags) does not show the coco experiments but if I do tasks = Task.get_tasks(tags=tags, task_name="coco") they are there. it seems that there is a limit on how many tasks get_tasks returns. indeed the length of the returned tasks is exactly 500 for tasks = Task.get_tasks(tags=tags). how can I solve the issue?

[erezalg]

Hi @davidenitti,

It's actually a funny limitation of the api calls to the backend. We cap the response to 500 tasks per page, but we don't check if there are more pages :) We should fix the clearml to aggregate the tasks across multiple pages and then serve it to you, it's an easy fix which I think we can get out quickly.

[davidenitti]

thanks

[mtruongle]

I think the projects = Task.get_projects() might be capped to 500 as well by the same limitation.

I only get 500 entries even though there are way more on our web interface. Should I open a new issue or can this be tackled in the same?

[jkhenning]

Hi @mtruongle,

Let's keep it under the same issue 🙂

[erezalg]

Hi @davidenitti , we've released clearml 1.4.0 with the fix. Please use Task.query_tasks() to get a list of all the Task IDs, then you can get the specific tasks with Task.get_tasks(). (We decided that Task.get_tasks() should only retrieve 500 as if you'd retrieve all of them it might take a REALLY long time :) )

Let us know if it work, I'll update here once we add support for Task.get_projects()

[erezalg]

Hi @mtruongle We've just released clearml V1.5.0 (pip install -U clearml==1.5.0) with a fix to Task.get_projects() which will now return all projects meeting a specific query.

Let me know if there's still an issue!

[jkhenning]

Hi @davidenitti, @mtruongle, closing this. Please re-open if required.