aim
aim copied to clipboard
aimhubio
Aim Performance Problems
There exist some performance problems when the total number of runs exceed the certain number such as 1000. As number of runs increase, the program is getting slower and slower. The metric graphics can't be reached and GUI shows the same error ''Something went wrong'' all the time. I haven't faced such an issue while using ''tensorboard''. What is the reason behind this situation, do you have any suggestion in order to have a stable performance for high number of runs❓
Hey @ugccelik! The number of tracked Runs is one of the factors affecting Aim performance. However, the 1000 runs should be fine. In fact, some Aim users reached 20.000 Runs without any performance issues.
In order to be able to debug the specific issue you faced, we'll need some more details.
- What version of Aim you are using?
- Are there any error messages/stack trace available in the terminal?
- What is the query you are running in Metrics page?
- On average, how many Run parameters are set?
- Any other information you think might be helpful 🙏
We are constantly working to improve Aim performance, both on UI and storage/queries. The upcoming 3.11 release contains some major improvements:
- Add caching mechanism for sequence queries to optimize query performance
- Use step random hash as a key for metric sequences
- Optimize the initial render time of Aim UI by using more lightweight font-family
- Improve the processing speed for tb to aim converter
@ugccelik thanks for reporting this issue. Performance is of utmost priority for us.
Additionally, any chance you could share your Statistics and Activity section on the home page?
Hello, thanks for fast reply. I am using version 3.10.3. In terminal, there is ''socket.send() raised exception.'' message.
@ugccelik Aim 3.11 has been shipped and above mentioned improvements are available. May I ask you to try the new package and see how it performs?