Jiyuan Qian
Jiyuan Qian
I'm encountering the same issue. Wondering if there's any update? Judging from the errors it looks like it's trying to load js from `/hub`, where the jupyterhub login service is,...
I guess it’s about absolute path vs relative path. VS code is using absolute paths which points to /hub as root path
@alberttorosyan I see. I was expecting the same behavior for metrics given logs was being live updated. It would be nice to have live updates enabled for metrics and system...
> > @alberttorosyan I see. I was expecting the same behavior for metrics given logs was being live updated. It would be nice to have live updates enabled for metrics...
> hey @jiyuanq! Thanks for the request. > Makes a total sense, and we had it on our plans as well, just didn't have the time yet to implement. >...
> Hey @jiyuanq. That is a great point! We have been thinking about adding a project-level settings page, where users will be able to whitelist/blacklist those configs, which aren't necessarily...
@alberttorosyan sure, I was running single GPU training with pytorch 1.10.0, pytorch lightning 1.5.10, aim 3.13.0, on python 3.8.10, and I was using the provided AimLogger. The training job was...
Also I'm wondering if using remote tracking server will make it easier to scale writes?
@alberttorosyan unfortunately this seems to happen to me pretty often. I had two more runs hitting the same error after around 40k mini batches of training. I checked the logs...
To share more information: 1. The two jobs started encountering the issue at around the same time 2. While training logs stopped updating at minibatch 24500-ish, it looks like aim...