IDSIA
Utility to move experiments between observer backends
First, great piece of software! I'm using it quite a bit to log the output of Keras runs.
I didn't see the on the issue list already, so I'd like to propose a potential feature and get feedback on it. I think it would be useful would have a simple utility to move experiments between observer backends. That way I can run experiments locally and log to a file oberserver, and then convert them to my mongo db instance for archiving later. Or when my mongodb instance is remote, I can export the series of experiments I'm interested in to a local fileobserver for processing. Or if my mongodb instance is slow, moving to an sql server / tinydb / filesystem observer or file for speed reasons.
The biggest issue I can see is de-conflicting the integer experiment ids between the backends. But beyond that, without looking at the code, this doesn't seem like it would be that hard..? I might take a crack at it myself if I get some free time. I've already written a utility to extract artifacts from mongo for a given run, that's at least part of the battle.
Thoughts? and/or pitfalls?
That sounds like a useful utility to have, and I agree: it shouldn't be hard. If you end up writing something that is robust, I'd be happy to merge it. Ideally, this would become part of a sacred commandline-tool and use a general interface to query different databases (both see #232). But since both of them might take a while to become part of sacred, I'd suggest putting it in a separate tools.py file.
Just dropping in to say obviously I've dropped the ball on this one. I've been trying for months to make time to do this but its just not happening. I won't be able to implement this any time soon as there's just too many other things going on right now. If someone else would like to step up, we all agree it it would be a useful feature.