sacred
sacred copied to clipboard
IDSIA
Generalize FileStorageObserver
To be able to write other file-like observers easily (such as S3 for example), I would like FileStorageObserver to be able to use arbitrary open/mkdir functions, or something similar. To avoid code duplication.
I have no experience with S3 so I can't really tell how much customization would be necessary. But you seem rather certain that it is just a matter of customizing a few functions. If you prepare a PR that makes these functions a parameter of the FileStorageObserver I'll happily merge it after my vacation.
This is a bit resurrected from the dead, but I actually wrote a (currently fairly simple, but working for my test cases) version of a S3 Observer in a fork of the repo, because it was something that came up as quite useful in a meeting my team had about improvements to Sacred functionality. (It made more sense to me to just implement it as a separate Observer class, rather than a parameterized version of FileStorageObserver, because the functionality around uploading and bucket logic is different enough, but I'm open to having my mind changed there)
More to the point, @Qwlouse, is separate S3 observer be something that the maintainers would be interested in having in the main library? If so, I'd be potentially interested in talking about what minimum functionality/level of robustness you'd want in such a thing, and seeing if what I have could be a start in the direction of something that could be merged in.
@decodyng I would be definitely delighted by an S3 observer. I believe it is something that many people could make good use of!
@JarnoRFB Cool! I'm currently trying to figure out the best way to design tests for the S3 syncing functionality (e.g. to what extent is it feasible or useful to mock up a S3 connection rather than having to create a real one through credentials, but in a way that actually tests things in a real way), but after I have that resolved/passing tox, I can create a PR with what I have and we can go from there, if that makes sense for you guys.
(Also, if you have experience with S3 mocking and have recommendations there, I'd love to hear those!)
@decodyng Great! I don't have much experience with mocking S3 or remote APIs in general, but I am often struggling with the same question. However, after googling a bit I found this https://github.com/spulec/moto which looks interesting. I would be fine with pulling this in a test dependency.
@JarnoRFB Yeah, I also found that in the course of some googling, and have found it to be a quite good fit for what I needed! Just opened a PR.