pgcontents
pgcontents copied to clipboard
quantopian
Pull HybridContentsManager out to own package
I have a use case for HybridContentsManager without pgcontents itself. Would you be willing to pull the relevant code/tests out into a separate package to allow installing without the postgres dependencies?
hey @jcrist!
My first instinct is that I'm not super excited about splitting this up into multiple repositories. This repo is generally pretty stable, and as you can see from https://github.com/quantopian/pgcontents/issues/28 I haven't had a ton of bandwidth lately for issue triage/packaging/release maintenance, so creating twice the amount of work for releases doesn't sound great.
That said, if you were willing to help out with some of that maintenance (at least for whatever new home HybridContentsManager found), I'd be willing to consider splitting it out, especially if you were willing to help out with getting CI/release tooling back in shape for the split repo. Without having put too much thought into it, my preference would probably be to maintain two packages in this repo: one named something like jupyter_contents_tools (containing hybrid contents and some of the shared utility code), and the other being pgcontents, which would depend on jupyter_contents_tools and add postgres-specific stuff. For testing purposes, I'd probably want to leave most or all of the tests in pgcontents to be able to verify that HybridContentsManager works with a nontrivial alternative contents manager.
That said, if you were willing to help out with some of that maintenance (at least for whatever new home HybridContentsManager found), I'd be willing to consider splitting it out, especially if you were willing to help out with getting CI/release tooling back in shape for the split repo.
I am willing to put in this effort, no problem.
Without having put too much thought into it, my preference would probably be to maintain two packages in this repo...
I'm fine with this approach. My one concern would be that having multiple packages per repo can make it a bit confusing where to file bugs (harder to google search), but agree that for development this will likely make things easier.
If the above plan makes sense with you, I'll try to get a PR worked up sometime next week. Thanks!
My one concern would be that having multiple packages per repo can make it a bit confusing where to file bugs (harder to google search),
I buy this concern, though I think in this specific case I'm not too worried about that. I see HybridContentsManager as a pretty advanced tool that's mostly going to be used by people doing something fairly unusual with their Jupyter deployment, so I'm less concerned about discoverability than I would be for a tool aiming for a wider appeal. We can also always decide to separate further in the future if it's warranted.
If the above plan makes sense with you, I'll try to get a PR worked up sometime next week. Thanks!
Sounds good! Let me know if I can be of help.
I second this request. The HybridContentsManager is useful when used in conjunction with s3contents. However, pgcontents won't install without postgres. I did pull out the necessary files for my deployment, but obviously if you make updates I won't be able to easily pull them in.
If you are trying to bring in broader appeal. Once HybridContentsManager is pulled out I'd suggest checking out https://github.com/danielfrg/s3contents to give wider applicability. Turns out in my use case, I have notebook on an ephemeral docker image on AWS, I "mount" a persistent "user" S3 location and a persistent "shared" S3 location to a local filesystem. Using HybridContentsManager I can make S3 buckets various "drives" in a collaborative environment.
HybridContentsManager I believe will be a very useful module as people are trying to use jupyter more and more in a cloud environment
We have the same use-case as @summerswallow above. We realised that, while whole pgcontents is not compatible with IPython 6, simply commenting out the check in ipycompat.py allows us to use the HybridContentManager with IPython 6.
Feels like another reason to have them separated.
I actually having the same issue and also looking for copy-paste across multi-region. Therefore I make a completely new package: https://github.com/lydian/multicontents
feel free to give it a try
@jcrist @summerswallow @cristiklein We had the same use case, so we created and are maintaining a hyrbidcontents fork. Feel free to try out and let me know if you have any feature requests!
