Should Microsoft Azure be Supported?

[tathey1]

Is Azure an option for storing precomputed data with cloud-volume? I see options for Google and Amazon but cannot find anything in the documentation about Azure.

[william-silversmith]

Hi Thomas,

We haven't had a need for trying azure yet so we never built the functionality. It shouldn't be too difficult to build it if the need is compelling. Do you have a lot of data on Azure?

Will

On Wed, Aug 12, 2020, 4:31 PM Thomas Athey [email protected] wrote:

Is Azure an option for storing precomputed data with cloud-volume? I see options for Google and Amazon but cannot find anything in the documentation about Azure.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/seung-lab/cloud-volume/issues/382, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATGQSKESYL3OXBUIFOBDE3SAL33HANCNFSM4P5IAFWQ .

[tathey1]

In my project, we are in the process of deciding whether to house a couple complete mouse brain images (~14TB/brain/channel) either on Azure, or on our OpenNeurodata AWS account. So not at the moment yet, on my end. But maybe my advisor @jovo has input.

[william-silversmith]

My impression was that all the major cloud services were essentially at parity cost-wise. It was a little tricky to figure it out, but I think this link says that Azure is about 8.6% cheaper than GCS on their hot tier as a recurring cost. You also need to consider egress and transaction fees too though, which can be substantial.

https://azure.microsoft.com/en-us/pricing/details/storage/blobs/

On Fri, Aug 14, 2020 at 2:15 PM Thomas Athey [email protected] wrote:

In my project, we are in the process of deciding whether to house a couple complete mouse brain images (~14TB/brain/channel) either on Azure, or on our OpenNeurodata AWS account. So not at the moment yet, on my end. But maybe my advisor @jovo https://github.com/jovo has input.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/seung-lab/cloud-volume/issues/382#issuecomment-674200842, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATGQSN4XUZ2PKOBBVL766DSAV5NNANCNFSM4P5IAFWQ .

[jovo]

thanks @william-silversmith. For all the open access data, we can store it for free here: https://registry.opendata.aws/open-neurodata/. But we also have some resources at Azure for private data that we work with, so are incentivized to store it there as well. I was hoping the cloud service providers would have figured out by now how to be agnostic to who owns the hard drives at this point.

[william-silversmith]

The nice thing about this library (and Dask's fsspec) is that sans-egress-fee lock-in, we're creating that basic interoperability in a way that doesn't depend on those companies (although "enterprises" probably use a lot of vendor-specific features like lifecycle rules, legal holds, versioning, special metadata, and other features we don't support).

It's not a huge trick to implement an azure connector. We tend to see more problems with parts of the software that seung-lab doesn't use frequently, however unit tests could possibly help with that in the case (unlike with Windows support, where I can't even test it without finding a paid copy of the OS). The major implementation details would reside in CloudFiles to support a new file interface, bucket pool, support for an az:// protocol or some such, and support for azure credentials:

https://github.com/seung-lab/cloud-files/blob/master/cloudfiles/connectionpools.py https://github.com/seung-lab/cloud-files/blob/master/cloudfiles/interfaces.py https://github.com/seung-lab/cloud-files/blob/master/cloudfiles/paths.py https://github.com/seung-lab/cloud-files/blob/master/cloudfiles/secrets.py https://github.com/seung-lab/cloud-files/blob/master/automated_test.py

...and in CloudVolume:

https://github.com/seung-lab/cloud-volume/blob/master/cloudvolume/paths.py https://github.com/seung-lab/cloud-volume/blob/master/cloudvolume/secrets.py

I'm (probably) going to be on vacation for a bit, but I'd be happy to review and release an update if someone can contribute the code.

[william-silversmith]

(Correction: I probably could use AppVeyor to get more comprehensive Windows testing, but that's for another day.)

[jovo]

that sounds terrible. who uses windows anymore anyway?