Jacob Danovitch

@jacobtomlinson So sorry, just getting to this now. You can use MPI in AzureML using a commandJob and specifying the `distribution.type` as MPI (docs [here](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-train-cli#distributed-training)). You need the `az` cli...

Fwiw, using `QAPipeline(reader='models/bert_qa_vGPU-sklearn.joblib', predict_batch_size=128, verbose_logging=True).to('cuda')` gives me very reasonable inference time over a pretty large set of documents. Obviously this might not be feasible for everyone, I'm just running this...

> Well we must consider production where we have 1000 users doing inference concurrently. > Also not everyone has GPU. For sure on the first point, it obviously doesn't scale....

Changing to binary mode worked for me: ```python with open(vocab_path, 'rb') as json_file: self.vocab = json.load(json_file) ```

I think this may be an issue on Swift storage as well. I get an output like this: ``` time="2022-05-27T15:12:18Z" level=error msg="bad S3 PutObject response" func="pkg/block/s3.(*Adapter).streamToS3" file="build/pkg/block/s3/adapter.go:250" error="s3 error: NotImplementedtx000000000000045e10e0e-006290ea51-385b2f-default385b2f-default-default"...

> Thanks for reporting this @jacobdanovitch! According to the compatibility matrix it _should_ be supported.. Can you share which version of Swift/Open Stack you're running? If it's a recent version,...

Okay, seems like putting a MinIO gateway in front of the Swift S3 endpoint solves this (temporarily). Minimal docker-compose based on [lakeFS'](https://github.com/treeverse/lakeFS/blob/master/deployments/compose/docker-compose.yml): ```yaml version: '3.7' services: postgres: image: postgres:11 container_name:...

Took a while to figure out but it's straightforward (even if not at all obvious): ```bash # fill these in KEYVAULT= AKS= AKS_RG= az keyvault set-policy \ -n $KEYVAULT \...

Still having this issue as well. Any chance anyone's found a solution?

> Thank you @geonux! Can you please confirm this issue could be closed? Or are there any changes still required on the lakeFS side? I haven't tried in a little...