allennlp
allennlp copied to clipboard
allenai
Predictor.from_path using in memory file instead of disk
We use openshift, and our enterprise sets limits on ephemeral storage. We need to be able to load a 1.4GB Model and all I see available is Predictor.from_path, which requires the model to be on disk before we can use it. They wont let us use something like S3FS at the moment, so we cant "mount" a network storage location. Is there a way we can take the model that we have loaded into memory as a binary file, and load it to the Predictor directly? We apparently have plenty of RAM to work with.
We asked the same to Spacy, and they had a response for us. https://github.com/explosion/spaCy/issues/3841
Hi @smiles3983, so if I'm understanding this correctly, you'd like to have something like a Predictor.from_bytes() method?
Hi @smiles3983, so if I'm understanding this correctly, you'd like to have something like a
Predictor.from_bytes()method?
Yes that would be great! Does it exist and I’m just not seeing it?
It does not exist at the moment, but it's feasible to implement. If you (or someone else) is interested in making a PR to implement this, we'd love to see it. Otherwise I could take a look next week.
Yeah, not sure our team will be able to do it, unless they want to do it from home on their own time :). Dont think we can clone outside our Intranet. We use Nexus to bring down packages. Were open source leeches haha.
It looks like torch has the ability to load with io.bytesIO. We will be pulling our model from an S3 bucket using s3.get_object(). So if we can convert that to what torch needs without touching the file system.
# Load tensor from io.BytesIO object
>>> with open('tensor.pt', 'rb') as f:
... buffer = io.BytesIO(f.read())
>>> torch.load(buffer)
Hi @epwalsh any update on this request... Even I would like to have this configuration.
Hi @krishna-anumula, I don't have the bandwidth to tackle this at the moment, but I'm happy to review a PR if you'd like to work on it.