Deploy model

[krrishdholakia]

Hi,

I'm trying to deploy this model via torch serve. But when i try and save the tokenizer:

tokenizer = AutoTokenizer.from_pretrained("valhalla/t5-base-e2e-qg") tokenizer.save_pretrained('/dummy_location')

it does not save a vocab.txt, instead saving:

'/t5_model/spiece.model', '/t5_model/special_tokens_map.json', '/t5_model/added_tokens.json' '/tokenizer_config.json'

How can i get the vocab.txt to use for deployment / please suggest an alternative method for deployment

[krrishdholakia]

hi @patil-suraj how can i deploy this model? i'd like to deploy to an endpoint and have a live site connected to it.

[patil-suraj]

hi @krrishdholakia ,

You can just use init the pipline on the server and wrap it behing an API endpoint. Let's say it's a simple flask app then a simple way could be

from flask import Flask, request, jsonify
from pipelines import pipeline

app = Flask(__name__)

nlp = pipeline("question-generation")

@app.route('/quegen', methods=["POST"])
def quegen():
    context = request.form['context']
    output = nlp(context)
    return jsonify(output)

Is there any specific reason for not using the pipeline object ?

[krrishdholakia]

hey @patil-suraj ,

was looking for a way which would allow me to do some more custom-stuff in the future (eg. multiple-choice question-answer generation on long documents etc.)

though i'm not too familiar with pipelines, would it still be the ideal way forward if i want to do things in a more custom fashion + if i want to move to using lambdas / cloud functions ?

[patil-suraj]

In that case have a look at this issue #21, let me know if you still have doubts.

[patil-suraj]

Also for T5 there is no vocab.txt since it uses sentencepiece, and when you call AutoTokenizer.from_pretrained("valhalla/t5-base-e2e-qg") the tokenizer is automatically cached, you might not need to save it separately.

[patil-suraj]

also note that the pipeline code should easy to change. Have look at the QGPipeline class, you can just take it and modify.