Add RoBERTa model for question answering using SQuADv2.0 dataset.

[QingtaoLi1]

trafficstars

[dnfadmin]

All CLA requirements met.

[GeorgeS2019]

@QingtaoLi1 thanks for submitting a PR related to TorchText or Pytorch Deep NLP/Pytorch for HuggingFace.

I wonder if there is need now to start thinking how to organize Torchsharp so NLP tutorials and use cases are more aligned with TorchText.

Just an unrelated question, have you looked into Huggingface GPT2?

[QingtaoLi1]

Just an unrelated question, have you looked into Huggingface GPT2?

I have not used GPT2 yet. The main difference between the GPTs and BERTs is that the former is a generative model which can output a sequence in an auto-regression format. So I believe many of code are very similar except for generation-related parts.

[GeorgeS2019]

@QingtaoLi1

This is an attempt to get GPT2 working using BlingFire GPT2 tokeniser and GPT2 ONNX with the preliminary generation-related parts which are the issue

If you have time and interest, hopefully, get your perspective of the challenges of getting the generation parts working.

[QingtaoLi1]

@GeorgeS2019 Sorry but I wonder what you would like to achieve with GPT2 or BlingFire/ONNX?

[GeorgeS2019]

RoBERT is for question and answer. GPT2 is for Text Generation or NLG, which is rare in .NET

I hope this answer your question.

Microsoft BlingFire...there should be a tokeniser for RoBERT...Do check out..better performance?

[GeorgeS2019]

@QingtaoLi1 updated the reply

[QingtaoLi1]

I see. You want to include GPT2 in this repo, right? I currently have other work to do; may seek for some time later.

And I know some are attempting to create infrastructure for general tokenizers. I guess your thoughts of re-organizing TorchSharpExamples to align with torchtext will have similar effect on NLP systems -- this will make it easier to build different NLP models for different tasks/datasets.

[GeorgeS2019]

The spirit of Torchshap is to empower. NET developers to do deep AI within .NET without the need to go back to python.

[QingtaoLi1]

I'm not turning back to python; I mean infrastructure in .NET world.

[QingtaoLi1]

An unrelated question: what do you think are the main obstacles for developers to build a deep AI system?

[GeorgeS2019]

@QingtaoLi1

https://devblogs.microsoft.com/dotnet/ml-net-june-updates-model-builder/#ml-net-survey-results

[GeorgeS2019]

@QingtaoLi1 https://github.com/dotnet/machinelearning/issues/5918

[QingtaoLi1]

Thanks very much!