swift-models icon indicating copy to clipboard operation
swift-models copied to clipboard

Improve BERT model

Open BradLarson opened this issue 4 years ago • 0 comments

After importing the initial BERT model into the repository, there are a few remaining to-do items to expand upon this work (extracted from Dan's comment):

  • [ ] Rewrite utilities for downloading/extracting data in this PR using unified ModelSupport APIs. Consider generalizing DatasetUtilities.downloadResource to support URLs that don't end in the zipped file name. The CoLA dataset URL is one such example. Alternatively, we could host the dataset elsewhere.

  • [ ] Improve code organization. Currently, most BERT code lives at the top-level in Models/Text. Models/Text could be better organized, like Models/ImageClassification.

  • [ ] Verify that BERT training converges, and that results match a reference implementation.

  • [ ] Verify that other BERT variants work. BERT variants like RoBERTa and albert were added in this PR but are untested.

BradLarson avatar Feb 14 '20 16:02 BradLarson