nlp-recipes icon indicating copy to clipboard operation
nlp-recipes copied to clipboard

Published 20 hours ago • verified publisher icon microsoft

[ASK] Merge preprocess_encoder_tokens and preprocess_classification_tokens

Open daden-ms opened this issue 5 years ago • 1 comments

Description

preprocess_encoder_tokens and preprocess_classification_tokens contain large chunk of same code and we should rewrite this part of not repeat the same code.

Other Comments

daden-ms avatar Aug 16 '19 16:08 daden-ms

good catch.

  • merge both functions into preprocess_sequence_tokens
  • function returns (tokens, input_ids, input_mask, token_type_ids)
  • update notebook references

saidbleik avatar Aug 16 '19 18:08 saidbleik