tokenizers icon indicating copy to clipboard operation
tokenizers copied to clipboard

Published 20 hours ago • verified publisher icon huggingface

Added ability to inspect a 'Sequence' decoder and the `AddedVocabulary`.

Open eaplatanios opened this issue 1 year ago • 2 comments

This PR is in similar spirit to #1341 and adds a couple more functions that allow one to construct a modified version of an existing Tokenizer. I've followed the existing style and conventions for newly introduced functions.

eaplatanios avatar Jan 22 '24 21:01 eaplatanios

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

HuggingFaceDocBuilderDev avatar Jan 22 '24 21:01 HuggingFaceDocBuilderDev

@ArthurZucker @Narsil gentle ping on this one and on #1444.

eaplatanios avatar Jan 31 '24 18:01 eaplatanios

@ArthurZucker @Narsil gentle ping about this PR. This PR should not be controversial and is in similar spirit to #1341 (and has the same motivation).

eaplatanios avatar Feb 27 '24 16:02 eaplatanios

really sorry about all the delays, lot happening on transformers, I'll free some time

ArthurZucker avatar Mar 22 '24 01:03 ArthurZucker

Also can you add tests for set and get? 🤗

ArthurZucker avatar Mar 26 '24 13:03 ArthurZucker

Sorry this fell through the cracks a bit over the past couple of weeks and I just saw the last couple of comments. Thanks for approving and merging this!

eaplatanios avatar Apr 11 '24 14:04 eaplatanios

Thanks for your contribution 🤗

ArthurZucker avatar Jun 11 '24 12:06 ArthurZucker