Add a vocabulary_size argument to WordPieceTokenizer

[mattdangerw]

We should add a vocabulary_size argument to the WordPieceTokenizer layer that forces the vocabulary size by truncating the passed in vocabulary if necessary.

Potential docstring:

        vocabulary_size: Force the vocabulary to be exactly `vocabulary_size`,
            by truncating the input vocabulary if necessary. This is not
            equivalent to retraining a word piece vocabulary from scratch, but
            can be useful for quick hyperparameter tuning.

[mattdangerw]

Some other notes:

• If the vocabulary_size argument is passed, calling layer.vocabulary_size() should always match what was passed.
• If the vocabulary file is shorted that the forced vocabulary size, we can log a warning. Warning: Setting vocab size to a larger value than the input vocabulary file. Some token ids will never be output from the tokenizer.

[blackhat-coder]

hi @mattdangerw, would love to work on this.

[mattdangerw]

Thank you!

[mattdangerw]

@blackhat-coder Any updates on this? This would actually be a useful hyperparmeter to tune in our first guide that could help reduce training time.

[blackhat-coder]

I'm sorry for the delay, been busy with school work. Would send in a PR asap.

[mattdangerw]

Thank you! Let me know if there are any question I can help with.

[blackhat-coder]

Hi @mattdangerw , how do I run tests ? currently written testcases

[mattdangerw]

Check out the environment and test running sections of our contributing guide.

https://github.com/keras-team/keras-nlp/blob/master/CONTRIBUTING.md#setting-up-an-environment

If something is broken or unclear there, let us know!

[aflah02]

Hey @blackhat-coder, are you still working on this?

[lordgavy01]

Hey @mattdangerw, can I contribute to this?

[jayam30]

Hey I would like to take this,

[atharvapurdue]

I would like to contribute to this

[sahusiddharth]

@mattdangerw, I would like to contribute on this

[sahusiddharth]

@jbischof

Context: inside the constructor(init) of WordPieceTokenizer we can assign the vocabulary in one of two ways

1.making changes in the code below, reading from file till the vocabulary size

if isinstance(vocabulary, str):
       self.vocabulary = [
                line.rstrip() for line in tf.io.gfile.GFile(vocabulary)
       ]
       elif isinstance(vocabulary, Iterable):
            # Make a copy.
            self.vocabulary = list(vocabulary)
       else:
            raise ValueError(
                "Vocabulary must be an file path or list of terms. "
                f"Received: vocabulary={vocabulary}"
            )

2. let it do it's thing here and make changes in all the functions and only passing the the slice of vocabulary list. something like below

def get_vocabulary(self) -> List[str]:
        """Get the tokenizer vocabulary as a list of strings tokens."""
        return self.vocabulary[:self.vocabulary_size]

something similar done in all the functions

Question: Can you give me some suggestions on how should I move forward?

[jbischof]

@sahusiddharth you probably want approach #1, otherwise there's no way to use this argument to load a vocabulary file too large to fit in memory.

[sahusiddharth]

@jbischof

Context: Task is to add an argument named vocabulary _size but there is a method with the same name in WordPieceTokenizer

def vocabulary_size(self) -> int:
         """Get the size of the tokenizer vocabulary."""
         return len(self.vocabulary)

Can you suggest a name other than vocabulary_size?

[man0045]

hi there is this issue still open ,i love to conteibute init

[sahusiddharth]

Hi @man0045,

this issue is already closed.