Nathan Fradet comments

Results 52 comments of


                                            Nathan Fradet

trafficstars

Error in training tokenizer "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf0 in position 0: invalid continuation byte"

@Yikai-Liao @lzqlzzq 👋 Sorry to ping you with another bug 😅 I'm not sure what's the exact cause of the problem here, only that symusic fails to bind a few...

Error in training tokenizer "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf0 in position 0: invalid continuation byte"

@lzqlzzq Thank you for your help! @ojus1 Great! I actually just added a `filter_dataset` method in #160 that foes exactly what its name suggests

Error in training tokenizer "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf0 in position 0: invalid continuation byte"

Thank you for the fix!

The Example_HuggingFace_Mistral_Transformer.ipynb notebook gives a ValueError when running trainer.train()

Hello, Apologies for these errors, I do not always update the notebook when some changes are made to the library. The error is occurring because the data collator is not...

The Example_HuggingFace_Mistral_Transformer.ipynb notebook gives a ValueError when running trainer.train()

Wonderful! And I would be happy to merge it! I just ran everything, and indeed there are a few other changes to make. Here is a notebook version with everything...

after tokenizing with trained tokenizer, the "tokens" array contains original tokens

Hi, This is a design choice (i.e. to only alter the ids) as the main purpose of encoding the sequence is to fed the ids to a model. If you...

ERROR:model.generate(input_tokens.ids, max_length=200)-------AttributeError: 'list' object has no attribute 'ids'

Hi, thank you for your kind comment! :) In most cases, models take input ids (i.e. integers which can be seen as the idx of the tokens in the vocabulary),...

The usage on Orchestra music

Thank you for your message! I am not sure to understand exactly what you are trying to accomplish. Is Orchestra a specific dataset? Or do you mean you are using...

The usage on Orchestra music

This is done automatically once the tokenizer has been trained. :) You can see the documentation of the `encode` (to tokenize files) and `encode_token_ids` (to apply BPE/Unigram to token ids...

pitch_intervals_max_time_dist expects bool instead of int

Hello, thank you for reporting this error. Indeed this argument is expected to be an integer. I fixed it in #240 and added support for float values as well (i.e....