Yuncong Cao comments

Repositories
Issues
Comments

Results 2 comments of


                                            Yuncong Cao

Add comprehensive tokenization tests, update diagram, and adjust code to handle edge cases

> call stack diagram for dataset and tokenization

Add comprehensive tokenization tests, update diagram, and adjust code to handle edge cases

Code changes based on tokenization tests 1. I updated ConversationTemplate.encode_conversation to drop any unpaired final message when there’s an odd count and return only the paired turns; if the first...