Questgen.ai
Questgen.ai copied to clipboard
ramsrigouthamg
"WARNING:transformers.tokenization_utils_base:Token indices ..." comes with main.BoolQGen() and main.AnswerPredictor()
Hi, First of all, thanks for your sharing work. While I use main.BoolQGen() and main.AnswerPredictor() in your Questgen package, I met warnings.
Input text: Other artisans made pottery, incense, mats, furniture, linen clothing, sandals, or jewelry.A few Egyptians traveled to the upper Nile to trade with other Africans.These traders took Egyptian products such as scrolls, linen, gold, and jewelry.They brought back exotic woods, animal skins, and live beasts.As Egypt grew, so did its need to organize.Egyptians created a government that divided the empire into 42 provinces.Many officials worked to keep the provinces running smoothly.Egypt also created an army to defend itself.One of the highest jobs in Egypt was to be a priest.Priests followed formal rituals and took care of the temples.Before entering a temple, a priest bathed and put on special linen garments and white sandals.Priests cleaned the sacred statues in temples, changed their clothes, and even fed them meals.Together, the priests and the ruler held ceremonies to please the gods.Egyptians believed that if the gods were angry, the Nile would not flood.As a result, crops would not grow, and people would die.So the ruler and the priests tried hard to keep the gods happy.By doing so, they hoped to maintain the social and political order.Slaves were at the bottom of society.In Egypt, people became slaves if they owed a debt, committed a crime, or were captured in war.Egyptian slaves were usually freed after a period of time.One exception was the slaves who had to work in the mines.Many died from the exhausting labor.Egypt was one of the best places in the ancient world to be a woman.Unlike other ancient African cultures, in Egyptian society men and women had fairly equal rights.For example, they could both own and manage their own property.The main job of most women was to care for their children and home, but some did other jobs too.Some women wove cloth.Others worked with their husbands in fields or workshops.Some women, such as Queen Tiy, even rose to important positions in the government.Children in Egypt played with toys such as dolls, animal figures, board games, and marbles.
Warning in output: WARNING:transformers.tokenization_utils_base:Token indices sequence length is longer than the specified maximum sequence length for this model (3962 > 512). Running this sequence through the model will result in indexing errors
I'd like to know ways to solve the problem. Sincerely.
Hi @alexcruz0202 ,
Split the text into smaller chunks. Keep each chunk around 200-300 words at the maximum.
@ramsrigouthamg
I've split the text into several chunks. Warning is not disappeared in main.AnswerPredictor()
Input text: The Greek historian knew what he was talking about. The Nile River fed Egyptian civilization for hundreds of years. The Longest River the Nile is 4,160 miles long—the world’s longest river. It begins near the equator in Africa and flows north to the Mediterranean Sea. In the south the Nile churns with cataracts.
Warning in output: WARNING:transformers.tokenization_utils_base:Token indices sequence length is longer than the specified maximum sequence length for this model (3962 > 512). Running this sequence through the model will result in indexing errors
Can you share your code? I've been working with Questgen for a week or so and haven't had that issue.
