PikaPikachu

The COCOPreprocessor.create_dataset wants a "num_threads" parameter, while COCOPreprocessor.minibatch did not pass "params.datasets_num_private_threads". By default, the dataset will use inter-thread pool for preprocessing, which leads to bad performance when inter-thread-num is...

How to pass hidden_states to llm directly, when using inflight batching?

2

Is there any method to pass hidden_states to llm directly, when using inflight batching? For example: In multimodal case, the image feature embedding is done by vision_tower and projector. Generally,...

this sample func seems like a bug...

6

没有考虑topk topp？ 而且至少应该算个softmax出来？ https://github.com/Infini-AI-Lab/TriForce/blob/164c8c0131cf49951eefdea89a3fbcccb8ca326b/utils/sampling.py#L64