baaivision
Implementation in generate_image()
Hi, I have a question regarding the image generation process, specifically the generate_image function at https://github.com/baaivision/Emu/blob/main/models/modeling_emu.py#L185
According to this function and the description in the paper, it forces the model to generate n_causal=32 <image> tokens consecutively.
In each decoding step, the function concatenates an outputs.hidden_state[-1][:, -n_causal:, :] is projected through the self.decoder.lm.stu_regress_head.
So my confusion is, what is the difference between directly appending n_causal=32 <image> tokens to the text prompts, and decoding the output at once?
Because each <image> token is identical and there is no storage of intermediate outputs.
Please correct me if have misunderstood.
The core idea here is that the image tokens should be generated in an auto-regressive manner which is the same as training, in which the latter token is generated depending on all previous tokens (including both prompt tokens and previously generated image result tokens).
The <image> token is just a special placeholder token which is only used in tokenizing the input sequence.
It is useful when calling generate_image for a batch of input to ensure that the resulting token sequence length is the same after appending image tokens. All <image> token features are replaced by image features of prompt_image and result_image before being fed into LMM to generate the next token.
The main difference between the current implementation and inferring 32 images tokens at once is that the current implementation is auto-regressive but the latter is not. More specifically, the generation process of i-th token of the latter method does not depends on the results of 0-th to i-1-th token which is inconsistent with the LMM inference process.
Besides, you can of course append one [IMG] and 31 <image> tokens at once. But you still need to call 32 times of inference and at each step you should replace first i <image> tokens by corresponding generated tokens to ensure the auto-regressive property.
