MiniCPM-V
MiniCPM-V copied to clipboard
OpenBMB
Project author team stay tuned: I found out that the llama3-V project is stealing a lot of academic work from MiniCPM-Llama3-V 2.5
Fellow MiniCPM-Llama3-V 2.5 project authors, a few days ago I discovered a shocking fact.There is a large amount of work in the llama3-V (https://github.com/mustafaaljadery/llama3v) project that is suspected to have been stolen from the MiniCPM-Llama3-V 2.5 project, and I raised my query in the GitHub project issue of llama3-v, and did not think that the The authors of Llama3-V quickly deleted my questionable post, and hid Llama3-V's Huggingface project page. I strongly question what they did, and I will release all the evidence next, and I urge you to pay attention to this fact.
this issue has been deleted by the author of llama3-V ( https://github.com/mustafaaljadery/llama3v )，I will expose all the evidence to expose the fact that the authors of llama3-v are a bunch of thieves!
Fact 1: The llama3-V project uses almost exactly the same model structure and code as the minicom-llama 3-v 2.5 project
Llama3-V has exactly the same model structure and config file as MiniCPM-Llama3-V 2.5, with only the difference in variable names. Left: MiniCPM-Llama3-V 2.5 Right: Llama3-V
Its code appears to be MiniCPM-Llama3-V 2.5's code with some reformatting and variable renaming, including but not limited to image slicing, tokenizer, resampler, and data loading. Just give some examples.
The author of Llama3-V refers to LLaVA-UHD for the architecture, and list difference (on ViT and LLM choice). What the author does not mention is that their specific implementation is identical to MiniCPM-Llama3-V 2.5, which is different from LLaVA-UHD in many ways, such as the spatial schema. Llama3-V also has the same tokenizer as MiniCPM-Llama3-V 2.5, including the special tokens newly defined by MiniCPM-Llama3-V 2.5.
Fact 2: When I questioned how the authors of llama3-v used MinicPM-Llama3-V2.5's tokenizer before the MinicPM-Llama3-V2.5 project was released, the authors of the llama3-v project began to lie.
The author of llama3-V project thought the tokenizer would be from here: https://huggingface.co/openbmb/MinicPM-V-2/blob/main/tokenizer.json Before llama3 MiniCPM released.
but the fact is that MinicPM-V-2's tokenizer is totally different from MinicPM-Llama3-V2.5，below is the two files in Huggingface. Obviously, they are not the same tokenizer file, and their file sizes are completely different.
And MinicPM-Llama3-v2.5's tokenizer is llama3 tokenizer plus miniCPM-v series model of a few special token composition, and MinicPM-v2 release are before llama3 open source
Fact 3: The author of llama3-V project afraid to face questioning, deleted the issue I filed at llama3-V questioning their stealing.
Also, it seems the author does not fully understand MiniCPM-Llama3-V 2.5's architecture or their own code. Perceiver resampler is a single-layer cross-attention, not a two-layer self-attention. Sigmoid activation of SigLIP is not used for training multimodal large language models. These activations are only used for pretraining SigLIP.
Llama3-V:
MiniCPM-Llama3-V 2.5:
Visual feature extraction doesn't need sigmoid activation.
Based on the above three facts, I think there is sufficient evidence to prove that the llama3-v project has stolen the academic achievements of the minicpm-llama 3-v 2.5 project, and I strongly suggest that the minicpm-llama 3-v 2.5 project's team go to the complaint to expose the llama3-v project authors' stealing and lying about academic misconduct, and so on a series of problems!
Hi @pzc163, Thank you for sharing this important information with us. We are deeply shocked and will be paying special attention to this matter. We will immediately launch an investigation to verify the above situation. Any new findings will be quickly disclosed to you, to the open-source community, and the public.
This situation sounds extremely serious. We never expected anything like this to happen. We hope the truth will come to light soon.
Adding two important piece of information：
- A few days ago, when I tried to run Llama3-V, I found their provided code could not work with their checkpoint from HuggingFace. Many issues about this have been posted on GitHub and HuggingFace, but no reply from the author yet. I changed the variable names in Llama3-V's model weights downloaded from HuggingFace to MiniCPM-Llama3-V 2.5's names, and surprisingly found that the model can be run with MiniCPM-V code successfully.
model.safetensors.index.json
2.Guess what you get if you add Gaussian noise(parameterized by a single scalar) to MiniCPM-Llama3-V 2.5's checkpoint?
new_dict = {} for k, v in model.state_dict().items(): torch.cuda.manual_seed_all(42) new_dict[k] = v + torch.randn_like(v) / 708 model.load_state_dict(new_dict)
That's crazy! You can actually get a new checkpoint, emm, so let's give this new checkpoint a new name and call it llama3-V, doesn't that sound great? At least the hash will be completely different from miniCPM-llama3-V2.5, right?
Thanks for the info. The inference fix and noise sound horrific. We are reproducing it and will test more on some in-house features.
The conclusion of our investigation:
- Llama3-V can be run using MiniCPM-Llama3-V 2.5's code and config.json after changing param names
- It behaves similarly to MiniCPM-Llama3-V 2.5 in unrevealed experimental features trained on in-house data, e.g., recognizing Tsinghua Bamboo Characters and GUIAgent
- It is somewhat similar to a noised version of MiniCPM-Llama3-V 2.5?
After receiving the issue from @yangzhizheng1on GitHub, we launched a serious investigation. We can obtain inference results correctly using Llama3-V checkpoint with MiniCPM-Llama3-V 2.5's code and config file following @yangzhizheng1's instruction on GitHub. Even more, we also surprisingly find that Llama3-V shows highly similar behaviors to MiniCPM-Llama3-V 2.5 in some unrevealed experimental features, which are trained on private in-house data, such as recognizing Tsinghua Bamboo Characters.
One of the experimental features of MiniCPM-Llama3-V 2.5 is recognizing Tsinghua Bamboo Characters (清华简), a very special and rare type of Chinese ancient characters written on bamboo during China's Warring States Period (475 BC-221 BC). These training images are recently scanned from unearthed cultural relics and annotated by our team, which has not been publicly released yet. Surprisingly, we find highly similar capabilities for Llama3-V in both good and bad cases.
For quantative results, we also tested several Llama3-based VLMs on 1K Bamboo Character images and compared the prediction exact match for each pair of models.
The overlaps between every two models are zero, whereas the overlaps between Llama3-V and MiniCPM-Llama3-V 2.5 achieve a surprising 87%. Moreover, MiniCPM-Llama3-V 2.5 and Llama3-V even share a similar error distribution. Llama3-V and MiniCPM-Llama3-V 2.5 make 236 and 194 wrong predictions respectively, while the overlapped part is 182. The MiniCPM-Llama3-V2.5-noisy obtained following @yangzhizheng1's instruction on GitHub shows nearly identical quantative results with Llama3-V. This is really confusing...
The same thing also happens to WebAgent, another unrevealed feature trained on in-house data. They even make identical errors in a WebAgent schema newly defined within our team...
Since the HuggingFace page of Llama3-V is removed now, we upload the checkpoint here (https://bit.ly/3yRFxYq). Since this model has received several thousands of downloads on HuggingFace, there should be independent copies to reproduce this.
Given these results, we are afraid it is hard to explain such unusual similarities as coincidences. We hope the authors can give an official explanation of the issue. We believe this is important for the common good of the open-source community.
You might want to report this to Stanford CS or Stanford itself. These are serious allegations and they appear (at a quick glance and to my non-expert eyes) to be well substantiated.
If the research team from Stanford University is proven to have plagiarized this MiniCPM-V project from Tsinghua University, they should feel ashamed, and also, MiniCPM-V project deserve an apology and acknowledgment.
You can consult the Dean of Stanford CS department to report misconducts. Refer to this policy:
https://doresearch.stanford.edu/policies/research-policy-handbook/conduct-research/research-misconduct-policy-allegations-investigations-and-reporting
Section 5: Individual Reporting Responsibility Any individual who believes an act of research misconduct has occurred or is occurring should notify the dean of the appropriate school.
The current Dean is likely Jennifer Widom: https://profiles.stanford.edu/jennifer-widom?tab=bio The one who has the most solid proof should notify her
https://web.archive.org/web/20240528201635/https://github.com/mustafaaljadery/llama3v/blob/main/model.safetensors.index.json
If you google Llama3v there have now been more than 1000 pages attributing the work. This has made a detrimental impact. Their actions seemed deliberately planned, aiming for rapid and extensive coverage in tech news with attention-grabbing assertions. This strategy can make the stolen credits be attributed to them quickly before the original authors even realize it. The authors may want to escalate this to their academic administrators immediately to prevent any further negative impact.
I am not sure if I understand correctly. Actually, the original project is an open-source project, so Llama3-V can use it, but they didn't comply with the open-source license?
I am not sure if I understand correctly. Actually, the original project is an open-source project, so Llama3-V can use it, but they didn't comply with the open-source license?
If I get it right, they did not acknowledge the original project at all, but instead, claimed this as their own "innovation" and promoted their "contribution" with large volumes on social media and technology news/blogs.
I am not sure if I understand correctly. Actually, the original project is an open-source project, so Llama3-V can use it, but they didn't comply with the open-source license?
If I get it right, they did not acknowledge the original project at all, but instead, claimed this as their own "innovation" and promoted their "contribution" with large volumes on social media and technology news/blogs.
"I understand your point, and indeed, from this perspective, their actions are reprehensible. However, from a legal standpoint, it seems they are only guilty of violating the license, which may constitute infringement."
I am not sure if I understand correctly. Actually, the original project is an open-source project, so Llama3-V can use it, but they didn't comply with the open-source license?
If I get it right, they did not acknowledge the original project at all, but instead, claimed this as their own "innovation" and promoted their "contribution" with large volumes on social media and technology news/blogs.
"I understand your point, and indeed, from this perspective, their actions are reprehensible. However, from a legal standpoint, it seems they are only guilty of violating the license, which may constitute infringement."
Totally agree with you, from a legal standpoint, they will not go to jail for this.
I am not sure if I understand correctly. Actually, the original project is an open-source project, so Llama3-V can use it, but they didn't comply with the open-source license?
If I get it right, they did not acknowledge the original project at all, but instead, claimed this as their own "innovation" and promoted their "contribution" with large volumes on social media and technology news/blogs.
"I understand your point, and indeed, from this perspective, their actions are reprehensible. However, from a legal standpoint, it seems they are only guilty of violating the license, which may constitute infringement."
Administrative measures can be taken from academia where policies for academic misconduct may apply, especially if they publish the work and aim for academic impacts.
The latest news, one of the authors Aksh Garg, has acknowledged that on his medium post
We realized that our architecture is very similar to OpenBMB’s “MiniCPM-Llama3-V 2.5...We have taken down our original model in respect to the authors.
The latest news, one of the authors Aksh Garg, has acknowledged that on his medium post
We realized that our architecture is very similar to OpenBMB’s “MiniCPM-Llama3-V 2.5...We have taken down our original model in respect to the authors.
You would hardly be satisfied with this kind of statement if it were your work being copied, with model weights deliberately altered with Gaussian noise and renamed, with a plotted and overwhelming coverage in news and social media (enhanced by eyes-grabbing "$500" statements in the headlines) etc... This goes beyond merely saying "the architecture is very similar to blah blah blah..." And guess what, they said you merely "beat us to the implementation."??? To be honest, one would be furious if they were the authors and saw the statement...
A thief is being tried in court, and this is his statement:
"I would like to thank the prosecutor for pressing charges. I realize that my belongings are very similar to those of the victim. To show my respect for him, I am relinquishing these belongings."
https://www.reddit.com/r/stanford/comments/1d75jns/comment/l6x0m4z/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button
@tangmingxing1988 you forgot the part before the trial where the thief takes a world tour extolling their amazing belongings :stuck_out_tongue_winking_eye:
@pzc163 @Cuiunbo @RylanSchaeffer
Given the two checkpoints, perhaps one can compute the diff in the weight to see the histogram? (even though it already seems like there is enough evidence)
