data-juicer

data-juicer copied to clipboard

support panda's student captioner model in our captioning mapper

8

yxdyc

As the title says, fix #186

yxdyc

[Feature Request] Implement more streamlined interfaces for users seeking minimal functionality (data_juicer.op.functional)

3

### Search before continuing 先搜索，再继续 - [X] I have searched the Data-Juicer issues and found no similar feature requests. 我已经搜索了 Data-Juicer 的 issue 列表但是没有发现类似的功能需求。 ### Description 描述 This proposal is...

yxdyc

stopwords_filter 为什么是过滤掉小于某个阈值的样本

2

### Before Asking 在提问之前 - [X] I have read the [README](https://github.com/alibaba/data-juicer/blob/main/README.md) carefully. 我已经仔细阅读了 [README](https://github.com/alibaba/data-juicer/blob/main/README_ZH.md) 上的操作指引。 - [X] I have pulled the latest code of main branch to run again and...

noforit

1. setup local actions runner on a GPU machine 2. use docker-compose to setup a cluster 3. add more unit tests - single process - multi process - multi process...

pan-x-c

hash calculate in ray deduplicator

4

### Before Asking 在提问之前 - [X] I have read the [README](https://github.com/alibaba/data-juicer/blob/main/README.md) carefully. 我已经仔细阅读了 [README](https://github.com/alibaba/data-juicer/blob/main/README_ZH.md) 上的操作指引。 - [X] I have pulled the latest code of main branch to run again and...

simplew2011

filter是否支持batch处理，以及怎么设置batch_size？

5

### Before Asking 在提问之前 - [X] I have read the [README](https://github.com/alibaba/data-juicer/blob/main/README.md) carefully. 我已经仔细阅读了 [README](https://github.com/alibaba/data-juicer/blob/main/README_ZH.md) 上的操作指引。 - [X] I have pulled the latest code of main branch to run again and...

Yang-QW

Potential performance Issue: Slow read_csv() Function with pandas 2.0.0

4

**Issue Description:** Hello. I have discovered a performance degradation in the `read_csv` function of pandas version below 2.0.1. And I notice some parts of the repository depend on pandas 2.0.0...

TendouArisu

Add GPT-4V as evaluator

6

- Initial version to enrich the multimodal evaluation features, using GPT4V API to assess models - Welcome further testing and refinement

drcege

Add tools for FVD, KVD, ISV, PRV, FID, KID, IS, PR eval for videos.

BeachWang