data-juicer

data-juicer copied to clipboard

为什么大部分的refined recipe都是用simhash去重？

1

### Before Asking 在提问之前 - [X] I have read the [README](https://github.com/alibaba/data-juicer/blob/main/README.md) carefully. 我已经仔细阅读了 [README](https://github.com/alibaba/data-juicer/blob/main/README_ZH.md) 上的操作指引。 - [X] I have pulled the latest code of main branch to run again and...

sherrytonger

1. add data executor hook 2. add data partition executor

BeachWang

报”error: Unrecognized arguments: -B -S -I -c“

2

### Before Asking 在提问之前 - [X] I have read the [README](https://github.com/alibaba/data-juicer/blob/main/README.md) carefully. 我已经仔细阅读了 [README](https://github.com/alibaba/data-juicer/blob/main/README_ZH.md) 上的操作指引。 - [X] I have pulled the latest code of main branch to run again and...

HaleYang

Op/video face ratio filters

1

Add video_face_ratio_filters

SYSUzhouting

Switch face detection to OpenCV

2

Switch face detections to `opencv` implementation. Remove dependency on `dlib`. **Note:** All opencv-related OPs cannot run in multiprocessing with `fork` mode, which should be considered during refactoring.

drcege

Add source tags to the newly generated images, videos, and audio in some mapper op.

chenhesen

About Quality Classifier

1

### Before Asking 在提问之前 - [X] I have read the [README](https://github.com/alibaba/data-juicer/blob/main/README.md) carefully. 我已经仔细阅读了 [README](https://github.com/alibaba/data-juicer/blob/main/README_ZH.md) 上的操作指引。 - [X] I have pulled the latest code of main branch to run again and...

koanho

我想自定义一个算子，这个算子可以同时处理三个字段，text_key="instruct","input","output"的方法貌似不成功

lihongxiacream

How to download the YoukuMPLUG video dataset?

8

the modelscope newestlib unable to download it any more. Alos, is there a simple way to download the videos?

lucasjinreal

[Bug]: 运行sandbox的时候显示ModuleNotFoundError: No module named 'tools.mm_eval'

2

### Before Reporting 报告之前 - [X] I have pulled the latest code of main branch to run again and the bug still existed. 我已经拉取了主分支上最新的代码，重新运行之后，问题仍不能解决。 - [X] I have read the...

Snow0111