data-juicer

data-juicer copied to clipboard

### Before Reporting 报告之前 - [X] I have pulled the latest code of main branch to run again and the bug still existed. 我已经拉取了主分支上最新的代码，重新运行之后，问题仍不能解决。 - [X] I have read the...

FailedNamed

### Before Reporting 报告之前 - [X] I have pulled the latest code of main branch to run again and the bug still existed. 我已经拉取了主分支上最新的代码，重新运行之后，问题仍不能解决。 - [X] I have read the...

luckystar1992

支持RangeSpecifiedFieldSelector使用指定字段的值域进行数据选择

1

目前的RangeSpecifiedFieldSelector类只支持通过百分位数和rank进行select，这并不符合直觉，最简单且最常用的方法应该是根据某个字段的值域进行选择，比如相似度大于某个阈值，PPL小于某个阈值等，本次PR支持这一功能。 此外，原来的process函数在逻辑判断时存在一定的问题（不支持某些情况下的缺省），lower_percentile和lower_rank不能同时为None，upper_percentile和upper_rank也不能同时为None，否则就不会进行select，这不适用于只有上界或者只有下界的情况，本次PR针对这一逻辑进行了优化。

2108038773

As the title says.

BeachWang

### Before Reporting 报告之前 - [X] I have pulled the latest code of main branch to run again and the bug still existed. 我已经拉取了主分支上最新的代码，重新运行之后，问题仍不能解决。 - [X] I have read the...

ForeverNewLee

Add calling api with agentscope to demos.

BeachWang

Add segment_mapper

2

Perform segment-anything on images (with FastSAM) and return the bounding box values. Hyperparameters: - imgsz: image resolution after image resizing - conf: confidence score threshold - iou: IoU (Intersection over...

Qirui-jiao

Add sentence_augmentation_mapper

2

Augment sentences using LLMs. Hyperparameters: - system_prompt: system prompt; - task_sentence: the instruction for the current task; - max_new_tokens: the maximum number of new tokens generated by the model; -...

Qirui-jiao

why often happen: One of the subprocesses has abruptly died during map operation?

3

### Before Asking 在提问之前 - [X] I have read the [README](https://github.com/alibaba/data-juicer/blob/main/README.md) carefully. 我已经仔细阅读了 [README](https://github.com/alibaba/data-juicer/blob/main/README_ZH.md) 上的操作指引。 - [X] I have pulled the latest code of main branch to run again and...

strongcc

Fix typos

co63oc