simplew2011

CHINESE detect error

5

```python from lingua import Language, LanguageDetectorBuilder languages = [Language.ENGLISH, Language.JAPANESE, Language.CHINESE, Language.KOREAN] # detector = LanguageDetectorBuilder.from_languages(*languages).build() # detector = LanguageDetectorBuilder.from_all_languages().with_preloaded_language_models().build() # detector = LanguageDetectorBuilder.from_languages(*languages).with_low_accuracy_mode().build() detector = LanguageDetectorBuilder.from_languages(*languages).build() s8 = "不想穷,就好好记住这篇文章...

Is there a plan for model transformation and deployment

modify generate_frame.m to python

1

```python import os import cv2 from tqdm import tqdm base_dir = '/home/simplew/dataset/sod/DHF1K' video_dir = os.path.join(base_dir, 'video') movies = [mov for mov in os.listdir(video_dir) if mov.endswith('.AVI')] for movie in tqdm(movies): image_dir...

Could not export Python function call 'BlurFunction'. Remove calls to Python functions before export. Did you forget to add @script or @script_method annotation? If this is a nn.ModuleList, add it...

how about make a ray executor to deduplication

1

- https://github.com/ChenghaoMou/text-dedup/blob/main/text_dedup/minhash_spark.py - reference：https://github.com/alibaba/data-juicer/blob/main/data_juicer/core/ray_executor.py - Ray is simpler and faster than Spark

pretrain model

2

pretrain model release?

service/base.py报错

7

serve/base.py 45行 from service import conf ImportError: cannot import name 'conf' conf是service的一个字典，import出错。

load model error

4


可否转为TensorRT

1

这个工程的h5模型可否转为TensorRT模型。

hash calculate in ray deduplicator

4

### Before Asking 在提问之前 - [X] I have read the [README](https://github.com/alibaba/data-juicer/blob/main/README.md) carefully. 我已经仔细阅读了 [README](https://github.com/alibaba/data-juicer/blob/main/README_ZH.md) 上的操作指引。 - [X] I have pulled the latest code of main branch to run again and...