关于创建数据处理函数问题

[AlexYoung757]

如题，如果我有多个任务：A、B、C、D那是不是需要创建多个类似A_cls这样的装饰器函数，如下所示： @preprocessing_fn def A_cls(params, mode): # get data # your work return input_list, target_list 能否根据具体任务动态的创建数据处理函数呢

[JayYip]

这个貌似是个python编程问题？可以参考 https://stackoverflow.com/questions/11291242/python-dynamically-create-function-at-runtime

On Thu, Nov 11, 2021, 16:26 DataAnalysist @.***> wrote:

如题，如果我有多个任务：A、B、C、D那是不是需要创建多个类似A_cls这样的装饰器函数，如下所示： @preprocessing_fn def A_cls(params, mode):

get data

your work

return input_list, target_list 能否根据具体任务动态的创建数据处理函数呢

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/JayYip/bert-multitask-learning/issues/93, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADS2OTDJATZOX7AXIHTQ4V3ULN44JANCNFSM5HZ33S4Q .

[AlexYoung757]

创建单任务数据处理函数

def create_fn(problem_list): # do some print("do work")

@preprocessing_fn
def example_cls(params, mode):
    train_texts, train_labels = [], []
    test_texts, test_labels = [],[]

    # 训练模型
    if (mode == TRAIN):
        input_list = train_texts
        target_list = train_labels
    else:
        input_list = test_texts
        target_list = test_labels
    
    return input_list, target_list

example_cls.__name__ = problem_list

return example_cls

like this？

[JayYip]

嗯, 看起来差不多, 可能这样构造会好一点

def create_preproc_fns(problem: str) -> Callable:
    @preprocessing_fn
    def example_cls(params, mode):
        train_texts, train_labels = [], []
        test_texts, test_labels = [],[]

        # 训练模型
        if (mode == TRAIN):
            input_list = train_texts
            target_list = train_labels
        else:
            input_list = test_texts
            target_list = test_labels
        
        return input_list, target_list
    example_cls.__name__ = problem
    return example_cls

fn_list = [create_preproc_fns(problem) for problem in ['a', 'b', 'c']]