FATE icon indicating copy to clipboard operation
FATE copied to clipboard

fate求交算法在1.9和1.11版本下的运行时间差异

Open enidastronomical opened this issue 1 year ago • 8 comments

我们测试了fate在1.92和1.11版本下的求交算法的运行时间 (本来准备测raw的,但是后面发现不支持raw,会默认改成ecdh算法) 但是测试结果显示fate1.11求交的时间是1.92的接近三倍,想请问下这是由于什么原因所导致的呢?

这是我们的作业配置

# host端
docker exec -it confs-9999_client_1 bash
cat > /tmp/upload.json <<EOF
{
    "file": "/data/projects/fate/examples/data/10m.csv",
    "id_delimiter": ",",
    "head": 1,
    "partition": 8,
    "namespace": "test",
    "table_name": "data_10m"
}
EOF
flow data upload -c /tmp/upload.json

# guest端
docker exec -it confs-10000_client_1 bash
cat > /tmp/upload.json <<EOF
{
    "file": "/data/projects/fate/examples/data/5m.csv",
    "id_delimiter": ",",
    "head": 1,
    "partition": 8,
    "namespace": "test",
    "table_name": "data_5m"
}
EOF
flow data upload -c /tmp/upload.json

cat > /tmp/dsl.json <<EOF
{
    "components": {
        "reader_0": {
            "module": "Reader",
            "output": {
                "data": [
                    "data"
                ]
            }
        },
        "data_transform_0": {
            "module": "DataTransform",
            "input": {
                "data": {
                    "data": [
                        "reader_0.data"
                    ]
                }
            },
            "output": {
                "data": [
                    "data"
                ],
                "model": [
                    "model"
                ]
            }
        },
        "intersection_0": {
            "module": "Intersection",
            "input": {
                "data": {
                    "data": [
                        "data_transform_0.data"
                    ]
                }
            },
            "output": {
                "data": [
                    "data"
                ]
            }
        }
    }
}
EOF

cat > /tmp/conf.json <<EOF
{
    "dsl_version": "2",
    "initiator": {
        "role": "guest",
        "party_id": 10000
    },
    "role": {
        "guest": [
            10000
        ],
        "host": [
            9999
        ]
    },
    "job_parameters": {
        "common": {
            "task_parallelism": 1,
            "auto_retries": 0,
            "computing_partitions": 8,
            "task_cores": 32
        }
    },
    "component_parameters": {
        "common": {
            "intersection_0": {
                "intersect_method": "raw",
                "sync_intersect_ids": true,
                "join_role": "host"
            }
        },
        "role": {
            "guest": {
                "0": {
                    "reader_0": {
                        "table": {
                            "name": "data_5m",
                            "namespace": "test"
                        }
                    },
                    "data_transform_0": {
                        "with_label": false,
                        "output_format": "dense"
                    }
                }
            },
            "host": {
                "0": {
                    "reader_0": {
                        "table": {
                            "name": "data_10m",
                            "namespace": "test"
                        }
                    },
                    "data_transform_0": {
                        "with_label": false,
                        "output_format": "dense"
                    }
                }
            }
        }
    }
}
EOF
flow job submit -c /tmp/conf.json -d /tmp/dsl.json

enidastronomical avatar Jun 28 '23 08:06 enidastronomical

1.11哪个版本呢?从数据表名看是guest: 10m vs host: 5m?

mgqa34 avatar Jun 29 '23 02:06 mgqa34

1.11哪个版本呢?从数据表名看是guest: 10m vs host: 5m?

是kube-fate docker 1.11 数据量是guest: 5m vs host: 10m

enidastronomical avatar Jun 30 '23 02:06 enidastronomical

这是对应1.11.1?

dylan-fan avatar Jun 30 '23 03:06 dylan-fan

这是对应1.11.1?

是的,是1.11.1

enidastronomical avatar Jun 30 '23 08:06 enidastronomical

在测试样例的里面你也出现这个情况吗?

HYC-create avatar Jul 06 '23 12:07 HYC-create

这是对应1.11.1?

是的,是1.11.1

想问一下你用的机子是什么配置

HYC-create avatar Jul 06 '23 13:07 HYC-create

我也测试了下,发现两个版本任务时间确实差很多,这个是结果:

版本 任务时间
1.11.1 01:07:13
1.9.2 00:26:14

不同版本的配置没有太大区别,不确定是哪里配置还是是代码更新的原因

owlet42 avatar Jul 17 '23 05:07 owlet42

请问不用flowboard的话,在哪里查看求交的结果呢

CNcorazon avatar Aug 09 '23 02:08 CNcorazon