Running failed when increase number of tree in homo secure boost alg.

[mahmoudsajjadi]

Describe the bug Running failed when I increase number of trees in homo secure boost algorithm.

To Reproduce Steps to reproduce the behavior: when I change num_trees from 10 to 50 it failed. (the training for num_trees=10 and "max_depth": 10 is complete) fateboard showed it failed in 2nd iteration.

homo_secureboost_0 = HomoSecureBoost(name="homo_secureboost_0", num_trees=50, task_type='classification', objective_param={"objective": "cross_entropy"}, tree_param={ "max_depth": 10 }, validation_freqs=1 )

Screenshots error in fateboard is as follows:

error after compiling:

Desktop (please complete the following information):

• OS: Ubuntu
• Browser Chrome

I will appreciate it if you could help me with this problem.

[talkingwallace]

This is caused by local database IO. This sometimes happens. Please rerun the job.

[ashar236]

Hi, I am trying to run pipeline examples. Can you please tell me the cause of the following error? Thanks.

pipeline.fit() 2022-10-05 22:18:18.158 | ERROR | main::1 - An error has been caught in function '', process 'MainProcess' (5634), thread 'MainThread' (140457537599296): Traceback (most recent call last):

File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/fate_client-1.9.0-py3.8.egg/pipeline/utils/invoker/job_submitter.py", line 46, in submit_job raise ValueError(f"retcode err, callback result is {result}")

ValueError: retcode err, callback result is {'retcode': 100, 'retmsg': "('Connection aborted.', BadStatusLine('\x00\x00\x18\x04\x00\x00\x00\x00\x00\x00\x04\x00?ÿÿ\x00\x05\x00?ÿÿ\x00\x06\x00\x00 \x00þ\x03\x00\x00\x00\x01\x00\x00\x04\x08\x00\x00\x00\x00\x00\x00?\x00\x00'))", 'traceback': ['Traceback (most recent call last):\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen\n httplib_response = self._make_request(\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 445, in _make_request\n six.raise_from(e, None)\n', ' File "", line 3, in raise_from\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 440, in _make_request\n httplib_response = conn.getresponse()\n', ' File "/data/projects/fate/env/python/miniconda/lib/python3.8/http/client.py", line 1348, in getresponse\n response.begin()\n', ' File "/data/projects/fate/env/python/miniconda/lib/python3.8/http/client.py", line 316, in begin\n version, status, reason = self._read_status()\n', ' File "/data/projects/fate/env/python/miniconda/lib/python3.8/http/client.py", line 298, in _read_status\n raise BadStatusLine(line)\n', 'http.client.BadStatusLine: \x00\x00\x18\x04\x00\x00\x00\x00\x00\x00\x04\x00?ÿÿ\x00\x05\x00?ÿÿ\x00\x06\x00\x00 \x00þ\x03\x00\x00\x00\x01\x00\x00\x04\x08\x00\x00\x00\x00\x00\x00?\x00\x00\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/requests/adapters.py", line 439, in send\n resp = conn.urlopen(\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen\n retries = retries.increment(\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 532, in increment\n raise six.reraise(type(error), error, _stacktrace)\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise\n raise value.with_traceback(tb)\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen\n httplib_response = self._make_request(\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 445, in _make_request\n six.raise_from(e, None)\n', ' File "", line 3, in raise_from\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 440, in _make_request\n httplib_response = conn.getresponse()\n', ' File "/data/projects/fate/env/python/miniconda/lib/python3.8/http/client.py", line 1348, in getresponse\n response.begin()\n', ' File "/data/projects/fate/env/python/miniconda/lib/python3.8/http/client.py", line 316, in begin\n version, status, reason = self._read_status()\n', ' File "/data/projects/fate/env/python/miniconda/lib/python3.8/http/client.py", line 298, in _read_status\n raise BadStatusLine(line)\n', "urllib3.exceptions.ProtocolError: ('Connection aborted.', BadStatusLine('\x00\x00\x18\x04\x00\x00\x00\x00\x00\x00\x04\x00?ÿÿ\x00\x05\x00?ÿÿ\x00\x06\x00\x00 \x00þ\x03\x00\x00\x00\x01\x00\x00\x04\x08\x00\x00\x00\x00\x00\x00?\x00\x00'))\n", '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/fate_client-1.9.0-py3.8.egg/flow_sdk/client/base.py", line 81, in _request\n response = self._http.send(prepped, stream=stream)\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/requests/sessions.py", line 655, in send\n r = adapter.send(request, **kwargs)\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/requests/adapters.py", line 498, in send\n raise ConnectionError(err, request=request)\n', "requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine('\x00\x00\x18\x04\x00\x00\x00\x00\x00\x00\x04\x00?ÿÿ\x00\x05\x00?ÿÿ\x00\x06\x00\x00 \x00þ\x03\x00\x00\x00\x01\x00\x00\x04\x08\x00\x00\x00\x00\x00\x00?\x00\x00'))\n"]}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "", line 1, in File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/fate_client-1.9.0-py3.8.egg/pipeline/backend/pipeline.py", line 568, in fit self._train_job_id, detail_info = self._job_invoker.submit_job(self._train_dsl, training_conf, callback_func) │ │ │ │ │ │ │ │ └ None │ │ │ │ │ │ │ └ {'dsl_version': 2, 'initiator': {'role': 'guest', 'party_id': 9999}, 'role': {'guest': [9999], 'host': [10000]}, 'job_paramet... │ │ │ │ │ │ └ {'components': {'reader_0': {'module': 'Reader', 'output': {'data': ['data']}, 'provider': 'fate_flow'}, 'data_transform_0': ... │ │ │ │ │ └ <pipeline.backend.pipeline.PipeLine object at 0x7fbe54246c10> │ │ │ │ └ <function JobInvoker.submit_job at 0x7fbe54236550> │ │ │ └ <pipeline.utils.invoker.job_submitter.JobInvoker object at 0x7fbe54258e20> │ │ └ <pipeline.backend.pipeline.PipeLine object at 0x7fbe54246c10> │ └ None └ <pipeline.backend.pipeline.PipeLine object at 0x7fbe54246c10> File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/fate_client-1.9.0-py3.8.egg/pipeline/utils/invoker/job_submitter.py", line 54, in submit_job raise ValueError("job submit failed, err msg: {}".format(result)) └ {'retcode': 100, 'retmsg': "('Connection aborted.', BadStatusLine('\x00\x00\x18\x04\x00\x00\x00\x00\x00\x00\x04\x...

ValueError: job submit failed, err msg: {'retcode': 100, 'retmsg': "('Connection aborted.', BadStatusLine('\x00\x00\x18\x04\x00\x00\x00\x00\x00\x00\x04\x00?ÿÿ\x00\x05\x00?ÿÿ\x00\x06\x00\x00 \x00þ\x03\x00\x00\x00\x01\x00\x00\x04\x08\x00\x00\x00\x00\x00\x00?\x00\x00'))", 'traceback': ['Traceback (most recent call last):\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen\n httplib_response = self._make_request(\n', ' File "/data/pr..

[talkingwallace]

Hi, I am trying to run pipeline examples. Can you please tell me the cause of the following error? Thanks.

pipeline.fit() 2022-10-05 22:18:18.158 | ERROR | main::1 - An error has been caught in function '', process 'MainProcess' (5634), thread 'MainThread' (140457537599296): Traceback (most recent call last):

File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/fate_client-1.9.0-py3.8.egg/pipeline/utils/invoker/job_submitter.py", line 46, in submit_job raise ValueError(f"retcode err, callback result is {result}")

ValueError: retcode err, callback result is {'retcode': 100, 'retmsg': "('Connection aborted.', BadStatusLine('\x00\x00\x18\x04\x00\x00\x00\x00\x00\x00\x04\x00?ÿÿ\x00\x05\x00?ÿÿ\x00\x06\x00\x00 \x00þ\x03\x00\x00\x00\x01\x00\x00\x04\x08\x00\x00\x00\x00\x00\x00?\x00\x00'))", 'traceback': ['Traceback (most recent call last):\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen\n httplib_response = self._make_request(\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 445, in _make_request\n six.raise_from(e, None)\n', ' File "", line 3, in raise_from\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 440, in _make_request\n httplib_response = conn.getresponse()\n', ' File "/data/projects/fate/env/python/miniconda/lib/python3.8/http/client.py", line 1348, in getresponse\n response.begin()\n', ' File "/data/projects/fate/env/python/miniconda/lib/python3.8/http/client.py", line 316, in begin\n version, status, reason = self._read_status()\n', ' File "/data/projects/fate/env/python/miniconda/lib/python3.8/http/client.py", line 298, in _read_status\n raise BadStatusLine(line)\n', 'http.client.BadStatusLine: \x00\x00\x18\x04\x00\x00\x00\x00\x00\x00\x04\x00?ÿÿ\x00\x05\x00?ÿÿ\x00\x06\x00\x00 \x00þ\x03\x00\x00\x00\x01\x00\x00\x04\x08\x00\x00\x00\x00\x00\x00?\x00\x00\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/requests/adapters.py", line 439, in send\n resp = conn.urlopen(\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen\n retries = retries.increment(\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 532, in increment\n raise six.reraise(type(error), error, _stacktrace)\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise\n raise value.with_traceback(tb)\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen\n httplib_response = self._make_request(\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 445, in _make_request\n six.raise_from(e, None)\n', ' File "", line 3, in raise_from\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 440, in _make_request\n httplib_response = conn.getresponse()\n', ' File "/data/projects/fate/env/python/miniconda/lib/python3.8/http/client.py", line 1348, in getresponse\n response.begin()\n', ' File "/data/projects/fate/env/python/miniconda/lib/python3.8/http/client.py", line 316, in begin\n version, status, reason = self._read_status()\n', ' File "/data/projects/fate/env/python/miniconda/lib/python3.8/http/client.py", line 298, in _read_status\n raise BadStatusLine(line)\n', "urllib3.exceptions.ProtocolError: ('Connection aborted.', BadStatusLine('\x00\x00\x18\x04\x00\x00\x00\x00\x00\x00\x04\x00?ÿÿ\x00\x05\x00?ÿÿ\x00\x06\x00\x00 \x00þ\x03\x00\x00\x00\x01\x00\x00\x04\x08\x00\x00\x00\x00\x00\x00?\x00\x00'))\n", '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/fate_client-1.9.0-py3.8.egg/flow_sdk/client/base.py", line 81, in _request\n response = self._http.send(prepped, stream=stream)\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/requests/sessions.py", line 655, in send\n r = adapter.send(request, **kwargs)\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/requests/adapters.py", line 498, in send\n raise ConnectionError(err, request=request)\n', "requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine('\x00\x00\x18\x04\x00\x00\x00\x00\x00\x00\x04\x00?ÿÿ\x00\x05\x00?ÿÿ\x00\x06\x00\x00 \x00þ\x03\x00\x00\x00\x01\x00\x00\x04\x08\x00\x00\x00\x00\x00\x00?\x00\x00'))\n"]}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "", line 1, in File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/fate_client-1.9.0-py3.8.egg/pipeline/backend/pipeline.py", line 568, in fit self._train_job_id, detail_info = self._job_invoker.submit_job(self._train_dsl, training_conf, callback_func) │ │ │ │ │ │ │ │ └ None │ │ │ │ │ │ │ └ {'dsl_version': 2, 'initiator': {'role': 'guest', 'party_id': 9999}, 'role': {'guest': [9999], 'host': [10000]}, 'job_paramet... │ │ │ │ │ │ └ {'components': {'reader_0': {'module': 'Reader', 'output': {'data': ['data']}, 'provider': 'fate_flow'}, 'data_transform_0': ... │ │ │ │ │ └ <pipeline.backend.pipeline.PipeLine object at 0x7fbe54246c10> │ │ │ │ └ <function JobInvoker.submit_job at 0x7fbe54236550> │ │ │ └ <pipeline.utils.invoker.job_submitter.JobInvoker object at 0x7fbe54258e20> │ │ └ <pipeline.backend.pipeline.PipeLine object at 0x7fbe54246c10> │ └ None └ <pipeline.backend.pipeline.PipeLine object at 0x7fbe54246c10> File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/fate_client-1.9.0-py3.8.egg/pipeline/utils/invoker/job_submitter.py", line 54, in submit_job raise ValueError("job submit failed, err msg: {}".format(result)) └ {'retcode': 100, 'retmsg': "('Connection aborted.', BadStatusLine('\x00\x00\x18\x04\x00\x00\x00\x00\x00\x00\x04\x...

ValueError: job submit failed, err msg: {'retcode': 100, 'retmsg': "('Connection aborted.', BadStatusLine('\x00\x00\x18\x04\x00\x00\x00\x00\x00\x00\x04\x00?ÿÿ\x00\x05\x00?ÿÿ\x00\x06\x00\x00 \x00þ\x03\x00\x00\x00\x01\x00\x00\x04\x08\x00\x00\x00\x00\x00\x00?\x00\x00'))", 'traceback': ['Traceback (most recent call last):\n', ' File "/data/projects/fate/env/python/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen\n httplib_response = self._make_request(\n', ' File "/data/pr..

It seems that it had some connection problems. Could you please create a new issue so that other developers can see your problem? Thanks!

[github-actions[bot]]

This issue has been marked as stale because it has been open for 365 days with no activity. If this issue is still relevant or if there is new information, please feel free to update or reopen it.

[github-actions[bot]]

This issue was closed because it has been inactive for 1 days since being marked as stale. If this issue is still relevant or if there is new information, please feel free to update or reopen it.