sedna
sedna copied to clipboard
kubeedge
Jointinferince：edge node cannot access port 5000 of big model
What happened:
After I use the command(ffmpeg -re -i /data/video/video.mp4 -vcodec libx264 -f rtsp rtsp://localhost/video) in the example to push the rtsp stream, the edge node will report an error：
[2022-04-07 13:36:50,742] client.py(49) [ERROR] - Error occurred while request http://helmet-detection-inference-example-cloud.default:5000/sedna/predict, Msg: HTTPConnectionPool(host='helmet-detection-inference-example-cloud.default', port=5000): Max retries exceeded with url: /sedna/predict (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f55f82c9e80>: Failed to establish a new connection: [Errno 110] Connection timed out',))
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/usr/local/lib/python3.6/dist-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/usr/local/lib/python3.6/dist-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 392, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.6/http/client.py", line 1264, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1310, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1259, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1038, in _send_output
self.send(msg)
File "/usr/lib/python3.6/http/client.py", line 976, in send
self.connect()
File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 187, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 172, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f55f82c9e80>: Failed to establish a new connection: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/usr/local/lib/python3.6/dist-packages/urllib3/util/retry.py", line 446, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='helmet-detection-inference-example-cloud.default', port=5000): Max retries exceeded with url: /sedna/predict (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f55f82c9e80>: Failed to establish a new connection: [Errno 110] Connection timed out',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/lib/sedna/service/client.py", line 37, in http_request
response = request(method=_method, url=url, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='helmet-detection-inference-example-cloud.default', port=5000): Max retries exceeded with url: /sedna/predict (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f55f82c9e80>: Failed to establish a new connection: [Errno 110] Connection timed out',))
What you expected to happen: Both the edge node and the cloud node work fine and get the same results as in the example How to reproduce it (as minimally and precisely as possible): According to the steps in the tutorial, after performing the last step of rtsp streaming operation, the edge node will report the above error Anything else we need to know?: Edge node will report another error before I do rtsp push：
[rtsp @ 0x71129c0] method DESCRIBE failed: 404 NOT FOUND
[ERROR:0] global /tmp/pip-req-build-6amqbhlx/opencv/modules/videoio/src/cap.cpp (140) open VIDEOIO(CV_IMAGES): raised OpenCV exception:
OpenCV(4.4.0) /tmp/pip-req-build-6amqbhlx/opencv/modules/videoio/src/cap_images.cpp:253: error: (-5:Bad argument) CAP_IMAGES: can't find starting number (in the name of file): rtsp://localhost/video in function 'icvExtractPattern'
Environment:
Sedna Version
$ kubectl get -n sedna deploy gm -o jsonpath='{.spec.template.spec.containers[0].image}'
#
kubeedge/sedna-gm:v0.4.3
$ kubectl get -n sedna ds lc -o jsonpath='{.spec.template.spec.containers[0].image}'
#
kubeedge/sedna-lc:v0.4.3
Kubernets Version
$ kubectl version
#
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:58:47Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8", GitCommit:"7061dbbf75f9f82e8ab21f9be7e8ffcaae8e0d44", GitTreeState:"clean", BuildDate:"2022-03-16T14:04:34Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
KubeEdge Version
$ cloudcore --version
# KubeEdge v1.9.2
$ edgecore --version
#
KubeEdge v1.9.2
CloudSide Environment:
Hardware configuration
$ lscpu
#
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
座： 1
NUMA 节点： 1
厂商 ID： GenuineIntel
CPU 系列： 6
型号： 85
型号名称： Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
步进： 4
CPU MHz： 2100.218
BogoMIPS： 4200.43
L1d 缓存： 32K
L1i 缓存： 32K
L2 缓存： 4096K
L3 缓存： 16384K
NUMA 节点0 CPU： 0-7
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 arat pku ospke md_clear spec_ctrl intel_stibp
OS
$ cat /etc/os-release
#
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
Kernel
$ uname -a
#
Linux master 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Others
EdgeSide Environment:
Hardware configuration
$ lscpu
#
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
座： 1
NUMA 节点： 1
厂商 ID： GenuineIntel
CPU 系列： 6
型号： 85
型号名称： Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
步进： 4
CPU MHz： 2099.999
BogoMIPS： 4199.99
L1d 缓存： 32K
L1i 缓存： 32K
L2 缓存： 4096K
L3 缓存： 16384K
NUMA 节点0 CPU： 0-7
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 arat pku ospke md_clear spec_ctrl intel_stibp
OS
$ cat /etc/os-release
#
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
Kernel
$ uname -a
#
Linux edge01 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Others
What happened: After I use the command(
ffmpeg -re -i /data/video/video.mp4 -vcodec libx264 -f rtsp rtsp://localhost/video) in the example to push the rtsp stream, the edge node will report an error：[2022-04-07 13:36:50,742] client.py(49) [ERROR] - Error occurred while request http://helmet-detection-inference-example-cloud.default:5000/sedna/predict, Msg: HTTPConnectionPool(host='helmet-detection-inference-example-cloud.default', port=5000): Max retries exceeded with url: /sedna/predict (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f55f82c9e80>: Failed to establish a new connection: [Errno 110] Connection timed out',)) Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 160, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw File "/usr/local/lib/python3.6/dist-packages/urllib3/util/connection.py", line 84, in create_connection raise err File "/usr/local/lib/python3.6/dist-packages/urllib3/util/connection.py", line 74, in create_connection sock.connect(sa) TimeoutError: [Errno 110] Connection timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 677, in urlopen chunked=chunked, File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 392, in _make_request conn.request(method, url, **httplib_request_kw) File "/usr/lib/python3.6/http/client.py", line 1264, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.6/http/client.py", line 1310, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.6/http/client.py", line 1259, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.6/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/lib/python3.6/http/client.py", line 976, in send self.connect() File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 187, in connect conn = self._new_conn() File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 172, in _new_conn self, "Failed to establish a new connection: %s" % e urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f55f82c9e80>: Failed to establish a new connection: [Errno 110] Connection timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 449, in send timeout=timeout File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 727, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/usr/local/lib/python3.6/dist-packages/urllib3/util/retry.py", line 446, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='helmet-detection-inference-example-cloud.default', port=5000): Max retries exceeded with url: /sedna/predict (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f55f82c9e80>: Failed to establish a new connection: [Errno 110] Connection timed out',)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/lib/sedna/service/client.py", line 37, in http_request response = request(method=_method, url=url, **kwargs) File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 530, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 643, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='helmet-detection-inference-example-cloud.default', port=5000): Max retries exceeded with url: /sedna/predict (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f55f82c9e80>: Failed to establish a new connection: [Errno 110] Connection timed out',))What you expected to happen: Both the edge node and the cloud node work fine and get the same results as in the example How to reproduce it (as minimally and precisely as possible): According to the steps in the tutorial, after performing the last step of rtsp streaming operation, the edge node will report the above error Anything else we need to know?: Edge node will report another error before I do rtsp push：
[rtsp @ 0x71129c0] method DESCRIBE failed: 404 NOT FOUND [ERROR:0] global /tmp/pip-req-build-6amqbhlx/opencv/modules/videoio/src/cap.cpp (140) open VIDEOIO(CV_IMAGES): raised OpenCV exception: OpenCV(4.4.0) /tmp/pip-req-build-6amqbhlx/opencv/modules/videoio/src/cap_images.cpp:253: error: (-5:Bad argument) CAP_IMAGES: can't find starting number (in the name of file): rtsp://localhost/video in function 'icvExtractPattern'Environment:
Sedna Version Kubernets Version KubeEdge Version CloudSide Environment:
Hardware configuration OS Kernel Others EdgeSide Environment:
Hardware configuration OS Kernel Others
have you solve this problem, i have sam question.edge node cannot access port 5000 of big model,so cannot get result from
big model
@JimmyYang20 edgemesh node on edge works fine withour reporting any error
@JimmyYang20 edgemesh node on edge works fine withour reporting any error have you solve this problem？ i have the same question.edge node cannot access port 5000 of big model,so cannot get result from big model？Thank you so much
I have sam question. I entered the docker, run the infer.py, and get the warning: Connection refused in request http://helmet-detection-inference-example-cloud.default:5000/sedna/predict
I also have a question, in the edgenode, I entered the little model container,and I run the infer.py, and get the warning: Connection refused in request http://helmet-detection-inference-example-cloud.default:5000/sedna/predict!!!!!!! Why!!!
