ARL icon indicating copy to clipboard operation
ARL copied to clipboard
verified publisher icon TophantTechnology

Python3.6.8请求一些https网站时报 ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:852) 错误

Open tarihub opened this issue 3 years ago • 2 comments

使用的是master分支最新 docker 版本 环境：MacOS 13.0.1，Docker Desktop 4.16.2，Docker Engine 20.10.22

在ARL层面，表现为获取不到任何站点，原本可以收集到很多信息的。 image

应该是在 ProbeHTTP 类的方法通过 requests 请求站时出现问题

进入容器内，通过 Python requests 请求该网站报错 image 完整报错

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 344, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 843, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 370, in connect
    ssl_context=context)
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 355, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket
    _context=self, _session=session)
  File "/usr/lib64/python3.6/ssl.py", line 776, in __init__
    self.do_handshake()
  File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:877)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 641, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='xxxxx.com', port=443): Max retries exceeded with url: /dashboard/ (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:877)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='xxxxx.com', port=443): Max retries exceeded with url: /dashboard/ (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:877)'),))
>>> print(res.content)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'res' is not defined
>>> ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:877)
  File "<stdin>", line 1
    ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:877)

运行时的 arl_work 日志，不过没抛出啥异常，有点奇怪，

[2023-01-25 19:43:29,556: INFO/ForkPoolWorker-7] app.celerytask.arl_task[b2f827ef-79a1-45f9-9dc5-2831c88e364b]: start check_http 0
[2023-01-25 19:43:29,559: INFO/ForkPoolWorker-7] app.celerytask.arl_task[b2f827ef-79a1-45f9-9dc5-2831c88e364b]: start check http 0
[2023-01-25 19:43:29,562: INFO/ForkPoolWorker-7] app.celerytask.arl_task[b2f827ef-79a1-45f9-9dc5-2831c88e364b]: end start check http 0 elapse 0.0030908584594726562
[2023-01-25 19:43:29,565: INFO/ForkPoolWorker-7] app.celerytask.arl_task[b2f827ef-79a1-45f9-9dc5-2831c88e364b]: end check_http result 0, elapse 0.00940394401550293
[2023-01-25 19:43:29,572: INFO/ForkPoolWorker-7] app.celerytask.arl_task[b2f827ef-79a1-45f9-9dc5-2831c88e364b]: start run fetch_site, <WebSiteFetch> task_id:63d1153cc1fbd98923260abc, sites: 0, available_sites:0
[2023-01-25 19:43:29,596: INFO/ForkPoolWorker-7] app.celerytask.arl_task[b2f827ef-79a1-45f9-9dc5-2831c88e364b]: start fetch site 0
[2023-01-25 19:43:29,601: INFO/ForkPoolWorker-7] app.celerytask.arl_task[b2f827ef-79a1-45f9-9dc5-2831c88e364b]: end fetch site elapse 0.005663871765136719
[2023-01-25 19:43:29,611: INFO/ForkPoolWorker-7] app.celerytask.arl_task[b2f827ef-79a1-45f9-9dc5-2831c88e364b]: end run fetch_site (0.03s), <WebSiteFetch> task_id:63d1153cc1fbd98923260abc, sites: 0, available_sites:0
[2023-01-25 19:43:29,614: INFO/ForkPoolWorker-7] app.celerytask.arl_task[b2f827ef-79a1-45f9-9dc5-2831c88e364b]: start run site_identify, <WebSiteFetch> task_id:63d1153cc1fbd98923260abc, sites: 0, available_sites:0
[2023-01-25 19:43:29,620: INFO/ForkPoolWorker-7] app.celerytask.arl_task[b2f827ef-79a1-45f9-9dc5-2831c88e364b]: start WebAnalyze 0
[2023-01-25 19:43:29,623: INFO/ForkPoolWorker-7] app.celerytask.arl_task[b2f827ef-79a1-45f9-9dc5-2831c88e364b]: end WebAnalyze elapse 0.0036220550537109375
[2023-01-25 19:43:29,628: INFO/ForkPoolWorker-7] app.celerytask.arl_task[b2f827ef-79a1-45f9-9dc5-2831c88e364b]: end run site_identify (0.01s), <WebSiteFetch> task_id:63d1153cc1fbd98923260abc, sites: 0, available_sites:0
[2023-01-25 19:43:29,633: INFO/ForkPoolWorker-7] app.celerytask.arl_task[b2f827ef-79a1-45f9-9dc5-2831c88e364b]: save_site_info site:0, <WebSiteFetch> task_id:63d1153cc1fbd98923260abc, sites: 0, available_sites:0
[2023-01-25 19:43:29,639: INFO/ForkPoolWorker-7] app.celerytask.arl_task[b2f827ef-79a1-45f9-9dc5-2831c88e364b]: start run site_capture, <WebSiteFetch> task_id:63d1153cc1fbd98923260abc, sites: 0, available_sites:0
[2023-01-25 19:43:29,655: INFO/ForkPoolWorker-7] app.celerytask.arl_task[b2f827ef-79a1-45f9-9dc5-2831c88e364b]: start screen shot 0
[2023-01-25 19:43:29,658: INFO/ForkPoolWorker-7] app.celerytask.arl_task[b2f827ef-79a1-45f9-9dc5-2831c88e364b]: end screen shot elapse 0.0029866695404052734
[2023-01-25 19:43:29,665: INFO/ForkPoolWorker-7] app.celerytask.arl_task[b2f827ef-79a1-45f9-9dc5-2831c88e364b]: end run site_capture (0.02s), <WebSiteFetch> task_id:63d1153cc1fbd98923260abc, sites: 0, available_sites:0
[2023-01-25 19:43:29,674: INFO/ForkPoolWorker-7] app.celerytask.arl_task[b2f827ef-79a1-45f9-9dc5-2831c88e364b]: start run site_spider, <WebSiteFetch> task_id:63d1153cc1fbd98923260abc, sites: 0, available_sites:0
[2023-01-25 19:43:29,683: INFO/ForkPoolWorker-7] app.celerytask.arl_task[b2f827ef-79a1-45f9-9dc5-2831c88e364b]: start site url spider entry_urls_list:0
[2023-01-25 19:43:29,686: INFO/ForkPoolWorker-7] app.celerytask.arl_task[b2f827ef-79a1-45f9-9dc5-2831c88e364b]: end site url spider (0.00s)
[2023-01-25 19:43:29,692: INFO/ForkPoolWorker-7] app.celerytask.arl_task[b2f827ef-79a1-45f9-9dc5-2831c88e364b]: end run site_spider (0.01s), <WebSiteFetch> task_id:63d1153cc1fbd98923260abc, sites: 0, available_sites:0

已尝试的解决方案（均无效）

  • https://stackoverflow.com/questions/33410577/python-requests-exceptions-sslerror-eof-occurred-in-violation-of-protocol
  • https://github.com/pypa/pip/issues/4205

我本机的Python3.6.8也是报这个错，升级至3.7，可正常请求 image

站点为项目URL，打码了，需要可私聊。问题应该是 Centos7 官方的Python3.6包编译安装一些编译选项导致 ssl 功能异常

师傅们打算怎么解决这个问题呀？目前我想到的

  1. 修改 Dockerfile 通过 yum 安装 Python3.6 为 编译安装 3.7 新点版本；
  2. 还是装 Python3.6，试试这里 https://github.com/pypa/pip/issues/4205#issuecomment-269735544 的编译选项

tarihub avatar Jan 25 '23 11:01 tarihub