apm-agent-python icon indicating copy to clipboard operation
apm-agent-python copied to clipboard

Python Agent unable to connect to APM-server

Open rodaddy opened this issue 5 years ago • 2 comments

Errors when trying to send message to APM Server from non framework python test application.

  • run the sample.py script from a 3.6.8 virtEnv
Click to expand Trace from script
```
/Users/Shared/Development/pythonEnvs/PyCurion/bin/python /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd.py --multiproc --qt-support=auto --client 127.0.0.1 --port 51312 --file /Users/Shared/Development/PyCurion/PyCurion/bin/testAPM.py
pydev debugger: process 78700 is connecting

Connected to pydev debugger (build 202.8194.22)
Failed to submit message: 'Unable to reach APM Server: HTTPSConnectionPool(host=\'elastic-apm.FQDN.biz\', port=8200): Max retries exceeded with url: /intake/v2/events (Caused by SSLError(\'Fingerprints did not match. Expected "13efb39a2f6654e8c67bd04f4c6d4c90cd6cab5091bcedc73787f6b77d3d3fe7", got "b\\\'81799d57e55d334ff35ab66358945ae4639bfd06e571ee9c431457e921e3960e\\\'".\')) (url: https://elastic-apm.FQDN.biz:8200/intake/v2/events)'
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 976, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 374, in connect
    assert_fingerprint(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 177, in assert_fingerprint
    raise SSLError(
urllib3.exceptions.SSLError: Fingerprints did not match. Expected "13efb39a2f6654e8c67bd04f4c6d4c90cd6cab5091bcedc73787f6b77d3d3fe7", got "b'81799d57e55d334ff35ab66358945ae4639bfd06e571ee9c431457e921e3960e'".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/Shared/Development/pythonEnvs/PyCurion/lib/python3.8/site-packages/elasticapm/transport/http.py", line 82, in send
response = self.http.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/poolmanager.py", line 336, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 752, in urlopen
return self.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 752, in urlopen
return self.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 752, in urlopen
return self.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 724, in urlopen
retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='elastic-apm.FQDN.biz', port=8200): Max retries exceeded with url: /intake/v2/events (Caused by SSLError('Fingerprints did not match. Expected "13efb39a2f6654e8c67bd04f4c6d4c90cd6cab5091bcedc73787f6b77d3d3fe7", got "b\'81799d57e55d334ff35ab66358945ae4639bfd06e571ee9c431457e921e3960e\'".'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/Shared/Development/pythonEnvs/PyCurion/lib/python3.8/site-packages/elasticapm/transport/base.py", line 228, in _flush
self.send(data)
  File "/Users/Shared/Development/pythonEnvs/PyCurion/lib/python3.8/site-packages/elasticapm/transport/http.py", line 96, in send
raise TransportException(message, data, print_trace=print_trace)
elasticapm.transport.exceptions.TransportException: Unable to reach APM Server: HTTPSConnectionPool(host='elastic-apm.FQDN.biz', port=8200): Max retries exceeded with url: /intake/v2/events (Caused by SSLError('Fingerprints did not match. Expected "13efb39a2f6654e8c67bd04f4c6d4c90cd6cab5091bcedc73787f6b77d3d3fe7", got "b\'81799d57e55d334ff35ab66358945ae4639bfd06e571ee9c431457e921e3960e\'".')) (url: https://elastic-apm.FQDN.biz:8200/intake/v2/events)

Process finished with exit code 0

```
  • The thing I not about the error is that the return is expecting a "str", but gets a "b'dif_str'"

Environment (please complete the following information)

  • OS: Testing on MacOS
  • Python version: 3.8.6
  • Framework and version [e.g. Django 2.1]: None
  • APM Server version: 7.10.1
  • Agent version: 5.10.1

Additional context

Add any other context about the problem here.

  • Agent config options

    Working Java Config
    • The APM & entire Elastic Stack are configured to use an entrusted-chain.crt for ssl
    • On the servers where we run the Java app that .crt was added to the java cacerts for the Java cert store(using keytool -import).
    JAVA_AGENT="-javaagent:/usr/share/java/tomcat/elastic-apm-agent-current.jar \
    -Delastic.apm.service_name=Support-test-JAVAapp01 \
    -Delastic.apm.server_urls=https://elastic-apm.FQDN.biz:8200 \
    -Delastic.apm.application_packages=com.XXX.**"
    
  • sample.py:

    Click to expand Simple test code
    from elasticapm import Client
    
    client = Client(service_name="PyCustom-test",
        server_url='https://elastic-apm.FQDN.biz:8200',
        server_cert='./entrust-chain.crt',
        framework_name='PyCustomTest')
    
    if __name__ == "__main__":
        client.capture_message('Test Message')
    

rodaddy avatar Jan 26 '21 20:01 rodaddy

Hey @rodaddy

I think the error message (urllib3.exceptions.SSLError: Fingerprints did not match. Expected "13efb...", got "b'81799...'") is due to the fact that urllib3 uses hexlify when creating the error message (here). hexlify returns a bytestring. The actual comparison uses bytestrings for both the given fingerprint and the certificate.

As for why the fingerprint doesn't match the expected value, I'm not quite sure. Does the entrust-chain.crt file contain -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- markers, and is the first certificate in the file your end user certificate?

beniwohli avatar Jan 27 '21 13:01 beniwohli

@rodaddy Any updates here?

basepi avatar Feb 22 '21 18:02 basepi