Session expired while scanning aws

[kushankurdas]

Bug Template

Title: *neo4j.exceptions.SessionExpired: Failed to read from defunct connection IPv4Address(('07be756b.databases.neo4j.io', 7687)) *

Description: running cartography in ec2 launched instance to observe relationship of my aws resources

To Reproduce:

cartography --neo4j-uri neo4j+s://07be756b.databases.neo4j.io --neo4j-password-prompt --neo4j-user neo4j

Logs: Unhandled exception during sync stage 'aws'Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/neo4j/io/_common.py", line 57, in _yield_messages buffer.receive(sock, 2) File "/usr/local/lib/python3.7/site-packages/neo4j/packstream.py", line 484, in receive raise OSError("No data")OSError: No dataThe above exception was the direct cause of the following exception:Traceback (most recent call last): File "/home/ec2-user/cartography-ec2-dev/cartography/sync.py", line 86, in run stage_func(neo4j_session, config) File "/home/ec2-user/cartography-ec2-dev/cartography/util.py", line 132, in timed return method(*args, **kwargs) File "/home/ec2-user/cartography-ec2-dev/cartography/intel/aws/__init__.py", line 244, in start_aws_ingestion requested_syncs, File "/home/ec2-user/cartography-ec2-dev/cartography/intel/aws/__init__.py", line 168, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/home/ec2-user/cartography-ec2-dev/cartography/intel/aws/__init__.py", line 63, in _sync_one_account RESOURCE_FUNCTIONS[func_name](**sync_args) File "/home/ec2-user/cartography-ec2-dev/cartography/util.py", line 132, in timed return method(*args, **kwargs) File "/home/ec2-user/cartography-ec2-dev/cartography/intel/aws/ec2/volumes.py", line 118, in sync_ebs_volumes load_volume_relationships(neo4j_session, transformed_data, update_tag) File "/home/ec2-user/cartography-ec2-dev/cartography/intel/aws/ec2/volumes.py", line 95, in load_volume_relationships aws_update_tag=aws_update_tag, File "/home/ec2-user/cartography-ec2-dev/cartography/experimental_neo4j_4x_support.py", line 109, in wrapper return run_func(query, *args, **kwargs) File "/usr/local/lib/python3.7/site-packages/neo4j/work/simple.py", line 218, in run self._bookmarks, **kwparameters File "/usr/local/lib/python3.7/site-packages/neo4j/work/result.py", line 117, in _run self._attach() File "/usr/local/lib/python3.7/site-packages/neo4j/work/result.py", line 206, in _attach self._connection.fetch_message() File "/usr/local/lib/python3.7/site-packages/neo4j/io/_common.py", line 188, in inner func(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/neo4j/io/_bolt4.py", line 283, in fetch_message details, summary_signature, summary_metadata = next(self.inbox) File "/usr/local/lib/python3.7/site-packages/neo4j/io/_common.py", line 83, in __next__ tag, fields = self.pop() File "/usr/local/lib/python3.7/site-packages/neo4j/io/_common.py", line 77, in pop return next(self._messages) File "/usr/local/lib/python3.7/site-packages/neo4j/io/_common.py", line 74, in _yield_messages self.on_error(error) File "/usr/local/lib/python3.7/site-packages/neo4j/io/__init__.py", line 561, in _set_defunct_read self._set_defunct(message, error=error, silent=silent) File "/usr/local/lib/python3.7/site-packages/neo4j/io/__init__.py", line 606, in _set_defunct raise SessionExpired(message) from errorneo4j.exceptions.SessionExpired: Failed to read from defunct connection IPv4Address(('07be756b.databases.neo4j.io', 7687)) (IPv4Address(('34.126.171.25', 7687))) | Unhandled exception during sync stage 'aws' Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/neo4j/io/_common.py", line 57, in _yield_messages buffer.receive(sock, 2) File "/usr/local/lib/python3.7/site-packages/neo4j/packstream.py", line 484, in receive raise OSError("No data") OSError: No data The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/ec2-user/cartography-ec2-dev/cartography/sync.py", line 86, in run stage_func(neo4j_session, config) File "/home/ec2-user/cartography-ec2-dev/cartography/util.py", line 132, in timed return method(*args, **kwargs) File "/home/ec2-user/cartography-ec2-dev/cartography/intel/aws/__init__.py", line 244, in start_aws_ingestion requested_syncs, File "/home/ec2-user/cartography-ec2-dev/cartography/intel/aws/__init__.py", line 168, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/home/ec2-user/cartography-ec2-dev/cartography/intel/aws/__init__.py", line 63, in _sync_one_account RESOURCE_FUNCTIONS[func_name](**sync_args) File "/home/ec2-user/cartography-ec2-dev/cartography/util.py", line 132, in timed return method(*args, **kwargs) File "/home/ec2-user/cartography-ec2-dev/cartography/intel/aws/ec2/volumes.py", line 118, in sync_ebs_volumes load_volume_relationships(neo4j_session, transformed_data, update_tag) File "/home/ec2-user/cartography-ec2-dev/cartography/intel/aws/ec2/volumes.py", line 95, in load_volume_relationships aws_update_tag=aws_update_tag, File "/home/ec2-user/cartography-ec2-dev/cartography/experimental_neo4j_4x_support.py", line 109, in wrapper return run_func(query, *args, **kwargs) File "/usr/local/lib/python3.7/site-packages/neo4j/work/simple.py", line 218, in run self._bookmarks, **kwparameters File "/usr/local/lib/python3.7/site-packages/neo4j/work/result.py", line 117, in _run self._attach() File "/usr/local/lib/python3.7/site-packages/neo4j/work/result.py", line 206, in _attach self._connection.fetch_message() File "/usr/local/lib/python3.7/site-packages/neo4j/io/_common.py", line 188, in inner func(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/neo4j/io/_bolt4.py", line 283, in fetch_message details, summary_signature, summary_metadata = next(self.inbox) File "/usr/local/lib/python3.7/site-packages/neo4j/io/_common.py", line 83, in __next__ tag, fields = self.pop() File "/usr/local/lib/python3.7/site-packages/neo4j/io/_common.py", line 77, in pop return next(self._messages) File "/usr/local/lib/python3.7/site-packages/neo4j/io/_common.py", line 74, in _yield_messages self.on_error(error) File "/usr/local/lib/python3.7/site-packages/neo4j/io/__init__.py", line 561, in _set_defunct_read self._set_defunct(message, error=error, silent=silent) File "/usr/local/lib/python3.7/site-packages/neo4j/io/__init__.py", line 606, in _set_defunct raise SessionExpired(message) from error neo4j.exceptions.SessionExpired: Failed to read from defunct connection IPv4Address(('07be756b.databases.neo4j.io', 7687)) (IPv4Address(('34.126.171.25', 7687)))

Screenshots:

If applicable, add screenshots to help explain your problem.

Please complete the following information::

• Cartography release version or commit hash [1e274e3e1e30e6558a688d70f4f274b5e78f8dc0]
• Python version: 2.7.18
• OS (EC2-Amazon Linux 2)

Additional context:

Add any other context about the problem here.

[ramonpetgrave64]

It looks like your using Neo4j Auroa (neo4j.io), I would start by contacting t heir support and checking the transaction logs for the query in load_volume_relationships.

[achantavy]

In addition to contacting Neo4j support, you may want to play around with our --neo4j-max-connection-lifetime argument. This can help with timed out sessions especially over a load balancer. See https://github.com/lyft/cartography/blob/7122301f65ffe328552631b1b93427c37d11b6e9/cartography/cli.py#L92-L101 and https://neo4j.com/docs/api/python-driver/current/api.html#max-connection-lifetime-ref.

[achantavy]

Closing this out since we haven't heard back, please reopen if you need more help. I think you'll have success with playing with that parameter I sent you.

[AlmogBaku]

Hi, I'm also facing this error (also using aura). @kushankurdas did you find a solution?

[kushankurdas]

I switched to self hosted version of neo4j https://neo4j.com/docs/operations-manual/current/installation/linux/debian/, worked for me since you can increase the memory limits while deploying container

# Raise memory limits:
- NEO4J_dbms_memory_pagecache_size=1G
- NEO4J_dbms.memory.heap.initial_size=1G
- NEO4J_dbms_memory_heap_max__size=1G

[AlmogBaku]

@achantavy can we reopen it? I think that in case of "session expired" we need to renew the session or something like that

[AlmogBaku]

see https://aura.support.neo4j.com/hc/en-us/articles/1500001173021-How-to-avoid-Session-Expired-Error-to-Neo4j-Aura#:~:text=Neo4j%20Aura%20has%20a%20hard,sent%20back%20from%20the%20server.

[AlmogBaku]

I'm working on a solution for that - wrapping up the session with a "session manager" that gives you a fresh session in case you got "session expired"

I'll send a PR later on today

[AlmogBaku]

I could reproduce the same issue from my local environment to the same cluster(Neo4j Aura at GCP).

Attached is the log file, as requested: https://gist.github.com/AlmogBaku/0761f1fb62a79019e8c65b187c8ddb93