EC2 get_images_in_use() failed with CypherSyntaxError when using Neo4J 5.16.0

[pvasilevich]

Title: EC2 get_images_in_use() failed with CypherSyntaxError when using Neo4J 5.16.0

Description: Cypher query inside method https://github.com/lyft/cartography/blob/master/cartography/intel/aws/ec2/images.py#L21 need to be rewritten in order to work with latest versions of Neo4J.

cartography/intel/aws/ec2/images.py", line 99, in sync_ec2_images images_in_use = get_images_in_use

To Reproduce: Launch Cartography on any AWS account. Specify credentials of Neo4J database with version 5.16 You will see the following error:

neo4j.exceptions.CypherSyntaxError: {code: Neo.ClientError.Statement.SyntaxError} {message: Aggregation column contains implicit grouping expressions. For example, in 'RETURN n.a, n.a + n.b + count()' the aggregation expression 'n.a + n.b + count()' includes the implicit grouping key 'n.b'. It may be possible to rewrite the query by extracting these grouping/aggregation expressions into a preceding WITH clause. Illegal expression(s): images (line 7, column 40 (offset: 321)) " WITH collect(DISTINCT lc.image_id)+images AS images"

Logs:

Traceback (most recent call last): File "/var/cartography/cartography/sync.py", line 111, in run stage_func(neo4j_session, config) File "/var/cartography/cartography/util.py", line 197, in timed return method(*args, **kwargs) File "/var/cartography/cartography/intel/aws/init.py", line 298, in start_aws_ingestion sync_successful = _sync_multiple_accounts( File "/var/cartography/cartography/intel/aws/init.py", line 169, in _sync_multiple_accounts _sync_one_account( File "/var/cartography/cartography/intel/aws/init.py", line 64, in _sync_one_account RESOURCE_FUNCTIONSfunc_name File "/var/cartography/cartography/util.py", line 197, in timed return method(*args, **kwargs) File "/var/cartography/cartography/intel/aws/ec2/images.py", line 99, in sync_ec2_images images_in_use = get_images_in_use(neo4j_session, region, current_aws_account_id) File "/var/cartography/cartography/util.py", line 197, in timed return method(*args, **kwargs) File "/var/cartography/cartography/intel/aws/ec2/images.py", line 35, in get_images_in_use results = neo4j_session.run(get_images_query, AWS_ACCOUNT_ID=current_aws_account_id, Region=region) File "/usr/local/lib/python3.9/site-packages/neo4j/work/simple.py", line 217, in run self._autoResult._run( File "/usr/local/lib/python3.9/site-packages/neo4j/work/result.py", line 115, in _run self._attach() File "/usr/local/lib/python3.9/site-packages/neo4j/work/result.py", line 204, in _attach self._connection.fetch_message() File "/usr/local/lib/python3.9/site-packages/neo4j/io/_common.py", line 187, in inner func(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/neo4j/io/_bolt4.py", line 306, in fetch_message response.on_failure(summary_metadata or {}) File "/usr/local/lib/python3.9/site-packages/neo4j/io/_common.py", line 245, in on_failure raise Neo4jError.hydrate(**metadata)

Screenshots: You can reproduce and debug this query even without cartography

Please complete the following information::

• Cartography 0.89.0 from ghcr.io/lyft/cartography:0.89.0

Additional context: Checked that the same query is working fine in Neo4J 3.5.25

[pvasilevich]

The proposal is to use the following query instead:

MATCH (:AWSAccount{id: "030371291164"})-[:RESOURCE]->(i:EC2Instance) WHERE i.region = "eu-west-1"
RETURN i.imageid as imageid
UNION
MATCH (:AWSAccount{id: "030371291164"})-[:RESOURCE]->(lc:LaunchConfiguration) WHERE lc.region = "eu-west-1"
RETURN lc.image_id as imageid
UNION
MATCH (:AWSAccount{id: "030371291164"})-[:RESOURCE]->(ltv:LaunchTemplateVersion) WHERE ltv.region = "eu-west-1"
RETURN ltv.image_id as imageid

Such query is working for both Neo4J 3.x and 5.x.

[pvasilevich]

Please accept pull request https://github.com/lyft/cartography/pull/1290 in order to fix this. I am currently using fork of Cartography with my fix, and this is the only issue found with Neo4J v5 yet.