Can't create AtlasProcess. AtlasException: "Referenced entity -1002 is not found"

[edobrynin-dodo]

What is your desired outcome Hi Will & all the contributors!

I'm trying to write a function to create/update AtlasProcess based on passed parameters. But for some reason I faced an exception, and I wasn't able to found something similar in google :(

What have you tried so far This is my function to create/update Atlas Process. If the process already exists - the function should add source_datasets list to Process' inputs. Otherwise the function should create Atlas Process from scratch.

def create_atlas_process(self, purview_client, name, qualified_name, source_datasets: list, target_dataset):

        existing_process = purview_client.get_entity(
            typeName="Process",
            qualifiedName=qualified_name
        )

        if existing_process:

            process_entity = existing_process["entities"][0]

            process_to_update = AtlasProcess(
                name=name,
                typeName="Process",
                qualified_name=qualified_name,
                inputs=None,  
                outputs=None,  
                guid=self.guid_tracker.get_guid()
            )

            existing_inputs = process_entity["attributes"]["inputs"]
            existing_outputs = process_entity["attributes"]["outputs"]

            process_to_update.inputs = existing_inputs + source_datasets
            process_to_update.outputs = existing_outputs

            self.__upload_atlas_entities(purview_client, entities_list=[process_to_update] + source_datasets)

        else:

            atlas_process = AtlasProcess(
                name=name,
                typeName="Process",
                qualified_name=qualified_name,
                inputs=source_datasets,
                outputs=[target_dataset],
                guid=-101, # I also tried GuidTracker.get_guid() here, but the result was the same :( 
            )

            # THE EXCEPTION WAS RAISED HERE
            self.__upload_atlas_entities(purview_client=purview_client, entities_list=[atlas_process])

Could you please help me with this? The exception message isn't very useful unfortunately :(

AtlasException: {"requestId":"f2ac6d45-22fd-4aeb-8929-23e92cf414d3","errorCode":"ATLAS-404-00-00A","errorMessage":"Referenced entity -1002 is not found"}

I cant't verify what entity (aka -1002) is the problem: is this a Process itself, our one of the inputs/outputs

Desktop (please complete the following information):

• OS: MacOS Monterey 12.3.1
• PyApacheAtlas Version == 0.13.1

Thank you in advance! Best regards, Evgenii

[wjohnson]

Hi, @edobrynin-dodo Thank you for using PyApacheAtlas!

This error means that you have a reference to a guid "-1002" (which was definitely generated by the GuidTracker) in your upload but the asset with guid "-1002" is not included in the upload.

Your else case doesn't include the upload of the source_datasets and target_dataset. Could you try using the same code you had as above: self.__upload_atlas_entities(purview_client, entities_list=[atlas_process] + source_datasets)

[wjohnson]

Closing out due to no follow up but please feel free to reopen if you are still having the same issue after looking at the above suggestion.