Instagram Fails to Sync - UserWarning: media does not allow field children

[lukearran]

Environment

• Airbyte version: 0.30.29-alpha
• OS Version / Instance: Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-110-generic x86_64)
• Deployment: Docker
• Source Connector and version: Instagram 0.1.9
• Destination Connector and version: Postgres 0.3.21
• Step where error happened: Sync job

Current Behavior

A job using the "Instagram" connector will unexpectedly stop syncing records to PostgreSQL due to the following error in the source connector. The process appears to get about halfway (4,000/7,000 records) through the process before the below error causes the process to stop.

/usr/local/lib/python3.7/site-packages/facebook_business/utils/api_utils.py:30: UserWarning: media does not allow field children

Expected Behavior

The job will successfully complete the job of syncing all streams from the Instagram source connector to a PostgreSQL destination connector.

Logs

Log download of attempt 1, 2 and 3

Steps to Reproduce

1. Create a new connection
2. Set the connection source to Instagram, with a valid account
3. Set the connection destination to PostgreSQL
4. Enable all streams
5. Enable basic Basic normalization
6. Start sync, and observe three failed attempts to sync

[lukearran]

Is this a known issue?

Kindest Regards

Luke

[harshithmullapudi]

Hey @lukearran can you confirm if removing the media stream the sync is successful?

[lukearran]

Hello @harshithmullapudi - sorry for the late response.

I've removed the "Media" stream, and after reset, the process continues to fail. However, we don't appear to have the Warning: media does not allow field children in the logs with the stream disabled.

Log download: https://tick9public.blob.core.windows.net/public-internet/github/instagram_child_issue_bug_logs_without_media_stream.zip

Kindest Regards

Luke

[harshithmullapudi]

Hey, this is interesting. I was actually able to run this perfectly looks to me like the issue is something else.

Could you try these

1. Upgrade to the latest version of Airbyte
2. Running the source in local

What I mean by local is you can

1. clone the repo
2. Inside airbyte-integrations/connectors/source-instagram
3. You can create a secrets folder and inside it a config.json
4. Then you can run python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog_without_stories.json and see if that is working

[YowanR]

@harshithmullapudi Is this issue still not repro? If so, can we go ahead and close it?

[harshithmullapudi]

Yeah I was not able to reproduce

[lukearran]

Hello @harshithmullapudi

I've upgraded Airbyte to 0.40.3 on a clean installation. We get the same UserWarning: media does not allow field children error with the Instagram connector with PostgreSQL. However, since upgrading, the logs include some more detail with response output from the API.

So it appears to be an issue with Meta/Instagram oAuth which we'll investigate further. Unless you're familiar with the cause of the below error?

  Message: Call was not successful
  Method:  GET
  Path:    https://graph.facebook.com/v11.0/REDACTED/insights
  Params:  {'metric': '["engagement","impressions","reach","saved","video_views"]'}

  Status:  400
  Response:
    {
      "error": {
        "message": "(#100) Incompatible metrics (engagement, impressions, video_views) with reel media",
        "type": "OAuthException",
        "code": 100,
        "fbtrace_id": "REDACTED"
      }
    }

[lukearran]

It appears this issue is related to the existing issue #14524. Feel free to close.

[harshithmullapudi]

Sure, thank you. Closing this in favour of #14524