tap-facebook isn't returning any data

[rafaelmariotti]

Hi,

this is the 1st time I'm using Singer, so please be patient with me :)

I'm trying to do something very simple: just pull my Facebook Ads data and save it as a CSV in my local. These are the commands I'm running:

pyenv virtualenv test-singer
pyenv activate test-singer
pip install --upgrade pip
pip install tap-facebook target-csv
tap-facebook --config facebook_config.json --properties facebook_properties.json --state facebook_state.json | target-csv --config csv_config.json

After I run my tap-facebook command, nothing happens... no file is created in my current directory, and there's no error message or anything, the command run successfully (check with echo $? command) but no file result was generated

These are my JSON files configured:

• facebook_config.json

{
    "start_date":"2020-01-01T00:00:00Z",
    "account_id":"XXX",
    "access_token":"XXX"
}

• facebook_state.json

{
    "adcreative": "2020-01-01T00:00:00Z",
    "ads": "2020-01-01T00:00:00Z",
    "adsets": "2020-01-01T00:00:00Z",
    "campaigns": "2020-01-01T00:00:00Z",
    "ads_insights": "2020-01-01T00:00:00Z",
    "ads_insights_age_and_gender": "2020-01-01T00:00:00Z",
    "ads_insights_country": "2020-01-01T00:00:00Z",
    "ads_insights_platform_and_device": "2020-01-01T00:00:00Z",
    "ads_insights_region": "2020-01-01T00:00:00Z",
    "ads_insights_dma": "2020-01-01T00:00:00Z",
    "ads_insights_hourly_advertiser": "2020-01-01T00:00:00Z"
}

• facebook_properties.json was generated by the following command: tap-facebook -c facebook_config.json --discover > facebook_properties.json

And here's my pip freeze:

aiohttp==2.3.10
async-timeout==3.0.1
attrs==16.3.0
backoff==1.8.0
backports.zoneinfo==0.2.1
certifi==2021.5.30
chardet==3.0.4
ciso8601==2.2.0
curlify==2.2.1
facebook-business==10.0.0
idna==2.7
idna-ssl==1.1.0
importlib-resources==5.2.2
jsonschema==2.6.0
multidict==5.1.0
pendulum==1.2.0
pycountry==20.7.3
python-dateutil==2.8.2
pytz==2018.4
pytzdata==2020.1
requests==2.20.0
simplejson==3.11.1
singer-python==5.10.0
six==1.16.0
tap-facebook==1.15.0
target-csv==0.2.0
typing-extensions==3.10.0.0
tzlocal==3.0
urllib3==1.24.3
yarl==1.6.3
zipp==3.5.0

Oh, and also I know that this should return data, since I already have a pipeline pulling data from Facebook through their API, so it's not because there's no data available ;)

any ideas?

[godiez]

I'm very new to singer packages, but I've been debugged the code and it seems a "problem" with this line (something related with the catalog), if you bypass it (i.e: adding a if True: statement), you will see that it works.

I'm going to investigate what is necessary to make it work.

[godiez]

Well, I've figured out to make it work. The "problem" is selected keyword in the metadata section.

Here is my properties.json file.

{ "stream": "campaigns", "tap_stream_id": "campaigns", "schema": { "properties": { "name": { "type": [ "null", "string" ] }, "objective": { "type": [ "null", "string" ] }, "id": { "type": [ "null", "string" ] }, "account_id": { "type": [ "null", "string" ] }, "effective_status": { "type": [ "null", "string" ] }, "buying_type": { "type": [ "null", "string" ] }, "spend_cap": { "type": [ "null", "number" ] }, "start_time": { "type": "string", "format": "date-time" }, "updated_time": { "type": "string", "format": "date-time" }, "ads": { "properties": { "data": { "items": { "properties": { "id": { "type": [ "null", "string" ] } }, "type": [ "null", "object" ] }, "type": [ "null", "array" ] } }, "type": [ "null", "object" ] }, "adlabels": { "type": [ "null", "array" ], "items": { "type": "object", "properties": { "id": { "type": "string" }, "name": { "type": "string" }, "created_time": { "type": "string", "format": "date-time" }, "updated_time": { "type": "string", "format": "date-time" } } } } }, "type": [ "null", "object" ] }, "metadata": [ { "breadcrumb": [], "metadata": { "selected": true, "table-key-properties": [ "id" ], "inclusion": "available" } }, { "breadcrumb": [ "properties", "name" ], "metadata": { "selected": true, "inclusion": "available" } }, { "breadcrumb": [ "properties", "objective" ], "metadata": { "selected": true, "inclusion": "available" } }, { "breadcrumb": [ "properties", "id" ], "metadata": { "selected": true, "inclusion": "automatic" } }, { "breadcrumb": [ "properties", "account_id" ], "metadata": { "selected": true, "inclusion": "available" } }, { "breadcrumb": [ "properties", "effective_status" ], "metadata": { "selected": true, "inclusion": "available" } }, { "breadcrumb": [ "properties", "buying_type" ], "metadata": { "selected": true, "inclusion": "available" } }, { "breadcrumb": [ "properties", "spend_cap" ], "metadata": { "selected": true, "inclusion": "available" } }, { "breadcrumb": [ "properties", "start_time" ], "metadata": { "selected": true, "inclusion": "available" } }, { "breadcrumb": [ "properties", "updated_time" ], "metadata": { "selected": true, "inclusion": "automatic" } }, { "breadcrumb": [ "properties", "ads" ], "metadata": { "selected": true, "inclusion": "available" } }, { "breadcrumb": [ "properties", "adlabels" ], "metadata": { "selected": true, "inclusion": "available" } } ] }

As you can see in the metadata section, I've added inside each property a "selected": true row. If you do that and run the following line tap-facebook --config facebook_config.json --properties facebook_properties.json --state facebook_state.json | target-csv -c config_csv.json the script works.

[WillAsh]

@godiez hey! how are you, can you help me please I'm stuck on my properties.json

[godiez]

@WillAsh Hi! Well, I can try. Open a new issue instead and I would see if I could help you.

[robg-eb]

For those struggling with this, I've found that using Meltano as a wrapper around this tap instead of just using Singer directly makes this entity/attribute selection process much easier. Instead of having to manually add "selected": true in the metadata of the properties.json file, you can just configure meltano.yml to do the following in the tap-facebook configuration:

select:
- ads.*

And you'll get all ads data selected, as an example.