Error with the target when paired with zendesk tap

[j-oflaherty]

Hello. I've run into an issue when trying to use this target alongside the zendesk tap, when the satisfaction_ratings stream provided by the latter is selected. Two errors appear to be happening at the same time, but only one happens within this target's files.

INFO Querying for satisfaction ratings between 2020-10-19T23:59:59Z and 2020-11-11T19:13:02Z
CRITICAL 'properties'
Traceback (most recent call last):
  File ".virtualenvs/target-postgres/bin/target-postgres", line 8, in <module>
    sys.exit(cli())
  File "~/singer_testing/.virtualenvs/target-postgres/lib/python3.8/site-packages/target_postgres/__init__.py", line 45, in cli
    main(args.config)
  File " ~/singer_testing/.virtualenvs/target-postgres/lib/python3.8/site-packages/target_postgres/__init__.py", line 39, in main
    target_tools.main(postgres_target)
  File "(directory)/singer_testing/.virtualenvs/target-postgres/lib/python3.8/site-packages/target_postgres/target_tools.py", line 28, in main
    stream_to_target(input_stream, target, config=config)
  File "(directory)/singer_testing/.virtualenvs/target-postgres/lib/python3.8/site-packages/target_postgres/target_tools.py", line 77, in stream_to_target
    raise e
  File "(directory)/singer_testing/.virtualenvs/target-postgres/lib/python3.8/site-packages/target_postgres/target_tools.py", line 58, in stream_to_target
    _line_handler(state_tracker,
  File "(directory)/singer_testing/.virtualenvs/target-postgres/lib/python3.8/site-packages/target_postgres/target_tools.py", line 123, in _line_handler
    buffered_stream = BufferedSingerStream(stream,
  File "(directory)/singer_testing/.virtualenvs/target-postgres/lib/python3.8/site-packages/target_postgres/singer_stream.py", line 47, in __init__
    self.update_schema(schema, key_properties)
  File "(directory)/singer_testing/.virtualenvs/target-postgres/lib/python3.8/site-packages/target_postgres/singer_stream.py", line 75, in update_schema
    properties = self.schema['properties']
KeyError: 'properties'

This is the first error that occurs. After a second error with the tap, this is what it's displayed:

BrokenPipeError: [Errno 32] Broken pipe
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe

I tried the same tap with the same config with the csv target, and it worked without a problem. I've tried other streams and they work fine, it's the only one I've run into a problem with.

PD: I'm running it from the terminal with

.virtualenvs/zendesk-tap/bin/tap-zendesk --config zendesk_config.json --catalog zendesk_test.json | .virtualenvs/targetpostgres/bin/target-postgres --config postgres_test_config.json 

[AlexanderMann]

@j-oflaherty sorry for just now getting back to you. Github's new notifications somehow buried this for me 😅

Are you still experiencing this problem? I don't know that we've had long lasting issues with Zendesk as of late, so I'd be a bit shocked if we can't get this working.

[j-oflaherty]

Hi @AlexanderMann,

Sorry for the delay,

The problem still persists. I've opened the issue in the zendesk tap and they pointed out it's probably a problem on the target's end. Is there any extra information i can provide to help you sort the problem out?

[AlexanderMann]

@j-oflaherty that's very odd...at this point, the best thing would be a test which reproduces this. You can check out our https://github.com/datamill-co/target-postgres/blob/master/tests/unit/test_sandbox.py for examples for how to add one.

Aside from that, redacted streams, schemas, and config are welcomed if you can share them.

[j-oflaherty]

@AlexanderMann I've managed to find and ''fix'' the problem. On the file singer_stream.py on line 69 the function json_simplify(schema) gives different structures for different streams. With the stream tickets the result is the following:

 {'type': ['object', 'null'],
  'properties': {'allow_attachments': {'type': ['boolean', 'null']}, 'allow_channelback': {'type': ['boolean', 'null']}, 'assignee_id': {'type': ['integer', 'null']}, 'brand_id': {'type': ['integer', 'null']}, 'created_at': {'type': ['string', 'null'], 'format': 'date-time'}, 'custom_fields': {'type': ['array', 'null'], 'items': {'type': ['object', 'null'], 'properties': {'id': {'type': ['integer', 'null']}, 'value': {'type': ['object'], 'properties': {}}}}}, 'description': {'type': ['string', 'null']}, 'due_at': {'type': ['string', 'null'], 'format': 'date-time'}, 'external_id': {'type': ['string', 'null']}, 'forum_topic_id': {'type': ['integer', 'null']}, 'generated_timestamp': {'type': ['integer', 'null']}, 'group_id': {'type': ['integer', 'null']}, 'has_incidents': {'type': ['boolean', 'null']}, 'id': {'type': ['integer', 'null']}, 'is_public': {'type': ['boolean', 'null']}, 'organization_id': {'type': ['integer', 'null']}, 'priority': {'type': ['string', 'null']}, 'problem_id': {'type': ['integer', 'null']}, 'raw_subject': {'type': ['string', 'null']}, 'recipient': {'type': ['string', 'null']}, 'requester_id': {'type': ['integer', 'null']}, 'satisfaction_rating': {'anyOf': [{'type': ['object', 'null'], 'properties': {'assignee_id': {'type': ['integer', 'null']}, 'comment': {'type': ['string', 'null']}, 'created_at': {'type': ['string', 'null'], 'format': 'date-time'}, 'group_id': {'type': ['integer', 'null']}, 'id': {'type': ['integer', 'null']}, 'reason': {'type': ['string', 'null']}, 'reason_id': {'type': ['integer', 'null']}, 'requester_id': {'type': ['integer', 'null']}, 'score': {'type': ['string', 'null']}, 'ticket_id': {'type': ['integer', 'null']}, 'updated_at': {'type': ['string', 'null'], 'format': 'date-time'}, 'url': {'type': ['string', 'null']}}}, {'type': ['string', 'null']}]}, 'status': {'type': ['string', 'null']}, 'subject': {'type': ['string', 'null']}, 'submitter_id': {'type': ['integer', 'null']}, 'tags': {'type': ['array', 'null'], 'items': {'type': ['string', 'null']}}, 'ticket_form_id': {'type': ['integer', 'null']}, 'type': {'type': ['string', 'null']}, 'updated_at': {'type': ['string', 'null'], 'format': 'date-time'}, 'url': {'type': ['string', 'null']}, 'via': {'type': ['object', 'null'], 'properties': {'channel': {'type': ['string', 'null']}, 'source': {'type': ['object', 'null'], 'properties': {'from': {'type': ['object', 'null'], 'properties': {'address': {'type': ['string', 'null']}, 'name': {'type': ['string', 'null']}, 'subject': {'type': ['string', 'null']}, 'ticket_id': {'type': ['integer', 'null']}}}, 'rel': {'type': ['string', 'null']}, 'to': {'type': ['object', 'null'], 'properties': {'address': {'type': ['string', 'null']}, 'name': {'type': ['string', 'null']}}}}}}}}} 

This way, in line 76, self.schema['properties'] does't raise a key error. When doing json_scheam.simplify on the satisfaction_ratings schema the result is the following structure:

{'anyOf': [ {'type': ['object', 'null'],
             'properties': {'assignee_id': {'type': ['integer', 'null']}, 'comment': {'type': ['string', 'null']}, 'created_at': {'type': ['string', 'null'], 'format': 'date-time'}, 'group_id': {'type': ['integer', 'null']}, 'id': {'type': ['integer', 'null']}, 'reason': {'type': ['string', 'null']}, 'reason_id': {'type': ['integer', 'null']}, 'requester_id': {'type': ['integer', 'null']}, 'score': {'type': ['string', 'null']}, 'ticket_id': {'type': ['integer', 'null']}, 'updated_at': {'type': ['string', 'null'], 'format': 'date-time'}, 'url': {'type': ['string', 'null']}}}, {'type': ['string', 'null']}]} 

In this case, on line 77 we get the KeyError. When changing line 70 to self.schema = json_schema.simplify(schema)['anyOf][0] it works fine, but now the rest of the streams fail. Adding a try/except seems to work fine.

[zopf]

I've been experiencing this same problem lately

[AlexanderMann]

@zopf @j-oflaherty do you have a link to these schemas? This definitely seems like a bug 🐛 we should be handling better for everyone. Additionally if you just have the work ready to merge, I'd be happy to entertain a pr contribution!

(sorry for the delays in response, work work has been really busy as of late)

[j-oflaherty]

@AlexanderMann I was able to make it work implementing a try and except to catch the error. I don't know if it's the best solution, but it hasn't given any errors so far...

I only added this on the file singer_stream.py on line 75:

try:
    properties = self.schema['properties']
except KeyError:
    self.schema = self.schema['anyOf'][0]
    properties = self.schema['properties']

I don't know what you mean by a link to the schemas. If it's catalogs you are refering to I just get it with the discover function from the zendesk tap.