sdk
sdk copied to clipboard
Published
20 hours ago •
meltano
meltano
bug: RESTStream not retrying in TimeoutError
Singer SDK Version
0.39
Is this a regression?
- [ ] Yes
Python Version
3.9
Bug scope
Taps (catalog, state, etc.)
Operating System
Linux
Description
We have a RESTStream that eventually fails due to a timeout error, but the SDK does not retry it with proper backoff (you can see it by the timestamp of the logs). The timeout error is unhandled. Is there any additional configuration needed in the tap code?
Here are the logs:
Logs
| @timestamp | event |
|---|---|
| 2024-09-03 22:31:13.687 | Environment 'prod' is active |
| 2024-09-03 22:31:14.190 | kind: password is deprecated for setting definitions in favour of sensitive: true, and is currently in use by the following settings of loader 'target-postgres': 'password'. Please open an issue or pull request to update the plugin definition on Meltano Hub at https://github.com/meltano/hub/blob/main/_data/meltano/loaders/target-postgres/transferwise.yml. |
| 2024-09-03 22:31:14.737 | Running extract & load... |
| 2024-09-03 22:31:15.075 | No state was found, complete import. |
| 2024-09-03 22:31:19.364 | 2024-09-03 22:31:19,363 | INFO |
| 2024-09-03 22:31:19.365 | 2024-09-03 22:31:19,363 | INFO |
| 2024-09-03 22:31:19.460 | time=2024-09-03 22:31:19 name=target_postgres level=INFO message=Table '"chamados"' exists |
| 2024-09-03 22:49:04.011 | 2024-09-03 22:49:04,009 | INFO |
| 2024-09-03 22:49:04.011 | 2024-09-03 22:49:04,010 | INFO |
| 2024-09-03 22:49:04.013 | 2024-09-03 22:49:04,010 | INFO |
| 2024-09-03 22:49:04.024 | 2024-09-03 22:49:04,010 | ERROR |
| 2024-09-03 22:49:04.025 | Traceback (most recent call last): |
| 2024-09-03 22:49:04.026 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 716, in urlopen |
| 2024-09-03 22:49:04.026 | httplib_response = self._make_request( |
| 2024-09-03 22:49:04.026 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 404, in _make_request |
| 2024-09-03 22:49:04.028 | self._validate_conn(conn) |
| 2024-09-03 22:49:04.029 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1061, in _validate_conn |
| 2024-09-03 22:49:04.030 | conn.connect() |
| 2024-09-03 22:49:04.031 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/connection.py", line 419, in connect |
| 2024-09-03 22:49:04.034 | self.sock = ssl_wrap_socket( |
| 2024-09-03 22:49:04.035 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 458, in ssl_wrap_socket |
| 2024-09-03 22:49:04.035 | ssl_sock = _ssl_wrap_socket_impl( |
| 2024-09-03 22:49:04.035 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 502, in _ssl_wrap_socket_impl |
| 2024-09-03 22:49:04.036 | return ssl_context.wrap_socket(sock, server_hostname=server_hostname) |
| 2024-09-03 22:49:04.036 | File "/usr/local/lib/python3.9/ssl.py", line 501, in wrap_socket |
| 2024-09-03 22:49:04.037 | return self.sslsocket_class._create( |
| 2024-09-03 22:49:04.037 | File "/usr/local/lib/python3.9/ssl.py", line 1074, in _create |
| 2024-09-03 22:49:04.038 | self.do_handshake() |
| 2024-09-03 22:49:04.038 | File "/usr/local/lib/python3.9/ssl.py", line 1343, in do_handshake |
| 2024-09-03 22:49:04.039 | self._sslobj.do_handshake() |
| 2024-09-03 22:49:04.039 | TimeoutError: [Errno 110] Connection timed out |
| 2024-09-03 22:49:04.040 | |
| 2024-09-03 22:49:04.040 | During handling of the above exception, another exception occurred: |
| 2024-09-03 22:49:04.040 | |
| 2024-09-03 22:49:04.040 | Traceback (most recent call last): |
| 2024-09-03 22:49:04.041 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/requests/adapters.py", line 667, in send |
| 2024-09-03 22:49:04.041 | resp = conn.urlopen( |
| 2024-09-03 22:49:04.041 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 802, in urlopen |
| 2024-09-03 22:49:04.042 | retries = retries.increment( |
| 2024-09-03 22:49:04.043 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/util/retry.py", line 552, in increment |
| 2024-09-03 22:49:04.044 | raise six.reraise(type(error), error, _stacktrace) |
| 2024-09-03 22:49:04.045 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/packages/six.py", line 769, in reraise |
| 2024-09-03 22:49:04.045 | raise value.with_traceback(tb) |
| 2024-09-03 22:49:04.050 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 716, in urlopen |
| 2024-09-03 22:49:04.051 | httplib_response = self._make_request( |
| 2024-09-03 22:49:04.052 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 404, in _make_request |
| 2024-09-03 22:49:04.053 | self._validate_conn(conn) |
| 2024-09-03 22:49:04.054 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1061, in _validate_conn |
| 2024-09-03 22:49:04.056 | conn.connect() |
| 2024-09-03 22:49:04.057 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/connection.py", line 419, in connect |
| 2024-09-03 22:49:04.058 | self.sock = ssl_wrap_socket( |
| 2024-09-03 22:49:04.059 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 458, in ssl_wrap_socket |
| 2024-09-03 22:49:04.059 | ssl_sock = _ssl_wrap_socket_impl( |
| 2024-09-03 22:49:04.060 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 502, in _ssl_wrap_socket_impl |
| 2024-09-03 22:49:04.060 | return ssl_context.wrap_socket(sock, server_hostname=server_hostname) |
| 2024-09-03 22:49:04.061 | File "/usr/local/lib/python3.9/ssl.py", line 501, in wrap_socket |
| 2024-09-03 22:49:04.061 | return self.sslsocket_class._create( |
| 2024-09-03 22:49:04.062 | File "/usr/local/lib/python3.9/ssl.py", line 1074, in _create |
| 2024-09-03 22:49:04.062 | self.do_handshake() |
| 2024-09-03 22:49:04.065 | File "/usr/local/lib/python3.9/ssl.py", line 1343, in do_handshake |
| 2024-09-03 22:49:04.068 | self._sslobj.do_handshake() |
| 2024-09-03 22:49:04.071 | urllib3.exceptions.ProtocolError: ('Connection aborted.', TimeoutError(110, 'Connection timed out')) |
| 2024-09-03 22:49:04.072 | |
| 2024-09-03 22:49:04.074 | During handling of the above exception, another exception occurred: |
| 2024-09-03 22:49:04.074 | |
| 2024-09-03 22:49:04.074 | Traceback (most recent call last): |
| 2024-09-03 22:49:04.076 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 1203, in sync |
| 2024-09-03 22:49:04.077 | for _ in self._sync_records(context=context): |
| 2024-09-03 22:49:04.079 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 1097, in _sync_records |
| 2024-09-03 22:49:04.079 | for idx, record_result in enumerate(self.get_records(current_context)): |
| 2024-09-03 22:49:04.080 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/singer_sdk/streams/rest.py", line 585, in get_records |
| 2024-09-03 22:49:04.080 | for record in self.request_records(context): |
| 2024-09-03 22:49:04.080 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/singer_sdk/streams/rest.py", line 390, in request_records |
| 2024-09-03 22:49:04.081 | prepared_request = self.prepare_request( |
| 2024-09-03 22:49:04.089 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/singer_sdk/streams/rest.py", line 363, in prepare_request |
| 2024-09-03 22:49:04.090 | return self.build_prepared_request( |
| 2024-09-03 22:49:04.094 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/singer_sdk/streams/rest.py", line 335, in build_prepared_request |
| 2024-09-03 22:49:04.121 | return self.requests_session.prepare_request(request) |
| 2024-09-03 22:49:04.121 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/requests/sessions.py", line 484, in prepare_request |
| 2024-09-03 22:49:04.122 | p.prepare( |
| 2024-09-03 22:49:04.122 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/requests/models.py", line 371, in prepare |
| 2024-09-03 22:49:04.123 | self.prepare_auth(auth, url) |
| 2024-09-03 22:49:04.123 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/requests/models.py", line 602, in prepare_auth |
| 2024-09-03 22:49:04.124 | r = auth(self) |
| 2024-09-03 22:49:04.124 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/singer_sdk/authenticators.py", line 155, in call |
| 2024-09-03 22:49:04.125 | return self.authenticate_request(r) |
| 2024-09-03 22:49:04.125 | File "/project/plugins/extractors/tap-americanet/tap_americanet/authenticator.py", line 121, in authenticate_request |
| 2024-09-03 22:49:04.126 | self.update_access_token() |
| 2024-09-03 22:49:04.126 | File "/project/plugins/extractors/tap-americanet/tap_americanet/authenticator.py", line 97, in update_access_token |
| 2024-09-03 22:49:04.126 | token_response = requests.post(self.auth_endpoint, json=auth_request_payload) |
| 2024-09-03 22:49:04.127 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/requests/api.py", line 115, in post |
| 2024-09-03 22:49:04.127 | return request("post", url, data=data, json=json, **kwargs) |
| 2024-09-03 22:49:04.128 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/requests/api.py", line 59, in request |
| 2024-09-03 22:49:04.129 | return session.request(method=method, url=url, **kwargs) |
| 2024-09-03 22:49:04.129 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/requests/sessions.py", line 589, in request |
| 2024-09-03 22:49:04.129 | resp = self.send(prep, **send_kwargs) |
| 2024-09-03 22:49:04.130 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/requests/sessions.py", line 703, in send |
| 2024-09-03 22:49:04.130 | r = adapter.send(request, **kwargs) |
| 2024-09-03 22:49:04.131 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/requests/adapters.py", line 682, in send |
| 2024-09-03 22:49:04.131 | raise ConnectionError(err, request=request) |
| 2024-09-03 22:49:04.131 | requests.exceptions.ConnectionError: ('Connection aborted.', TimeoutError(110, 'Connection timed out')) |
| 2024-09-03 22:49:04.132 | Traceback (most recent call last): |
| 2024-09-03 22:49:04.133 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 716, in urlopen |
| 2024-09-03 22:49:04.133 | httplib_response = self._make_request( |
| 2024-09-03 22:49:04.134 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 404, in _make_request |
| 2024-09-03 22:49:04.134 | self._validate_conn(conn) |
| 2024-09-03 22:49:04.135 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1061, in _validate_conn |
| 2024-09-03 22:49:04.135 | conn.connect() |
| 2024-09-03 22:49:04.135 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/connection.py", line 419, in connect |
| 2024-09-03 22:49:04.136 | self.sock = ssl_wrap_socket( |
| 2024-09-03 22:49:04.136 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 458, in ssl_wrap_socket |
| 2024-09-03 22:49:04.136 | ssl_sock = _ssl_wrap_socket_impl( |
| 2024-09-03 22:49:04.137 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 502, in _ssl_wrap_socket_impl |
| 2024-09-03 22:49:04.137 | return ssl_context.wrap_socket(sock, server_hostname=server_hostname) |
| 2024-09-03 22:49:04.137 | File "/usr/local/lib/python3.9/ssl.py", line 501, in wrap_socket |
| 2024-09-03 22:49:04.138 | return self.sslsocket_class._create( |
| 2024-09-03 22:49:04.138 | File "/usr/local/lib/python3.9/ssl.py", line 1074, in _create |
| 2024-09-03 22:49:04.139 | self.do_handshake() |
| 2024-09-03 22:49:04.139 | File "/usr/local/lib/python3.9/ssl.py", line 1343, in do_handshake |
| 2024-09-03 22:49:04.139 | self._sslobj.do_handshake() |
| 2024-09-03 22:49:04.140 | TimeoutError: [Errno 110] Connection timed out |
| 2024-09-03 22:49:04.140 | |
| 2024-09-03 22:49:04.140 | During handling of the above exception, another exception occurred: |
| 2024-09-03 22:49:04.141 | |
| 2024-09-03 22:49:04.141 | Traceback (most recent call last): |
| 2024-09-03 22:49:04.141 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/requests/adapters.py", line 667, in send |
| 2024-09-03 22:49:04.142 | resp = conn.urlopen( |
| 2024-09-03 22:49:04.142 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 802, in urlopen |
| 2024-09-03 22:49:04.142 | retries = retries.increment( |
| 2024-09-03 22:49:04.143 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/util/retry.py", line 552, in increment |
| 2024-09-03 22:49:04.143 | raise six.reraise(type(error), error, _stacktrace) |
| 2024-09-03 22:49:04.144 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/packages/six.py", line 769, in reraise |
| 2024-09-03 22:49:04.144 | raise value.with_traceback(tb) |
| 2024-09-03 22:49:04.144 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 716, in urlopen |
| 2024-09-03 22:49:04.145 | httplib_response = self._make_request( |
| 2024-09-03 22:49:04.145 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 404, in _make_request |
| 2024-09-03 22:49:04.145 | self._validate_conn(conn) |
| 2024-09-03 22:49:04.146 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1061, in _validate_conn |
| 2024-09-03 22:49:04.146 | conn.connect() |
| 2024-09-03 22:49:04.146 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/connection.py", line 419, in connect |
| 2024-09-03 22:49:04.147 | self.sock = ssl_wrap_socket( |
| 2024-09-03 22:49:04.147 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 458, in ssl_wrap_socket |
| 2024-09-03 22:49:04.153 | ssl_sock = _ssl_wrap_socket_impl( |
| 2024-09-03 22:49:04.153 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 502, in _ssl_wrap_socket_impl |
| 2024-09-03 22:49:04.154 | return ssl_context.wrap_socket(sock, server_hostname=server_hostname) |
| 2024-09-03 22:49:04.154 | File "/usr/local/lib/python3.9/ssl.py", line 501, in wrap_socket |
| 2024-09-03 22:49:04.155 | return self.sslsocket_class._create( |
| 2024-09-03 22:49:04.155 | File "/usr/local/lib/python3.9/ssl.py", line 1074, in _create |
| 2024-09-03 22:49:04.156 | self.do_handshake() |
| 2024-09-03 22:49:04.156 | File "/usr/local/lib/python3.9/ssl.py", line 1343, in do_handshake |
| 2024-09-03 22:49:04.157 | self._sslobj.do_handshake() |
| 2024-09-03 22:49:04.221 | urllib3.exceptions.ProtocolError: ('Connection aborted.', TimeoutError(110, 'Connection timed out')) |
| 2024-09-03 22:49:04.221 | |
| 2024-09-03 22:49:04.222 | During handling of the above exception, another exception occurred: |
| 2024-09-03 22:49:04.222 | |
| 2024-09-03 22:49:04.223 | Traceback (most recent call last): |
| 2024-09-03 22:49:04.223 | File "/project/.meltano/extractors/tap-americanet/venv/bin/tap-americanet", line 8, in |
| 2024-09-03 22:49:04.223 | sys.exit(Tapamericanet.cli()) |
| 2024-09-03 22:49:04.224 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/click/core.py", line 1157, in call |
| 2024-09-03 22:49:04.224 | return self.main(*args, **kwargs) |
| 2024-09-03 22:49:04.224 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/click/core.py", line 1078, in main |
| 2024-09-03 22:49:04.225 | rv = self.invoke(ctx) |
| 2024-09-03 22:49:04.226 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/singer_sdk/plugin_base.py", line 80, in invoke |
| 2024-09-03 22:49:04.226 | return super().invoke(ctx) |
| 2024-09-03 22:49:04.227 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/click/core.py", line 1434, in invoke |
| 2024-09-03 22:49:04.227 | return ctx.invoke(self.callback, **ctx.params) |
| 2024-09-03 22:49:04.228 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/click/core.py", line 783, in invoke |
| 2024-09-03 22:49:04.228 | return __callback(*args, **kwargs) |
| 2024-09-03 22:49:04.229 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 507, in invoke |
| 2024-09-03 22:49:04.229 | tap.sync_all() |
| 2024-09-03 22:49:04.229 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 466, in sync_all |
| 2024-09-03 22:49:04.230 | stream.sync() |
| 2024-09-03 22:49:04.230 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 1203, in sync |
| 2024-09-03 22:49:04.230 | for _ in self._sync_records(context=context): |
| 2024-09-03 22:49:04.231 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 1097, in _sync_records |
| 2024-09-03 22:49:04.231 | for idx, record_result in enumerate(self.get_records(current_context)): |
| 2024-09-03 22:49:04.234 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/singer_sdk/streams/rest.py", line 585, in get_records |
| 2024-09-03 22:49:04.234 | for record in self.request_records(context): |
| 2024-09-03 22:49:04.235 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/singer_sdk/streams/rest.py", line 390, in request_records |
| 2024-09-03 22:49:04.235 | prepared_request = self.prepare_request( |
| 2024-09-03 22:49:04.235 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/singer_sdk/streams/rest.py", line 363, in prepare_request |
| 2024-09-03 22:49:04.236 | return self.build_prepared_request( |
| 2024-09-03 22:49:04.237 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/singer_sdk/streams/rest.py", line 335, in build_prepared_request |
| 2024-09-03 22:49:04.237 | return self.requests_session.prepare_request(request) |
| 2024-09-03 22:49:04.238 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/requests/sessions.py", line 484, in prepare_request |
| 2024-09-03 22:49:04.239 | p.prepare( |
| 2024-09-03 22:49:04.239 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/requests/models.py", line 371, in prepare |
| 2024-09-03 22:49:04.240 | self.prepare_auth(auth, url) |
| 2024-09-03 22:49:04.240 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/requests/models.py", line 602, in prepare_auth |
| 2024-09-03 22:49:04.241 | r = auth(self) |
| 2024-09-03 22:49:04.241 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/singer_sdk/authenticators.py", line 155, in call |
| 2024-09-03 22:49:04.241 | return self.authenticate_request(r) |
| 2024-09-03 22:49:04.242 | File "/project/plugins/extractors/tap-americanet/tap_americanet/authenticator.py", line 121, in authenticate_request |
| 2024-09-03 22:49:04.242 | self.update_access_token() |
| 2024-09-03 22:49:04.243 | File "/project/plugins/extractors/tap-americanet/tap_americanet/authenticator.py", line 97, in update_access_token |
| 2024-09-03 22:49:04.243 | token_response = requests.post(self.auth_endpoint, json=auth_request_payload) |
| 2024-09-03 22:49:04.244 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/requests/api.py", line 115, in post |
| 2024-09-03 22:49:04.244 | return request("post", url, data=data, json=json, **kwargs) |
| 2024-09-03 22:49:04.245 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/requests/api.py", line 59, in request |
| 2024-09-03 22:49:04.245 | return session.request(method=method, url=url, **kwargs) |
| 2024-09-03 22:49:04.245 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/requests/sessions.py", line 589, in request |
| 2024-09-03 22:49:04.246 | resp = self.send(prep, **send_kwargs) |
| 2024-09-03 22:49:04.246 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/requests/sessions.py", line 703, in send |
| 2024-09-03 22:49:04.247 | r = adapter.send(request, **kwargs) |
| 2024-09-03 22:49:04.248 | File "/project/.meltano/extractors/tap-americanet/venv/lib/python3.9/site-packages/requests/adapters.py", line 682, in send |
| 2024-09-03 22:49:04.248 | raise ConnectionError(err, request=request) |
| 2024-09-03 22:49:04.249 | requests.exceptions.ConnectionError: ('Connection aborted.', TimeoutError(110, 'Connection timed out')) |
| 2024-09-03 22:49:04.555 | Incremental state has been updated at 2024-09-03 22:49:04.555135+00:00. |
| 2024-09-03 22:49:04.648 | Extraction failed |
| 2024-09-03 22:49:04.649 | ELT could not be completed: Extractor failed. |
| 2024-09-03 22:49:04.649 | For more detailed log messages re-run the command using 'meltano --log-level=debug ...' CLI flag. |
| 2024-09-03 22:49:04.649 | Note that you can also check the generated log file at '/project/.meltano/logs/elt/2024-09-03T223113--tap-americanet--target-postgres/28ece168-1b10-44d5-9a4a-04cd670e7dd6/elt.log'. |
| 2024-09-03 22:49:04.650 | For more information on debugging and logging: https://docs.meltano.com/reference/command-line-interface#debugging |
| 2024-09-03 22:49:04.677 | |
| 2024-09-03 22:49:04.677 | |
| 2024-09-03 22:49:04.677 | |
| 2024-09-03 22:49:04.677 | |
| 2024-09-03 22:49:04.677 | |
| 2024-09-03 22:49:04.677 |
Code
TAP:
class Tapamericanet(Tap):
"""americanet tap class."""
name = "tap-americanet"
config_jsonschema = th.PropertiesList(
th.Property(
"username",
th.StringType,
required=True,
description="Americanet username"
),
th.Property(
"password",
th.StringType,
required=True,
description="Americanet password"
),
th.Property(
"api_url",
th.StringType,
default="<default url here>",
description="The url for the API service"
),
).to_dict()
def discover_streams(self) -> List[Stream]:
"""Return a list of discovered streams."""
return [stream_class(tap=self) for stream_class in STREAM_TYPES]
Stream:
class ChamadosStream(americanetStream):
"""Chamados stream."""
name = "chamados"
primary_keys = ["Num_Protocolo"]
replication_key = None
schema_filepath = SCHEMAS_DIR / "chamados.json"
records_jsonpath = "$.Chamados[*]"
def prepare_request_payload(
self, context: Optional[dict], next_page_token: Optional[Any]
) -> Optional[dict]:
"""Prepare the data payload for the REST API request.
"""
now = datetime.now() - timedelta(hours=3) # now in UTC-3
payload: dict = {}
# payload config here:
return payload
Link to Slack/Linen
No response
