airbyte
airbyte copied to clipboard
airbytehq
[source-appsflyer] host name issue
Connector Name
source-appsflyer
Connector Version
0.1.0
What step the error happened?
During the sync
Relevant information
while setting up the source as appsflyer i am receiving hostname error
Relevant log output
2024-04-18 12:56:04 [32mINFO[m i.a.w.t.TemporalAttemptExecution(get):142 - Docker volume job log path: /tmp/workspace/633ce658-9d26-47fa-8505-01c0c1bf2fec/0/logs.log
2024-04-18 12:56:04 [32mINFO[m i.a.w.t.TemporalAttemptExecution(get):147 - Executing worker wrapper. Airbyte version: 0.42.1
2024-04-18 12:56:04 [32mINFO[m i.a.a.c.AirbyteApiClient(retryWithJitter):179 - Attempt 0 to save workflow id for cancellation
2024-04-18 12:56:04 [32mINFO[m i.a.c.i.LineGobbler(voidCall):149 -
2024-04-18 12:56:04 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-04-18 12:56:04 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-04-18 12:56:04 [32mINFO[m i.a.c.i.LineGobbler(voidCall):149 - ----- START CHECK -----
2024-04-18 12:56:04 [32mINFO[m i.a.c.i.LineGobbler(voidCall):149 -
2024-04-18 12:56:04 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-04-18 12:56:04 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-04-18 12:56:04 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-04-18 12:56:04 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2024-04-18 12:56:04 [32mINFO[m i.a.c.i.LineGobbler(voidCall):149 - Checking if airbyte/source-appsflyer:0.1.0 exists...
2024-04-18 12:56:04 [32mINFO[m i.a.c.i.LineGobbler(voidCall):149 - airbyte/source-appsflyer:0.1.0 was found locally.
2024-04-18 12:56:04 [32mINFO[m i.a.w.p.DockerProcessFactory(create):130 - Creating docker container = source-appsflyer-check-633ce658-9d26-47fa-8505-01c0c1bf2fec-0-jkxlf with resources io.airbyte.config.ResourceRequirements@30222420[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] and allowedHosts null
2024-04-18 12:56:04 [32mINFO[m i.a.w.p.DockerProcessFactory(create):175 - Preparing command: docker run --rm --init -i -w /data/633ce658-9d26-47fa-8505-01c0c1bf2fec/0 --log-driver none --name source-appsflyer-check-633ce658-9d26-47fa-8505-01c0c1bf2fec-0-jkxlf --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-appsflyer:0.1.0 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e USE_STREAM_CAPABLE_STATE=true -e FIELD_SELECTION_WORKSPACES= -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e FEATURE_FLAG_CLIENT= -e AIRBYTE_VERSION=0.42.1 -e WORKER_JOB_ID=633ce658-9d26-47fa-8505-01c0c1bf2fec airbyte/source-appsflyer:0.1.0 check --config source_config.json
2024-04-18 12:56:04 [32mINFO[m i.a.w.i.VersionedAirbyteStreamFactory(create):110 - Reading messages from protocol version 0.2.0
2024-04-18 12:56:06 [1;31mERROR[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):190 - Check failed
2024-04-18 12:56:06 [32mINFO[m i.a.w.g.DefaultCheckConnectionWorker(run):120 - Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@39fef777[status=failed,message=ConnectionError(MaxRetryError("HTTPSConnectionPool(host='hq.appsflyer.com', port=443): Max retries exceeded with url: /export/id******86/partners_report/v5?api_token=********************KHqg&from=2024-04-18&to=2024-04-18&timezone=UTC (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f26fb1c3fd0>: Failed to establish a new connection: [Errno -2] Name does not resolve'))"))]
2024-04-18 12:56:06 [32mINFO[m i.a.w.t.TemporalAttemptExecution(get):169 - Stopping cancellation check scheduling...
2024-04-18 12:56:06 [32mINFO[m i.a.c.i.LineGobbler(voidCall):149 -
2024-04-18 12:56:06 [32mINFO[m i.a.c.i.LineGobbler(voidCall):149 - ----- END CHECK -----
2024-04-18 12:56:06 [32mINFO[m i.a.c.i.LineGobbler(voidCall):149 -
Contribute
- [ ] Yes, I want to contribute
Yeah, source-appsflyer needs some cleanup. It's a community source, so we won't be able to get to it and fix it quickly. It looks like Appsflyer might've deprecated hq.appsflyer.com.
Looks like the new host is simply hq1.appsflyer.com, so it looks like an easy fix?
Also there is some specific problem with Organic-in-app-events stream, I changed all urls to a new version in connector and when tested other endpoints work fine but Organic-in-app-events stream always raises an error, what might be a cause of this error, shouldn't it work as fine as In-app-events?
Encountered an exception while reading stream organic_in_app_events
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 129, in read
stream_is_available, reason = stream_instance.check_availability(logger, self)
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/core.py", line 325, in check_availability
return self.availability_strategy.check_availability(self, logger, source)
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/availability_strategy.py", line 56, in check_availability
is_available, reason = self.handle_http_error(stream, logger, source, error)
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/availability_strategy.py", line 85, in handle_http_error
raise error
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/availability_strategy.py", line 50, in check_availability
get_first_record_for_slice(stream, stream_slice)
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/utils/stream_helper.py", line 40, in get_first_record_for_slice
return next(records_for_slice)
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 482, in read_records
yield from self._read_pages(
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 498, in _read_pages
request, response = self._fetch_next_page(stream_slice, stream_state, next_page_token)
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 524, in _fetch_next_page
response = self._send_request(request, request_kwargs)
File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 422, in _send_request
return backoff_handler(user_backoff_handler)(request, request_kwargs)
File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry
ret = target(*args, **kwargs)
