call back url error

[4estone]

hi all, Env: CKAN 2.9, os: Centos7 I got an error when pushinf file on datastore with xloader: `Express Load starting: /dataset/test-2-9/resource/d891053e-1291-431b-bd2a-6f4c390376d4

Fetching from: https://t-opendata67.alsace.eu/dataset/09802715-e21e-46b3-bd31-e433ab56f596/resource/d891053e-1291-431b-bd2a-6f4c390376d4/download/b_apa_dom_canton_2015.csv

URL error: HTTPSConnectionPool(host='t-opendata67.alsace.eu', port=443): Max retries exceeded with url: /dataset/09802715-e21e-46b3-bd31-e433ab56f596/resource/d891053e-1291-431b-bd2a-6f4c390376d4/download/b_apa_dom_canton_2015.csv (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4dd81bd828>: Failed to establish a new connection: [Errno 111] Connection refused',))

End of log` It's not an issue, but I realy want to get some clue or information about solving this problem.

Thanks Sylvain

[kowh-ai]

Does that hostname:port look right?

[viladesuso]

@4estone Hi!! We have de same problem. How did you solve it ??

Env. CKAN 2.9 Dockerized Xloader

"Express Load starting: /dataset/primerapruebarequests/resource/5cf85e8a-caa1-47bb-ab17-e4c4f4fcbe78" "Fetching from: https://ckan..com/dataset/c6dfdebe-6313-4f20-be77-6ef39775bc93/resource/5cf85e8a-caa1-47bb-ab17-e4c4f4fcbe78/download/****.csv" "Error: b"HTTPSConnectionPool(host='ckan..com', port=443): Max retries exceeded with url: /dataset/c6dfdebe-6313-4f20-be77-6ef39775bc93/resource/5cf85e8a-caa1-47bb-ab17-e4c4f4fcbe78/download/.csv (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fc5042eb250>: Failed to establish a new connection: [Errno -2] Name does not resolve')) status=None url=https://ckan..com/dataset/c6dfdebe-6313-4f20-be77-6ef39775bc93/resource/5cf85e8a-caa1-47bb-ab17-e4c4f4fcbe78/download/*****.csv response=None"

When we use a url with https(nginx - CKAN_SITE_URL="https://ckan.*******.com"), xloader can't download the file.

1. If we use a external URL with IP for upload the resource, xloader works fine.
2. If we use a external URL from other CKAN. Error is next: "CKANEXT__XLOADER__SSL_VERIFY=false" "Error: b"HTTPSConnectionPool(host='www.externalCkanURL.com', port=443): Max retries exceeded with url: /dataset/e4d87fcb-c224-4820-9e53-752fe7f5caf2/resource/06bea2fa-d016-4f3b-99ba-1f8a8551d0df/download/.csv (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)'))) status=None url=https://www.externalCkanURL.com/dataset/e4d87fcb-c224-4820-9e53-752fe7f5caf2/resource/06bea2fa-d016-4f3b-99ba-1f8a8551d0df/download/.csv response=None"

[ThrawnCA]

It sounds like your XLoader doesn't recognise the SSL certificate of the site you're downloading from?

[pcbimon]

i have to run on localhost but i have same issue HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /dataset/a346e00e-a6d4-4f93-9091-2d691b1dfc3c/resource/d248fdea-e49b-424a-9ef9-5ba911c6c738/download/book1test.xlsx (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffeb5da690>: Failed to establish a new connection: [Errno 111] Connection refused',

• i can download file from this url.

[ThrawnCA]

If you're running on localhost and still getting connection errors, that's definitely odd. Is your XLoader on the same machine?

[pcbimon]

@ThrawnCA yes. i have XLoader in same machine and run on docker compose container that include ckan nginx

[duttonw]

Hi @pcbimon is this from pypi or from github? Could you try the qld-gov-au fork, https://github.com/qld-gov-au/ckanext-xloader/tree/1.0.1-qgov.12 and see if you get the same results.

[ThrawnCA]

@ThrawnCA yes. i have XLoader in same machine and run on docker compose container that include ckan nginx

In Docker? Is the XLoader in the same container as the web application?

[pcbimon]

@duttonw i fixed by not use localhost because i have to build ckan in container with nginx container reverse proxy when. i use localhost in ckan it's not call nginx reverse proxy but call localhost ckan container that my fault.

[pcbimon]

@ThrawnCA if you stuck in same issue please check your web server app service that can call from external container.

[duttonw]

Sounds like you should change the ckan.url to the nginx box url/port to have it go through the correct routing. For non local dev's this should use the external dns domain maybe the cdn frontdoor for caching ability.

For example's that we run for github action testing, see: docker/test-OpenData.ini, docker-compose.yml and Dockerfile-template.ckan Do note that this uses .ahoy.yml to wrap things but these could easily be shell scripts or github action calls.