clearml-server icon indicating copy to clipboard operation
clearml-server copied to clipboard

Can't access agent-services

Open nikneural opened this issue 2 years ago • 8 comments
trafficstars

% Total % Received % Xferd Average Speed Time Time Time Current 2023-01-18 16:41:55 Dload Upload Total Spent Left Speed 100 294 100 294 0 0 49000 0 --:--:-- --:--:-- --:--:-- 49000 2023-01-18 16:41:55 http://fileserver:8081 http://webserver:8080 http://clearml-apiserver:8008 2023-01-18 16:41:56 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv 2023-01-18 16:41:56 {"meta":{"id":"7ae388d14fe1460f8a4117abc23f2868","trx":"7ae388d14fe1460f8a4117abc23f2868","endpoint":{"name":"debug.ping","requested_version":"2.23","actual_version":"1.0"},"result_code":200,"result_subcode":0,"result_msg":"OK","error_stack":"","error_data":{}},"data":{"msg":"ClearML server"}}Using environment access key CLEARML_API_ACCESS_KEY=RTLEDYVSE68275W4FUGY 2023-01-18 16:41:56 Using environment secret key CLEARML_API_SECRET_KEY=******** 2023-01-18 16:41:56 2023-01-18 16:41:56 clearml_agent: ERROR: Failed getting token (error 401 from http://clearml-apiserver:8008): Unauthorized (invalid credentials) (failed to locate provided credentials)

It is my agent-services in docker-compose.yml:

`agent-services: networks: - backend container_name: clearml-agent-services image: allegroai/clearml-agent-services:latest deploy: restart_policy: condition: on-failure privileged: true environment: CLEARML_HOST_IP: ${CLEARML_HOST_IP} CLEARML_WEB_HOST: http://webserver:8080 CLEARML_API_HOST: http://clearml-apiserver:8008 CLEARML_FILES_HOST: http://fileserver:8081 CLEARML_API_ACCESS_KEY: "***********" CLEARML_API_SECRET_KEY: "****************" CLEARML_AGENT_GIT_USER: ${CLEARML_AGENT_GIT_USER} CLEARML_AGENT_GIT_PASS: ${CLEARML_AGENT_GIT_PASS} CLEARML_AGENT_UPDATE_VERSION: ${CLEARML_AGENT_UPDATE_VERSION:->=0.17.0} CLEARML_AGENT_DEFAULT_BASE_DOCKER: "ubuntu:18.04" AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID:-} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY:-} AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION:-} AZURE_STORAGE_ACCOUNT: ${AZURE_STORAGE_ACCOUNT:-} AZURE_STORAGE_KEY: ${AZURE_STORAGE_KEY:-} GOOGLE_APPLICATION_CREDENTIALS: ${GOOGLE_APPLICATION_CREDENTIALS:-} CLEARML_WORKER_ID: "clearml-services" CLEARML_AGENT_DOCKER_HOST_MOUNT: "/opt/clearml/agent:/root/.clearml" SHUTDOWN_IF_NO_ACCESS_KEY: 1 volumes: - /var/run/docker.sock:/var/run/docker.sock - /opt/clearml/agent:/root/.clearml depends_on: - apiserver entrypoint: > bash -c "curl --retry 10 --retry-delay 10 --retry-connrefused 'http://apiserver:8008/debug.ping' && /usr/agent/entrypoint.sh"

networks: backend: driver: bridge frontend: driver: bridge `

nikneural avatar Jan 18 '23 13:01 nikneural

Did you configure CLEARML_API_ACCESS_KEY and CLEARML_API_SECRET_KEY using key/secret you generated in the WebApp Profile section?

jkhenning avatar Jan 18 '23 15:01 jkhenning

Did you configure CLEARML_API_ACCESS_KEY and CLEARML_API_SECRET_KEY using key/secret you generated in the WebApp Profile section?

Yeap

nikneural avatar Jan 18 '23 15:01 nikneural

And the access key showing in the log matches the one you specified?

jkhenning avatar Jan 18 '23 15:01 jkhenning

And the access key showing in the log matches the one you specified?

Yes, I see the key that I passed

nikneural avatar Jan 18 '23 15:01 nikneural

OK, can you try issuing this command using the same credentials? curl -u "<key>:<secret>" http://clearml-apiserver:8008/auth.login

Make sure you use the correct apiserver address.

jkhenning avatar Jan 19 '23 06:01 jkhenning

OK, can you try issuing this command using the same credentials? curl -u "<key>:<secret>" http://clearml-apiserver:8008/auth.login

Make sure you use the correct apiserver address.

I got this:

{"meta":{"id":"1a7a7406920249f0b2fd69cf6cc921cb","trx":"1a7a7406920249f0b2fd69cf6cc921cb","endpoint":{"name":"auth.login","requested_version":"2.23","actual_version":"1.0"},"result_code":200,"result_subcode":0,"result_msg":"OK","error_stack":"","error_data":{}},"data":{"token":"BIG TOKEN"}

nikneural avatar Jan 19 '23 17:01 nikneural

I'm having the same problem. @jkhenning what do you mean by webapp section?

lambdaofgod avatar Feb 20 '23 16:02 lambdaofgod

I meant where did you take the key and secret from 🙂

jkhenning avatar Feb 20 '23 18:02 jkhenning