agent-lightning AgentOps: Authentication failed: Authentication failed: 504

I’m encountering an authentication issue when running the sql_agent.py script in the examples/spider directory. The error message is: AgentOps: Authentication failed: Authentication failed: 504 Has anyone else encountered this issue before? Any advice on how to resolve it would be greatly appreciated.

Aug 13 '25 11:08 yangdongdong2000

How did you run it? Detailed traceback please.

Aug 13 '25 16:08 ultmaster

How did you run the script? Please provide the command.

Aug 25 '25 09:08 ultmaster

This error occurs when I running the spider example, i followed the instruction to start ray cluster by running the script ../../scripts/restart_ray.sh, then when i start running sql_agent.py, the terminal start to show the following error: [ERROR DETAIL] (lightning-env) root@poissongpu008:/home/user1/project/agent-lightning/examples/spider# VERL_API_BASE=http://localhost:9999/ python sql_agent.py 2025-08-27 11:10:43,680 [INFO] (Process-5449 agentlightning.config) Instantiating LitSQLAgent with args: {'trained_agents': 'write', 'val_temperature': None, 'max_turns': 3, 'table_info_truncate': 2048, 'execution_truncate': 2048} 2025-08-27 11:10:43,680 [INFO] (Process-5449 agentlightning.config) Instantiating Trainer with args: {'dev': False, 'n_workers': 1, 'max_tasks': None, 'daemon': True, 'tracer': None, 'triplet_exporter': None} 2025-08-27 11:10:43,680 [INFO] (Process-5449 agentlightning.instrumentation.agentops) AgentOpsServerManager initialized. 2025-08-27 11:10:44,050 [WARNING] (Process-5449 main) Environment variable OPENAI_API_BASE is not set. Using default value 'https://api.openai.com/v1'. 2025-08-27 11:10:44,052 [INFO] (Process-5449 agentlightning.trainer) Initializing Trainer... 2025-08-27 11:10:44,052 [INFO] (Process-5449 agentlightning.trainer) Initializing AgentLightningClient with endpoint: http://localhost:9999/ 2025-08-27 11:10:44,052 [INFO] (Process-5449 agentlightning.instrumentation.agentops) Starting AgentOps local server on port 48119... 2025-08-27 11:10:44,060 [INFO] (Process-5449 agentlightning.instrumentation.agentops) AgentOps local server process (PID: 5519) started, targeting port 48119.

Serving Flask app 'agentlightning.instrumentation.agentops'
Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
Running on http://127.0.0.1:48119 Press CTRL+C to quit 2025-08-27 11:10:44,561 [INFO] (Process-5449 agentlightning.trainer) Trainer main initialization complete. 2025-08-27 11:10:44,561 [INFO] (Process-5449 agentlightning.trainer) Running with n_workers=1 (synchronous in main process). 2025-08-27 11:10:44,561 [INFO] (Process-5449 agentlightning.trainer) [Worker 0] Setting up trainer environment... 2025-08-27 11:10:44,561 [INFO] (Process-5449 agentlightning.tracer.agentops) [Worker 0] Setting up tracer... 2025-08-27 11:10:44,888 [INFO] (Process-5449 agentlightning.instrumentation.agentops) Patched newer version of agentops using handle_chat_attributes /opt/conda/envs/lightning-env/lib/python3.11/site-packages/agentlightning/instrumentation/init.py:53: UserWarning: litellm is not installed. It's therefore not instrumented. warnings.warn("litellm is not installed. It's therefore not instrumented.") /opt/conda/envs/lightning-env/lib/python3.11/site-packages/agentlightning/instrumentation/init.py:60: UserWarning: vllm is not installed. It's therefore not instrumented. warnings.warn("vllm is not installed. It's therefore not instrumented.") 2025-08-27 11:10:44,888 [INFO] (Process-5449 agentlightning.tracer.agentops) [Worker 0] Instrumentation applied. 2025-08-27 11:10:44,888 [INFO] (Process-5449 agentlightning.tracer.agentops) [Worker 0] Env var set: AGENTOPS_API_KEY=dummy 2025-08-27 11:10:44,888 [INFO] (Process-5449 agentlightning.tracer.agentops) [Worker 0] Env var set: AGENTOPS_API_ENDPOINT=http://localhost:48119 2025-08-27 11:10:44,888 [INFO] (Process-5449 agentlightning.tracer.agentops) [Worker 0] Env var set: AGENTOPS_APP_URL=http://localhost:48119/notavailable 2025-08-27 11:10:44,888 [INFO] (Process-5449 agentlightning.tracer.agentops) [Worker 0] Env var set: AGENTOPS_EXPORTER_ENDPOINT=http://localhost:48119/traces 🖇 AgentOps: Authentication failed: 504 - Perhaps an invalid API key? 🖇 AgentOps: Authentication failed: Authentication failed: 504 2025-08-27 11:10:55,435 [ERROR] (Process-5449 agentlightning.trainer) Unhandled exception in fit method. Traceback (most recent call last): File "/opt/conda/envs/lightning-env/lib/python3.11/site-packages/agentlightning/trainer.py", line 247, in fit num_tasks = self._worker_main_loop(agent, 0, is_async) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/lightning-env/lib/python3.11/site-packages/agentlightning/trainer.py", line 168, in _worker_main_loop self._initialize_worker_env(worker_id) File "/opt/conda/envs/lightning-env/lib/python3.11/site-packages/agentlightning/trainer.py", line 199, in _initialize_worker_env self.tracer.init_worker(worker_id) File "/opt/conda/envs/lightning-env/lib/python3.11/site-packages/agentlightning/tracer/agentops.py", line 125, in init_worker agentops.init() File "/opt/conda/envs/lightning-env/lib/python3.11/site-packages/agentops/init.py", line 173, in init return client.init(**init_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/lightning-env/lib/python3.11/site-packages/agentops/client/client.py", line 105, in init response = self.api.v3.fetch_auth_token(self.config.api_key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/lightning-env/lib/python3.11/site-packages/agentops/client/api/versions/v3.py", line 43, in fetch_auth_token raise ApiServerException(error_msg) agentops.exceptions.ApiServerException: Authentication failed: 504 2025-08-27 11:10:55,439 [INFO] (Process-5449 agentlightning.trainer) Cleaning up Trainer... 2025-08-27 11:10:55,439 [INFO] (Process-5449 agentlightning.instrumentation.agentops) Stopping AgentOps local server (PID: 5519)... 2025-08-27 11:10:55,453 [INFO] (Process-5449 agentlightning.instrumentation.agentops) AgentOps local server stopped. 2025-08-27 11:10:55,453 [INFO] (Process-5449 agentlightning.tracer.agentops) AgentOps server stopped. 2025-08-27 11:10:55,453 [INFO] (Process-5449 agentlightning.trainer) Trainer main cleanup complete.

[PACKADES DETAIL ] Package Version

accelerate 1.10.1 agentlightning 0.1.2 agentops 0.4.18 aiohappyeyeballs 2.6.1 aiohttp 3.12.15 aiohttp-cors 0.8.1 aiosignal 1.4.0 annotated-types 0.7.0 antlr4-python3-runtime 4.9.3 anyio 4.10.0 astor 0.8.1 attrs 25.3.0 autogen-agentchat 0.7.4 autogen-core 0.7.4 autogen-ext 0.7.4 blake3 1.0.5 blinker 1.9.0 cachetools 5.5.2 cbor2 5.7.0 certifi 2025.8.3 cffi 1.17.1 charset-normalizer 3.4.3 click 8.2.1 cloudpickle 3.1.1 codetiming 1.4.0 colorful 0.5.7 compressed-tensors 0.10.2 cupy-cuda12x 13.6.0 dataclasses-json 0.6.7 datasets 4.0.0 depyf 0.19.0 dill 0.3.8 diskcache 5.6.3 distlib 0.4.0 distro 1.9.0 dnspython 2.7.0 einops 0.8.1 email_validator 2.2.0 fastapi 0.116.1 fastapi-cli 0.0.8 fastapi-cloud-cli 0.1.5 fastrlock 0.8.3 filelock 3.19.1 flash_attn 2.8.3 Flask 3.1.2 frozenlist 1.7.0 fsspec 2025.3.0 gguf 0.17.1 gitdb 4.0.12 GitPython 3.1.45 google-api-core 2.25.1 google-auth 2.40.3 googleapis-common-protos 1.70.0 graphviz 0.21 greenlet 3.2.4 grpcio 1.74.0 h11 0.16.0 hf-xet 1.1.8 httpcore 1.0.9 httpdbg 1.2.6 httptools 0.6.4 httpx 0.28.1 httpx-sse 0.4.1 huggingface-hub 0.34.4 hydra-core 1.3.2 idna 3.10 importlib_metadata 8.7.0 interegular 0.3.3 itsdangerous 2.2.0 Jinja2 3.1.6 jiter 0.10.0 jsonpatch 1.33 jsonpointer 3.0.0 jsonref 1.1.0 jsonschema 4.25.1 jsonschema-specifications 2025.4.1 langchain 0.3.27 langchain-community 0.3.28 langchain-core 0.3.75 langchain-text-splitters 0.3.9 langgraph 0.6.6 langgraph-checkpoint 2.1.1 langgraph-prebuilt 0.6.4 langgraph-sdk 0.2.3 langsmith 0.4.18 lark 1.2.2 llguidance 0.7.30 llvmlite 0.44.0 lm-format-enforcer 0.10.12 markdown-it-py 4.0.0 MarkupSafe 3.0.2 marshmallow 3.26.1 mcp 1.13.1 mdurl 0.1.2 mistral_common 1.8.4 mpmath 1.3.0 msgpack 1.1.1 msgspec 0.19.0 multidict 6.6.4 multiprocess 0.70.16 mypy_extensions 1.1.0 networkx 3.5 ninja 1.13.0 numba 0.61.2 numpy 1.26.4 nvidia-cublas-cu12 12.6.4.1 nvidia-cuda-cupti-cu12 12.6.80 nvidia-cuda-nvrtc-cu12 12.6.77 nvidia-cuda-runtime-cu12 12.6.77 nvidia-cudnn-cu12 9.5.1.17 nvidia-cufft-cu12 11.3.0.4 nvidia-cufile-cu12 1.11.1.6 nvidia-curand-cu12 10.3.7.77 nvidia-cusolver-cu12 11.7.1.2 nvidia-cusparse-cu12 12.5.4.2 nvidia-cusparselt-cu12 0.6.3 nvidia-nccl-cu12 2.26.2 nvidia-nvjitlink-cu12 12.6.85 nvidia-nvtx-cu12 12.6.77 omegaconf 2.3.0 openai 1.102.0 openai-harmony 0.0.4 opencensus 0.11.4 opencensus-context 0.1.3 opencv-python-headless 4.12.0.88 opentelemetry-api 1.36.0 opentelemetry-exporter-otlp-proto-common 1.36.0 opentelemetry-exporter-otlp-proto-http 1.36.0 opentelemetry-exporter-prometheus 0.57b0 opentelemetry-instrumentation 0.57b0 opentelemetry-proto 1.36.0 opentelemetry-sdk 1.36.0 opentelemetry-semantic-conventions 0.57b0 ordered-set 4.1.0 orjson 3.11.2 ormsgpack 1.10.0 outlines_core 0.2.10 packaging 24.2 pandas 2.3.2 partial-json-parser 0.2.1.1.post6 peft 0.17.1 pillow 11.3.0 pip 25.1 platformdirs 4.3.8 prometheus_client 0.22.1 prometheus-fastapi-instrumentator 7.1.0 propcache 0.3.2 proto-plus 1.26.1 protobuf 5.29.5 psutil 7.0.0 py-cpuinfo 9.0.0 py-spy 0.4.1 pyarrow 21.0.0 pyasn1 0.6.1 pyasn1_modules 0.4.2 pybase64 1.4.2 pybind11 3.0.1 pycountry 24.6.1 pycparser 2.22 pydantic 2.11.7 pydantic_core 2.33.2 pydantic-extra-types 2.10.5 pydantic-settings 2.10.1 Pygments 2.19.2 pylatexenc 2.10 python-dateutil 2.9.0.post0 python-dotenv 1.1.1 python-json-logger 3.3.0 python-multipart 0.0.20 pytz 2025.2 pyvers 0.1.0 PyYAML 6.0.2 pyzmq 27.0.2 ray 2.48.0 referencing 0.36.2 regex 2025.7.34 requests 2.32.5 requests-toolbelt 1.0.0 rich 14.1.0 rich-toolkit 0.15.0 rignore 0.6.4 rpds-py 0.27.0 rsa 4.9.1 safetensors 0.6.2 scipy 1.16.1 sentencepiece 0.2.1 sentry-sdk 2.35.0 setproctitle 1.3.6 setuptools 78.1.1 shellingham 1.5.4 six 1.17.0 smart_open 7.3.0.post1 smmap 5.0.2 sniffio 1.3.1 soundfile 0.13.1 soxr 0.5.0.post1 SQLAlchemy 2.0.43 sqlparse 0.5.3 sse-starlette 3.0.2 starlette 0.47.3 sympy 1.14.0 tenacity 9.1.2 tensordict 0.9.1 termcolor 2.4.0 tiktoken 0.11.0 tokenizers 0.21.4 torch 2.7.1 torchaudio 2.7.1 torchdata 0.11.0 torchvision 0.22.1 tqdm 4.67.1 transformers 4.55.4 triton 3.3.1 typer 0.16.1 typing_extensions 4.14.1 typing-inspect 0.9.0 typing-inspection 0.4.1 tzdata 2025.2 urllib3 2.5.0 uvicorn 0.35.0 uvloop 0.21.0 verl 0.5.0 virtualenv 20.34.0 vllm 0.10.1.1 wandb 0.21.1 watchfiles 1.1.0 websockets 15.0.1 Werkzeug 3.1.3 wheel 0.45.1 wrapt 1.17.3 xformers 0.0.31 xgrammar 0.1.21 xxhash 3.5.0 yarl 1.20.1 zipp 3.23.0 zstandard 0.24.0

[PYTHON VERSION] 3.11.13

Aug 27 '25 03:08 yangdongdong2000

I think the agentops is not properly configured even after setting the environment varaibles. Not sure why.

Could you get into agentops.client.Client here and print what self.config looks like?

https://github.com/AgentOps-AI/agentops/blob/817f554936431dddfc2ae75b30ad252812d63628/agentops/client/client.py#L151

Aug 29 '25 06:08 ultmaster

Thanks for reply. I upgraded the agentops package from 0.4.18 to 0.4.20 to address this issue. However, I noticed that the pyproject.toml file specifies agentops<=0.4.18, which is quite confusing.

Aug 29 '25 08:08 yangdongdong2000

agentops break a lot of times when upgrading. So we pin it to avoid unexpected upgrade. Will upgrade the pin.

Aug 31 '25 01:08 ultmaster