latest seems to be not working right now

[washburnjoe]

Deploying AWX via operator for the past few days is failing on quay.io/ansible/awx-ee:latest Getting an ImagePullBackOff. I am getting the issue from multiple locations.

I noticed that latest is double the size version 0.6.0. The default awx-demo deployment in the operator install instructions try and pull latest

https://github.com/ansible/awx-operator

[Wildeone1]

Confirm also seeing same ImagePullBackOff when trying to use quay.io/ansible/awx-ee:latest.

Last had a successful run on 2022-12-13 11:54 GMT when pulling latest, whatever version that latest constituted at quay.io/ansible/awx-ee:latest at the time.

Consider this another vote towards needing more versioned releases than just 0.6.0 and latest.

[akus062381]

@washburnjoe, thank you for your submission. Is this something you are still running up against today?

FYI, we are also tracking the tagging of AWX-EE images here: https://github.com/ansible/awx-ee/issues/125

[shrutebattlestargalactica]

@akus062381 I am currently seeing issues with the latest image. I am on the latest AWX version and run AWX-operator in EKS k8s version 1.24. I use an external Postgres db in RDS.

here is a trace back when running an inventory sync job:

"error", "traceback": "Traceback (most recent call last):\n File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py", line 589, in run\n res = receptor_job.run()\n File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py", line 319, in run\n res = self._run_internal(receptor_ctl)\n File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py", line 394, in _run_internal\n res = processor_future.result()\n File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 446, in result\n return self.__get_result()\n File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result\n raise self._exception\n File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/utils/common.py", line 1214, in wrapper_cleanup_new_process\n return func(*args, **kwargs)\n File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py", line 453, in processor\n return ansible_runner.interface.run(\n File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/interface.py", line 210, in run\n r.run()\n File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/streaming.py", line 254, in run\n self.artifacts_callback(data)\n File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/streaming.py", line 233, in artifacts_callback\n unstream_dir(self._input, length, self.artifact_dir)\n File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/utils/streaming.py", line 71, in unstream_dir\n data = source.read(chunk_size)\n File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/utils/base64io.py", line 282, in read\n results.write(base64.b64decode(data))\n File "/usr/lib64/python3.9/base64.py", line 87, in b64decode\n return binascii.a2b_base64(s)\nbinascii.Error: Invalid base64-encoded string: number of data characters (246137) cannot be 1 more than a multiple of 4\n" }

[shrutebattlestargalactica]

@shanemcd or @fosterseth Hi there - any chance you can help us with the awx-ee:latest image? I have also just tried using the newest image as well 21.11.0 and still seeing these python errors - here is another one:

"started": "2023-01-24T04:10:54.029713+00:00", "finished": "2023-01-24T04:11:03.662866+00:00", "status": "error", "traceback": "Traceback (most recent call last):\n File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py", line 589, in run\n res = receptor_job.run()\n File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py", line 319, in run\n res = self._run_internal(receptor_ctl)\n File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py", line 394, in _run_internal\n res = processor_future.result()\n File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 446, in result\n return self.__get_result()\n File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result\n raise self._exception\n File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/utils/common.py", line 1214, in wrapper_cleanup_new_process\n return func(*args, **kwargs)\n File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py", line 453, in processor\n return ansible_runner.interface.run(\n File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/interface.py", line 210, in run\n r.run()\n File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/streaming.py", line 254, in run\n self.artifacts_callback(data)\n File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/streaming.py", line 233, in artifacts_callback\n unstream_dir(self._input, length, self.artifact_dir)\n File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/utils/streaming.py", line 71, in unstream_dir\n data = source.read(chunk_size)\n File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/utils/base64io.py", line 282, in read\n results.write(base64.b64decode(data))\n File "/usr/lib64/python3.9/base64.py", line 87, in b64decode\n return binascii.a2b_base64(s)\nbinascii.Error: Incorrect padding\n" } thanks for your help

[leetcarey]

How did you resolve this, also getting error