bundle-kubeflow
bundle-kubeflow copied to clipboard
canonical
Test the UATs for the 1.9 release on Microk8s
Context
The UAT tests should be run on any new kubeflow bundle prior to release
What needs to get done
- execute the UATs on the kubeflow 1.9 release
Definition of Done
- UATs are passing for kubeflow 1.9 release
Thank you for reporting us your feedback!
The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5276.
This message was autogenerated
![syncronize-issues-to-jira[bot] avatar](https://avatars.githubusercontent.com/in/308591?v=4 "Published by a pub.dev verified publisher"){kind=small}
UATs in 1.9/beta
Identified issues:
- https://github.com/canonical/katib-operators/issues/211
- https://github.com/canonical/kubeflow-dashboard-operator/issues/197#issuecomment-2221204138
I ran UATs with tox -ve kubeflow-local, these are the results:
============================================================================ short test summary info =============================================================================
FAILED driver/test_kubeflow_workloads.py::test_kubeflow_workloads - Failed: Something went wrong while running Job test-kubeflow/test-kubeflow. Please inspect the attached logs for more info...
==================================================================== 1 failed, 1 passed in 1988.01s (0:33:08) ====================================================================
kubeflow-local: exit 1 (1989.54 seconds) /home/ubuntu/charmed-kubeflow-uats> pytest -vv --tb native /home/ubuntu/charmed-kubeflow-uats/driver/ -s --filter 'not mlflow' --model kubeflow pid=591250
kubeflow-local: FAIL code 1 (1989.59=setup[0.05]+cmd[1989.54] seconds)
evaluation failed :( (1989.68 seconds)
According to the logs, the Katib integration test is failing:
------------------------------ Captured log call -------------------------------
INFO test_notebooks:test_notebooks.py:44 Running katib-integration.ipynb...
ERROR test_notebooks:test_notebooks.py:58 Cell In[8], line 8, in assert_experiment_succeeded(client, experiment)
1 @retry(
2 wait=wait_exponential(multiplier=2, min=1, max=10),
3 stop=stop_after_attempt(30),
4 reraise=True,
5 )
6 def assert_experiment_succeeded(client, experiment):
7 """Wait for the Katib Experiment to complete successfully."""
----> 8 assert client.is_experiment_succeeded(name=experiment), f"Katib Experiment was not successful."
AssertionError: Katib Experiment was not successful.
=========================== short test summary info ============================
FAILED test_notebooks.py::test_notebook[katib-integration] - Failed: AssertionError: Katib Experiment was not successful.
============ 1 failed, 4 passed, 4 deselected in 1940.13s (0:32:20) ============
FAILED
------------------------------------------------------------------------------- live log teardown --------------------------------------------------------------------------------
INFO test_kubeflow_workloads:test_kubeflow_workloads.py:82 Deleting Profile test-kubeflow...
INFO httpx:_client.py:1013 HTTP Request: DELETE https://172.31.15.25:16443/apis/kubeflow.org/v1/profiles/test-kubeflow "HTTP/1.1 200 OK"
INFO test_kubeflow_workloads:test_kubeflow_workloads.py:141 Deleting Job test-kubeflow/test-kubeflow...
INFO httpx:_client.py:1013 HTTP Request: DELETE https://172.31.15.25:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
Looking a bit more into the logs, I can see the following:
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-8.2.2, pluggy-1.5.0 -- /opt/conda/bin/python3.8
cachedir: .pytest_cache
rootdir: /tests
configfile: pytest.ini
plugins: anyio-3.6.2
collecting ... collected 9 items / 4 deselected / 5 selected
test_notebooks.py::test_notebook[katib-integration]
-------------------------------- live log call ---------------------------------
INFO test_notebooks:test_notebooks.py:44 Running katib-integration.ipynb...
ERROR test_notebooks:test_notebooks.py:58 Cell In[8], line 8, in assert_experiment_succeeded(client, experiment)
1 @retry(
2 wait=wait_exponential(multiplier=2, min=1, max=10),
3 stop=stop_after_attempt(30),
4 reraise=True,
5 )
6 def assert_experiment_succeeded(client, experiment):
7 """Wait for the Katib Experiment to complete successfully."""
----> 8 assert client.is_experiment_succeeded(name=experiment), f"Katib Experiment was not successful."
AssertionError: Katib Experiment was not successful.
FAILED [ 20%]
test_notebooks.py::test_notebook[kfp-v1-integration]
-------------------------------- live log call ---------------------------------
INFO test_notebooks:test_notebooks.py:44 Running kfp-v1-integration.ipynb...
PASSED [ 40%]
test_notebooks.py::test_notebook[kfp-v2-integration]
-------------------------------- live log call ---------------------------------
INFO test_notebooks:test_notebooks.py:44 Running kfp-v2-integration.ipynb...
PASSED [ 60%]
test_notebooks.py::test_notebook[kserve-integration]
-------------------------------- live log call ---------------------------------
INFO test_notebooks:test_notebooks.py:44 Running kserve-integration.ipynb...
PASSED [ 80%]
test_notebooks.py::test_notebook[training-integration]
-------------------------------- live log call ---------------------------------
INFO test_notebooks:test_notebooks.py:44 Running training-integration.ipynb...
PASSED [100%]
=================================== FAILURES ===================================
_______________________ test_notebook[katib-integration] _______________________
test_notebook = '/tests/notebooks/katib/katib-integration.ipynb'
@pytest.mark.ipynb
@pytest.mark.parametrize(
# notebook - ipynb file to execute
"test_notebook",
NOTEBOOKS.values(),
ids=NOTEBOOKS.keys(),
)
def test_notebook(test_notebook):
"""Test Notebook Generic Wrapper."""
os.chdir(os.path.dirname(test_notebook))
with open(test_notebook) as nb:
notebook = nbformat.read(nb, as_version=nbformat.NO_CONVERT)
ep = ExecutePreprocessor(
timeout=-1, kernel_name="python3", on_notebook_start=install_python_requirements
)
ep.skip_cells_with_tag = "pytest-skip"
try:
log.info(f"Running {os.path.basename(test_notebook)}...")
output_notebook, _ = ep.preprocess(notebook, {"metadata": {"path": "./"}})
# persist the notebook output to the original file for debugging purposes
save_notebook(output_notebook, test_notebook)
except CellExecutionError as e:
# handle underlying error
pytest.fail(f"Notebook execution failed with {e.ename}: {e.evalue}")
for cell in output_notebook.cells:
metadata = cell.get("metadata", dict)
if "raises-exception" in metadata.get("tags", []):
for cell_output in cell.outputs:
if cell_output.output_type == "error":
# extract the error message from the cell output
log.error(format_error_message(cell_output.traceback))
> pytest.fail(cell_output.traceback[-1])
E Failed: AssertionError: Katib Experiment was not successful.
/tests/test_notebooks.py:59: Failed
Preliminary tests for beta
- Deployed
juju kubeflow --channel 1.9/beta --trust - Configured
dex-authandoidc-gatekeeper'spublic-url = http://dex-auth.kubeflow.svc:5556 - Configured
dex-auth'sstatic-usernameandstatic-password - Waited for about 10 minutes and checked the status of the model:
ubuntu@ip-172-31-15-25:~$ juju status
Model Controller Cloud/Region Version SLA Timestamp
kubeflow uk8s-343 microk8s/localhost 3.4.4 unsupported 20:23:35Z
App Version Status Scale Charm Channel Rev Address Exposed Message
admission-webhook active 1 admission-webhook latest/beta 328 10.152.183.124 no
argo-controller active 1 argo-controller latest/beta 526 10.152.183.183 no
dex-auth active 1 dex-auth latest/beta 507 10.152.183.141 no
envoy active 1 envoy latest/beta 231 10.152.183.126 no
istio-ingressgateway active 1 istio-gateway latest/beta 1048 10.152.183.69 no
istio-pilot active 1 istio-pilot latest/beta 1013 10.152.183.23 no
jupyter-controller active 1 jupyter-controller latest/beta 1002 10.152.183.175 no
jupyter-ui active 1 jupyter-ui latest/beta 925 10.152.183.41 no
katib-controller active 1 katib-controller latest/beta 690 10.152.183.35 no
katib-db 8.0.36-0ubuntu0.22.04.1 active 1 mysql-k8s 8.0/stable 153 10.152.183.129 no
katib-db-manager active 1 katib-db-manager latest/beta 653 10.152.183.50 no
katib-ui active 1 katib-ui latest/beta 657 10.152.183.217 no
kfp-api active 1 kfp-api latest/beta 1466 10.152.183.91 no
kfp-db 8.0.36-0ubuntu0.22.04.1 active 1 mysql-k8s 8.0/stable 153 10.152.183.80 no
kfp-metadata-writer active 1 kfp-metadata-writer latest/beta 524 10.152.183.59 no
kfp-persistence active 1 kfp-persistence latest/beta 1473 10.152.183.42 no
kfp-profile-controller active 1 kfp-profile-controller latest/beta 1431 10.152.183.130 no
kfp-schedwf active 1 kfp-schedwf latest/beta 1484 10.152.183.180 no
kfp-ui active 1 kfp-ui latest/beta 1467 10.152.183.229 no
kfp-viewer active 1 kfp-viewer latest/beta 1499 10.152.183.77 no
kfp-viz active 1 kfp-viz latest/beta 1417 10.152.183.219 no
knative-eventing active 1 knative-eventing latest/beta 441 10.152.183.111 no
knative-operator active 1 knative-operator latest/beta 416 10.152.183.134 no
knative-serving active 1 knative-serving latest/beta 442 10.152.183.75 no
kserve-controller active 1 kserve-controller latest/beta 397 10.152.183.132 no
kubeflow-dashboard active 1 kubeflow-dashboard latest/beta 600 10.152.183.32 no
kubeflow-profiles active 1 kubeflow-profiles latest/beta 393 10.152.183.221 no
kubeflow-roles active 1 kubeflow-roles latest/beta 225 10.152.183.150 no
kubeflow-volumes active 1 kubeflow-volumes latest/beta 314 10.152.183.28 no
metacontroller-operator active 1 metacontroller-operator latest/beta 280 10.152.183.61 no
minio res:oci-image@5102166 active 1 minio latest/beta 334 10.152.183.21 no
mlmd active 1 mlmd latest/beta 201 10.152.183.197 no
oidc-gatekeeper active 1 oidc-gatekeeper latest/beta 396 10.152.183.43 no
pvcviewer-operator active 1 pvcviewer-operator latest/beta 118 10.152.183.253 no
seldon-controller-manager active 1 seldon-core latest/beta 691 10.152.183.236 no
tensorboard-controller active 1 tensorboard-controller latest/beta 307 10.152.183.18 no
tensorboards-web-app active 1 tensorboards-web-app latest/beta 295 10.152.183.211 no
training-operator active 1 training-operator latest/beta 483 10.152.183.215 no
- Using the LB tried logging into the dashboard:
- I was able to log in and navigate the dashboard (all components seem to be working)
- I tried creating a notebook, connect to it and use it - it works
- I tried creating a Pipelines experiment, create a run and a recurring run - it works
- Looked into volumes and using the pvc viewer, I was able to navigate directories
Preliminary tests indicate the 1.9/beta bundle works just fine.
I have run the UATs on a Kubeflow deployment from latest/edge using the driver on a VM with the following versions:
- Ubuntu 22.04
- microk8s v1.29.5
- Juju 3.4.4
The final result is 1 successful and 1 failed test, here are the complete logs from the test run.
It seems that is an incompatibility during the installation of the packages specified in requirements.txt. This is an excerpt of the logs:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
kfp 2.8.0 requires kubernetes<27,>=8.0.0, but you have kubernetes 30.1.0 which is incompatible.
kfp 2.8.0 requires protobuf<5,>=4.21.1, but you have protobuf 3.20.3 which is incompatible.
kfp 2.8.0 requires urllib3<2.0.0, but you have urllib3 2.2.2 which is incompatible.
kubeflow-katib 0.17.0 requires protobuf<5,>=4.21.12, but you have protobuf 3.20.3 which is incompatible.
kfp-pipeline-spec 0.3.0 requires protobuf<5,>=4.21.1, but you have protobuf 3.20.3 which is incompatible.
After doing some research, I see that upstream kserve has pinned protobuf<4.0, as seen here. This creates an incompatibility with the versions kfp, kubeflow-katib, and kfp-pipeline-spec that are used in Kubeflow 1.9.
After further examination, the package conflicts are not actually showstoppers: The jupyter-scipy:v1.9.0 image has specific versions of the packages kfp, kubeflow-katib, and kfp-pipeline-spec preinstalled (among others). When a .ipynb test installs a different version of the dependencies (like kubernetes), an error is displayed to the stderr. However, those packages are not used in that test, and the tests do finish successfully. When we do want to use kfp, kubeflow-katib, and kfp-pipeline-spec, we specify the correct version of the dependencies, so no conflicts are actually created.
This is the preinstalled packages and their version from a newly created jupyter-scipy:v1.9.0 image:
(base) jovyan@test-image:~$ pip list
Package Version
------------------------- --------------
altair 5.3.0
anyio 4.4.0
archspec 0.2.3
argon2-cffi 23.1.0
argon2-cffi-bindings 21.2.0
arrow 1.3.0
asttokens 2.4.1
async-lru 2.0.4
attrs 23.2.0
Babel 2.14.0
beautifulsoup4 4.12.3
bleach 6.1.0
bokeh 3.3.4
boltons 24.0.0
Bottleneck 1.4.0
Brotli 1.1.0
cached-property 1.5.2
cachetools 5.3.3
certifi 2024.6.2
cffi 1.16.0
charset-normalizer 3.3.2
click 8.1.7
cloudpickle 3.0.0
colorama 0.4.6
comm 0.2.2
conda 24.5.0
conda-libmamba-solver 24.1.0
conda-package-handling 2.3.0
conda_package_streaming 0.10.0
contourpy 1.2.1
cycler 0.12.1
cytoolz 0.12.3
dask 2024.5.1
dask-expr 1.1.1
debugpy 1.8.2
decorator 5.1.1
defusedxml 0.7.1
dill 0.3.8
distributed 2024.5.1
distro 1.9.0
docstring_parser 0.16
entrypoints 0.4
et-xmlfile 1.1.0
exceptiongroup 1.2.0
executing 2.0.1
fastjsonschema 2.20.0
fonttools 4.53.0
fqdn 1.5.1
frozendict 2.4.4
fsspec 2024.6.1
gitdb 4.0.11
GitPython 3.1.43
gmpy2 2.1.5
google-api-core 2.19.1
google-auth 2.31.0
google-cloud-core 2.4.1
google-cloud-storage 2.17.0
google-crc32c 1.5.0
google-resumable-media 2.7.1
googleapis-common-protos 1.63.2
greenlet 3.0.3
h11 0.14.0
h2 4.1.0
h5py 3.11.0
hpack 4.0.0
httpcore 1.0.5
httpx 0.27.0
hyperframe 6.0.1
idna 3.7
imagecodecs 2024.6.1
imageio 2.34.2
importlib_metadata 8.0.0
importlib_resources 6.4.0
ipykernel 6.29.5
ipympl 0.9.4
ipython 8.26.0
ipython_genutils 0.2.0
ipywidgets 8.1.3
isoduration 20.11.0
jedi 0.19.1
Jinja2 3.1.4
joblib 1.4.2
json5 0.9.25
jsonpatch 1.33
jsonpointer 3.0.0
jsonschema 4.22.0
jsonschema-specifications 2023.12.1
jupyter_client 8.6.2
jupyter_core 5.7.2
jupyter-events 0.10.0
jupyter-lsp 2.2.5
jupyter_server 2.14.1
jupyter-server-mathjax 0.2.6
jupyter_server_terminals 0.5.3
jupyterlab 4.2.1
jupyterlab_git 0.50.1
jupyterlab_pygments 0.3.0
jupyterlab_server 2.27.2
jupyterlab_widgets 3.0.11
kfp 2.7.0
kfp-pipeline-spec 0.3.0
kfp-server-api 2.0.5
kiwisolver 1.4.5
kubernetes 26.1.0
lazy_loader 0.4
libmambapy 1.5.8
llvmlite 0.43.0
locket 1.0.0
lz4 4.3.3
mamba 1.5.8
MarkupSafe 2.1.5
matplotlib 3.8.4
matplotlib-inline 0.1.7
menuinst 2.1.1
mistune 3.0.2
mpmath 1.3.0
msgpack 1.0.8
munkres 1.1.4
nbclient 0.10.0
nbconvert 7.16.4
nbdime 4.0.1
nbformat 5.10.4
nest_asyncio 1.6.0
networkx 3.3
notebook 7.2.0
notebook_shim 0.2.4
numba 0.60.0
numexpr 2.10.0
numpy 1.26.4
oauthlib 3.2.2
openpyxl 3.1.4
overrides 7.7.0
packaging 24.1
pandas 2.1.4
pandocfilters 1.5.0
parso 0.8.4
partd 1.4.2
patsy 0.5.6
pexpect 4.9.0
pickleshare 0.7.5
pillow 10.4.0
pip 24.0
pkgutil_resolve_name 1.3.10
platformdirs 4.2.2
pluggy 1.5.0
ply 3.11
prometheus_client 0.20.0
prompt_toolkit 3.0.47
proto-plus 1.24.0
protobuf 4.25.3
psutil 6.0.0
ptyprocess 0.7.0
pure-eval 0.2.2
py-cpuinfo 9.0.0
pyarrow 16.1.0
pyarrow-hotfix 0.6
pyasn1 0.6.0
pyasn1_modules 0.4.0
pycosat 0.6.6
pycparser 2.22
Pygments 2.18.0
pyparsing 3.1.2
PyQt5 5.15.9
PyQt5-sip 12.12.2
PySocks 1.7.1
python-dateutil 2.9.0
python-json-logger 2.0.7
pytz 2024.1
PyWavelets 1.6.0
PyYAML 6.0.1
pyzmq 26.0.3
referencing 0.35.1
requests 2.32.3
requests-oauthlib 2.0.0
requests-toolbelt 0.10.1
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rpds-py 0.18.1
rsa 4.9
ruamel.yaml 0.18.6
ruamel.yaml.clib 0.2.8
scikit-image 0.22.0
scikit-learn 1.3.2
scipy 1.11.3
seaborn 0.13.2
Send2Trash 1.8.3
setuptools 70.1.1
sip 6.7.12
six 1.16.0
smmap 5.0.1
sniffio 1.3.1
sortedcontainers 2.4.0
soupsieve 2.5
SQLAlchemy 2.0.30
stack-data 0.6.2
statsmodels 0.14.2
sympy 1.12.1
tables 3.9.2
tabulate 0.9.0
tblib 3.0.0
terminado 0.18.1
threadpoolctl 3.5.0
tifffile 2024.7.2
tinycss2 1.3.0
toml 0.10.2
tomli 2.0.1
toolz 0.12.1
tornado 6.4.1
tqdm 4.66.4
traitlets 5.14.3
truststore 0.8.0
types-python-dateutil 2.9.0.20240316
typing_extensions 4.12.2
typing-utils 0.1.0
tzdata 2024.1
uri-template 1.3.0
urllib3 1.26.19
vincent 0.4.4
wcwidth 0.2.13
webcolors 24.6.0
webencodings 0.5.1
websocket-client 1.8.0
wheel 0.43.0
widgetsnbextension 4.0.11
xlrd 2.0.1
xyzservices 2024.6.0
zict 3.0.0
zipp 3.19.2
zstandard 0.22.0
After manually running the tests one by one on a JupyterLab notebook, I got these logs from the kserve-integration.ipynb:
kubectl describe pods -n admin sklearn-iris-predictor-00001-deployment-8d64bfddc-c2r7p
Name: sklearn-iris-predictor-00001-deployment-8d64bfddc-c2r7p
Namespace: admin
Priority: 0
Service Account: default
Node: <none>
Labels: app=sklearn-iris-predictor-00001
component=predictor
pod-template-hash=8d64bfddc
service.istio.io/canonical-name=sklearn-iris-predictor
service.istio.io/canonical-revision=sklearn-iris-predictor-00001
serving.knative.dev/configuration=sklearn-iris-predictor
serving.knative.dev/configurationGeneration=1
serving.knative.dev/configurationUID=7b7e5f61-803e-4a18-b2e8-e94f1f580e6f
serving.knative.dev/revision=sklearn-iris-predictor-00001
serving.knative.dev/revisionUID=7c6a2d83-01b3-4b36-a1b6-7dee7bac9852
serving.knative.dev/service=sklearn-iris-predictor
serving.knative.dev/serviceUID=5dfb37d9-5f32-46a5-929e-a57b94fbddc3
serving.kserve.io/inferenceservice=sklearn-iris
Annotations: autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
autoscaling.knative.dev/min-scale: 1
internal.serving.kserve.io/storage-initializer-sourceuri: gs://kfserving-examples/models/sklearn/1.0/model
prometheus.kserve.io/path: /metrics
prometheus.kserve.io/port: 8080
serving.knative.dev/creator: system:serviceaccount:kubeflow:kserve-controller
serving.kserve.io/enable-metric-aggregation: false
serving.kserve.io/enable-prometheus-scraping: false
sidecar.istio.io/inject: false
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/sklearn-iris-predictor-00001-deployment-8d64bfddc
Init Containers:
storage-initializer:
Image: charmedkubeflow/storage-initializer:0.13.0-70e4564
Port: <none>
Host Port: <none>
Args:
gs://kfserving-examples/models/sklearn/1.0/model
/mnt/models
Limits:
cpu: 1
memory: 1Gi
Requests:
cpu: 100m
memory: 100Mi
Environment: <none>
Mounts:
/mnt/models from kserve-provision-location (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-87rsq (ro)
Containers:
kserve-container:
Image: index.docker.io/charmedkubeflow/sklearnserver@sha256:851d519f718692eff76f815e3bcda062a2ec219d57e59903bc0d0ee3264e27a4
Port: 8080/TCP
Host Port: 0/TCP
Args:
--model_name=sklearn-iris
--model_dir=/mnt/models
--http_port=8080
Limits:
cpu: 1
memory: 2Gi
Requests:
cpu: 1
memory: 2Gi
Environment:
PORT: 8080
K_REVISION: sklearn-iris-predictor-00001
K_CONFIGURATION: sklearn-iris-predictor
K_SERVICE: sklearn-iris-predictor
Mounts:
/mnt/models from kserve-provision-location (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-87rsq (ro)
queue-proxy:
Image: gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:89e6f90141f1b63405883fbb4de0d3b6d80f8b77e530904c4d29bdcd1dc5a167
Ports: 8022/TCP, 9090/TCP, 9091/TCP, 8012/TCP, 8112/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
SeccompProfile: RuntimeDefault
Requests:
cpu: 25m
Readiness: http-get http://:8012/ delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
SERVING_NAMESPACE: admin
SERVING_SERVICE: sklearn-iris-predictor
SERVING_CONFIGURATION: sklearn-iris-predictor
SERVING_REVISION: sklearn-iris-predictor-00001
QUEUE_SERVING_PORT: 8012
QUEUE_SERVING_TLS_PORT: 8112
CONTAINER_CONCURRENCY: 0
REVISION_TIMEOUT_SECONDS: 300
REVISION_RESPONSE_START_TIMEOUT_SECONDS: 0
REVISION_IDLE_TIMEOUT_SECONDS: 0
SERVING_POD: sklearn-iris-predictor-00001-deployment-8d64bfddc-c2r7p (v1:metadata.name)
SERVING_POD_IP: (v1:status.podIP)
SERVING_LOGGING_CONFIG:
SERVING_LOGGING_LEVEL:
SERVING_REQUEST_LOG_TEMPLATE: {"httpRequest": {"requestMethod": "{{.Request.Method}}", "requestUrl": "{{js .Request.RequestURI}}", "requestSize": "{{.Request.ContentLength}}", "status": {{.Response.Code}}, "responseSize": "{{.Response.Size}}", "userAgent": "{{js .Request.UserAgent}}", "remoteIp": "{{js .Request.RemoteAddr}}", "serverIp": "{{.Revision.PodIP}}", "referer": "{{js .Request.Referer}}", "latency": "{{.Response.Latency}}s", "protocol": "{{.Request.Proto}}"}, "traceId": "{{index .Request.Header "X-B3-Traceid"}}"}
SERVING_ENABLE_REQUEST_LOG: false
SERVING_REQUEST_METRICS_BACKEND: prometheus
SERVING_REQUEST_METRICS_REPORTING_PERIOD_SECONDS: 5
TRACING_CONFIG_BACKEND: none
TRACING_CONFIG_ZIPKIN_ENDPOINT:
TRACING_CONFIG_DEBUG: false
TRACING_CONFIG_SAMPLE_RATE: 0.1
USER_PORT: 8080
SYSTEM_NAMESPACE: knative-serving
METRICS_DOMAIN: knative.dev/internal/serving
SERVING_READINESS_PROBE: {"tcpSocket":{"port":8080,"host":"127.0.0.1"},"successThreshold":1}
ENABLE_PROFILING: false
SERVING_ENABLE_PROBE_REQUEST_LOG: false
METRICS_COLLECTOR_ADDRESS:
HOST_IP: (v1:status.hostIP)
ENABLE_HTTP2_AUTO_DETECTION: false
ROOT_CA:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-87rsq (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
kube-api-access-87rsq:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
kserve-provision-location:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 26s (x2 over 5m52s) default-scheduler 0/1 nodes are available: 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
Insufficient memory (on a local VM with 16GiB of memory) seemed to be the the culprit. Testing the UATs on a t2.2xlarge instance on AWS was successful. Here are the logs:
kubeflow-local: find interpreter for spec PythonSpec(path=/home/ubuntu/py38/bin/python3.8)
kubeflow-local: proposed PythonInfo(spec=CPython3.8.19.final.0-64, system=/usr/bin/python3.8, exe=/home/ubuntu/py38/bin/python3.8, platform=linux, version='3.8.19 (default, Apr 6 2024, 17:58:10) \n[GCC 11.4.0]', encoding_fs_io=utf-8-utf-8)
kubeflow-local: commands[0]> pytest -vv --tb native /home/ubuntu/charmed-kubeflow-uats/driver/ -s --filter 'not mlflow' --model kubeflow
=========================================== test session starts ===========================================
platform linux -- Python 3.8.19, pytest-7.4.3, pluggy-1.3.0 -- /home/ubuntu/charmed-kubeflow-uats/.tox/kubeflow-local/bin/python
cachedir: .tox/kubeflow-local/.pytest_cache
rootdir: /home/ubuntu/charmed-kubeflow-uats
configfile: pyproject.toml
plugins: asyncio-0.21.1, operator-0.31.0, anyio-4.0.0
asyncio: mode=strict
collected 2 items
driver/test_kubeflow_workloads.py::test_create_profile
--------------------------------------------- live log setup ----------------------------------------------
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
INFO test_kubeflow_workloads:test_kubeflow_workloads.py:69 Creating Profile test-kubeflow...
INFO httpx:_client.py:1013 HTTP Request: POST https://127.0.0.1:16443/apis/kubeflow.org/v1/profiles "HTTP/1.1 201 Created"
---------------------------------------------- live log call ----------------------------------------------
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/kubeflow.org/v1/profiles/test-kubeflow "HTTP/1.1 200 OK"
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/api/v1/namespaces/test-kubeflow "HTTP/1.1 404 Not Found"
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/api/v1/namespaces/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:32 Waiting for namespace test-kubeflow to become 'Active': phase == Active
PASSED
driver/test_kubeflow_workloads.py::test_kubeflow_workloads
---------------------------------------------- live log call ----------------------------------------------
INFO test_kubeflow_workloads:test_kubeflow_workloads.py:110 Starting Kubernetes Job test-kubeflow/test-kubeflow to run notebook tests...
INFO httpx:_client.py:1013 HTTP Request: POST https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs "HTTP/1.1 201 Created"
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == not ready)
INFO utils:utils.py:40 Retrying in 2 seconds (attempts: 1)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 4 seconds (attempts: 2)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 8 seconds (attempts: 3)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 16 seconds (attempts: 4)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 5)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 6)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 7)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 8)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 9)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 10)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 11)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 12)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 13)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 14)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 15)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 16)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 17)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 18)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 19)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 20)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 21)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 22)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 23)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 24)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 25)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 26)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 27)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 28)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 29)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 30)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 31)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 32)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 33)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 34)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 35)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 36)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 37)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 38)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 39)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 40)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 41)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 42)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 43)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 44)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 45)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 46)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 47)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 48)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 49)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 50)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 51)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 52)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 53)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 54)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 55)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 56)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 57)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 58)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 59)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 60)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 61)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 62)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO utils:utils.py:40 Retrying in 32 seconds (attempts: 63)
INFO httpx:_client.py:1013 HTTP Request: GET https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO utils:utils.py:69 Job test-kubeflow/test-kubeflow completed successfully!
INFO test_kubeflow_workloads:test_kubeflow_workloads.py:135 Fetching Job logs...
##### test-kubeflow container logs #####
============================= test session starts ==============================
platform linux -- Python 3.11.9, pytest-8.3.2, pluggy-1.5.0 -- /opt/conda/bin/python3.11
cachedir: .pytest_cache
rootdir: /tests
configfile: pytest.ini
plugins: anyio-4.4.0
collecting ... collected 9 items / 4 deselected / 5 selected
test_notebooks.py::test_notebook[katib-integration]
-------------------------------- live log call ---------------------------------
INFO test_notebooks:test_notebooks.py:44 Running katib-integration.ipynb...
PASSED [ 20%]
test_notebooks.py::test_notebook[kfp-v1-integration]
-------------------------------- live log call ---------------------------------
INFO test_notebooks:test_notebooks.py:44 Running kfp-v1-integration.ipynb...
PASSED [ 40%]
test_notebooks.py::test_notebook[kfp-v2-integration]
-------------------------------- live log call ---------------------------------
INFO test_notebooks:test_notebooks.py:44 Running kfp-v2-integration.ipynb...
PASSED [ 60%]
test_notebooks.py::test_notebook[kserve-integration]
-------------------------------- live log call ---------------------------------
INFO test_notebooks:test_notebooks.py:44 Running kserve-integration.ipynb...
PASSED [ 80%]
test_notebooks.py::test_notebook[training-integration]
-------------------------------- live log call ---------------------------------
INFO test_notebooks:test_notebooks.py:44 Running training-integration.ipynb...
PASSED [100%]
================= 5 passed, 4 deselected in 1903.56s (0:31:43) =================
PASSED
-------------------------------------------- live log teardown --------------------------------------------
INFO test_kubeflow_workloads:test_kubeflow_workloads.py:82 Deleting Profile test-kubeflow...
INFO httpx:_client.py:1013 HTTP Request: DELETE https://127.0.0.1:16443/apis/kubeflow.org/v1/profiles/test-kubeflow "HTTP/1.1 200 OK"
INFO test_kubeflow_workloads:test_kubeflow_workloads.py:141 Deleting Job test-kubeflow/test-kubeflow...
INFO httpx:_client.py:1013 HTTP Request: DELETE https://127.0.0.1:16443/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
===================================== 2 passed in 1923.72s (0:32:03) ======================================
kubeflow-local: exit 0 (1924.75 seconds) /home/ubuntu/charmed-kubeflow-uats> pytest -vv --tb native /home/ubuntu/charmed-kubeflow-uats/driver/ -s --filter 'not mlflow' --model kubeflow pid=577857
kubeflow-local: OK (1924.80=setup[0.05]+cmd[1924.75] seconds)
congratulations :) (1924.89 seconds)