kserve-controller fails to start after upgrading from Charmed Kubeflow 1.6 to 1.7

[ColmBhandal]

Needs reproduction. Based off Joshua's Discourse post.

After upgrading from Charmed Kubeflow 1.6 to 1.7, the kserve-controller fails to start and reports this error:

ERROR juju.worker.uniter.operation runhook.go:153 hook “install” (via hook dispatching script: dispatch) failed: exit status 1
The web interface can still be accessed. However, when users attempt to log in, a CSRF check failed error is displayed.

Also, this error may occur if the login form is opened in multiple tabs: Please attempt to log in again.

Here is the kserve-controller log:

2023-06-13T13:27:12.906Z [container-agent] 
2023-06-13 13:27:12 INFO juju.worker.uniter resolver.go:155 awaiting error resolution for “install” hook 
2023-06-13T13:27:17.340Z [pebble] Check “readiness” failure 1320 (threshold 3): received non-20x status code 418 
2023-06-13T13:27:27.342Z [pebble] Check “readiness” failure 1321 (threshold 3): received non-20x status code 418 
2023-06-13T13:27:37.343Z [pebble] Check “readiness” failure 1322 (threshold 3): received non-20x status code 418 
2023-06-13T13:27:47.343Z [pebble] Check “readiness” failure 1323 (threshold 3): received non-20x status code 418 
2023-06-13T13:27:57.342Z [pebble] Check “readiness” failure 1324 (threshold 3): received non-20x status code 418 
2023-06-13T13:28:07.343Z [pebble] Check “readiness” failure 1325 (threshold 3): received non-20x status code 418