kubeblocks
kubeblocks copied to clipboard
Published
20 hours ago •
apecloud
apecloud
[BUG] weaviate cluster status is always updating sometimes after retsart /vscale ops
Describe the bug weaviate cluster status is always updating sometimes after retsart ops
kbcli version
Kubernetes: v1.26.3
KubeBlocks: 0.8.0-beta.52
kbcli: 0.8.0-beta.54
To Reproduce Steps to reproduce the behavior:
- create cluster
- config
kbcli cluster configure weaviate-czjxqj --auto-approve --set QUERY_DEFAULTS_LIMIT=150 --components weaviate --config-spec weaviate-env-template --namespace default
- See error
kubectl get cluster weaviate-czjxqj
NAME CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS AGE
weaviate-czjxqj weaviate weaviate-1.18.0 WipeOut Updating 84m
➜ ~
➜ ~ kubectl get pod -l app.kubernetes.io/instance=weaviate-czjxqj
NAME READY STATUS RESTARTS AGE
weaviate-czjxqj-weaviate-0 1/1 Running 0 74m
weaviate-czjxqj-weaviate-1 0/1 Running 0 75m
describe pod
➜ ~ kubectl describe pod weaviate-czjxqj-weaviate-1
Name: weaviate-czjxqj-weaviate-1
Namespace: default
Priority: 0
Node: minikube/192.168.49.2
Start Time: Fri, 12 Jan 2024 09:56:18 +0800
Labels: app.kubernetes.io/component=weaviate
app.kubernetes.io/instance=weaviate-czjxqj
app.kubernetes.io/managed-by=kubeblocks
app.kubernetes.io/name=weaviate
app.kubernetes.io/version=
apps.kubeblocks.io/component-name=weaviate
controller-revision-hash=weaviate-czjxqj-weaviate-56787995dd
statefulset.kubernetes.io/pod-name=weaviate-czjxqj-weaviate-1
Annotations: apps.kubeblocks.io/component-replicas: 2
config.kubeblocks.io/restart-weaviate-env-template: 55854dc959
kubeblocks.io/restart: 2024-01-12T01:51:08Z
Status: Running
IP: 10.244.1.20
IPs:
IP: 10.244.1.20
Controlled By: StatefulSet/weaviate-czjxqj-weaviate
Containers:
weaviate:
Container ID: docker://14f6d8c861a0757f2297afe899a62343cba265183674f98b02b5ec0a10a96dad
Image: docker.io/semitechnologies/weaviate:1.19.6
Image ID: docker-pullable://semitechnologies/weaviate@sha256:6bd9b062b8fe9a3dd33f3c0706f83f7ff28a2b4de7e3bc43971385ca838d4034
Ports: 8080/TCP, 2112/TCP, 7000/TCP, 7001/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP
Command:
/bin/sh
-c
idx=${KB_POD_NAME##*-}
while true; do
if [ "$idx" == "0" ]; then
break
fi
# ensure that the first pod is up and running before starting the others
wget -q -O - http://${CLUSTER_JOIN}:8080/v1/meta
if [ $? -eq 0 ]; then
break
fi
sleep 1
done
export $(cat /weaviate-env/envs | xargs)
/bin/weaviate --host 0.0.0.0 --port "8080" --scheme http --config-file /weaviate-config/conf.yaml --read-timeout=60s --write-timeout=60s
State: Running
Started: Fri, 12 Jan 2024 09:56:26 +0800
Ready: False
Restart Count: 0
Limits:
cpu: 200m
memory: 644245094400m
Requests:
cpu: 200m
memory: 644245094400m
Liveness: http-get http://:8080/v1/.well-known/live delay=30s timeout=3s period=10s #success=1 #failure=30
Readiness: http-get http://:8080/v1/.well-known/ready delay=30s timeout=3s period=10s #success=1 #failure=3
Startup: http-get http://:8080/v1/.well-known/ready delay=30s timeout=3s period=10s #success=1 #failure=3
Environment Variables from:
weaviate-czjxqj-weaviate-env ConfigMap Optional: false
weaviate-czjxqj-weaviate-rsm-env ConfigMap Optional: false
Environment:
KB_POD_NAME: weaviate-czjxqj-weaviate-1 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_POD_FQDN: $(KB_POD_NAME).weaviate-czjxqj-weaviate-headless.$(KB_NAMESPACE).svc
CLUSTER_DATA_BIND_PORT: 7001
CLUSTER_GOSSIP_BIND_PORT: 7000
GOGC: 100
PROMETHEUS_MONITORING_ENABLED: true
PROMETHEUS_MONITORING_PORT: 2112
QUERY_MAXIMUM_RESULTS: 100000
REINDEX_VECTOR_DIMENSIONS_AT_STARTUP: false
TRACK_VECTOR_DIMENSIONS: false
PERSISTENCE_DATA_PATH: /var/lib/weaviate
DEFAULT_VECTORIZER_MODULE: none
CLUSTER_HOSTNAME: $(KB_POD_NAME)
CLUSTER_JOIN: $(KB_0_HOSTNAME).$(KB_NAMESPACE).svc.cluster.local
Mounts:
/var/lib/weaviate from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s2rch (ro)
/weaviate-config from weaviate-config (rw)
/weaviate-env from weaviate-env (rw)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-weaviate-czjxqj-weaviate-1
ReadOnly: false
weaviate-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: weaviate-czjxqj-weaviate-weaviate-config-template
Optional: false
weaviate-env:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: weaviate-czjxqj-weaviate-weaviate-env-template
Optional: false
kube-api-access-s2rch:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: kb-data=true:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 12s (x503 over 74m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 503
logs pod
➜ ~ kubectl logs weaviate-czjxqj-weaviate-1
{"hostname":"http://[::]:8080","modules":{},"version":"1.19.6"}
{"action":"config_load","config_file_path":"/weaviate-config/conf.yaml","level":"info","msg":"Usage of the weaviate.conf.json file is deprecated and will be removed in the future. Please use environment variables.","time":"2024-01-12T01:56:27Z"}
{"deprecation":{"apiType":"Configuration","id":"config-files","locations":["--config-file=\"\""],"mitigation":"Configure Weaviate using environment variables.","msg":"use of deprecated command line argument --config-file","sinceTime":"2020-09-08T09:46:00.000Z","sinceVersion":"0.22.16","status":"deprecated"},"level":"warning","msg":"use of deprecated command line argument --config-file","time":"2024-01-12T01:56:27Z"}
{"action":"startup","default_vectorizer_module":"none","level":"info","msg":"the default vectorizer modules is set to \"none\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-01-12T01:56:27Z"}
{"action":"startup","auto_schema_enabled":true,"level":"info","msg":"auto schema enabled setting is set to \"true\"","time":"2024-01-12T01:56:27Z"}
{"level":"warning","msg":" memberlist: Refuting a suspect message (from: weaviate-czjxqj-weaviate-1)","time":"2024-01-12T01:56:27Z"}
{"action":"grpc_startup","level":"info","msg":"grpc server listening at [::]:50051","time":"2024-01-12T01:56:27Z"}
{"action":"restapi_management","level":"info","msg":"Serving weaviate at http://[::]:8080","time":"2024-01-12T01:56:27Z"}
{"level":"info","msg":" memberlist: Suspect weaviate-czjxqj-weaviate-0 has failed, no acks received","time":"2024-01-12T01:57:29Z"}
{"level":"info","msg":" memberlist: Suspect weaviate-czjxqj-weaviate-0 has failed, no acks received","time":"2024-01-12T01:57:31Z"}
{"level":"info","msg":" memberlist: Marking weaviate-czjxqj-weaviate-0 as failed, suspect timeout reached (0 peer confirmations)","time":"2024-01-12T01:57:33Z"}
{"level":"info","msg":" memberlist: Suspect weaviate-czjxqj-weaviate-0 has failed, no acks received","time":"2024-01-12T01:57:35Z"}
➜ ~
test logs: https://github.com/apecloud/kubeblocks/actions/runs/7486455798/job/20377751287
Expected behavior weaviate cluster status is Running.
Screenshots If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
Additional context Add any other context about the problem here.
This issue has been marked as stale because it has been open for 30 days with no activity
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Same error returned after vscale weaviate
`kbcli cluster vscale weaviate-ikqhyw --auto-approve --components weaviate --cpu 200m --memory 0.6Gi --namespace default `
OpsRequest weaviate-ikqhyw-verticalscaling-zxx4j created successfully, you can view the progress:
kbcli cluster describe-ops weaviate-ikqhyw-verticalscaling-zxx4j -n default
`kubectl get cluster weaviate-ikqhyw -o yaml --namespace default `
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
annotations:
kubeblocks.io/ops-request: '[{"name":"weaviate-ikqhyw-verticalscaling-zxx4j","type":"VerticalScaling"}]'
kubeblocks.io/reconcile: "2024-03-08T09:42:22.744358578Z"
creationTimestamp: "2024-03-08T09:33:33Z"
finalizers:
- cluster.kubeblocks.io/finalizer
generation: 3
labels:
app.kubernetes.io/instance: weaviate-ikqhyw
clusterdefinition.kubeblocks.io/name: weaviate
clusterversion.kubeblocks.io/name: weaviate-1.18.0
name: weaviate-ikqhyw
namespace: default
resourceVersion: "277402"
uid: 16e351a7-db9c-4f36-b53d-ee75643edf73
spec:
affinity:
podAntiAffinity: Preferred
tenancy: SharedNode
clusterDefinitionRef: weaviate
clusterVersionRef: weaviate-1.18.0
componentSpecs:
- classDefRef:
class: ""
componentDefRef: weaviate
monitor: false
name: weaviate
noCreatePDB: false
replicas: 2
resources:
limits:
cpu: 200m
memory: 644245094400m
requests:
cpu: 200m
memory: 644245094400m
rsmTransformPolicy: ToSts
serviceAccountName: kb-weaviate-ikqhyw
volumeClaimTemplates:
- name: data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
monitor: {}
resources:
cpu: "0"
memory: "0"
storage:
size: "0"
terminationPolicy: Halt
status:
clusterDefGeneration: 2
components:
weaviate:
phase: Updating
podsReady: false
podsReadyTime: "2024-03-08T09:36:11Z"
conditions:
- lastTransitionTime: "2024-03-08T09:33:33Z"
message: 'The operator has started the provisioning of Cluster: weaviate-ikqhyw'
observedGeneration: 3
reason: PreCheckSucceed
status: "True"
type: ProvisioningStarted
- lastTransitionTime: "2024-03-08T09:33:33Z"
message: Successfully applied for resources
observedGeneration: 3
reason: ApplyResourcesSucceed
status: "True"
type: ApplyResources
- lastTransitionTime: "2024-03-08T09:36:33Z"
message: 'pods are not ready in Components: [weaviate], refer to related component
message in Cluster.status.components'
reason: ReplicasNotReady
status: "False"
type: ReplicasReady
- lastTransitionTime: "2024-03-08T09:36:33Z"
message: 'pods are unavailable in Components: [weaviate], refer to related component
message in Cluster.status.components'
reason: ComponentsNotReady
status: "False"
type: Ready
observedGeneration: 3
phase: Updating
➜ ~ kubectl describe cluster weaviate-ikqhyw
Name: weaviate-ikqhyw
Namespace: default
Labels: app.kubernetes.io/instance=weaviate-ikqhyw
clusterdefinition.kubeblocks.io/name=weaviate
clusterversion.kubeblocks.io/name=weaviate-1.18.0
Annotations: kubeblocks.io/ops-request: [{"name":"weaviate-ikqhyw-verticalscaling-zxx4j","type":"VerticalScaling"}]
kubeblocks.io/reconcile: 2024-03-08T09:47:22.673708779Z
API Version: apps.kubeblocks.io/v1alpha1
Kind: Cluster
Metadata:
Creation Timestamp: 2024-03-08T09:33:33Z
Finalizers:
cluster.kubeblocks.io/finalizer
Generation: 3
Managed Fields:
API Version: apps.kubeblocks.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
f:app.kubernetes.io/instance:
f:spec:
.:
f:affinity:
.:
f:podAntiAffinity:
f:tenancy:
f:clusterDefinitionRef:
f:clusterVersionRef:
f:monitor:
f:resources:
.:
f:cpu:
f:memory:
f:storage:
.:
f:size:
f:terminationPolicy:
Manager: kbcli
Operation: Update
Time: 2024-03-08T09:35:06Z
API Version: apps.kubeblocks.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:clusterDefGeneration:
f:components:
.:
f:weaviate:
.:
f:phase:
f:podsReady:
f:podsReadyTime:
f:conditions:
f:observedGeneration:
f:phase:
Manager: manager
Operation: Update
Subresource: status
Time: 2024-03-08T09:36:34Z
API Version: apps.kubeblocks.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubeblocks.io/ops-request:
f:kubeblocks.io/reconcile:
f:finalizers:
.:
v:"cluster.kubeblocks.io/finalizer":
f:labels:
.:
f:clusterdefinition.kubeblocks.io/name:
f:clusterversion.kubeblocks.io/name:
f:spec:
f:componentSpecs:
Manager: manager
Operation: Update
Time: 2024-03-08T09:47:22Z
Resource Version: 281700
UID: 16e351a7-db9c-4f36-b53d-ee75643edf73
Spec:
Affinity:
Pod Anti Affinity: Preferred
Tenancy: SharedNode
Cluster Definition Ref: weaviate
Cluster Version Ref: weaviate-1.18.0
Component Specs:
Class Def Ref:
Class:
Component Def Ref: weaviate
Monitor: false
Name: weaviate
No Create PDB: false
Replicas: 2
Resources:
Limits:
Cpu: 200m
Memory: 644245094400m
Requests:
Cpu: 200m
Memory: 644245094400m
Rsm Transform Policy: ToSts
Service Account Name: kb-weaviate-ikqhyw
Volume Claim Templates:
Name: data
Spec:
Access Modes:
ReadWriteOnce
Resources:
Requests:
Storage: 1Gi
Monitor:
Resources:
Cpu: 0
Memory: 0
Storage:
Size: 0
Termination Policy: Halt
Status:
Cluster Def Generation: 2
Components:
Weaviate:
Phase: Updating
Pods Ready: false
Pods Ready Time: 2024-03-08T09:36:11Z
Conditions:
Last Transition Time: 2024-03-08T09:33:33Z
Message: The operator has started the provisioning of Cluster: weaviate-ikqhyw
Observed Generation: 3
Reason: PreCheckSucceed
Status: True
Type: ProvisioningStarted
Last Transition Time: 2024-03-08T09:33:33Z
Message: Successfully applied for resources
Observed Generation: 3
Reason: ApplyResourcesSucceed
Status: True
Type: ApplyResources
Last Transition Time: 2024-03-08T09:36:33Z
Message: pods are not ready in Components: [weaviate], refer to related component message in Cluster.status.components
Reason: ReplicasNotReady
Status: False
Type: ReplicasReady
Last Transition Time: 2024-03-08T09:36:33Z
Message: pods are unavailable in Components: [weaviate], refer to related component message in Cluster.status.components
Reason: ComponentsNotReady
Status: False
Type: Ready
Observed Generation: 3
Phase: Updating
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ComponentPhaseTransition 13m cluster-controller component is Creating
Normal HorizontalScale 12m (x2 over 12m) component-controller start horizontal scale component weaviate of cluster weaviate-ikqhyw from 1 to 2
Normal ComponentPhaseTransition 11m (x2 over 12m) cluster-controller component is Running
Normal AllReplicasReady 11m (x2 over 12m) cluster-controller all pods of components are ready, waiting for the probe detection successful
Normal ClusterReady 11m (x2 over 12m) cluster-controller Cluster: weaviate-ikqhyw is ready, current phase is Running
Normal Running 11m (x2 over 12m) cluster-controller Cluster: weaviate-ikqhyw is ready, current phase is Running
Normal ApplyResourcesSucceed 10m (x3 over 13m) cluster-controller Successfully applied for resources
Normal PreCheckSucceed 10m (x3 over 13m) cluster-controller The operator has started the provisioning of Cluster: weaviate-ikqhyw
Normal ComponentPhaseTransition 10m (x2 over 12m) cluster-controller component is Updating
Warning ReplicasNotReady 10m (x2 over 12m) cluster-controller pods are not ready in Components: [weaviate], refer to related component message in Cluster.status.components
Warning ComponentsNotReady 10m (x2 over 12m) cluster-controller pods are unavailable in Components: [weaviate], refer to related component message in Cluster.status.components
Warning Unhealthy 9m16s (x3 over 9m36s) event-controller Pod weaviate-ikqhyw-weaviate-0: Readiness probe failed: HTTP probe failed with statuscode: 503
Warning Unhealthy 2s (x19 over 7m52s) event-controller Pod weaviate-ikqhyw-weaviate-1: Readiness probe failed: HTTP probe failed with statuscode: 503
➜ ~
➜ ~ k get pod | grep weaviate-ikqhyw
weaviate-ikqhyw-weaviate-0 1/1 Running 0 9m1s
weaviate-ikqhyw-weaviate-1 0/1 Running 0 10m
➜ ~ k describe pod weaviate-ikqhyw-weaviate-1
Name: weaviate-ikqhyw-weaviate-1
Namespace: default
Priority: 0
Node: gke-yjtest-default-pool-cdaf391b-6f4l/10.128.15.214
Start Time: Fri, 08 Mar 2024 17:37:02 +0800
Labels: app.kubernetes.io/component=weaviate
app.kubernetes.io/instance=weaviate-ikqhyw
app.kubernetes.io/managed-by=kubeblocks
app.kubernetes.io/name=weaviate
app.kubernetes.io/version=
apps.kubeblocks.io/cluster-uid=16e351a7-db9c-4f36-b53d-ee75643edf73
apps.kubeblocks.io/component-name=weaviate
clusterdefinition.kubeblocks.io/name=weaviate
clusterversion.kubeblocks.io/name=weaviate-1.18.0
controller-revision-hash=weaviate-ikqhyw-weaviate-65594856c6
statefulset.kubernetes.io/pod-name=weaviate-ikqhyw-weaviate-1
Annotations: apps.kubeblocks.io/component-replicas: 2
Status: Running
IP: 10.104.4.108
IPs:
IP: 10.104.4.108
Controlled By: StatefulSet/weaviate-ikqhyw-weaviate
Containers:
weaviate:
Container ID: containerd://962994b3880d4f13c3cfe5e82fa097f511ee85a6a4d5425609e5251fbf9f9338
Image: docker.io/semitechnologies/weaviate:1.19.6
Image ID: docker.io/semitechnologies/weaviate@sha256:6bd9b062b8fe9a3dd33f3c0706f83f7ff28a2b4de7e3bc43971385ca838d4034
Ports: 8080/TCP, 2112/TCP, 7000/TCP, 7001/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP
Command:
/bin/sh
-c
idx=${KB_POD_NAME##*-}
while true; do
if [ "$idx" == "0" ]; then
break
fi
# ensure that the first pod is up and running before starting the others
wget -q -O - http://${CLUSTER_JOIN}:8080/v1/meta
if [ $? -eq 0 ]; then
break
fi
sleep 1
done
export $(cat /weaviate-env/envs | xargs)
/bin/weaviate --host 0.0.0.0 --port "8080" --scheme http --config-file /weaviate-config/conf.yaml --read-timeout=60s --write-timeout=60s
State: Running
Started: Fri, 08 Mar 2024 17:37:13 +0800
Ready: False
Restart Count: 0
Limits:
cpu: 200m
memory: 644245094400m
Requests:
cpu: 200m
memory: 644245094400m
Liveness: http-get http://:8080/v1/.well-known/live delay=30s timeout=3s period=10s #success=1 #failure=30
Readiness: http-get http://:8080/v1/.well-known/ready delay=30s timeout=3s period=10s #success=1 #failure=3
Startup: http-get http://:8080/v1/.well-known/ready delay=30s timeout=3s period=10s #success=1 #failure=3
Environment Variables from:
weaviate-ikqhyw-weaviate-env ConfigMap Optional: false
weaviate-ikqhyw-weaviate-rsm-env ConfigMap Optional: false
Environment:
KB_POD_NAME: weaviate-ikqhyw-weaviate-1 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_POD_FQDN: $(KB_POD_NAME).weaviate-ikqhyw-weaviate-headless.$(KB_NAMESPACE).svc
CLUSTER_DATA_BIND_PORT: 7001
CLUSTER_GOSSIP_BIND_PORT: 7000
GOGC: 100
PROMETHEUS_MONITORING_ENABLED: true
PROMETHEUS_MONITORING_PORT: 2112
QUERY_MAXIMUM_RESULTS: 100000
REINDEX_VECTOR_DIMENSIONS_AT_STARTUP: false
TRACK_VECTOR_DIMENSIONS: false
PERSISTENCE_DATA_PATH: /var/lib/weaviate
DEFAULT_VECTORIZER_MODULE: none
CLUSTER_HOSTNAME: $(KB_POD_NAME)
CLUSTER_JOIN: $(KB_0_HOSTNAME).$(KB_NAMESPACE).svc.cluster.local
Mounts:
/var/lib/weaviate from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bswnz (ro)
/weaviate-config from weaviate-config (rw)
/weaviate-env from weaviate-env (rw)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-weaviate-ikqhyw-weaviate-1
ReadOnly: false
weaviate-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: weaviate-ikqhyw-weaviate-weaviate-config-template
Optional: false
weaviate-env:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: weaviate-ikqhyw-weaviate-weaviate-env-template
Optional: false
kube-api-access-bswnz:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: kb-data=true:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 11m default-scheduler Successfully assigned default/weaviate-ikqhyw-weaviate-1 to gke-yjtest-default-pool-cdaf391b-6f4l
Normal Pulled 10m kubelet Container image "docker.io/semitechnologies/weaviate:1.19.6" already present on machine
Normal Created 10m kubelet Created container weaviate
Normal Started 10m kubelet Started container weaviate
Warning Unhealthy 44s (x59 over 9m14s) kubelet Readiness probe failed: HTTP probe failed with statuscode: 503
➜ ~ k logs weaviate-ikqhyw-weaviate-1
{"hostname":"http://[::]:8080","modules":{},"version":"1.19.6"}
{"action":"config_load","config_file_path":"/weaviate-config/conf.yaml","level":"info","msg":"Usage of the weaviate.conf.json file is deprecated and will be removed in the future. Please use environment variables.","time":"2024-03-08T09:37:13Z"}
{"deprecation":{"apiType":"Configuration","id":"config-files","locations":["--config-file=\"\""],"mitigation":"Configure Weaviate using environment variables.","msg":"use of deprecated command line argument --config-file","sinceTime":"2020-09-08T09:46:00.000Z","sinceVersion":"0.22.16","status":"deprecated"},"level":"warning","msg":"use of deprecated command line argument --config-file","time":"2024-03-08T09:37:13Z"}
{"action":"startup","default_vectorizer_module":"none","level":"info","msg":"the default vectorizer modules is set to \"none\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-03-08T09:37:13Z"}
{"action":"startup","auto_schema_enabled":true,"level":"info","msg":"auto schema enabled setting is set to \"true\"","time":"2024-03-08T09:37:13Z"}
{"level":"warning","msg":" memberlist: Refuting a suspect message (from: weaviate-ikqhyw-weaviate-1)","time":"2024-03-08T09:37:13Z"}
{"action":"grpc_startup","level":"info","msg":"grpc server listening at [::]:50051","time":"2024-03-08T09:37:13Z"}
{"action":"restapi_management","level":"info","msg":"Serving weaviate at http://[::]:8080","time":"2024-03-08T09:37:14Z"}
{"level":"info","msg":" memberlist: Suspect weaviate-ikqhyw-weaviate-0 has failed, no acks received","time":"2024-03-08T09:38:50Z"}
{"level":"info","msg":" memberlist: Suspect weaviate-ikqhyw-weaviate-0 has failed, no acks received","time":"2024-03-08T09:38:52Z"}
{"level":"info","msg":" memberlist: Marking weaviate-ikqhyw-weaviate-0 as failed, suspect timeout reached (0 peer confirmations)","time":"2024-03-08T09:38:54Z"}
{"level":"info","msg":" memberlist: Suspect weaviate-ikqhyw-weaviate-0 has failed, no acks received","time":"2024-03-08T09:38:56Z"}
➜ ~
Still failed on 0.9 https://github.com/apecloud/kubeblocks/actions/runs/8599504622/job/23563829638 pod cannot be started after restart
➜ ~ k get pod
NAME READY STATUS RESTARTS AGE
weaviate-cgwsgi-weaviate-0 1/1 Running 0 4m40s
weaviate-cgwsgi-weaviate-1 0/1 Running 0 6m4s
➜ ~ k logs weaviate-cgwsgi-weaviate-1
{"hostname":"http://[::]:8080","modules":{},"version":"1.19.6"}
{"action":"config_load","config_file_path":"/weaviate-config/conf.yaml","level":"info","msg":"Usage of the weaviate.conf.json file is deprecated and will be removed in the future. Please use environment variables.","time":"2024-04-09T09:56:07Z"}
{"deprecation":{"apiType":"Configuration","id":"config-files","locations":["--config-file=\"\""],"mitigation":"Configure Weaviate using environment variables.","msg":"use of deprecated command line argument --config-file","sinceTime":"2020-09-08T09:46:00.000Z","sinceVersion":"0.22.16","status":"deprecated"},"level":"warning","msg":"use of deprecated command line argument --config-file","time":"2024-04-09T09:56:07Z"}
{"action":"startup","default_vectorizer_module":"none","level":"info","msg":"the default vectorizer modules is set to \"none\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-04-09T09:56:07Z"}
{"action":"startup","auto_schema_enabled":true,"level":"info","msg":"auto schema enabled setting is set to \"true\"","time":"2024-04-09T09:56:07Z"}
{"action":"grpc_startup","level":"info","msg":"grpc server listening at [::]:50051","time":"2024-04-09T09:56:07Z"}
{"action":"restapi_management","level":"info","msg":"Serving weaviate at http://[::]:8080","time":"2024-04-09T09:56:07Z"}
{"level":"info","msg":" memberlist: Suspect weaviate-cgwsgi-weaviate-0 has failed, no acks received","time":"2024-04-09T09:57:16Z"}
{"level":"info","msg":" memberlist: Suspect weaviate-cgwsgi-weaviate-0 has failed, no acks received","time":"2024-04-09T09:57:19Z"}
{"level":"info","msg":" memberlist: Marking weaviate-cgwsgi-weaviate-0 as failed, suspect timeout reached (0 peer confirmations)","time":"2024-04-09T09:57:20Z"}
{"level":"info","msg":" memberlist: Suspect weaviate-cgwsgi-weaviate-0 has failed, no acks received","time":"2024-04-09T09:57:22Z"}
➜ ~ k logs weaviate-cgwsgi-weaviate-0
{"action":"config_load","config_file_path":"/weaviate-config/conf.yaml","level":"info","msg":"Usage of the weaviate.conf.json file is deprecated and will be removed in the future. Please use environment variables.","time":"2024-04-09T09:57:17Z"}
{"deprecation":{"apiType":"Configuration","id":"config-files","locations":["--config-file=\"\""],"mitigation":"Configure Weaviate using environment variables.","msg":"use of deprecated command line argument --config-file","sinceTime":"2020-09-08T09:46:00.000Z","sinceVersion":"0.22.16","status":"deprecated"},"level":"warning","msg":"use of deprecated command line argument --config-file","time":"2024-04-09T09:57:17Z"}
{"action":"startup","default_vectorizer_module":"none","level":"info","msg":"the default vectorizer modules is set to \"none\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-04-09T09:57:17Z"}
{"action":"startup","auto_schema_enabled":true,"level":"info","msg":"auto schema enabled setting is set to \"true\"","time":"2024-04-09T09:57:17Z"}
{"action":"grpc_startup","level":"info","msg":"grpc server listening at [::]:50051","time":"2024-04-09T09:57:17Z"}
{"action":"restapi_management","level":"info","msg":"Serving weaviate at http://[::]:8080","time":"2024-04-09T09:57:18Z"}
➜ ~