dataset-viewer
dataset-viewer copied to clipboard
Published
20 hours ago •
huggingface
huggingface
feat(chart): expose all services through ALB
- [x] disable nginx proxy
- [x] expose
/onapiinstances - [x] expose
/adminonadmininstances - [x] expose
/rowsonrowsinstances - [x] expose
/searchonsearchinstances - [x] expose
/filteronsearchinstances - [x] expose
/sseonsseApiinstances - [ ] expose
/openapi.jsonon github - [ ] customise 404.html error page ?
ArgoCD Diff for commit f3b5e64
Updated at 5/15/2024, 9:48:03 PM CEST
App: datasets-server-prod
YAML generation: Success 🟢
App sync status: Out of Sync ⚠️
===== apps/Deployment datasets-server/prod-datasets-server-admin ======
--- /tmp/argocd-diff152185749/prod-datasets-server-admin-live.yaml 2024-05-15 19:48:01.417888718 +0000
+++ /tmp/argocd-diff152185749/prod-datasets-server-admin 2024-05-15 19:48:01.413888671 +0000
@@ -456,7 +456,7 @@
value: "9"
- name: ADMIN_UVICORN_PORT
value: "8080"
- image: huggingface/datasets-server-services-admin:sha-2c69f29
+ image: huggingface/datasets-server-services-admin:sha-58c3ce9
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 30
===== apps/Deployment datasets-server/prod-datasets-server-api ======
--- /tmp/argocd-diff3072877043/prod-datasets-server-api-live.yaml 2024-05-15 19:48:01.437888953 +0000
+++ /tmp/argocd-diff3072877043/prod-datasets-server-api 2024-05-15 19:48:01.433888906 +0000
@@ -468,7 +468,7 @@
value: "9"
- name: API_UVICORN_PORT
value: "8080"
- image: huggingface/datasets-server-services-api:sha-2c69f29
+ image: huggingface/datasets-server-services-api:sha-58c3ce9
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 30
===== apps/Deployment datasets-server/prod-datasets-server-rows ======
--- /tmp/argocd-diff2521970332/prod-datasets-server-rows-live.yaml 2024-05-15 19:48:01.461889236 +0000
+++ /tmp/argocd-diff2521970332/prod-datasets-server-rows 2024-05-15 19:48:01.457889188 +0000
@@ -501,7 +501,7 @@
value: "8080"
- name: ROWS_INDEX_MAX_ARROW_DATA_IN_MEMORY
value: "300_000_000"
- image: huggingface/datasets-server-services-rows:sha-2c69f29
+ image: huggingface/datasets-server-services-rows:sha-58c3ce9
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 30
===== apps/Deployment datasets-server/prod-datasets-server-search ======
--- /tmp/argocd-diff4259709711/prod-datasets-server-search-live.yaml 2024-05-15 19:48:01.477889424 +0000
+++ /tmp/argocd-diff4259709711/prod-datasets-server-search 2024-05-15 19:48:01.477889424 +0000
@@ -477,7 +477,7 @@
value: /tmp/duckdb-extensions
- name: HF_HUB_ENABLE_HF_TRANSFER
value: "1"
- image: huggingface/datasets-server-services-search:sha-2c69f29
+ image: huggingface/datasets-server-services-search:sha-58c3ce9
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 30
===== apps/Deployment datasets-server/prod-datasets-server-sse-api ======
--- /tmp/argocd-diff1974212367/prod-datasets-server-sse-api-live.yaml 2024-05-15 19:48:01.489889565 +0000
+++ /tmp/argocd-diff1974212367/prod-datasets-server-sse-api 2024-05-15 19:48:01.489889565 +0000
@@ -315,7 +315,7 @@
value: "1"
- name: API_UVICORN_PORT
value: "8080"
- image: huggingface/datasets-server-services-sse-api:sha-2c69f29
+ image: huggingface/datasets-server-services-sse-api:sha-58c3ce9
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 30
===== apps/Deployment datasets-server/prod-datasets-server-storage-admin ======
--- /tmp/argocd-diff2447079320/prod-datasets-server-storage-admin-live.yaml 2024-05-15 19:48:01.493889613 +0000
+++ /tmp/argocd-diff2447079320/prod-datasets-server-storage-admin 2024-05-15 19:48:01.493889613 +0000
@@ -1,284 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
- annotations:
- deployment.kubernetes.io/revision: "302"
- kubectl.kubernetes.io/last-applied-configuration: |
- {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"prod-datasets-server-storage-admin","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","argocd.argoproj.io/instance":"datasets-server-prod","helm.sh/chart":"datasets-server"},"name":"prod-datasets-server-storage-admin","namespace":"datasets-server"},"spec":{"progressDeadlineSeconds":600,"replicas":1,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"prod-datasets-server-storage-admin","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","helm.sh/chart":"datasets-server"}},"template":{"metadata":{"labels":{"app.kubernetes.io/component":"prod-datasets-server-storage-admin","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","helm.sh/chart":"datasets-server"}},"spec":{"containers":[{"image":"huggingface/datasets-server-services-storage-admin:sha-2c69f29","imagePullPolicy":"IfNotPresent","name":"prod-datasets-server-storage-admin","resources":{"limits":{"cpu":4,"memory":"4Gi"},"requests":{"cpu":4,"memory":"4Gi"}},"securityContext":{"runAsGroup":0,"runAsNonRoot":false,"runAsUser":0},"volumeMounts":[{"mountPath":"/storage/parquet-metadata","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":false,"subPath":"datasets-server/prod/parquet-metadata/"},{"mountPath":"/volumes/parquet-metadata","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":false}]}],"initContainers":[{"args":["chown 1000:3000 /mounted-path;"],"command":["/bin/sh","-c"],"image":"ubuntu:focal","imagePullPolicy":"IfNotPresent","name":"prepare-parquet-metadata","securityContext":{"runAsGroup":0,"runAsNonRoot":false,"runAsUser":0},"volumeMounts":[{"mountPath":"/mounted-path","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":false,"subPath":"datasets-server/prod/parquet-metadata/"}]}],"nodeSelector":{"role-datasets-server":"true"},"tolerations":[],"volumes":[{"name":"volume-parquet-metadata","persistentVolumeClaim":{"claimName":"datasets-server-parquet-pvc"}}]}}}}
- generation: 308
- labels:
- app.kubernetes.io/component: prod-datasets-server-storage-admin
- app.kubernetes.io/instance: prod
- app.kubernetes.io/managed-by: Helm
- app.kubernetes.io/name: prod-datasets-server
- argocd.argoproj.io/instance: datasets-server-prod
- helm.sh/chart: datasets-server
- managedFields:
- - apiVersion: apps/v1
- fieldsType: FieldsV1
- fieldsV1:
- f:metadata:
- f:labels:
- .: {}
- f:app.kubernetes.io/component: {}
- f:app.kubernetes.io/instance: {}
- f:app.kubernetes.io/managed-by: {}
- f:app.kubernetes.io/name: {}
- f:helm.sh/chart: {}
- f:spec:
- f:progressDeadlineSeconds: {}
- f:revisionHistoryLimit: {}
- f:selector: {}
- f:strategy:
- f:rollingUpdate:
- .: {}
- f:maxSurge: {}
- f:maxUnavailable: {}
- f:type: {}
- f:template:
- f:metadata:
- f:labels:
- .: {}
- f:app.kubernetes.io/component: {}
- f:app.kubernetes.io/instance: {}
- f:app.kubernetes.io/managed-by: {}
- f:app.kubernetes.io/name: {}
- f:helm.sh/chart: {}
- f:spec:
- f:containers:
- k:{"name":"prod-datasets-server-storage-admin"}:
- .: {}
- f:imagePullPolicy: {}
- f:name: {}
- f:resources:
- .: {}
- f:limits: {}
- f:requests: {}
- f:securityContext:
- .: {}
- f:runAsGroup: {}
- f:runAsNonRoot: {}
- f:runAsUser: {}
- f:terminationMessagePath: {}
- f:terminationMessagePolicy: {}
- f:volumeMounts: {}
- f:dnsPolicy: {}
- f:nodeSelector: {}
- f:restartPolicy: {}
- f:schedulerName: {}
- f:securityContext: {}
- f:terminationGracePeriodSeconds: {}
- f:volumes: {}
- manager: helm
- operation: Update
- time: "2023-01-30T14:28:11Z"
- - apiVersion: apps/v1
- fieldsType: FieldsV1
- fieldsV1:
- f:metadata:
- f:annotations:
- f:kubectl.kubernetes.io/last-applied-configuration: {}
- f:labels:
- f:argocd.argoproj.io/instance: {}
- f:spec:
- f:template:
- f:spec:
- f:containers:
- k:{"name":"prod-datasets-server-storage-admin"}:
- f:image: {}
- f:resources:
- f:limits:
- f:cpu: {}
- f:memory: {}
- f:requests:
- f:cpu: {}
- f:memory: {}
- f:volumeMounts:
- k:{"mountPath":"/storage/parquet-metadata"}:
- .: {}
- f:mountPath: {}
- f:mountPropagation: {}
- f:name: {}
- f:subPath: {}
- k:{"mountPath":"/volumes/parquet-metadata"}:
- .: {}
- f:mountPath: {}
- f:mountPropagation: {}
- f:name: {}
- f:initContainers:
- .: {}
- k:{"name":"prepare-parquet-metadata"}:
- .: {}
- f:args: {}
- f:command: {}
- f:image: {}
- f:imagePullPolicy: {}
- f:name: {}
- f:resources: {}
- f:securityContext:
- .: {}
- f:runAsGroup: {}
- f:runAsNonRoot: {}
- f:runAsUser: {}
- f:terminationMessagePath: {}
- f:terminationMessagePolicy: {}
- f:volumeMounts:
- .: {}
- k:{"mountPath":"/mounted-path"}:
- .: {}
- f:mountPath: {}
- f:mountPropagation: {}
- f:name: {}
- f:subPath: {}
- f:volumes:
- k:{"name":"volume-parquet-metadata"}:
- .: {}
- f:name: {}
- f:persistentVolumeClaim:
- .: {}
- f:claimName: {}
- manager: argocd-controller
- operation: Update
- time: "2024-05-15T13:47:31Z"
- - apiVersion: apps/v1
- fieldsType: FieldsV1
- fieldsV1:
- f:metadata:
- f:annotations:
- .: {}
- f:deployment.kubernetes.io/revision: {}
- f:status:
- f:availableReplicas: {}
- f:conditions:
- .: {}
- k:{"type":"Available"}:
- .: {}
- f:lastTransitionTime: {}
- f:lastUpdateTime: {}
- f:message: {}
- f:reason: {}
- f:status: {}
- f:type: {}
- k:{"type":"Progressing"}:
- .: {}
- f:lastTransitionTime: {}
- f:lastUpdateTime: {}
- f:message: {}
- f:reason: {}
- f:status: {}
- f:type: {}
- f:observedGeneration: {}
- f:readyReplicas: {}
- f:replicas: {}
- f:updatedReplicas: {}
- manager: kube-controller-manager
- operation: Update
- subresource: status
- time: "2024-05-15T13:49:52Z"
- name: prod-datasets-server-storage-admin
- namespace: datasets-server
- resourceVersion: "906942385"
- uid: 6fde887e-4f58-4f71-8ad0-d08a555e2498
-spec:
- progressDeadlineSeconds: 600
- replicas: 1
- revisionHistoryLimit: 10
- selector:
- matchLabels:
- app.kubernetes.io/component: prod-datasets-server-storage-admin
- app.kubernetes.io/instance: prod
- app.kubernetes.io/managed-by: Helm
- app.kubernetes.io/name: prod-datasets-server
- helm.sh/chart: datasets-server
- strategy:
- rollingUpdate:
- maxSurge: 25%
- maxUnavailable: 25%
- type: RollingUpdate
- template:
- metadata:
- creationTimestamp: null
- labels:
- app.kubernetes.io/component: prod-datasets-server-storage-admin
- app.kubernetes.io/instance: prod
- app.kubernetes.io/managed-by: Helm
- app.kubernetes.io/name: prod-datasets-server
- helm.sh/chart: datasets-server
- spec:
- containers:
- - image: huggingface/datasets-server-services-storage-admin:sha-2c69f29
- imagePullPolicy: IfNotPresent
- name: prod-datasets-server-storage-admin
- resources:
- limits:
- cpu: "4"
- memory: 4Gi
- requests:
- cpu: "4"
- memory: 4Gi
- securityContext:
- runAsGroup: 0
- runAsNonRoot: false
- runAsUser: 0
- terminationMessagePath: /dev/termination-log
- terminationMessagePolicy: File
- volumeMounts:
- - mountPath: /storage/parquet-metadata
- mountPropagation: None
- name: volume-parquet-metadata
- subPath: datasets-server/prod/parquet-metadata/
- - mountPath: /volumes/parquet-metadata
- mountPropagation: None
- name: volume-parquet-metadata
- dnsPolicy: ClusterFirst
- initContainers:
- - args:
- - chown 1000:3000 /mounted-path;
- command:
- - /bin/sh
- - -c
- image: ubuntu:focal
- imagePullPolicy: IfNotPresent
- name: prepare-parquet-metadata
- resources: {}
- securityContext:
- runAsGroup: 0
- runAsNonRoot: false
- runAsUser: 0
- terminationMessagePath: /dev/termination-log
- terminationMessagePolicy: File
- volumeMounts:
- - mountPath: /mounted-path
- mountPropagation: None
- name: volume-parquet-metadata
- subPath: datasets-server/prod/parquet-metadata/
- nodeSelector:
- role-datasets-server: "true"
- restartPolicy: Always
- schedulerName: default-scheduler
- securityContext: {}
- terminationGracePeriodSeconds: 30
- volumes:
- - name: volume-parquet-metadata
- persistentVolumeClaim:
- claimName: datasets-server-parquet-pvc
-status:
- availableReplicas: 1
- conditions:
- - lastTransitionTime: "2024-04-23T11:13:47Z"
- lastUpdateTime: "2024-04-23T11:13:47Z"
- message: Deployment has minimum availability.
- reason: MinimumReplicasAvailable
- status: "True"
- type: Available
- - lastTransitionTime: "2024-04-10T08:28:24Z"
- lastUpdateTime: "2024-05-15T13:49:52Z"
- message: ReplicaSet "prod-datasets-server-storage-admin-6f865bdbb4" has successfully
- progressed.
- reason: NewReplicaSetAvailable
- status: "True"
- type: Progressing
- observedGeneration: 308
- readyReplicas: 1
- replicas: 1
- updatedReplicas: 1
===== apps/Deployment datasets-server/prod-datasets-server-worker-heavy ======
--- /tmp/argocd-diff3300701052/prod-datasets-server-worker-heavy-live.yaml 2024-05-15 19:48:01.521889942 +0000
+++ /tmp/argocd-diff3300701052/prod-datasets-server-worker-heavy 2024-05-15 19:48:01.517889894 +0000
@@ -681,7 +681,7 @@
value: "1"
- name: WORKER_UVICORN_PORT
value: "8080"
- image: huggingface/datasets-server-services-worker:sha-2c69f29
+ image: huggingface/datasets-server-services-worker:sha-58c3ce9
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 30
===== apps/Deployment datasets-server/prod-datasets-server-worker-light ======
--- /tmp/argocd-diff1690184756/prod-datasets-server-worker-light-live.yaml 2024-05-15 19:48:01.545890225 +0000
+++ /tmp/argocd-diff1690184756/prod-datasets-server-worker-light 2024-05-15 19:48:01.541890177 +0000
@@ -681,7 +681,7 @@
value: "1"
- name: WORKER_UVICORN_PORT
value: "8080"
- image: huggingface/datasets-server-services-worker:sha-2c69f29
+ image: huggingface/datasets-server-services-worker:sha-58c3ce9
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 30
===== apps/Deployment datasets-server/prod-datasets-server-worker-medium ======
--- /tmp/argocd-diff3621563949/prod-datasets-server-worker-medium-live.yaml 2024-05-15 19:48:01.569890507 +0000
+++ /tmp/argocd-diff3621563949/prod-datasets-server-worker-medium 2024-05-15 19:48:01.565890459 +0000
@@ -681,7 +681,7 @@
value: "1"
- name: WORKER_UVICORN_PORT
value: "8080"
- image: huggingface/datasets-server-services-worker:sha-2c69f29
+ image: huggingface/datasets-server-services-worker:sha-58c3ce9
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 30
===== batch/CronJob datasets-server/prod-datasets-server-job-backfill ======
--- /tmp/argocd-diff962223924/prod-datasets-server-job-backfill-live.yaml 2024-05-15 19:48:01.593890790 +0000
+++ /tmp/argocd-diff962223924/prod-datasets-server-job-backfill 2024-05-15 19:48:01.593890790 +0000
@@ -256,7 +256,7 @@
value: backfill
- name: LOG_LEVEL
value: debug
- image: huggingface/datasets-server-jobs-cache_maintenance:sha-2c69f29
+ image: huggingface/datasets-server-jobs-cache_maintenance:sha-58c3ce9
imagePullPolicy: IfNotPresent
name: prod-datasets-server-backfill
resources:
===== batch/CronJob datasets-server/prod-datasets-server-job-backfill-retryable-errors ======
--- /tmp/argocd-diff466884636/prod-datasets-server-job-backfill-retryable-errors-live.yaml 2024-05-15 19:48:01.605890931 +0000
+++ /tmp/argocd-diff466884636/prod-datasets-server-job-backfill-retryable-errors 2024-05-15 19:48:01.601890883 +0000
@@ -256,7 +256,7 @@
value: backfill-retryable-errors
- name: LOG_LEVEL
value: debug
- image: huggingface/datasets-server-jobs-cache_maintenance:sha-2c69f29
+ image: huggingface/datasets-server-jobs-cache_maintenance:sha-58c3ce9
imagePullPolicy: IfNotPresent
name: prod-datasets-server-backfill-retryable-errors
resources:
===== batch/CronJob datasets-server/prod-datasets-server-job-cache-metrics-collector ======
--- /tmp/argocd-diff3166647213/prod-datasets-server-job-cache-metrics-collector-live.yaml 2024-05-15 19:48:01.613891025 +0000
+++ /tmp/argocd-diff3166647213/prod-datasets-server-job-cache-metrics-collector 2024-05-15 19:48:01.609890977 +0000
@@ -188,7 +188,7 @@
optional: false
- name: CACHE_MAINTENANCE_ACTION
value: collect-cache-metrics
- image: huggingface/datasets-server-jobs-cache_maintenance:sha-2c69f29
+ image: huggingface/datasets-server-jobs-cache_maintenance:sha-58c3ce9
imagePullPolicy: IfNotPresent
name: prod-datasets-server-cache-metrics-collector
resources:
===== batch/CronJob datasets-server/prod-datasets-server-job-post-messages ======
--- /tmp/argocd-diff757697292/prod-datasets-server-job-post-messages-live.yaml 2024-05-15 19:48:01.625891166 +0000
+++ /tmp/argocd-diff757697292/prod-datasets-server-job-post-messages 2024-05-15 19:48:01.621891119 +0000
@@ -211,7 +211,7 @@
value: post-messages
- name: LOG_LEVEL
value: info
- image: huggingface/datasets-server-jobs-cache_maintenance:sha-2c69f29
+ image: huggingface/datasets-server-jobs-cache_maintenance:sha-58c3ce9
imagePullPolicy: IfNotPresent
name: prod-datasets-server-post-messages
resources:
===== batch/CronJob datasets-server/prod-datasets-server-job-queue-metrics-collector ======
--- /tmp/argocd-diff562115790/prod-datasets-server-job-queue-metrics-collector-live.yaml 2024-05-15 19:48:01.633891260 +0000
+++ /tmp/argocd-diff562115790/prod-datasets-server-job-queue-metrics-collector 2024-05-15 19:48:01.629891213 +0000
@@ -188,7 +188,7 @@
optional: false
- name: CACHE_MAINTENANCE_ACTION
value: collect-queue-metrics
- image: huggingface/datasets-server-jobs-cache_maintenance:sha-2c69f29
+ image: huggingface/datasets-server-jobs-cache_maintenance:sha-58c3ce9
imagePullPolicy: IfNotPresent
name: prod-datasets-server-queue-metrics-collector
resources:
===== networking.k8s.io/Ingress datasets-server/prod-datasets-server ======
--- /tmp/argocd-diff844062500/prod-datasets-server-live.yaml 2024-05-15 19:48:01.641891354 +0000
+++ /tmp/argocd-diff844062500/prod-datasets-server 2024-05-15 19:48:01.641891354 +0000
@@ -1,110 +0,0 @@
-apiVersion: networking.k8s.io/v1
-kind: Ingress
-metadata:
- annotations:
- alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:707930574880:certificate/971187a3-2baa-40e5-bcae-94d6ec55cd24
- alb.ingress.kubernetes.io/group.name: datasets-server
- alb.ingress.kubernetes.io/group.order: "100"
- alb.ingress.kubernetes.io/healthcheck-path: /healthcheck
- alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80, "HTTPS": 443}]'
- alb.ingress.kubernetes.io/load-balancer-name: hub-datasets-server-prod
- alb.ingress.kubernetes.io/scheme: internet-facing
- alb.ingress.kubernetes.io/tags: Env=prod,Project=datasets-server,Terraform=true
- alb.ingress.kubernetes.io/target-node-labels: role-datasets-server=true
- kubectl.kubernetes.io/last-applied-configuration: |
- {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"alb.ingress.kubernetes.io/certificate-arn":"arn:aws:acm:us-east-1:707930574880:certificate/971187a3-2baa-40e5-bcae-94d6ec55cd24","alb.ingress.kubernetes.io/group.name":"datasets-server","alb.ingress.kubernetes.io/group.order":"100","alb.ingress.kubernetes.io/healthcheck-path":"/healthcheck","alb.ingress.kubernetes.io/listen-ports":"[{\"HTTP\": 80, \"HTTPS\": 443}]","alb.ingress.kubernetes.io/load-balancer-name":"hub-datasets-server-prod","alb.ingress.kubernetes.io/scheme":"internet-facing","alb.ingress.kubernetes.io/tags":"Env=prod,Project=datasets-server,Terraform=true","alb.ingress.kubernetes.io/target-node-labels":"role-datasets-server=true","kubernetes.io/ingress.class":"alb"},"labels":{"app.kubernetes.io/component":"prod-datasets-server-reverse-proxy","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","argocd.argoproj.io/instance":"datasets-server-prod","helm.sh/chart":"datasets-server"},"name":"prod-datasets-server","namespace":"datasets-server"},"spec":{"rules":[{"host":"datasets-server.huggingface.co","http":{"paths":[{"backend":{"service":{"name":"prod-datasets-server-reverse-proxy","port":{"name":"http"}}},"path":"/","pathType":"Prefix"}]}}],"tls":[{"hosts":["datasets-server.huggingface.co"]}]}}
- kubernetes.io/ingress.class: alb
- finalizers:
- - group.ingress.k8s.aws/datasets-server
- generation: 1
- labels:
- app.kubernetes.io/component: prod-datasets-server-reverse-proxy
- app.kubernetes.io/instance: prod
- app.kubernetes.io/managed-by: Helm
- app.kubernetes.io/name: prod-datasets-server
- argocd.argoproj.io/instance: datasets-server-prod
- helm.sh/chart: datasets-server
- managedFields:
- - apiVersion: networking.k8s.io/v1
- fieldsType: FieldsV1
- fieldsV1:
- f:metadata:
- f:annotations:
- .: {}
- f:alb.ingress.kubernetes.io/certificate-arn: {}
- f:alb.ingress.kubernetes.io/group.name: {}
- f:alb.ingress.kubernetes.io/healthcheck-path: {}
- f:alb.ingress.kubernetes.io/listen-ports: {}
- f:alb.ingress.kubernetes.io/load-balancer-name: {}
- f:alb.ingress.kubernetes.io/scheme: {}
- f:alb.ingress.kubernetes.io/tags: {}
- f:alb.ingress.kubernetes.io/target-node-labels: {}
- f:kubernetes.io/ingress.class: {}
- f:labels:
- .: {}
- f:app.kubernetes.io/component: {}
- f:app.kubernetes.io/instance: {}
- f:app.kubernetes.io/managed-by: {}
- f:app.kubernetes.io/name: {}
- f:helm.sh/chart: {}
- f:spec:
- f:rules: {}
- f:tls: {}
- manager: helm
- operation: Update
- time: "2023-01-30T14:28:11Z"
- - apiVersion: networking.k8s.io/v1
- fieldsType: FieldsV1
- fieldsV1:
- f:status:
- f:loadBalancer:
- f:ingress: {}
- manager: controller
- operation: Update
- subresource: status
- time: "2023-01-30T14:28:13Z"
- - apiVersion: networking.k8s.io/v1
- fieldsType: FieldsV1
- fieldsV1:
- f:metadata:
- f:finalizers:
- .: {}
- v:"group.ingress.k8s.aws/datasets-server": {}
- manager: controller
- operation: Update
- time: "2023-11-09T10:02:59Z"
- - apiVersion: networking.k8s.io/v1
- fieldsType: FieldsV1
- fieldsV1:
- f:metadata:
- f:annotations:
- f:alb.ingress.kubernetes.io/group.order: {}
- f:kubectl.kubernetes.io/last-applied-configuration: {}
- f:labels:
- f:argocd.argoproj.io/instance: {}
- manager: argocd-controller
- operation: Update
- time: "2024-04-12T13:13:48Z"
- name: prod-datasets-server
- namespace: datasets-server
- resourceVersion: "855572143"
- uid: db417bc6-dd5e-4013-8a0a-1754933c8f23
-spec:
- rules:
- - host: datasets-server.huggingface.co
- http:
- paths:
- - backend:
- service:
- name: prod-datasets-server-reverse-proxy
- port:
- name: http
- path: /
- pathType: Prefix
- tls:
- - hosts:
- - datasets-server.huggingface.co
-status:
- loadBalancer:
- ingress:
- - hostname: hub-datasets-server-prod-1473732607.us-east-1.elb.amazonaws.com
===== networking.k8s.io/Ingress datasets-server/prod-datasets-server-search ======
--- /tmp/argocd-diff2498906332/prod-datasets-server-search-live.yaml 2024-05-15 19:48:01.645891402 +0000
+++ /tmp/argocd-diff2498906332/prod-datasets-server-search 2024-05-15 19:48:01.645891402 +0000
@@ -4,7 +4,7 @@
annotations:
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:707930574880:certificate/971187a3-2baa-40e5-bcae-94d6ec55cd24
alb.ingress.kubernetes.io/group.name: datasets-server
- alb.ingress.kubernetes.io/group.order: "1"
+ alb.ingress.kubernetes.io/group.order: "3"
alb.ingress.kubernetes.io/healthcheck-path: /healthcheck
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80, "HTTPS": 443}]'
alb.ingress.kubernetes.io/load-balancer-name: hub-datasets-server-prod
@@ -104,6 +104,16 @@
name: http
path: /search
pathType: Prefix
+ - host: datasets-server.huggingface.co
+ http:
+ paths:
+ - backend:
+ service:
+ name: prod-datasets-server-search
+ port:
+ name: http
+ path: /filter
+ pathType: Prefix
tls:
- hosts:
- datasets-server.huggingface.co
===== networking.k8s.io/Ingress datasets-server/prod-datasets-server-rows ======
--- /tmp/argocd-diff527063497/prod-datasets-server-rows-live.yaml 2024-05-15 19:48:01.657891542 +0000
+++ /tmp/argocd-diff527063497/prod-datasets-server-rows 2024-05-15 19:48:01.657891542 +0000
@@ -0,0 +1,38 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ annotations:
+ alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:707930574880:certificate/971187a3-2baa-40e5-bcae-94d6ec55cd24
+ alb.ingress.kubernetes.io/group.name: datasets-server
+ alb.ingress.kubernetes.io/group.order: "2"
+ alb.ingress.kubernetes.io/healthcheck-path: /healthcheck
+ alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80, "HTTPS": 443}]'
+ alb.ingress.kubernetes.io/load-balancer-name: hub-datasets-server-prod
+ alb.ingress.kubernetes.io/scheme: internet-facing
+ alb.ingress.kubernetes.io/tags: Env=prod,Project=datasets-server,Terraform=true
+ alb.ingress.kubernetes.io/target-node-labels: role-datasets-server-rows=true
+ kubernetes.io/ingress.class: alb
+ labels:
+ app.kubernetes.io/component: prod-datasets-server-rows
+ app.kubernetes.io/instance: prod
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/name: prod-datasets-server
+ argocd.argoproj.io/instance: datasets-server-prod
+ helm.sh/chart: datasets-server
+ name: prod-datasets-server-rows
+ namespace: datasets-server
+spec:
+ rules:
+ - host: datasets-server.huggingface.co
+ http:
+ paths:
+ - backend:
+ service:
+ name: prod-datasets-server-rows
+ port:
+ name: http
+ path: /rows
+ pathType: Prefix
+ tls:
+ - hosts:
+ - datasets-server.huggingface.co
===== networking.k8s.io/Ingress datasets-server/prod-datasets-server-admin ======
--- /tmp/argocd-diff197958742/prod-datasets-server-admin-live.yaml 2024-05-15 19:48:01.657891542 +0000
+++ /tmp/argocd-diff197958742/prod-datasets-server-admin 2024-05-15 19:48:01.657891542 +0000
@@ -0,0 +1,38 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ annotations:
+ alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:707930574880:certificate/971187a3-2baa-40e5-bcae-94d6ec55cd24
+ alb.ingress.kubernetes.io/group.name: datasets-server
+ alb.ingress.kubernetes.io/group.order: "1"
+ alb.ingress.kubernetes.io/healthcheck-path: /healthcheck
+ alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80, "HTTPS": 443}]'
+ alb.ingress.kubernetes.io/load-balancer-name: hub-datasets-server-prod
+ alb.ingress.kubernetes.io/scheme: internet-facing
+ alb.ingress.kubernetes.io/tags: Env=prod,Project=datasets-server,Terraform=true
+ alb.ingress.kubernetes.io/target-node-labels: role-datasets-server=true
+ kubernetes.io/ingress.class: alb
+ labels:
+ app.kubernetes.io/component: prod-datasets-server-admin
+ app.kubernetes.io/instance: prod
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/name: prod-datasets-server
+ argocd.argoproj.io/instance: datasets-server-prod
+ helm.sh/chart: datasets-server
+ name: prod-datasets-server-admin
+ namespace: datasets-server
+spec:
+ rules:
+ - host: datasets-server.huggingface.co
+ http:
+ paths:
+ - backend:
+ service:
+ name: prod-datasets-server-admin
+ port:
+ name: http
+ path: /admin
+ pathType: Prefix
+ tls:
+ - hosts:
+ - datasets-server.huggingface.co
===== networking.k8s.io/Ingress datasets-server/prod-datasets-server-api ======
--- /tmp/argocd-diff2585252315/prod-datasets-server-api-live.yaml 2024-05-15 19:48:01.661891590 +0000
+++ /tmp/argocd-diff2585252315/prod-datasets-server-api 2024-05-15 19:48:01.661891590 +0000
@@ -0,0 +1,51 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ annotations:
+ alb.ingress.kubernetes.io/actions.metrics-unauthorized: '{"type":"fixed-response","fixedResponseConfig":{"contentType":"text/plain","statusCode":"401","messageBody":"401
+ Unauthorized"}}'
+ alb.ingress.kubernetes.io/actions.openapi-redirect: '{"Type":"redirect","RedirectConfig":{"Host":"raw.githubusercontent.com","Path":"/huggingface/dataset-viewer/main/docs/source/openapi.json","Port":"443","Protocol":"HTTPS","Query":"#{query}","StatusCode":"HTTP_307"}}'
+ alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:707930574880:certificate/971187a3-2baa-40e5-bcae-94d6ec55cd24
+ alb.ingress.kubernetes.io/group.name: datasets-server
+ alb.ingress.kubernetes.io/group.order: "100"
+ alb.ingress.kubernetes.io/healthcheck-path: /healthcheck
+ alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80, "HTTPS": 443}]'
+ alb.ingress.kubernetes.io/load-balancer-name: hub-datasets-server-prod
+ alb.ingress.kubernetes.io/scheme: internet-facing
+ alb.ingress.kubernetes.io/tags: Env=prod,Project=datasets-server,Terraform=true
+ alb.ingress.kubernetes.io/target-node-labels: role-datasets-server-api=true
+ kubernetes.io/ingress.class: alb
+ labels:
+ app.kubernetes.io/component: prod-datasets-server-api
+ app.kubernetes.io/instance: prod
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/name: prod-datasets-server
+ argocd.argoproj.io/instance: datasets-server-prod
+ helm.sh/chart: datasets-server
+ name: prod-datasets-server-api
+ namespace: datasets-server
+spec:
+ rules:
+ - host: datasets-server.huggingface.co
+ http:
+ paths:
+ - backend:
+ service:
+ name: prod-datasets-server-api
+ port:
+ name: http
+ path: /
+ pathType: Prefix
+ - backend:
+ serviceName: openapi-redirect
+ servicePort: use-annotation
+ path: /openapi.json
+ pathType: Exact
+ - backend:
+ serviceName: metrics-unauthorized
+ servicePort: use-annotation
+ path: /metrics
+ pathType: Exact
+ tls:
+ - hosts:
+ - datasets-server.huggingface.co
===== networking.k8s.io/Ingress datasets-server/prod-datasets-server-sse-api ======
--- /tmp/argocd-diff636853492/prod-datasets-server-sse-api-live.yaml 2024-05-15 19:48:01.661891590 +0000
+++ /tmp/argocd-diff636853492/prod-datasets-server-sse-api 2024-05-15 19:48:01.661891590 +0000
@@ -0,0 +1,38 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ annotations:
+ alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:707930574880:certificate/971187a3-2baa-40e5-bcae-94d6ec55cd24
+ alb.ingress.kubernetes.io/group.name: datasets-server
+ alb.ingress.kubernetes.io/group.order: "4"
+ alb.ingress.kubernetes.io/healthcheck-path: /healthcheck
+ alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80, "HTTPS": 443}]'
+ alb.ingress.kubernetes.io/load-balancer-name: hub-datasets-server-prod
+ alb.ingress.kubernetes.io/scheme: internet-facing
+ alb.ingress.kubernetes.io/tags: Env=prod,Project=datasets-server,Terraform=true
+ alb.ingress.kubernetes.io/target-node-labels: role-datasets-server=true
+ kubernetes.io/ingress.class: alb
+ labels:
+ app.kubernetes.io/component: prod-datasets-server-sse-api
+ app.kubernetes.io/instance: prod
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/name: prod-datasets-server
+ argocd.argoproj.io/instance: datasets-server-prod
+ helm.sh/chart: datasets-server
+ name: prod-datasets-server-sse-api
+ namespace: datasets-server
+spec:
+ rules:
+ - host: datasets-server.huggingface.co
+ http:
+ paths:
+ - backend:
+ service:
+ name: prod-datasets-server-sse-api
+ port:
+ name: http
+ path: /sse
+ pathType: Prefix
+ tls:
+ - hosts:
+ - datasets-server.huggingface.co
App: datasets-server-staging
YAML generation: Success 🟢
App sync status: Out of Sync ⚠️
===== apps/Deployment datasets-server/staging-datasets-server-admin ======
--- /tmp/argocd-diff3433744982/staging-datasets-server-admin-live.yaml 2024-05-15 19:48:02.897906136 +0000
+++ /tmp/argocd-diff3433744982/staging-datasets-server-admin 2024-05-15 19:48:02.897906136 +0000
@@ -449,7 +449,7 @@
value: "1"
- name: ADMIN_UVICORN_PORT
value: "8080"
- image: huggingface/datasets-server-services-admin:sha-2c69f29
+ image: huggingface/datasets-server-services-admin:sha-58c3ce9
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 30
===== apps/Deployment datasets-server/staging-datasets-server-api ======
--- /tmp/argocd-diff2505071376/staging-datasets-server-api-live.yaml 2024-05-15 19:48:02.913906325 +0000
+++ /tmp/argocd-diff2505071376/staging-datasets-server-api 2024-05-15 19:48:02.913906325 +0000
@@ -453,7 +453,7 @@
value: "1"
- name: API_UVICORN_PORT
value: "8080"
- image: huggingface/datasets-server-services-api:sha-2c69f29
+ image: huggingface/datasets-server-services-api:sha-58c3ce9
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 30
===== apps/Deployment datasets-server/staging-datasets-server-rows ======
--- /tmp/argocd-diff1360180051/staging-datasets-server-rows-live.yaml 2024-05-15 19:48:02.941906654 +0000
+++ /tmp/argocd-diff1360180051/staging-datasets-server-rows 2024-05-15 19:48:02.937906607 +0000
@@ -507,7 +507,7 @@
value: "8080"
- name: ROWS_INDEX_MAX_ARROW_DATA_IN_MEMORY
value: "300_000_000"
- image: huggingface/datasets-server-services-rows:sha-2c69f29
+ image: huggingface/datasets-server-services-rows:sha-58c3ce9
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 30
===== apps/Deployment datasets-server/staging-datasets-server-search ======
--- /tmp/argocd-diff1287647326/staging-datasets-server-search-live.yaml 2024-05-15 19:48:02.961906889 +0000
+++ /tmp/argocd-diff1287647326/staging-datasets-server-search 2024-05-15 19:48:02.957906842 +0000
@@ -482,7 +482,7 @@
value: /tmp/duckdb-extensions
- name: HF_HUB_ENABLE_HF_TRANSFER
value: "1"
- image: huggingface/datasets-server-services-search:sha-2c69f29
+ image: huggingface/datasets-server-services-search:sha-58c3ce9
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 30
===== apps/Deployment datasets-server/staging-datasets-server-sse-api ======
--- /tmp/argocd-diff2867408035/staging-datasets-server-sse-api-live.yaml 2024-05-15 19:48:02.973907031 +0000
+++ /tmp/argocd-diff2867408035/staging-datasets-server-sse-api 2024-05-15 19:48:02.969906983 +0000
@@ -319,7 +319,7 @@
value: "1"
- name: API_UVICORN_PORT
value: "8080"
- image: huggingface/datasets-server-services-sse-api:sha-2c69f29
+ image: huggingface/datasets-server-services-sse-api:sha-58c3ce9
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 30
===== apps/Deployment datasets-server/staging-datasets-server-storage-admin ======
--- /tmp/argocd-diff2970146402/staging-datasets-server-storage-admin-live.yaml 2024-05-15 19:48:02.977907077 +0000
+++ /tmp/argocd-diff2970146402/staging-datasets-server-storage-admin 2024-05-15 19:48:02.973907031 +0000
@@ -1,282 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
- annotations:
- deployment.kubernetes.io/revision: "341"
- kubectl.kubernetes.io/last-applied-configuration: |
- {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"staging-datasets-server-storage-admin","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","argocd.argoproj.io/instance":"datasets-server-staging","helm.sh/chart":"datasets-server"},"name":"staging-datasets-server-storage-admin","namespace":"datasets-server"},"spec":{"progressDeadlineSeconds":600,"replicas":1,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"staging-datasets-server-storage-admin","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","helm.sh/chart":"datasets-server"}},"template":{"metadata":{"labels":{"app.kubernetes.io/component":"staging-datasets-server-storage-admin","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","helm.sh/chart":"datasets-server"}},"spec":{"containers":[{"image":"huggingface/datasets-server-services-storage-admin:sha-2c69f29","imagePullPolicy":"IfNotPresent","name":"staging-datasets-server-storage-admin","resources":{"limits":{"cpu":1,"memory":"256Mi"},"requests":{"cpu":"50m","memory":"64Mi"}},"securityContext":{"runAsGroup":0,"runAsNonRoot":false,"runAsUser":0},"volumeMounts":[{"mountPath":"/storage/parquet-metadata","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":false,"subPath":"datasets-server/staging/parquet-metadata/"},{"mountPath":"/volumes/parquet-metadata","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":false}]}],"initContainers":[{"args":["chown 1000:3000 /mounted-path;"],"command":["/bin/sh","-c"],"image":"ubuntu:focal","imagePullPolicy":"IfNotPresent","name":"prepare-parquet-metadata","securityContext":{"runAsGroup":0,"runAsNonRoot":false,"runAsUser":0},"volumeMounts":[{"mountPath":"/mounted-path","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":false,"subPath":"datasets-server/staging/parquet-metadata/"}]}],"nodeSelector":{},"tolerations":[],"volumes":[{"name":"volume-parquet-metadata","persistentVolumeClaim":{"claimName":"datasets-server-parquet-pvc"}}]}}}}
- generation: 341
- labels:
- app.kubernetes.io/component: staging-datasets-server-storage-admin
- app.kubernetes.io/instance: staging
- app.kubernetes.io/managed-by: Helm
- app.kubernetes.io/name: staging-datasets-server
- argocd.argoproj.io/instance: datasets-server-staging
- helm.sh/chart: datasets-server
- managedFields:
- - apiVersion: apps/v1
- fieldsType: FieldsV1
- fieldsV1:
- f:spec:
- f:template:
- f:metadata:
- f:annotations:
- .: {}
- f:kubectl.kubernetes.io/restartedAt: {}
- manager: k9s
- operation: Update
- time: "2023-11-13T16:14:52Z"
- - apiVersion: apps/v1
- fieldsType: FieldsV1
- fieldsV1:
- f:metadata:
- f:annotations:
- .: {}
- f:kubectl.kubernetes.io/last-applied-configuration: {}
- f:labels:
- .: {}
- f:app.kubernetes.io/component: {}
- f:app.kubernetes.io/instance: {}
- f:app.kubernetes.io/managed-by: {}
- f:app.kubernetes.io/name: {}
- f:argocd.argoproj.io/instance: {}
- f:helm.sh/chart: {}
- f:spec:
- f:progressDeadlineSeconds: {}
- f:replicas: {}
- f:revisionHistoryLimit: {}
- f:selector: {}
- f:strategy:
- f:rollingUpdate:
- .: {}
- f:maxSurge: {}
- f:maxUnavailable: {}
- f:type: {}
- f:template:
- f:metadata:
- f:labels:
- .: {}
- f:app.kubernetes.io/component: {}
- f:app.kubernetes.io/instance: {}
- f:app.kubernetes.io/managed-by: {}
- f:app.kubernetes.io/name: {}
- f:helm.sh/chart: {}
- f:spec:
- f:containers:
- k:{"name":"staging-datasets-server-storage-admin"}:
- .: {}
- f:image: {}
- f:imagePullPolicy: {}
- f:name: {}
- f:resources:
- .: {}
- f:limits:
- .: {}
- f:cpu: {}
- f:memory: {}
- f:requests:
- .: {}
- f:cpu: {}
- f:memory: {}
- f:securityContext:
- .: {}
- f:runAsGroup: {}
- f:runAsNonRoot: {}
- f:runAsUser: {}
- f:terminationMessagePath: {}
- f:terminationMessagePolicy: {}
- f:volumeMounts:
- .: {}
- k:{"mountPath":"/storage/parquet-metadata"}:
- .: {}
- f:mountPath: {}
- f:mountPropagation: {}
- f:name: {}
- f:subPath: {}
- k:{"mountPath":"/volumes/parquet-metadata"}:
- .: {}
- f:mountPath: {}
- f:mountPropagation: {}
- f:name: {}
- f:dnsPolicy: {}
- f:initContainers:
- .: {}
- k:{"name":"prepare-parquet-metadata"}:
- .: {}
- f:args: {}
- f:command: {}
- f:image: {}
- f:imagePullPolicy: {}
- f:name: {}
- f:resources: {}
- f:securityContext:
- .: {}
- f:runAsGroup: {}
- f:runAsNonRoot: {}
- f:runAsUser: {}
- f:terminationMessagePath: {}
- f:terminationMessagePolicy: {}
- f:volumeMounts:
- .: {}
- k:{"mountPath":"/mounted-path"}:
- .: {}
- f:mountPath: {}
- f:mountPropagation: {}
- f:name: {}
- f:subPath: {}
- f:restartPolicy: {}
- f:schedulerName: {}
- f:securityContext: {}
- f:terminationGracePeriodSeconds: {}
- f:volumes:
- .: {}
- k:{"name":"volume-parquet-metadata"}:
- .: {}
- f:name: {}
- f:persistentVolumeClaim:
- .: {}
- f:claimName: {}
- manager: argocd-controller
- operation: Update
- time: "2024-05-15T13:44:59Z"
- - apiVersion: apps/v1
- fieldsType: FieldsV1
- fieldsV1:
- f:metadata:
- f:annotations:
- f:deployment.kubernetes.io/revision: {}
- f:status:
- f:availableReplicas: {}
- f:conditions:
- .: {}
- k:{"type":"Available"}:
- .: {}
- f:lastTransitionTime: {}
- f:lastUpdateTime: {}
- f:message: {}
- f:reason: {}
- f:status: {}
- f:type: {}
- k:{"type":"Progressing"}:
- .: {}
- f:lastTransitionTime: {}
- f:lastUpdateTime: {}
- f:message: {}
- f:reason: {}
- f:status: {}
- f:type: {}
- f:observedGeneration: {}
- f:readyReplicas: {}
- f:replicas: {}
- f:updatedReplicas: {}
- manager: kube-controller-manager
- operation: Update
- subresource: status
- time: "2024-05-15T13:45:25Z"
- name: staging-datasets-server-storage-admin
- namespace: datasets-server
- resourceVersion: "364841609"
- uid: 972d142e-bf95-4aa3-ba25-b5f742dbe41d
-spec:
- progressDeadlineSeconds: 600
- replicas: 1
- revisionHistoryLimit: 10
- selector:
- matchLabels:
- app.kubernetes.io/component: staging-datasets-server-storage-admin
- app.kubernetes.io/instance: staging
- app.kubernetes.io/managed-by: Helm
- app.kubernetes.io/name: staging-datasets-server
- helm.sh/chart: datasets-server
- strategy:
- rollingUpdate:
- maxSurge: 25%
- maxUnavailable: 25%
- type: RollingUpdate
- template:
- metadata:
- annotations:
- kubectl.kubernetes.io/restartedAt: "2023-11-13T17:14:52+01:00"
- creationTimestamp: null
- labels:
- app.kubernetes.io/component: staging-datasets-server-storage-admin
- app.kubernetes.io/instance: staging
- app.kubernetes.io/managed-by: Helm
- app.kubernetes.io/name: staging-datasets-server
- helm.sh/chart: datasets-server
- spec:
- containers:
- - image: huggingface/datasets-server-services-storage-admin:sha-2c69f29
- imagePullPolicy: IfNotPresent
- name: staging-datasets-server-storage-admin
- resources:
- limits:
- cpu: "1"
- memory: 256Mi
- requests:
- cpu: 50m
- memory: 64Mi
- securityContext:
- runAsGroup: 0
- runAsNonRoot: false
- runAsUser: 0
- terminationMessagePath: /dev/termination-log
- terminationMessagePolicy: File
- volumeMounts:
- - mountPath: /storage/parquet-metadata
- mountPropagation: None
- name: volume-parquet-metadata
- subPath: datasets-server/staging/parquet-metadata/
- - mountPath: /volumes/parquet-metadata
- mountPropagation: None
- name: volume-parquet-metadata
- dnsPolicy: ClusterFirst
- initContainers:
- - args:
- - chown 1000:3000 /mounted-path;
- command:
- - /bin/sh
- - -c
- image: ubuntu:focal
- imagePullPolicy: IfNotPresent
- name: prepare-parquet-metadata
- resources: {}
- securityContext:
- runAsGroup: 0
- runAsNonRoot: false
- runAsUser: 0
- terminationMessagePath: /dev/termination-log
- terminationMessagePolicy: File
- volumeMounts:
- - mountPath: /mounted-path
- mountPropagation: None
- name: volume-parquet-metadata
- subPath: datasets-server/staging/parquet-metadata/
- restartPolicy: Always
- schedulerName: default-scheduler
- securityContext: {}
- terminationGracePeriodSeconds: 30
- volumes:
- - name: volume-parquet-metadata
- persistentVolumeClaim:
- claimName: datasets-server-parquet-pvc
-status:
- availableReplicas: 1
- conditions:
- - lastTransitionTime: "2024-05-15T12:05:12Z"
- lastUpdateTime: "2024-05-15T12:05:12Z"
- message: Deployment has minimum availability.
- reason: MinimumReplicasAvailable
- status: "True"
- type: Available
- - lastTransitionTime: "2024-04-16T14:35:58Z"
- lastUpdateTime: "2024-05-15T13:45:25Z"
- message: ReplicaSet "staging-datasets-server-storage-admin-c7dd44dcc" has successfully
- progressed.
- reason: NewReplicaSetAvailable
- status: "True"
- type: Progressing
- observedGeneration: 341
- readyReplicas: 1
- replicas: 1
- updatedReplicas: 1
===== apps/Deployment datasets-server/staging-datasets-server-worker-all ======
--- /tmp/argocd-diff2479868724/staging-datasets-server-worker-all-live.yaml 2024-05-15 19:48:03.001907360 +0000
+++ /tmp/argocd-diff2479868724/staging-datasets-server-worker-all 2024-05-15 19:48:02.997907313 +0000
@@ -680,7 +680,7 @@
value: "1"
- name: WORKER_UVICORN_PORT
value: "8080"
- image: huggingface/datasets-server-services-worker:sha-2c69f29
+ image: huggingface/datasets-server-services-worker:sha-58c3ce9
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 30
===== apps/Deployment datasets-server/staging-datasets-server-worker-light ======
--- /tmp/argocd-diff3042316029/staging-datasets-server-worker-light-live.yaml 2024-05-15 19:48:03.029907690 +0000
+++ /tmp/argocd-diff3042316029/staging-datasets-server-worker-light 2024-05-15 19:48:03.021907596 +0000
@@ -680,7 +680,7 @@
value: "1"
- name: WORKER_UVICORN_PORT
value: "8080"
- image: huggingface/datasets-server-services-worker:sha-2c69f29
+ image: huggingface/datasets-server-services-worker:sha-58c3ce9
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 30
===== batch/CronJob datasets-server/staging-datasets-server-job-cache-metrics-collector ======
--- /tmp/argocd-diff1821327112/staging-datasets-server-job-cache-metrics-collector-live.yaml 2024-05-15 19:48:03.045907877 +0000
+++ /tmp/argocd-diff1821327112/staging-datasets-server-job-cache-metrics-collector 2024-05-15 19:48:03.045907877 +0000
@@ -186,7 +186,7 @@
optional: false
- name: CACHE_MAINTENANCE_ACTION
value: collect-cache-metrics
- image: huggingface/datasets-server-jobs-cache_maintenance:sha-2c69f29
+ image: huggingface/datasets-server-jobs-cache_maintenance:sha-58c3ce9
imagePullPolicy: IfNotPresent
name: staging-datasets-server-cache-metrics-collector
resources:
===== batch/CronJob datasets-server/staging-datasets-server-job-post-messages ======
--- /tmp/argocd-diff2392510189/staging-datasets-server-job-post-messages-live.yaml 2024-05-15 19:48:03.053907972 +0000
+++ /tmp/argocd-diff2392510189/staging-datasets-server-job-post-messages 2024-05-15 19:48:03.053907972 +0000
@@ -210,7 +210,7 @@
value: post-messages
- name: LOG_LEVEL
value: info
- image: huggingface/datasets-server-jobs-cache_maintenance:sha-2c69f29
+ image: huggingface/datasets-server-jobs-cache_maintenance:sha-58c3ce9
imagePullPolicy: IfNotPresent
name: staging-datasets-server-post-messages
resources:
===== batch/CronJob datasets-server/staging-datasets-server-job-queue-metrics-collector ======
--- /tmp/argocd-diff2416487229/staging-datasets-server-job-queue-metrics-collector-live.yaml 2024-05-15 19:48:03.065908114 +0000
+++ /tmp/argocd-diff2416487229/staging-datasets-server-job-queue-metrics-collector 2024-05-15 19:48:03.061908066 +0000
@@ -187,7 +187,7 @@
optional: false
- name: CACHE_MAINTENANCE_ACTION
value: collect-queue-metrics
- image: huggingface/datasets-server-jobs-cache_maintenance:sha-2c69f29
+ image: huggingface/datasets-server-jobs-cache_maintenance:sha-58c3ce9
imagePullPolicy: IfNotPresent
name: staging-datasets-server-queue-metrics-collector
resources:
===== networking.k8s.io/Ingress datasets-server/staging-datasets-server ======
--- /tmp/argocd-diff770952525/staging-datasets-server-live.yaml 2024-05-15 19:48:03.073908208 +0000
+++ /tmp/argocd-diff770952525/staging-datasets-server 2024-05-15 19:48:03.069908160 +0000
@@ -1,95 +0,0 @@
-apiVersion: networking.k8s.io/v1
-kind: Ingress
-metadata:
- annotations:
- alb.ingress.kubernetes.io/group.name: datasets-server
- alb.ingress.kubernetes.io/healthcheck-path: /healthcheck
- alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80, "HTTPS": 443}]'
- alb.ingress.kubernetes.io/load-balancer-name: hub-datasets-server-staging
- alb.ingress.kubernetes.io/scheme: internet-facing
- alb.ingress.kubernetes.io/tags: Env=staging,Project=datasets-server,Terraform=true
- kubectl.kubernetes.io/last-applied-configuration: |
- {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"alb.ingress.kubernetes.io/group.name":"datasets-server","alb.ingress.kubernetes.io/healthcheck-path":"/healthcheck","alb.ingress.kubernetes.io/listen-ports":"[{\"HTTP\": 80, \"HTTPS\": 443}]","alb.ingress.kubernetes.io/load-balancer-name":"hub-datasets-server-staging","alb.ingress.kubernetes.io/scheme":"internet-facing","alb.ingress.kubernetes.io/tags":"Env=staging,Project=datasets-server,Terraform=true","kubernetes.io/ingress.class":"alb"},"labels":{"app.kubernetes.io/component":"staging-datasets-server-reverse-proxy","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","argocd.argoproj.io/instance":"datasets-server-staging","helm.sh/chart":"datasets-server"},"name":"staging-datasets-server","namespace":"datasets-server"},"spec":{"rules":[{"host":"datasets-server.us.dev.moon.huggingface.tech","http":{"paths":[{"backend":{"service":{"name":"staging-datasets-server-reverse-proxy","port":{"name":"http"}}},"path":"/","pathType":"Prefix"}]}}],"tls":[{"hosts":["datasets-server.us.dev.moon.huggingface.tech"]}]}}
- kubernetes.io/ingress.class: alb
- finalizers:
- - group.ingress.k8s.aws/datasets-server
- generation: 3
- labels:
- app.kubernetes.io/component: staging-datasets-server-reverse-proxy
- app.kubernetes.io/instance: staging
- app.kubernetes.io/managed-by: Helm
- app.kubernetes.io/name: staging-datasets-server
- argocd.argoproj.io/instance: datasets-server-staging
- helm.sh/chart: datasets-server
- managedFields:
- - apiVersion: networking.k8s.io/v1
- fieldsType: FieldsV1
- fieldsV1:
- f:metadata:
- f:finalizers:
- .: {}
- v:"group.ingress.k8s.aws/datasets-server": {}
- manager: controller
- operation: Update
- time: "2023-06-19T12:40:25Z"
- - apiVersion: networking.k8s.io/v1
- fieldsType: FieldsV1
- fieldsV1:
- f:status:
- f:loadBalancer:
- f:ingress: {}
- manager: controller
- operation: Update
- subresource: status
- time: "2023-06-19T15:40:55Z"
- - apiVersion: networking.k8s.io/v1
- fieldsType: FieldsV1
- fieldsV1:
- f:metadata:
- f:annotations:
- .: {}
- f:alb.ingress.kubernetes.io/group.name: {}
- f:alb.ingress.kubernetes.io/healthcheck-path: {}
- f:alb.ingress.kubernetes.io/listen-ports: {}
- f:alb.ingress.kubernetes.io/load-balancer-name: {}
- f:alb.ingress.kubernetes.io/scheme: {}
- f:alb.ingress.kubernetes.io/tags: {}
- f:kubectl.kubernetes.io/last-applied-configuration: {}
- f:kubernetes.io/ingress.class: {}
- f:labels:
- .: {}
- f:app.kubernetes.io/component: {}
- f:app.kubernetes.io/instance: {}
- f:app.kubernetes.io/managed-by: {}
- f:app.kubernetes.io/name: {}
- f:argocd.argoproj.io/instance: {}
- f:helm.sh/chart: {}
- f:spec:
- f:rules: {}
- f:tls: {}
- manager: argocd-controller
- operation: Update
- time: "2023-10-26T16:55:52Z"
- name: staging-datasets-server
- namespace: datasets-server
- resourceVersion: "249549276"
- uid: 6d9a264e-cbf1-4e6f-807c-718cee12e6e5
-spec:
- rules:
- - host: datasets-server.us.dev.moon.huggingface.tech
- http:
- paths:
- - backend:
- service:
- name: staging-datasets-server-reverse-proxy
- port:
- name: http
- path: /
- pathType: Prefix
- tls:
- - hosts:
- - datasets-server.us.dev.moon.huggingface.tech
-status:
- loadBalancer:
- ingress:
- - hostname: hub-datasets-server-dev-163133244.us-east-1.elb.amazonaws.com
===== networking.k8s.io/Ingress datasets-server/staging-datasets-server-search ======
--- /tmp/argocd-diff430099096/staging-datasets-server-search-live.yaml 2024-05-15 19:48:03.077908254 +0000
+++ /tmp/argocd-diff430099096/staging-datasets-server-search 2024-05-15 19:48:03.077908254 +0000
@@ -3,6 +3,7 @@
metadata:
annotations:
alb.ingress.kubernetes.io/group.name: datasets-server
+ alb.ingress.kubernetes.io/group.order: "3"
alb.ingress.kubernetes.io/healthcheck-path: /healthcheck
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80, "HTTPS": 443}]'
alb.ingress.kubernetes.io/load-balancer-name: hub-datasets-server-staging
@@ -86,6 +87,16 @@
name: http
path: /search
pathType: Prefix
+ - host: datasets-server.us.dev.moon.huggingface.tech
+ http:
+ paths:
+ - backend:
+ service:
+ name: staging-datasets-server-search
+ port:
+ name: http
+ path: /filter
+ pathType: Prefix
tls:
- hosts:
- datasets-server.us.dev.moon.huggingface.tech
===== networking.k8s.io/Ingress datasets-server/staging-datasets-server-api ======
--- /tmp/argocd-diff441313762/staging-datasets-server-api-live.yaml 2024-05-15 19:48:03.085908348 +0000
+++ /tmp/argocd-diff441313762/staging-datasets-server-api 2024-05-15 19:48:03.085908348 +0000
@@ -0,0 +1,36 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ annotations:
+ alb.ingress.kubernetes.io/group.name: datasets-server
+ alb.ingress.kubernetes.io/group.order: "100"
+ alb.ingress.kubernetes.io/healthcheck-path: /healthcheck
+ alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80, "HTTPS": 443}]'
+ alb.ingress.kubernetes.io/load-balancer-name: hub-datasets-server-staging
+ alb.ingress.kubernetes.io/scheme: internet-facing
+ alb.ingress.kubernetes.io/tags: Env=staging,Project=datasets-server,Terraform=true
+ kubernetes.io/ingress.class: alb
+ labels:
+ app.kubernetes.io/component: staging-datasets-server-api
+ app.kubernetes.io/instance: staging
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/name: staging-datasets-server
+ argocd.argoproj.io/instance: datasets-server-staging
+ helm.sh/chart: datasets-server
+ name: staging-datasets-server-api
+ namespace: datasets-server
+spec:
+ rules:
+ - host: datasets-server.us.dev.moon.huggingface.tech
+ http:
+ paths:
+ - backend:
+ service:
+ name: staging-datasets-server-api
+ port:
+ name: http
+ path: /
+ pathType: Prefix
+ tls:
+ - hosts:
+ - datasets-server.us.dev.moon.huggingface.tech
===== networking.k8s.io/Ingress datasets-server/staging-datasets-server-rows ======
--- /tmp/argocd-diff2224623219/staging-datasets-server-rows-live.yaml 2024-05-15 19:48:03.089908396 +0000
+++ /tmp/argocd-diff2224623219/staging-datasets-server-rows 2024-05-15 19:48:03.089908396 +0000
@@ -0,0 +1,36 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ annotations:
+ alb.ingress.kubernetes.io/group.name: datasets-server
+ alb.ingress.kubernetes.io/group.order: "2"
+ alb.ingress.kubernetes.io/healthcheck-path: /healthcheck
+ alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80, "HTTPS": 443}]'
+ alb.ingress.kubernetes.io/load-balancer-name: hub-datasets-server-staging
+ alb.ingress.kubernetes.io/scheme: internet-facing
+ alb.ingress.kubernetes.io/tags: Env=staging,Project=datasets-server,Terraform=true
+ kubernetes.io/ingress.class: alb
+ labels:
+ app.kubernetes.io/component: staging-datasets-server-rows
+ app.kubernetes.io/instance: staging
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/name: staging-datasets-server
+ argocd.argoproj.io/instance: datasets-server-staging
+ helm.sh/chart: datasets-server
+ name: staging-datasets-server-rows
+ namespace: datasets-server
+spec:
+ rules:
+ - host: datasets-server.us.dev.moon.huggingface.tech
+ http:
+ paths:
+ - backend:
+ service:
+ name: staging-datasets-server-rows
+ port:
+ name: http
+ path: /rows
+ pathType: Prefix
+ tls:
+ - hosts:
+ - datasets-server.us.dev.moon.huggingface.tech
===== networking.k8s.io/Ingress datasets-server/staging-datasets-server-admin ======
--- /tmp/argocd-diff1487748644/staging-datasets-server-admin-live.yaml 2024-05-15 19:48:03.089908396 +0000
+++ /tmp/argocd-diff1487748644/staging-datasets-server-admin 2024-05-15 19:48:03.089908396 +0000
@@ -0,0 +1,36 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ annotations:
+ alb.ingress.kubernetes.io/group.name: datasets-server
+ alb.ingress.kubernetes.io/group.order: "1"
+ alb.ingress.kubernetes.io/healthcheck-path: /healthcheck
+ alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80, "HTTPS": 443}]'
+ alb.ingress.kubernetes.io/load-balancer-name: hub-datasets-server-staging
+ alb.ingress.kubernetes.io/scheme: internet-facing
+ alb.ingress.kubernetes.io/tags: Env=staging,Project=datasets-server,Terraform=true
+ kubernetes.io/ingress.class: alb
+ labels:
+ app.kubernetes.io/component: staging-datasets-server-admin
+ app.kubernetes.io/instance: staging
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/name: staging-datasets-server
+ argocd.argoproj.io/instance: datasets-server-staging
+ helm.sh/chart: datasets-server
+ name: staging-datasets-server-admin
+ namespace: datasets-server
+spec:
+ rules:
+ - host: datasets-server.us.dev.moon.huggingface.tech
+ http:
+ paths:
+ - backend:
+ service:
+ name: staging-datasets-server-admin
+ port:
+ name: http
+ path: /admin
+ pathType: Prefix
+ tls:
+ - hosts:
+ - datasets-server.us.dev.moon.huggingface.tech
===== networking.k8s.io/Ingress datasets-server/staging-datasets-server-sse-api ======
--- /tmp/argocd-diff2318268446/staging-datasets-server-sse-api-live.yaml 2024-05-15 19:48:03.093908443 +0000
+++ /tmp/argocd-diff2318268446/staging-datasets-server-sse-api 2024-05-15 19:48:03.093908443 +0000
@@ -0,0 +1,36 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ annotations:
+ alb.ingress.kubernetes.io/group.name: datasets-server
+ alb.ingress.kubernetes.io/group.order: "100"
+ alb.ingress.kubernetes.io/healthcheck-path: /healthcheck
+ alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80, "HTTPS": 443}]'
+ alb.ingress.kubernetes.io/load-balancer-name: hub-datasets-server-staging
+ alb.ingress.kubernetes.io/scheme: internet-facing
+ alb.ingress.kubernetes.io/tags: Env=staging,Project=datasets-server,Terraform=true
+ kubernetes.io/ingress.class: alb
+ labels:
+ app.kubernetes.io/component: staging-datasets-server-sse-api
+ app.kubernetes.io/instance: staging
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/name: staging-datasets-server
+ argocd.argoproj.io/instance: datasets-server-staging
+ helm.sh/chart: datasets-server
+ name: staging-datasets-server-sse-api
+ namespace: datasets-server
+spec:
+ rules:
+ - host: datasets-server.us.dev.moon.huggingface.tech
+ http:
+ paths:
+ - backend:
+ service:
+ name: staging-datasets-server-sse-api
+ port:
+ name: http
+ path: /sse
+ pathType: Prefix
+ tls:
+ - hosts:
+ - datasets-server.us.dev.moon.huggingface.tech
| Legend | Status |
|---|---|
| ✅ | The app is synced in ArgoCD, and diffs you see are solely from this PR. |
| ⚠️ | The app is out-of-sync in ArgoCD, and the diffs you see include those changes plus any from this PR. |
| 🛑 | There was an error generating the ArgoCD diffs due to changes in this PR. |
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
expose assets and cached-assets ? Should not be served by API anymore right ?
Yes, it should be served by S3+CloudFront if I am not wrong
@severo can you please tell me if openapi.json endpoint is still necessary ?
And if we have to customize the 404 error page ?
Yes, we need it somewhere. Not necessarily at that URL though, if it's an issue. re 404: not important at all
re: openapi.json. Ideally, it would be served from the docs. See the previous discussions here: https://github.com/huggingface/dataset-viewer/commit/cf8042da9a2325a1d5706dbecded9b187d5cc8db and here: https://github.com/huggingface/doc-builder/issues/312#issuecomment-1675099444. So: not a problem at all if we move its URL, we would just have to update it in the docs.
re: openapi.json. Ideally, it would be served from the docs. See the previous discussions here: cf8042d and here: huggingface/doc-builder#312 (comment). So: not a problem at all if we move its URL, we would just have to update it in the docs.
We can do the redirect with a custom annotation. So let me know if you want to move it or not ;)
I'm wondering: would it make sense to have all the paths in the same ingress config file? or is it better to have one file per service?
Yes we can. Do you think it will be easier for you? IMO it's better to keep it separate, in case you want to have a dedicated URL (ex: api.datasets.hf.co)
We can do the redirect with a custom annotation. So let me know if you want to move it or not ;)
Please let's keep it unchanged if it's possible, and we can move it later in a dedicated PR
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
We should finish this PR. Happy to give you a hand @rtrompier
Thx for the reminder. I plan to finish it in the week.