clickhouse-backup
clickhouse-backup copied to clipboard
Published
20 hours ago •
Altinity
Altinity
`upload` causes OOM killed - the whole zipped backup is less than 1GB
I give 2GB memory limit to the backup container. and it is oom killed... the whole zipped backup is less than 1GB!
Then, what if the clickhouse has 500GB data, Will this backup program needs 500GB or memory? :(
related logs:
2021/06/29 00:54:26 info done backup=2021-06-29T00-54-25 operation=create
0 / 766500453 0.00% 1.88 MiB / 730.99 MiB 0.26% 02m29s 1.88 MiB / 730.99 MiB 0.26% 02m28s 6.10 MiB / 730.99 MiB 0.83% 01m33s 6.10 MiB / 730.99 MiB 0.83% 01m33s 7.54 MiB / 730.99 MiB 1.03% 01m53s 9.71 MiB / 730.99 MiB 1.33% 01m42s 17.91 MiB / 730.99 MiB 2.45% 01m02s 17.91 MiB / 730.99 MiB 2.45% 01m02s 17.91 MiB / 730.99 MiB 2.45% 01m02s 30.72 MiB / 730.99 MiB 4.20% 00m49s 30.72 MiB / 730.99 MiB 4.20% 00m49s 30.72 MiB / 730.99 MiB 4.20% 00m49s 30.72 MiB / 730.99 MiB 4.20% 00m49s 30.72 MiB / 730.99 MiB 4.20% 00m49s 35.07 MiB / 730.99 MiB 4.80% 01m02s 44.88 MiB / 730.99 MiB 6.14% 00m51s 44.88 MiB / 730.99 MiB 6.14% 00m51s 91.44 MiB / 730.99 MiB 12.51% 00m26s 91.44 MiB / 730.99 MiB 12.51% 00m25s 91.44 MiB / 730.99 MiB 12.51% 00m25s 91.44 MiB / 730.99 MiB 12.51% 00m25s 91.44 MiB / 730.99 MiB 12.51% 00m25s 91.44 MiB / 730.99 MiB 12.51% 00m25s 91.44 MiB / 730.99 MiB 12.51% 00m24s 91.44 MiB / 730.99 MiB 12.51% 00m24s 91.44 MiB / 730.99 MiB 12.51% 00m24s 91.44 MiB / 730.99 MiB 12.51% 00m24s 91.44 MiB / 730.99 MiB 12.51% 00m24s 93.82 MiB / 730.99 MiB 12.84% 00m40s 99.51 MiB / 730.99 MiB 13.61% 00m39s 103.27 MiB / 730.99 MiB 14.13% 00m38s 116.73 MiB / 730.99 MiB 15.97% 00m34s 117.80 MiB / 730.99 MiB 16.12% 00m35s 118.03 MiB / 730.99 MiB 16.15% 00m35s 122.65 MiB / 730.99 MiB 16.78% 00m35s 122.65 MiB / 730.99 MiB 16.78% 00m35s 130.80 MiB / 730.99 MiB 17.89% 00m34s 130.80 MiB / 730.99 MiB 17.89% 00m34s 130.80 MiB / 730.99 MiB 17.89% 00m34s 132.53 MiB / 730.99 MiB 18.13% 00m37s 136.57 MiB / 730.99 MiB 18.68% 00m36s 143.08 MiB / 730.99 MiB 19.57% 00m35s 143.08 MiB / 730.99 MiB 19.57% 00m34s 172.76 MiB / 730.99 MiB 23.63% 00m28s 172.76 MiB / 730.99 MiB 23.63% 00m28s 172.76 MiB / 730.99 MiB 23.63% 00m28s 172.76 MiB / 730.99 MiB 23.63% 00m28s 172.76 MiB / 730.99 MiB 23.63% 00m27s 173.31 MiB / 730.99 MiB 23.71% 00m32s 173.98 MiB / 730.99 MiB 23.80% 00m32s 178.16 MiB / 730.99 MiB 24.37% 00m31s 179.37 MiB / 730.99 MiB 24.54% 00m32s 180.08 MiB / 730.99 MiB 24.64% 00m32s 184.86 MiB / 730.99 MiB 25.29% 00m32s 194.25 MiB / 730.99 MiB 26.57% 00m30s 194.25 MiB / 730.99 MiB 26.57% 00m30s 213.11 MiB / 730.99 MiB 29.15% 00m27s 213.11 MiB / 730.99 MiB 29.15% 00m27s 214.78 MiB / 730.99 MiB 29.38% 00m28s 214.99 MiB / 730.99 MiB 29.41% 00m29s 220.04 MiB / 730.99 MiB 30.10% 00m28s 228.83 MiB / 730.99 MiB 31.30% 00m27s 228.83 MiB / 730.99 MiB 31.30% 00m27s 228.83 MiB / 730.99 MiB 31.30% 00m26s 285.65 MiB / 730.99 MiB 39.08% 00m20s 285.65 MiB / 730.99 MiB 39.08% 00m20s 285.65 MiB / 730.99 MiB 39.08% 00m20s 285.65 MiB / 730.99 MiB 39.08% 00m19s 285.65 MiB / 730.99 MiB 39.08% 00m19s 285.65 MiB / 730.99 MiB 39.08% 00m19s 286.41 MiB / 730.99 MiB 39.18% 00m22s 286.60 MiB / 730.99 MiB 39.21% 00m22s 291.08 MiB / 730.99 MiB 39.82% 00m22s 291.08 MiB / 730.99 MiB 39.82% 00m21s 291.73 MiB / 730.99 MiB 39.91% 00m22s 306.18 MiB / 730.99 MiB 41.89% 00m21s 306.18 MiB / 730.99 MiB 41.89% 00m21s 306.18 MiB / 730.99 MiB 41.89% 00m20s 306.18 MiB / 730.99 MiB 41.89% 00m20s 306.18 MiB / 730.99 MiB 41.89% 00m20s 331.24 MiB / 730.99 MiB 45.31% 00m19s 331.24 MiB / 730.99 MiB 45.31% 00m19s 331.24 MiB / 730.99 MiB 45.31% 00m19s 331.24 MiB / 730.99 MiB 45.31% 00m19s 331.24 MiB / 730.99 MiB 45.31% 00m18s 331.24 MiB / 730.99 MiB 45.31% 00m18s 331.24 MiB / 730.99 MiB 45.31% 00m18s 331.24 MiB / 730.99 MiB 45.31% 00m18s 331.24 MiB / 730.99 MiB 45.31% 00m18s 331.24 MiB / 730.99 MiB 45.31% 00m17s 338.68 MiB / 730.99 MiB 46.33% 00m20s 347.27 MiB / 730.99 MiB 47.51% 00m20s 347.27 MiB / 730.99 MiB 47.51% 00m20s 347.27 MiB / 730.99 MiB 47.51% 00m20s 347.27 MiB / 730.99 MiB 47.51% 00m19s 355.86 MiB / 730.99 MiB 48.68% 00m20s 355.86 MiB / 730.99 MiB 48.68% 00m20s 355.86 MiB / 730.99 MiB 48.68% 00m20s 356.55 MiB / 730.99 MiB 48.78% 00m20s 357.56 MiB / 730.99 MiB 48.91% 00m20s 357.56 MiB / 730.99 MiB 48.91% 00m20s 364.10 MiB / 730.99 MiB 49.81% 00m20s 378.03 MiB / 730.99 MiB 51.71% 00m19s 468.97 MiB / 730.99 MiB 64.15% 00m11s 468.97 MiB / 730.99 MiB 64.15% 00m11s 468.97 MiB / 730.99 MiB 64.15% 00m11s 468.97 MiB / 730.99 MiB 64.15% 00m10s 468.97 MiB / 730.99 MiB 64.15% 00m10s 470.54 MiB / 730.99 MiB 64.37% 00m12s 480.78 MiB / 730.99 MiB 65.77% 00m11s 487.36 MiB / 730.99 MiB 66.67% 00m10s 487.36 MiB / 730.99 MiB 66.67% 00m10s 498.81 MiB / 730.99 MiB 68.24% 00m10s 498.81 MiB / 730.99 MiB 68.24% 00m10s 498.81 MiB / 730.99 MiB 68.24% 00m10s 498.81 MiB / 730.99 MiB 68.24% 00m09s 501.06 MiB / 730.99 MiB 68.55% 00m10s 504.41 MiB / 730.99 MiB 69.00% 00m10s 504.41 MiB / 730.99 MiB 69.00% 00m10s 504.41 MiB / 730.99 MiB 69.00% 00m10s 505.43 MiB / 730.99 MiB 69.14% 00m10s 513.27 MiB / 730.99 MiB 70.22% 00m10s 513.27 MiB / 730.99 MiB 70.22% 00m10s 513.31 MiB / 730.99 MiB 70.22% 00m09s 553.42 MiB / 730.99 MiB 75.71% 00m08s 553.42 MiB / 730.99 MiB 75.71% 00m08s 553.42 MiB / 730.99 MiB 75.71% 00m07s 553.42 MiB / 730.99 MiB 75.71% 00m07s 553.42 MiB / 730.99 MiB 75.71% 00m07s 553.42 MiB / 730.99 MiB 75.71% 00m07s 557.18 MiB / 730.99 MiB 76.22% 00m08s 562.04 MiB / 730.99 MiB 76.89% 00m08s 562.04 MiB / 730.99 MiB 76.89% 00m07s 570.79 MiB / 730.99 MiB 78.08% 00m07s 571.97 MiB / 730.99 MiB 78.25% 00m07s 578.17 MiB / 730.99 MiB 79.09% 00m07s 578.17 MiB / 730.99 MiB 79.09% 00m07s 578.17 MiB / 730.99 MiB 79.09% 00m06s 608.97 MiB / 730.99 MiB 83.31% 00m05s 608.97 MiB / 730.99 MiB 83.31% 00m05s 608.97 MiB / 730.99 MiB 83.31% 00m05s 608.97 MiB / 730.99 MiB 83.31% 00m05s 608.97 MiB / 730.99 MiB 83.31% 00m04s 612.62 MiB / 730.99 MiB 83.81% 00m05s 616.44 MiB / 730.99 MiB 84.33% 00m05s 636.84 MiB / 730.99 MiB 87.12% 00m04s 639.15 MiB / 730.99 MiB 87.44% 00m04s 648.11 MiB / 730.99 MiB 88.66% 00m03s 695.42 MiB / 730.99 MiB 95.13% 00m01s 695.42 MiB / 730.99 MiB 95.13% 00m01s 695.42 MiB / 730.99 MiB 95.13% 00m01s 695.42 MiB / 730.99 MiB 95.13% 695.42 MiB / 730.99 MiB 95.13% 695.42 MiB / 730.99 MiB 95.13% 695.42 MiB / 730.99 MiB 95.13% 695.42 MiB / 730.99 MiB 95.13% 696.36 MiB / 730.99 MiB 95.26% 00m01s 699.75 MiB / 730.99 MiB 95.73% 00m01s 701.74 MiB / 730.99 MiB 96.00% 00m01s 705.47 MiB / 730.99 MiB 96.51% 00m01s 719.75 MiB / 730.99 MiB 98.46% 719.75 MiB / 730.99 MiB 98.46% 719.75 MiB / 730.99 MiB 98.46% 719.75 MiB / 730.99 MiB 98.46% 719.99 MiB / 730.99 MiB 98.50% 720.58 MiB / 730.99 MiB 98.58% 727.81 MiB / 730.99 MiB 99.56% 727.81 MiB / 730.99 MiB 99.56% 730.98 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 730.99 MiB / 730.99 MiB 100.00% 51s
2021/06/29 00:55:19 info done backup=2021-06-29T00-54-25 duration=51.776s operation=upload size=731.78MiB table=plusequalone_main.request_info
0 / 81471966 0.00% 5.16 MiB / 77.70 MiB 6.63% 00m02s 5.16 MiB / 77.70 MiB 6.63% 00m02s 6.65 MiB / 77.70 MiB 8.56% 00m06s 19.78 MiB / 77.70 MiB 25.46% 00m02s 22.80 MiB / 77.70 MiB 29.34% 00m02s 27.40 MiB / 77.70 MiB 35.26% 00m02s 28.87 MiB / 77.70 MiB 37.16% 00m02s 40.29 MiB / 77.70 MiB 51.85% 00m01s 40.29 MiB / 77.70 MiB 51.85% 00m01s 40.29 MiB / 77.70 MiB 51.85% 00m01s 40.29 MiB / 77.70 MiB 51.85% 40.30 MiB / 77.70 MiB 51.87% 00m02s 42.17 MiB / 77.70 MiB 54.28% 00m02s 44.09 MiB / 77.70 MiB 56.75% 00m02s 49.56 MiB / 77.70 MiB 63.79% 00m01s 50.28 MiB / 77.70 MiB 64.71% 00m01s 70.89 MiB / 77.70 MiB 91.24% 70.89 MiB / 77.70 MiB 91.24% 70.89 MiB / 77.70 MiB 91.24% 72.84 MiB / 77.70 MiB 93.75% 72.84 MiB / 77.70 MiB 93.75% 72.84 MiB / 77.70 MiB 93.75% 74.00 MiB / 77.70 MiB 95.24% 74.00 MiB / 77.70 MiB 95.24% 77.19 MiB / 77.70 MiB 99.34% 77.70 MiB / 77.70 MiB 100.00% 77.70 MiB / 77.70 MiB 100.00% 77.70 MiB / 77.70 MiB 100.00% 77.70 MiB / 77.70 MiB 100.00% 77.70 MiB / 77.70 MiB 100.00% 77.70 MiB / 77.70 MiB 100.00% 77.70 MiB / 77.70 MiB 100.00% 77.70 MiB / 77.70 MiB 100.00% 77.70 MiB / 77.70 MiB 100.00% 6s
Killed
as you can see it is oom killed:
You are welcome! Looking forward to the bugfix - we use in production environment ;)
@fzyzcjy sorry for late reply, is bug still relevant to you?
Could you try to latest https://github.com/Altinity/clickhouse-backup/releases/tag/1.1.1? Which remote storage type do you use?
Could you share your config.yml with masked private credentials?
@Slach Hi thanks for the reply! I am not recently working on ops, but will try the latest version soon.
Btw should I use that forked repo, or the original repo?
I use S3 and env var (using k8s), the env var are shown below.
- name: backup
securityContext: {{- toYaml .Values.securityContext | nindent 16 }}
image: "{{ .Values.hg.imagePrefix }}{{ .Values.images.backup.repository }}:{{ .Values.images.backup.tag }}"
imagePullPolicy: {{ .Values.images.backup.pullPolicy }}
# ref https://redis.io/topics/rediscli#remote-backups-of-rdb-files
command:
- /bin/sh
- -c
- |
set -eu
clickhouse-backup server
env:
- name: GENERAL_BACKUPS_TO_KEEP_LOCAL
value: {{ .Values.backup.general.backupsToKeepLocal | quote }}
- name: GENERAL_BACKUPS_TO_KEEP_REMOTE
value: {{ .Values.backup.general.backupsToKeepRemote | quote }}
- name: CLICKHOUSE_USERNAME
value: default
- name: CLICKHOUSE_PASSWORD
valueFrom:
secretKeyRef:
name: {{ template "clickhouse.fullname" . }}
key: backup-clickhouse-password
- name: CLICKHOUSE_HOST
value: {{ include "clickhouse.fullname" . }}
- name: CLICKHOUSE_PORT
value: '9000'
- name: CLICKHOUSE_DATA_PATH
value: '/var/lib/clickhouse'
- name: CLICKHOUSE_SKIP_TABLES
value: {{ .Values.backup.clickhouse.skipTables }}
- name: S3_ACCESS_KEY
value: {{ .Values.backup.ak }}
- name: S3_SECRET_KEY
valueFrom:
secretKeyRef:
name: {{ template "clickhouse.fullname" . }}
key: backup-cloud-sk
- name: S3_BUCKET
value: {{ .Values.backup.bucket }}
- name: S3_PATH
value: {{ .Values.backup.path }}
- name: S3_ENDPOINT
value: {{ .Values.backup.endpoint }}
- name: S3_REGION
value: {{ .Values.backup.region }}
- name: S3_COMPRESSION_FORMAT
value: gzip
- name: API_LISTEN
value: "0.0.0.0:{{ .Values.backup.port }}"
- name: API_ENABLE_METRICS
value: "true"
your configuration looks OK. you can use forked repo if ready to adopt often path releases, or prefer original repo to more stable releases
please report if OOM gone away
@fzyzcjy any news? Is OOM still present for your workload? Could you try the latest https://github.com/AlexAkulov/clickhouse-backup/releases/tag/v1.2.0?
Sorry I am not doing ops recently. I will tell you as soon aa i get back to ops.
@fzyzcjy is issue still relevant for you? could you try to use https://github.com/AlexAkulov/clickhouse-backup/releases/tag/v1.3.1
Thanks for the information. Sorry I have not been doing ops for some time. I will check the latest version when getting back to ops.