thanos.shipper.json: no such file or directory with Ceph S3

[huongbn]

trafficstars

Hi guys, I'm testing with thanos sidecar. I;m using it to uplaod metrics to Ceph S3 but not success. Can you help me with this problem?

Enviroment:

• Debian 9
• Prometheus 2.14
• Thanos 0.11.0

Thanos-sidecar configuration:

thanos sidecar
--tsdb.path /var/lib/prometheus/
--prometheus.url "http://localhost:9090"
--http-address 0.0.0.0:19191
--grpc-address 0.0.0.0:19090
--objstore.config-file /etc/thanos/s3.yml

S3 configuration: type: S3 config: bucket: 'bucket-s3' endpoint: 'my-domain' region: 'xxx' access_key: 'xxx' insecure: false signature_version2: false encrypt_sse: false secret_key: 'xxx' put_user_metadata: {} http_config: idle_conn_timeout: 90s response_header_timeout: 2m insecure_skip_verify: false trace: enable: false part_size: 134217728

Log with thanos-sidecar Mar 03 11:02:32 cc8-3 thanos[10449]: level=info ts=2020-03-03T04:02:32.927503371Z caller=sidecar.go:355 msg="starting sidecar" Mar 03 11:02:32 cc8-3 thanos[10449]: level=info ts=2020-03-03T04:02:32.928034202Z caller=reloader.go:157 component=reloader msg="started watching config file and non-recursively rule dirs for changes" cfg= out= dirs= Mar 03 11:02:32 cc8-3 thanos[10449]: level=info ts=2020-03-03T04:02:32.928605046Z caller=intrumentation.go:52 msg="changing probe status" status=ready Mar 03 11:02:32 cc8-3 thanos[10449]: level=info ts=2020-03-03T04:02:32.929115956Z caller=grpc.go:105 service=gRPC/server component=sidecar msg="listening for StoreAPI gRPC" address=0.0.0.0:19090 Mar 03 11:02:32 cc8-3 thanos[10449]: level=info ts=2020-03-03T04:02:32.929588137Z caller=intrumentation.go:64 msg="changing probe status" status=healthy Mar 03 11:02:32 cc8-3 thanos[10449]: level=info ts=2020-03-03T04:02:32.929950701Z caller=http.go:56 service=http/server component=sidecar msg="listening for requests and metrics" address=0.0.0.0:19191 Mar 03 11:02:32 cc8-3 thanos[10449]: level=info ts=2020-03-03T04:02:32.935395496Z caller=sidecar.go:219 msg="successfully loaded prometheus external labels" external_labels="{replicas="X", thanos_env="lab_1", thanos_team="huongnv"}" Mar 03 11:02:32 cc8-3 thanos[10449]: level=info ts=2020-03-03T04:02:32.935431113Z caller=intrumentation.go:52 msg="changing probe status" status=ready Mar 03 11:02:34 cc8-3 thanos[10449]: level=warn ts=2020-03-03T04:02:34.929833873Z caller=shipper.go:336 msg="updating meta file failed" err="open /var/lib/prometheus/thanos.shipper.json.tmp: permission denied" Mar 03 11:02:34 cc8-3 thanos[10449]: level=warn ts=2020-03-03T04:02:34.929949009Z caller=sidecar.go:344 msg="reading timestamps failed" err="read shipper meta file: open /var/lib/prometheus/thanos.shipper.json: no such file or directory"

[daixiang0]

permission denied Does sidecar have permoission to write under /var/lib/prometheus?

[huongbn]

permission denied Does sidecar have permoission to write under /var/lib/prometheus?

yep, I have set permission with prometheus, but I have a problem with hardlink same issue 2210

[GiedriusS]

Hey! Operation not supported is quite different from Permission denied & no such file or directory. But how could you get such an error if the user/group under which Thanos Sidecar is running has access according to your previous message?

[stale[bot]]

This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions.

[didenko-yv]

I have the same situation with GCS, I can not to upload metrics woth Google Cloud Service Account. I'm using thanos-sidecar with bitnami/kube-prometheus and all of my attempts failed(

level=info ts=2022-04-26T08:18:08.545019331Z caller=options.go:27 protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
level=info ts=2022-04-26T08:18:08.545320118Z caller=factory.go:49 msg="loading bucket configuration"
level=info ts=2022-04-26T08:18:08.545657394Z caller=sidecar.go:357 msg="starting sidecar"
level=info ts=2022-04-26T08:18:08.545724231Z caller=reloader.go:199 component=reloader msg="nothing to be watched"
level=info ts=2022-04-26T08:18:08.545780216Z caller=intrumentation.go:75 msg="changing probe status" status=healthy
level=info ts=2022-04-26T08:18:08.545866529Z caller=intrumentation.go:56 msg="changing probe status" status=ready
level=info ts=2022-04-26T08:18:08.545835961Z caller=http.go:73 service=http/server component=sidecar msg="listening for requests and metrics" address=0.0.0.0:10902
level=info ts=2022-04-26T08:18:08.546080743Z caller=grpc.go:131 service=gRPC/server component=sidecar msg="listening for serving gRPC" address=0.0.0.0:10901
level=info ts=2022-04-26T08:18:08.546118865Z caller=tls_config.go:195 service=http/server component=sidecar msg="TLS is disabled." http2=false
level=info ts=2022-04-26T08:18:08.551006915Z caller=sidecar.go:179 msg="successfully loaded prometheus version"
level=info ts=2022-04-26T08:18:08.560611683Z caller=sidecar.go:201 msg="successfully loaded prometheus external labels" external_labels="{prometheus=\"default/monitoring-kube-prometheus-prometheus\", prometheus_replica=\"prometheus-monitoring-kube-prometheus-prometheus-0\"}"
level=warn ts=2022-04-26T08:18:10.545996638Z caller=shipper.go:239 msg="reading meta file failed, will override it" err="failed to read /prometheus/thanos.shipper.json: open /prometheus/thanos.shipper.json: no such file or directory"

This is my helm values for chart

prometheus:
  disableCompaction: true
  thanos:
    create: true
    objectStorageConfig:
      secretKey: "thanos-config"
      secretName: "thanos-config"

What can be the reason such error?

[mv-yurchenko]

Is someone found how to solve this issue?

[GiedriusS]

Perhaps you know a reliable way to reproduce this @mv-yurchenko ?

[mv-yurchenko]

I used config like and got empty bucket

prometheus: disableCompaction: true thanos: create: true objectStorageConfig: secretKey: "thanos-config" secretName: "thanos-config"

[didenko-yv]

I solved this issue with changing bitnami/kube-prometheus chart version to 6.5.0. I did not investigate the topic in depth, but probably it was issue with thanos container in newer versions

[Nafalgar]

I had the same issue as @didenko-yv above but using Azure Blob Storage. I was using the Quay image v0.26.0, reverting to v0.23.0 removed the error.

[mark4409]

I'm also getting the error with Azure Blob Storage on v0.26.0...

level=warn ts=2022-07-07T09:11:33.607892496Z caller=shipper.go:239 msg="reading meta file failed, will override it" err="failed to read /prometheus/thanos.shipper.json: open /prometheus/thanos.shipper.json: no such file or directory"

[mark4409]

The file thanos.shipper.json does exist on the volume and is owned by root however has 644 permissions. The volume itself is used by the default prometheus user 'nobody' (from the latest prometheus image). It does appear as though the sidecar is running as root - should that be the case? I haven't specified that in my compose file. Or maybe the message is simply referring to the fact it hasn't been created yet and so it gets created at that point?

Edit: Yes I think that's the case. The message only appears once the first time its ran. If I stop the container then start it again, I don't get the message as the file has already been created. Please ignore my message.

[stale[bot]]

Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[richie-tt]

Hello,

On version quay.io/thanos/thanos:v0.28.0 the problem still exists

level=info ts=2022-10-06T13:39:27.155982527Z caller=options.go:26 protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
level=info ts=2022-10-06T13:39:27.156465338Z caller=factory.go:50 msg="loading bucket configuration"
level=info ts=2022-10-06T13:39:27.156667713Z caller=sidecar.go:361 msg="starting sidecar"
level=info ts=2022-10-06T13:39:27.156806737Z caller=intrumentation.go:75 msg="changing probe status" status=healthy
level=info ts=2022-10-06T13:39:27.156822317Z caller=http.go:73 service=http/server component=sidecar msg="listening for requests and metrics" address=:10902
level=info ts=2022-10-06T13:39:27.157037602Z caller=tls_config.go:195 service=http/server component=sidecar msg="TLS is disabled." http2=false
level=info ts=2022-10-06T13:39:27.157437812Z caller=reloader.go:199 component=reloader msg="nothing to be watched"
level=info ts=2022-10-06T13:39:27.157490133Z caller=intrumentation.go:56 msg="changing probe status" status=ready
level=info ts=2022-10-06T13:39:27.157523114Z caller=grpc.go:131 service=gRPC/server component=sidecar msg="listening for serving gRPC" address=:10901
level=info ts=2022-10-06T13:39:27.159297797Z caller=sidecar.go:179 msg="successfully loaded prometheus version"
level=info ts=2022-10-06T13:39:27.257491254Z caller=sidecar.go:201 msg="successfully loaded prometheus external labels" external_labels="{prometheus=\"monitoring/dev-prometheus-stack-prometheus\", prometheus_replica=\"dev-prometheus-stack-prometheus-0\"}"
level=warn ts=2022-10-06T13:39:29.158196455Z caller=shipper.go:239 msg="reading meta file failed, will override it" err="failed to read /prometheus/thanos.shipper.json: open /prometheus/thanos.shipper.json: no such file or directory"

btw. why are you asking for any update, if the issue was reported on 3 March 2020 and I don't see any attempt to solve it? or I miss something

I checked all these versions and all of them have the same problem

• v0.28.0
• v0.27.0
• v0.26.0
• v0.25.0

/prometheus $ ls -la
total 24
drwxrwsrwx    4 root     2000            97 Oct  6 14:01 .
drwxr-xr-x    1 root     root            35 Oct  6 14:00 ..
drwxr-sr-x    2 1000     2000             6 Oct  6 14:00 chunks_head
-rw-r--r--    1 1000     2000             0 Oct  6 14:00 lock
-rw-r--r--    1 1000     2000         20001 Oct  6 14:01 queries.active
-rw-r--r--    1 1000     2000            37 Oct  6 14:01 thanos.shipper.json
drwxr-sr-x    2 1000     2000            22 Oct  6 14:00 wal
/prometheus $ id
uid=1000 gid=2000 groups=2000

The last working version is v0.24.0

level=info ts=2022-10-06T14:02:36.056319704Z caller=options.go:27 protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
level=info ts=2022-10-06T14:02:36.056815456Z caller=factory.go:49 msg="loading bucket configuration"
level=info ts=2022-10-06T14:02:36.057024442Z caller=sidecar.go:355 msg="starting sidecar"
level=info ts=2022-10-06T14:02:36.057214246Z caller=intrumentation.go:60 msg="changing probe status" status=healthy
level=info ts=2022-10-06T14:02:36.057241437Z caller=http.go:63 service=http/server component=sidecar msg="listening for requests and metrics" address=:10902
level=info ts=2022-10-06T14:02:36.057638376Z caller=tls_config.go:195 service=http/server component=sidecar msg="TLS is disabled." http2=false
level=info ts=2022-10-06T14:02:36.057937004Z caller=reloader.go:199 component=reloader msg="nothing to be watched"
level=info ts=2022-10-06T14:02:36.058044916Z caller=intrumentation.go:48 msg="changing probe status" status=ready
level=info ts=2022-10-06T14:02:36.05820174Z caller=grpc.go:131 service=gRPC/server component=sidecar msg="listening for serving gRPC" address=:10901
level=info ts=2022-10-06T14:02:36.15501157Z caller=sidecar.go:179 msg="successfully loaded prometheus version"
level=info ts=2022-10-06T14:02:36.165833343Z caller=sidecar.go:201 msg="successfully loaded prometheus external labels" external_labels="{prometheus=\"monitoring/dev-prometheus-stack-prometheus\", prometheus_replica=\"prometheus-dev-prometheus-stack-prometheus-0\"}"
level=info ts=2022-10-06T14:02:36.165872674Z caller=intrumentation.go:48 msg="changing probe status" status=ready

/prometheus $ ls -la
total 24
drwxrwsrwx    4 root     2000            97 Oct  6 14:03 .
drwxr-xr-x    1 root     root            35 Oct  6 14:02 ..
drwxr-sr-x    2 1000     2000             6 Oct  6 14:02 chunks_head
-rw-r--r--    1 1000     2000             0 Oct  6 14:02 lock
-rw-r--r--    1 1000     2000         20001 Oct  6 14:03 queries.active
-rw-r--r--    1 1000     2000            37 Oct  6 14:03 thanos.shipper.json
drwxr-sr-x    2 1000     2000            22 Oct  6 14:02 wal
/prometheus $ if
> ^C
/prometheus $ id
uid=1000 gid=2000 groups=2000

[Ryu-Z]

v2.28.0 prometheus-operator, the same question

[yeya24]

msg="reading meta file failed, will override it" err="failed to read /prometheus/thanos.shipper.json: open /prometheus/thanos.shipper.json: no such file or directory"

This log is expected if it is a brand new TSDB folder and there is no thanos.shipper.json before. If it doesn't exist sidecar will try to create the file. If it cannot open or write the file due to permission, please check the deployment configuration since it is your environment specific rather than a Thanos issue

[Ryu-Z]

thanks, now i see thanos sidercar is working ok, this log is expected kubectl exec -it prometheus-k8s-1 -c thanos-sidecar -n monitoring sh / $ ls -la prometheus/ total 24 drwxrwsrwx 10 root 2000 281 Oct 13 02:02 . drwxr-xr-x 1 root root 35 Oct 12 14:39 .. drwxr-sr-x 3 1000 2000 68 Oct 12 17:39 01GF6MEZ17WF7BQ21QZ2JK67XT drwxr-sr-x 3 1000 2000 68 Oct 12 19:00 01GF6S2QWYHMRJ4G0NHE0GDZDY drwxr-sr-x 3 1000 2000 68 Oct 12 21:00 01GF6ZYF4YW9P9K270V3RZF92D drwxr-sr-x 3 1000 2000 68 Oct 12 23:00 01GF76T6CYXRVS7CES985FC9K3 drwxr-sr-x 3 1000 2000 68 Oct 13 01:00 01GF7DNXMYDE76FSHZ5F5PY88G drwxr-sr-x 2 1000 2000 34 Oct 13 01:00 chunks_head -rw-r--r-- 1 1000 2000 0 Oct 12 14:39 lock -rw-r--r-- 1 1000 2000 20001 Oct 13 02:03 queries.active drwxr-s--- 3 1000 2000 20 Oct 12 17:39 thanos -rw-r--r-- 1 1000 2000 196 Oct 13 02:02 thanos.shipper.json drwxr-sr-x 3 1000 2000 113 Oct 13 01:34 wal / $ id uid=1000 gid=0(root) groups=2000

[yeya24]

Hello,

On version quay.io/thanos/thanos:v0.28.0 the problem still exists

level=info ts=2022-10-06T13:39:27.155982527Z caller=options.go:26 protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
level=info ts=2022-10-06T13:39:27.156465338Z caller=factory.go:50 msg="loading bucket configuration"
level=info ts=2022-10-06T13:39:27.156667713Z caller=sidecar.go:361 msg="starting sidecar"
level=info ts=2022-10-06T13:39:27.156806737Z caller=intrumentation.go:75 msg="changing probe status" status=healthy
level=info ts=2022-10-06T13:39:27.156822317Z caller=http.go:73 service=http/server component=sidecar msg="listening for requests and metrics" address=:10902
level=info ts=2022-10-06T13:39:27.157037602Z caller=tls_config.go:195 service=http/server component=sidecar msg="TLS is disabled." http2=false
level=info ts=2022-10-06T13:39:27.157437812Z caller=reloader.go:199 component=reloader msg="nothing to be watched"
level=info ts=2022-10-06T13:39:27.157490133Z caller=intrumentation.go:56 msg="changing probe status" status=ready
level=info ts=2022-10-06T13:39:27.157523114Z caller=grpc.go:131 service=gRPC/server component=sidecar msg="listening for serving gRPC" address=:10901
level=info ts=2022-10-06T13:39:27.159297797Z caller=sidecar.go:179 msg="successfully loaded prometheus version"
level=info ts=2022-10-06T13:39:27.257491254Z caller=sidecar.go:201 msg="successfully loaded prometheus external labels" external_labels="{prometheus=\"monitoring/dev-prometheus-stack-prometheus\", prometheus_replica=\"dev-prometheus-stack-prometheus-0\"}"
level=warn ts=2022-10-06T13:39:29.158196455Z caller=shipper.go:239 msg="reading meta file failed, will override it" err="failed to read /prometheus/thanos.shipper.json: open /prometheus/thanos.shipper.json: no such file or directory"

btw. why are you asking for any update, if the issue was reported on 3 March 2020 and I don't see any attempt to solve it? or I miss something

I checked all these versions and all of them have the same problem

• v0.28.0
• v0.27.0
• v0.26.0
• v0.25.0

/prometheus $ ls -la
total 24
drwxrwsrwx    4 root     2000            97 Oct  6 14:01 .
drwxr-xr-x    1 root     root            35 Oct  6 14:00 ..
drwxr-sr-x    2 1000     2000             6 Oct  6 14:00 chunks_head
-rw-r--r--    1 1000     2000             0 Oct  6 14:00 lock
-rw-r--r--    1 1000     2000         20001 Oct  6 14:01 queries.active
-rw-r--r--    1 1000     2000            37 Oct  6 14:01 thanos.shipper.json
drwxr-sr-x    2 1000     2000            22 Oct  6 14:00 wal
/prometheus $ id
uid=1000 gid=2000 groups=2000

The last working version is v0.24.0

level=info ts=2022-10-06T14:02:36.056319704Z caller=options.go:27 protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
level=info ts=2022-10-06T14:02:36.056815456Z caller=factory.go:49 msg="loading bucket configuration"
level=info ts=2022-10-06T14:02:36.057024442Z caller=sidecar.go:355 msg="starting sidecar"
level=info ts=2022-10-06T14:02:36.057214246Z caller=intrumentation.go:60 msg="changing probe status" status=healthy
level=info ts=2022-10-06T14:02:36.057241437Z caller=http.go:63 service=http/server component=sidecar msg="listening for requests and metrics" address=:10902
level=info ts=2022-10-06T14:02:36.057638376Z caller=tls_config.go:195 service=http/server component=sidecar msg="TLS is disabled." http2=false
level=info ts=2022-10-06T14:02:36.057937004Z caller=reloader.go:199 component=reloader msg="nothing to be watched"
level=info ts=2022-10-06T14:02:36.058044916Z caller=intrumentation.go:48 msg="changing probe status" status=ready
level=info ts=2022-10-06T14:02:36.05820174Z caller=grpc.go:131 service=gRPC/server component=sidecar msg="listening for serving gRPC" address=:10901
level=info ts=2022-10-06T14:02:36.15501157Z caller=sidecar.go:179 msg="successfully loaded prometheus version"
level=info ts=2022-10-06T14:02:36.165833343Z caller=sidecar.go:201 msg="successfully loaded prometheus external labels" external_labels="{prometheus=\"monitoring/dev-prometheus-stack-prometheus\", prometheus_replica=\"prometheus-dev-prometheus-stack-prometheus-0\"}"
level=info ts=2022-10-06T14:02:36.165872674Z caller=intrumentation.go:48 msg="changing probe status" status=ready

/prometheus $ ls -la
total 24
drwxrwsrwx    4 root     2000            97 Oct  6 14:03 .
drwxr-xr-x    1 root     root            35 Oct  6 14:02 ..
drwxr-sr-x    2 1000     2000             6 Oct  6 14:02 chunks_head
-rw-r--r--    1 1000     2000             0 Oct  6 14:02 lock
-rw-r--r--    1 1000     2000         20001 Oct  6 14:03 queries.active
-rw-r--r--    1 1000     2000            37 Oct  6 14:03 thanos.shipper.json
drwxr-sr-x    2 1000     2000            22 Oct  6 14:02 wal
/prometheus $ if
> ^C
/prometheus $ id
uid=1000 gid=2000 groups=2000

I think it is actually working properly for you, right? When you did ls the directory you can see the thanos.shipper.json file created. The error log is expected if the folder doesn't contain this file.

I will close this issue since it is not a Thanos bug.

[nebed]

I still had this issue on v0.28.0 and only downgrading to v0.24.0 fixed it. this issue is with shipping metrics to Azure blob storage. shipper.json file exists but it doesn't get read by thanos sidecar

@yeya24 @richie-tt

[yeya24]

this issue is with shipping logs to Azure blob storage.

I am confused. Shipping metrics or logs? But anyway, which object storage you are using doesn't matter. Shipper josn is in your local TSDB path.

shipper.json file exists but it doesn't get read by thanos sidecar

How do you know the file exists before? If you see the file exists but still see the warn log, it is because the file was created by Thanos sidecar at that time.

[5cat]

I can confirm this issue still exist with thanos version 0.30.2. Downgrading it to v0.24.0 fixed the issue initially, then when I upgraded back (and have Prometheus storage persisted) it worked. Can confirm the validity of this https://github.com/thanos-io/thanos/issues/2211#issuecomment-1276429139

[pablode-cs]

Downgrading 0.28.1 to v0.24.0 fixed too

[gzOne2Free]

Change signature_version2: false to true may solve your issue. I met same issue before, the different thing is I miss this line in thanos.yaml. when I append signature_version2: true to thanos.yaml, problem solved.