"Failed to fetch" - Snapshots and Doctor

[jordanhandy]

Snapshot viewer and Doctor both give me "failed to fetch" errors.

• Repositories are healthy
• Backups are succeeding
• Docker logs show 200s when accessing various paths

Where could this be coming from?

Thanks

[jordanhandy]

Update:

Added reverse proxy in front of Zerobyte. Message has changed to "No snapshots available", and lists a 404 in browser Network calls

[nicotsx]

Hello @jordanhandy, I'll need more info to help you debug your issue. If you don't see any error in the zerobyte docker logs it's probably your setup.

Where are you running zerobyte, which reverse proxy? What config?

Please share a minimal docker-compose.yml that reproduces the error on your side

[jordanhandy]

Hi @nicotsx , I'll try to be as verbose as possible here:

• Docker compose is below
• I spin this up, access the container via local IP, set up:
  • repository:
    • rclone:pcloud
  • volume:
    • SMB local share -- files are visible
  • backup:
    • force run initial backup
• Backup succeeds, and I can see the files in the source destination
• Now going to either the backup job snapshot view or the repository snapshot view, I get an ERR -- empty response warning in the console for the endpoint http://<localIp:port>/api/v1/repositories/<repository>/snapshots?backupId=1

Now, switching gears a little bit, I always put all of my services behind reverse proxy so I get SSL and domain name resolution.

• DNS is run through AdGuard Home
• The "upstream" endpoints that serve DNS are 2 raspberry Pis running Pi-Hole, each resolving using Unbound.
  • For local dns resolution, i.e. https://myservice.local.my.domain, I have local DNS records configured in Pi-Hole where the A records resolve to my reverse proxy
• I use Zoraxy as a reverse proxy, and Zerobyte is located at zerobyte.local.my.domain
• I try from this address to view snapshots, and instead this time get a 404 (address not found) for https://zerobyte.local.my.domain/api/v1/repositories/<repository>/snapshots?backupId=1

This makes sense, if my Docker network can't resolve local domains, so I also add the "DNS" key in the compose file to force it to use a specific resolver. It would also make sense that it can't resolve my proxied path, if it had no concept of my Base URL... maybe a config option for the future... but I thought I'd give it a try.

For some other information:

• I use Proxmox and have LXC containers and docker containers both
• All use local DNS resolution, and I've never encountered this issue before
• I also use Backrest, and I'm able to view snapshots through that app -- yes, completely different application, but it also is a Restic browser, so maybe some similarities under the hood?

Docker Compose

Didn't change much except timezone, added a dns key, and pointed to my rclone config

services:
  zerobyte:
    image: ghcr.io/nicotsx/zerobyte:v0.17
    container_name: zerobyte
    restart: unless-stopped
    dns:
      - 192.168.2.217
    cap_add:
      - SYS_ADMIN
    ports:
      - "4096:4096"
    devices:
      - /dev/fuse:/dev/fuse
    environment:
      - TZ=America/Toronto  # Set your timezone here
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/lib/zerobyte:/var/lib/zerobyte
      - /root/.config/rclone:/root/.config/rclone
      

Thanks for the help

[nicotsx]

Thanks for all the details, unfortunately I don't have an explanation to this. Could you try enable debug logs (with the environment variable LOG_LEVEL=debug) and check if you see anything suspicious?

The only plausible explanation I have here is possible a timeout either from your browser or your server. Listing snapshot on remotes can take very long depending on the provider.

Also, it's probably not this, but just for the sake of trying it, could you replace the DNS entry with something like 1.1.1.1

[jordanhandy]

Hi @nicotsx ,

I was able to test my scenario with a different rclone mount, and this one worked successfully (only initially) showing snapshots. After some time, they also "failed to fetch"

I believe it may be related to the timeout you mentioned. Can we manually set this value?

[MuktadirHassan]

Facing same issue here with google drive. Here are debug logs.

<-- GET /api/v1/repositories/rclone-drive-backup/snapshots?backupId=1
debug > [Mutex] Released shared lock for repo 811d5fee-e334-ab8d-d128ca48a132: snapshots (held for 14476ms)
--> GET /api/v1/repositories/rclone-drive-backup/snapshots?backupId=1 200 14s
debug > [Mutex] Released shared lock for repo 811d5fee-e334-ab8d-d128ca48a132: ls:a9359277 (held for 15426ms)
--> GET /api/v1/repositories/rclone-drive-backup/snapshots/a9359277/files?path=%2F 200 15s
<-- GET /api/v1/repositories/rclone-drive-backup/snapshots?backupId=1
<-- GET /api/v1/repositories/rclone-drive-backup/snapshots/318c47ec/files?path=%2F
debug > [Mutex] Released shared lock for repo 811d5fee-e334-ab8d-d128ca48a132: ls:318c47ec (held for 13355ms)
--> GET /api/v1/repositories/rclone-drive-backup/snapshots/318c47ec/files?path=%2F 200 13s
debug > [Mutex] Released shared lock for repo 811d5fee-e334-ab8d-d128ca48a132: snapshots (held for 14534ms)
--> GET /api/v1/repositories/rclone-drive-backup/snapshots?backupId=1 200 15s
debug > [Mutex] Released shared lock for repo 811d5fee-e334-ab8d-d128ca48a132: snapshots (held for 12169ms)
--> GET /api/v1/repositories/rclone-drive-backup/snapshots?backupId=1 200 12s
<-- GET /api/v1/repositories/rclone-drive-backup/snapshots?backupId=1
<-- GET /api/v1/repositories/rclone-drive-backup/snapshots/318c47ec/files?path=%2F
debug > [Mutex] Released shared lock for repo 811d5fee-e334-ab8d-d128ca48a132: ls:318c47ec (held for 16346ms)

For me the api call (/snapshots) only seemed to succeed once sometimes, then never again.

[nicotsx]

I was able to reproduce by lowering the server idle timeout to a few seconds. The default is 10s so it means your repo is extremely slow to respond. Here's what I'll do. I will bump the default idle timeout to 60 seconds and I'll add an environment variable config if someone what to go even further