attic_server::error: Database error: Query Error: error returned from database: (code: 6410) disk I/O error

[voidus]

Hey folks,

I'm getting the error from the when running in github actions. My flake.lock lists attic with revision fbe252a5c21febbe920c025560cbd63b20e24f3b

The github actions log shows:

error: unable to download 'https://nix-cache.<my-server>.tld/my-cache/nar/<some-nar-hash>.nar': HTTP error 500

       response body:

       {"code":500,"error":"InternalServerError","message":"The server encountered an internal error or misconfiguration."}

The systemd journal shows:

Feb 16 21:13:44 electra atticd[1751]: 2024-02-16T21:13:44.884923Z ERROR attic_server::error: Database error: Query Error: error returned from database: (code: 6410) disk I/O error
Feb 16 21:13:44 electra atticd[1751]:    0: attic_server::api::binary_cache::get_nar
Feb 16 21:13:44 electra atticd[1751]:              at server/src/api/binary_cache.rs:167
Feb 16 21:13:44 electra atticd[1751]:    1: tower_http::trace::make_span::request
Feb 16 21:13:44 electra atticd[1751]:            with method=GET uri=/cannahaus/nar/qs1mrabfkk50p5alxf9cml1qvfgrxc7x.nar version=HTTP/1.1
Feb 16 21:13:44 electra atticd[1751]:              at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tower-http-0.4.4/src/trace/make_span.rs:109
Feb 16 21:13:44 electra atticd[1751]:    2: tokio::task::runtime.spawn
Feb 16 21:13:44 electra atticd[1751]:            with kind=task task.name= task.id=326290 loc.file="/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/hyper-0.14.27>
Feb 16 21:13:44 electra atticd[1751]:              at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.34.0/src/util/trace.rs:17
Feb 16 21:13:44 electra atticd[1751]: 2024-02-16T21:13:44.887729Z ERROR tower_http::trace::on_failure: response failed classification=Status code: 500 Internal Server Error latency=21 ms

This repeats a bunch of times. There is nothing related, journalctl -k is not showing anything either.

The relevant nixos config:

{
  config,
  pkgs,
  ...
}: let
  host = "nix-cache.<my-server>.tld";
  local-port = "8080";

  inherit (pkgs.lib) concatStringsSep;
in {
  sops.secrets = {
    atticd_ssh_private_key = {owner = config.services.atticd.user;};
    atticd_credentials_file = {owner = config.services.atticd.user;};
    atticd_cifs_environmentfile = {};
  };

  environment.systemPackages = [
    pkgs.cifs-utils
  ];

  systemd.services.atticd = {
    serviceConfig = {
      DynamicUser = pkgs.lib.mkForce false;
    };
    unitConfig = {
      RequiresMountsFor = "/var/lib/atticd/storage";
    };
  };

  users = let
    u = config.services.atticd.user;
  in {
    groups.${u} = {};
    users.${u} = {
      isSystemUser = true;
      group = config.users.groups.${u}.name;
      extraGroups = [config.users.groups.keys.name];
    };
  };

  services = {
    atticd = {
      enable = true;

      credentialsFile = config.sops.secrets.atticd_credentials_file.path;

      settings = {
        listen = "[::]:${local-port}";

        allowed-hosts = [host];
        api-endpoint = "https://${host}/";

        garbage-collection.default-retention-period = "6 months";

        # Data chunking
        #
        # Warning: If you change any of the values here, it will be
        # difficult to reuse existing chunks for newly-uploaded NARs
        # since the cutpoints will be different. As a result, the
        # deduplication ratio will suffer for a while after the change.
        chunking = {
          # The minimum NAR size to trigger chunking
          #
          # If 0, chunking is disabled entirely for newly-uploaded NARs.
          # If 1, all NARs are chunked.
          nar-size-threshold = 64 * 1024; # 64 KiB

          # The preferred minimum size of a chunk, in bytes
          min-size = 16 * 1024; # 16 KiB

          # The preferred average size of a chunk, in bytes
          avg-size = 64 * 1024; # 64 KiB

          # The preferred maximum size of a chunk, in bytes
          max-size = 256 * 1024; # 256 KiB
        };
      };
    };

    nginx = {
      enable = true;
      recommendedProxySettings = true;
      recommendedTlsSettings = true;
      clientMaxBodySize = "1g";
      virtualHosts.${host} = {
        enableACME = true;
        forceSSL = true;
        locations = {
          "/" = {
            proxyPass = "http://[::]:${local-port}";
          };
        };
      };
    };
  };
}

I'm somewhat at a loss how to debug this. I've tried adding RUST_LOG=debug to the credentials file, but I don't think it changed anything.

There's a few other services running on the machine, so let me know if anything else might be relevant.

[voidus]

Just set everything up on a new VM to make sure, but this still happens. I think I'll have to set up a bucket instead :/