Expose more connection parameters to users + define reasonable defaults

[johansmitsnl]

I'm restoring files from the backup (rclone+sftp). Sometimes it panics:

[INFO] total restore size: 2.3 GiB
[00:01:16] restoring file contents...     ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  11.99 MiB/2.35 GiB   49.92 KiB/s  (ETA 4h)The application panicked (crashed).
Message:  called `Result::unwrap()` on an `Err` value: RusticError(Rest(BackoffError(Permanent(reqwest::Error { kind: Status(404), url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(127.0.0.1)), port: Some(33267), path: "/data/2718a0219090412984e200b92c7a6079952db0f3e677127e0de31b6c31f31116", query: None, fragment: None } }))))
Location: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rustic_core-0.1.2/src/commands/restore.rs:496

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  <empty backtrace>

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.
fish: Job 1, 'env RUST_BACKTRACE=full rustic …' terminated by signal SIGABRT (Abort)

It is making to much connections. Running with debug:

[INFO] rclone output: <7>DEBUG : pacer: low level retry 10/10 (error couldn't initialise SFTP: ssh: subsystem request failed)

[INFO] rclone output: <7>DEBUG : data/29/2952ec748c8d6a0549af7d2e9b95075cdc212dc35f9c74ea1fc63d23db596c35: GET request error: stat failed: stat: couldn't initialise SFTP: ssh: subsystem request failed

[johansmitsnl]

I did some searching and it is making to much connections. I see there is a default constant of 20 defined in the core lib src/commands/restore.rs. Reducing it to less then 10 (in my Hetzner case) makes it work perfect.

Can this made an option so that you can steer the amount of readers you can support? I also found #61 and #72. So this is related.

[jvandenbroek]

I have the same problem, also rclone+sftp with Hetzner:

Message:  called `Result::unwrap()` on an `Err` value: backoff failed: Permanent(reqwest::Error { kind: Sta
tus(404), url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4
(127.0.0.1)), port: Some(36595), path: "/data/6f31da73b4a59d23778f734df05c1a58a5fa697569486fb0f6d9b2d08c999
d9b", query: None, fragment: None } })                                                                     
                                                                                                           
Caused by:                                                                                                 
    HTTP status client error (404 Not Found) for url (http://127.0.0.1:36595/data/6f31da73b4a59d23778f734df
05c1a58a5fa697569486fb0f6d9b2d08c999d9b) 

While I agree the number of connections / threads should be configurable, I don't think it should panic on such occurrence. Instead I'd expect it to temporary back off and try again later.

[aawsome]

I removed the error label because the abort is a different story (and I think it wouldn't help you to let rustic just gracefully exit in this case).

The main problem is that rclone does return a permanent http error code in some cases where the error might be non-permanent.

We could add a configuration which allows rustic to retry for permanent errors - this IMO is a feature request. Better would be to open a rclone issue to ask if they can treat this case better.

Did you try rustic 0.7.0 and sftp using opendal, see https://github.com/rustic-rs/rustic/blob/main/config/services/sftp_hetzner_sbox.toml? This circumvents the rclone issue.

[simonsan]

https://github.com/rustic-rs/rustic_core/pull/155 starts to implement some logic regarding that topic, but only for the OpenDAL-supported backends.

[k9withabone]

For the opendal backend, it would also be nice to be able to set a bandwidth limit via a ThrottleLayer. That's the only thing that's stopping me from using the opendal backend instead of the rclone one.

[aawsome]

@k9withabone has been implemented in https://github.com/rustic-rs/rustic_core/pull/216 (and should be available in the next nightly build)

[k9withabone]

Thanks! I'll test it out.