restic Stuck upload to B2

This issue is opened in relation with this Debian bug.

Basically, I use BackBlaze to do backupus for several years now using the same script and did never encounter any problem so far. (Thanks, restic is a very cool tool which improves my life by making me feelmuch more secure.)

A few days ago the backup script started to stale.

I checked with several -v that it is not actually sending anything. Just that the time progresses indefinitely but the backup never completes.

I did remove a bunch of snapshots with prune, which freed 20GB but even that got stale at the repacking stage.

I don't think it's a problem with quotas because I didn't receive any alert from BaclBlaze whereas they are enabled.

Happy to provide any additional info which could help.

Sep 18 '24 21:09 shindere

Hi, it would help to have complete the command line you used to be able to reproduce the bug, and maybe an extract of the end of the DEBUG_LOG you got (removing sensible information of course).

Sep 19 '24 08:09 legrostdg

Thanks. I notice that the bug has disappeared.

I am currently using another ISP so I don't know whether it's because of that, or because there was another intermitent problem and the fact that this works now from another ISP is just a coincidence.

What I can say, though, is that no update happened onmy system between when it worked and when it started to not work.

I will be back to the "not-working" ISP at the beginning of next week and will report back here if it still does not work.

It feels good to be able to backupu again!!

Sep 21 '24 09:09 shindere

I confirm thingsnowwork again, evenon theISP wherethey didn't work last week.

Sep 23 '24 16:09 shindere

The problem has re-appeared without anything having changed on the machine.

I did capture a log but it's 241089 lines long and, with just a braille device, I find it super hard to find the relevant part.

Any hint on a text to lookfor and/or a way to sanitize the log before sharing?

Oct 07 '24 20:10 shindere

One more observation is that the behaviour seems to depend on which IP address is used.

Indeed, when connecting from home (fixed IP) the command gets stale, while when using another network, say the one ofmy mobile phone (which is itself not connected to my home network, Imade sure of that by disconnectingthe wireless and turning off my router), fromthere the backup works.

Oct 13 '24 06:10 shindere

This issue reminds me a lot of https://github.com/restic/restic/issues/3636 . There the problem was that one of the backblaze servers ended up in a DNS blocklist. The b2 backend then unfortunately kept retrying uploads endlessly.

The most interesting parts of the debug log are probably the parts where restic sends the requests to the B2 servers. If you search for lines that include logger.(*Backend).Save then the corresponding http request should be logged a few lines later.

An alternative which you can try is to use the S3 backend to upload to B2. It provides better error handling and might also just avoid the issue that block your B2 backups. B2 also provides an S3 API (see the Warning box right at the start of the B2 section in the restic docs ). See the S3 backend docs for further details on setting up the S3 backend.

The necessary settings for S3 using B2 are something like the following:

export RESTIC_REPOSITORY=s3:https://s3.eu-central-003.backblazeb2.com/bucket-name
export AWS_ACCESS_KEY_ID=id of the access key with read-write permissions
export AWS_SECRET_ACCESS_KEY=secret part of the access key

Oct 16 '24 20:10 MichaelEischer

Many thanks for sharing your thought, Michael!

This seems to indicate that I didn't think about sharing my last findings here, for which I apologize.

I now believe that the issue I reported here is actually not at all restic or BackBlaze related.

Indeed, I could observe similar problems when trying to upload an MP3 file of approximately 125MB over ssh. I tried theupload on 3 different machines located in 3 different countries and consistently observed the same problem, namely that the transfer rate continuously decreases until the nconnection becomes stalled.

I then tried to see whether connecting to my router either over Wireless or over Ethernet made a difference and it didn't.

I guess the next step will be to figure out whether the problem comes from the router itself or somehow from the ISP.

Meanwhile I am able to do the backups using my phone's 4G network access. Not ideal but quite an improvement over the previous situation in which I was not able to do the backups at all.

Again, many thanks for having shared your thoughts, I very much appreciate.

Oct 16 '24 20:10 shindere

Is there anything left to do here from our side? Or can we close the issue?

Oct 28 '24 19:10 MichaelEischer

Michael Eischer (2024/10/28 12:13 -0700):

Is there anything left to do here from our side? Or can we close the issue?

Thanks for asking and sorry I didn't share the conclusion spontaneously.

What fixed the problem was to stop the DSL model for 10 minutes and restart it.

The 10 minutes do matter as I did restart the modem before but without the 10 minutes break.

I think this issue can be closed, as well as the associated Debian bug report.

Thanks!

Oct 29 '24 09:10 shindere