Borg taking 2 days to complete backup as it is Replaying segments

[edsonsbj]

Have you checked borgbackup docs, FAQ, and open GitHub issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

BUG / ISSUE

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

borg 1.4.0

Operating system (distribution) and version.

Ubuntu Server 24.04 LTS

Hardware / network configuration, and filesystems used.

ext4 rclone v.167 fuse

How much data is handled by borg?

1226579 files

Full borg commandline that lead to the problem (leave away excludes and passwords)

#!/bin/bash

SCRIPT_DIR=$(cd $(dirname "${BASH_SOURCE[0]}") && pwd)
CONFIG="$SCRIPT_DIR/BackupRestore-Nextcloud-rem.conf"

# Check if config file exists
if [ ! -f "$CONFIG" ]; then
    echo "ERROR: Configuration file $CONFIG cannot be found!"
    echo "Please make sure that a configuration file '$CONFIG' is present in the main directory of the scripts."
    echo "This file can be created automatically using the setup.sh script."
    exit 1
fi

source "$CONFIG"

# Create a log file to record command outputs
touch "$LogFile"
exec > >(tee -a "$LogFile")
exec 2>&1

# Function for error messages
errorecho() {
    cat <<< "$@" 1>&2
}

## ---------------------------------- TESTS ------------------------------ #
# Check if the script is being executed by root or with sudo
if [ $EUID -ne 0 ]; then
   echo "========== This script needs to be executed as root or with sudo. ==========" 
   exit 1
fi

# -------------------------------FUNCTIONS----------------------------------------- #
BORG_OPTS="--verbose --filter AME --list --progress --stats --show-rc --compression lz4 --exclude-caches"

# Start Rclone Mount
systemctl start borgbackup.service

borg break-lock $BORG_REPO

# Function to Nextcloud Maintenance Mode
nextcloud_enable() {
    # Enabling Maintenance Mode
	sudo -u www-data php $NextcloudConfig/occ maintenance:mode --on
}

nextcloud_disable() {
    # Disabling Nextcloud Maintenance Mode
	sudo -u www-data php $NextcloudConfig/occ maintenance:mode --off
}

# Function to WebServer Stop Start
stop_webserver() {
    # Stop Web Server
	systemctl stop $webserverServiceName
}

start_webserver() {
    # Stop Web Server
	systemctl start $webserverServiceName
}

prune() {
    info "Pruning repository"

    # Use the subcoming `prune` to keep 7 days, 4 per week and 6 per month
    # files of this machine.The prefix '{hostname}-' is very important for
    # limits PLA's operation to files in this machine and does not apply to
    # Files of other machines too:

    borg prune --list --progress --show-rc --keep-daily 7 --keep-weekly 4 --keep-monthly 6

}

# Function to backup Nextcloud settings
nextcloud_settings() {
    echo "========== Backing up Nextcloud settings $( date )... =========="
    echo ""

    nextcloud_enable
    stop_webserver

   	# Export the database.
	mysqldump --quick -n --host=localhost $NextcloudDatabase --user=$DBUser --password=$DBPassword > "$NextcloudConfig/nextclouddb.sql"

    # Backup
    borg create $BORG_OPTS ::'NextcloudConfigs-{now:%Y%m%d-%H%M}' $NextcloudConfig --exclude $NextcloudDataDir

    # Remove the database 
    rm "$NextcloudConfig/nextclouddb.sql"

    start_webserver
    nextcloud_disable
}

# Function to backup Nextcloud DATA folder
nextcloud_data() {
    # Filters for Inclusion Exclusion Borg
    BorgFilters="./nc-patterns.lst"

    # Create a file with the delete standards Borg Inclusion
    tee -a "$BorgFilters" <<EOF > /dev/null 2>&1
P sh
R /

# DO NOT LOOK IN THESE FOLDERS
! proc

# DIRECTORIES TO BE EXCLUDED FROM BACKUP  
- $NextcloudDataDir/*/files_trashbin

# DIRECTORIES FOR BACKUP
+ $NextcloudDataDir/

# DO NOT INCLUDE ANY MORE FILES
- **
EOF

    echo "========== Backing up Nextcloud DATA folder $( date )...=========="
    echo ""

    nextcloud_enable

    borg create $BORG_OPTS --patterns-from "$BorgFilters" ::'NextcloudData-{now:%Y%m%d-%H%M}'

    rm "$BorgFilters"

    nextcloud_disable
}

# Function to perform a complete Nextcloud backup
nextcloud_complete() {
    echo "========== Backing up Nextcloud $( date )... =========="
    echo ""

    nextcloud_enable
    stop_webserver

   	# Export the database.
	mysqldump --quick -n --host=localhost $NextcloudDatabase --user=$DBUser --password=$DBPassword > "$NextcloudConfig/nextclouddb.sql"

    # Backup
    borg create $BORG_OPTS ::'NextcloudFull-{now:%Y%m%d-%H%M}' $NextcloudConfig $NextcloudDataDir --exclude "$NextcloudDataDir/*/files_trashbin"

    # Remove the database
    rm "$NextcloudConfig/nextclouddb.sql"

    start_webserver
    nextcloud_disable
}

# Check if an option was passed as an argument
if [[ ! -z ${1:-""} ]]; then    # Execute the corresponding Backup option
    case $1 in
        1)
            nextcloud_settings
            ;;
        2)
            nextcloud_data
            ;;
        3)
            nextcloud_complete
            ;;
        *)
            echo "Invalid option!"
            ;;
    esac
else
    # Display the menu to choose the Backup option
    echo "Choose a Backup option:"
    echo "1. Backup Nextcloud configurations and database."
    echo "2. Backup only the Nextcloud data folder. Useful if the folder is stored elsewhere."
    echo "3. Backup Nextcloud configurations, database, and data folder."
    echo "4. To go out."

    # Read the option entered by the user
    read option

    # Execute the corresponding Backup option
    case $option in
        1)
            nextcloud_settings
            ;;
        2)
            nextcloud_data
            ;;
        3)
            nextcloud_complete
            ;;
        4)
            echo "Leaving the script."
            exit 0
            ;;
        *)
            echo "Invalid option!"
            ;;
    esac
fi

    # Sleep for 90 minutes before unmounting the drive
sleep 5400

    # Stop Rclone Mount
systemctl stop borgbackup.service

Describe the problem you're observing.

After a broken update of a program, I had to reinstall the server again and, to my surprise, since 01/06, my backups have become extremely slow, starting to take about 3 to 4 hours to complete the backup, when they complete at all.

For about a month now, things have only gotten worse, because instead of 3 to 4 hours on a given day, it gives an error that the mount is not available and ends the backup halfway through and the next day it starts the backup normally but it only finishes the next day or takes up to 2 days and within the log the information contained is that it is recreating the cache and then it starts the backup then stops halfway and starts to recreate the cache again.

I have already made several attempts, such as changing the cache storage, disabling the cache, some changes in rclone and even the cloud where the backup is stored. I have changed and nothing seems to have any effect.

Before the problem with the server, I was running Ubuntu 22.04 LTS and borg. 1.2.8

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

The problem occurs when running the Borg Create command where it takes a long time in an rclone assembly, taking up to 2 to 3 days to perform a backup of just a few MB's.

Include any warning/errors/backtraces from the system logs

Log File Exceeds