npbackup icon indicating copy to clipboard operation
npbackup copied to clipboard
verified publisher icon netinvent

.conf specifies 2 different repos, npbackup only backs up the "default" via TaskScheduler [Windows]

Open chrispersonal opened this issue 1 year ago • 4 comments

This may be a restic question, or a misunderstanding of how things are supposed to work with restic/npbackup. Please just let me know if that is the case as I am not a restic power-user.

Goal: I want 1 instance of npbackup to manage 2 different repos (each backs up a different file set to a different repo location).

Windows npbackup-3.0.0-rc6-pub-3.11-x86 Taskscheduler > Start a program: C:\...\npbackup-gui\npbackup-gui.exe -c "C:\...\npbackup-gui\npbackup.conf --backup --run-as-cli

Shortened .conf file. Notice "default" and "DrivePool" repos, each backs up a different fileset to different repo locations:

audience: public
repos:
  default:
    repo_uri:  [Location 1]    
    repo_group: default_group
    backup_opts:
      paths:
      - C:/.../Documents
      exclude_patterns:
    repo_opts:
      retention_policy: {}
      repo_password: [redacted]
    prometheus: {}
    env: {}
    is_protected: false
  DrivePool:
    repo_uri: [Location 2]
    repo_group: default_group
    backup_opts:
      paths:
      - K:/
      exclude_patterns:
    repo_opts:
      retention_policy: {}
      repo_password: [redacted]
    prometheus: {}
    env: {}
    is_protected: false
groups:
  default_group:

At this point, I would expect whenever --backup is executed via the TaskScheduler it would generate snapshots according to the retention policy (in this case, the default_group's policy).

However, the repo "DrivePool" is now going on several days of not being snapshotted (but the "default" repo is working just fine).

  • DrivePool repo shows "too old" in GUI with a date/time of 2024-09-25 15:33:09.

cli.log example output:

2024-09-26 15:34:10,426 :: INFO :: npbackup v3.0.0-rc6-pub-3.11-x64 2024091601 - Copyright (C) 2022-2024 NetInvent running as ...
2024-09-26 15:34:10,577 :: INFO :: Loaded config 85503A77 in C:\...npbackup-gui\npbackup.conf
2024-09-26 15:34:10,692 :: INFO :: No extra information for repos default found
2024-09-26 15:34:10,692 :: INFO :: No extra information for repos DrivePool found
2024-09-26 15:34:11,310 :: INFO :: Searching for a backup newer than 1 day, 0:00:00 ago
2024-09-26 15:34:14,104 :: INFO :: Snapshots listed successfully
2024-09-26 15:34:14,186 :: INFO :: No recent backup found in repo default. Newest is from 2024-09-25 09:37:57.360200-04:00
2024-09-26 15:34:14,186 :: INFO :: Runner took 2.875824 seconds for has_recent_snapshot
2024-09-26 15:34:14,186 :: INFO :: Running backup of ['C:/.../Documents'] to repo default
2024-09-26 15:34:15,290 :: INFO :: Using VSS snapshot to backup
2024-09-26 15:41:14,692 :: INFO :: Backend finished with success
2024-09-26 15:41:14,783 :: INFO :: Processed 122.4 MB of data
2024-09-26 15:41:14,784 :: INFO :: Operation finished with success
2024-09-26 15:41:14,810 :: INFO :: Runner took 423.501894 seconds for backup
2024-09-26 15:41:14,812 :: INFO :: 
using parent snapshot f354506e
creating VSS snapshot for [c:\]
successfully created snapshot for [c:\]

Files:           0 new,     1 changed,   969 unmodified
Dirs:            0 new,     5 changed,    49 unmodified
Added to the repository: 42.883 KiB (9.615 KiB stored)

processed 970 files, 116.720 MiB in 6:48
snapshot ec5461fc saved

2024-09-26 15:41:14,812 :: INFO :: Operation finished
2024-09-26 15:41:14,984 :: INFO :: ExecTime = 0:07:04.578924, finished, state is: success.

2024-09-26 21:34:04,064 :: INFO :: npbackup v3.0.0-rc6-pub-3.11-x64 2024091601 - Copyright (C) 2022-2024 NetInvent running as ...
2024-09-26 21:34:04,197 :: INFO :: Loaded config 85503A77 in C:\...npbackup-gui\npbackup.conf
2024-09-26 21:34:04,300 :: INFO :: No extra information for repos default found
2024-09-26 21:34:04,301 :: INFO :: No extra information for repos DrivePool found
2024-09-26 21:34:04,970 :: INFO :: Searching for a backup newer than 1 day, 0:00:00 ago
2024-09-26 21:34:09,437 :: INFO :: Snapshots listed successfully
2024-09-26 21:34:09,509 :: INFO :: Recent snapshot ec5461fc of 2024-09-26T15:34:15.5574383-04:00 exists !
2024-09-26 21:34:09,510 :: INFO :: Most recent backup in repo default is from 2024-09-26 15:34:15.557438-04:00
2024-09-26 21:34:09,510 :: INFO :: Runner took 4.541332 seconds for has_recent_snapshot
2024-09-26 21:34:09,511 :: INFO :: No backup necessary
2024-09-26 21:34:09,512 :: INFO :: Runner took 4.621391 seconds for backup
2024-09-26 21:34:09,512 :: INFO :: Operation finished
2024-09-26 21:34:09,623 :: INFO :: ExecTime = 0:00:05.584242, finished, state is: success.

2024-09-27 03:34:10,109 :: INFO :: npbackup v3.0.0-rc6-pub-3.11-x64 2024091601 - Copyright (C) 2022-2024 NetInvent running as ...
2024-09-27 03:34:10,421 :: INFO :: Loaded config 85503A77 in C:\...\npbackup-gui\npbackup.conf
2024-09-27 03:34:10,533 :: INFO :: No extra information for repos default found
2024-09-27 03:34:10,533 :: INFO :: No extra information for repos DrivePool found
2024-09-27 03:34:10,778 :: INFO :: Searching for a backup newer than 1 day, 0:00:00 ago
2024-09-27 03:34:14,857 :: INFO :: Snapshots listed successfully
2024-09-27 03:34:14,872 :: INFO :: Recent snapshot ec5461fc of 2024-09-26T15:34:15.5574383-04:00 exists !
2024-09-27 03:34:14,873 :: INFO :: Most recent backup in repo default is from 2024-09-26 15:34:15.557438-04:00
2024-09-27 03:34:14,873 :: INFO :: Runner took 4.094786 seconds for has_recent_snapshot
2024-09-27 03:34:14,873 :: INFO :: No backup necessary
2024-09-27 03:34:14,874 :: INFO :: Runner took 4.141108 seconds for backup
2024-09-27 03:34:14,875 :: INFO :: Operation finished
2024-09-27 03:34:14,986 :: INFO :: ExecTime = 0:00:04.901028, finished, state is: success.

I am reading the log as npbackup is seeing both repos (ie. "no extra information..." entries), but then it lists snapshots and sees that a recent snapshot exists (I imagine only for "Default" repo?) and then doesn't go and check the next repo and it's snapshot status. The second repo "DrivePool" never gets analyzed as needing to also be snapshotted.

Any ideas why this would be or how to adjust for it?

chrispersonal avatar Sep 27 '24 13:09 chrispersonal

Indeed, currently NPBackup task creation GUI only cretes a task for the current repository. If you want both repositories to be handled, you'll have to manually modifiy the task by changing --backup to --group-operation backup --repo-group default_group which will then execute the tasks for every repo in the group default_group.

This is linked to #61. I'll have the GUI part to this sorted out before v3 final.

deajan avatar Sep 28 '24 15:09 deajan

Thank you. Would these options also apply with the --housekeeping flag?

chrispersonal avatar Sep 28 '24 19:09 chrispersonal

All operations that NPBackup does can be batched via --group-operation, --housekeeping included ;)

deajan avatar Sep 28 '24 19:09 deajan

Currently implementing task scheduler GUI for all tasks types and all repos in operation center. Stay tuned for next RC that will have a nice GUI to handle these ;)

deajan avatar Sep 30 '24 22:09 deajan

So this is getting way harder than I thought. Discussing with windows task scheduler is a real pain, especially via powershell which decides it shall shrink important information by replacing it with ellipses...

Anyway, I won't improve that GUI part until next release, since it involves getting rid of task scheduler completely, and running npbackup with it's own task scheduler, which implies running as service.

So for now, I added a remark that any other than default task must be configured manually.

deajan avatar Oct 25 '24 13:10 deajan

Finally a part of this has been put into latest 3.0.0rc8, which allows task creation for various repos / repo groups. Just not the best UI I can come up, but it works. Closing this in favor of #105

deajan avatar Oct 29 '24 15:10 deajan