Bug: skip_dir path may be deleted from remote drive

[dalemartin]

Describe the bug

A remote SharePoint folder may be deleted even if a skip_dir pattern matches it.

Operating System Details

Using a custom Linux kernel based on 6.6.51.


$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 12 (bookworm)
Release:	12
Codename:	bookworm

Client Installation Method

From 3rd Party Source (PPA, OpenSuSE Build Service etc)

OneDrive Account Type

Business | Office365

What is your OneDrive Application Version

onedrive v2.5.2-1+np1+1.1

What is your OneDrive Application Configuration

$ onedrive --confdir="~/.config/onedrive/accounts/SharePoint_SDI_Documents" --display-config
Reading configuration file: /home/dmartin/.config/onedrive/accounts/SharePoint_SDI_Documents/config
Configuration file successfully loaded
WARNING: D-Bus message bus daemon is not available; GUI notifications are disabled
Application version                          = onedrive v2.5.2-1+np1+1.1
Compiled with                                = LDC 2100
User Application Config path                 = /home/dmartin/.config/onedrive/accounts/SharePoint_SDI_Documents
System Application Config path               = /etc/onedrive
Applicable Application 'config' location     = /home/dmartin/.config/onedrive/accounts/SharePoint_SDI_Documents/config
Configuration file found in config location  = true - using 'config' file values to override application defaults
Applicable 'sync_list' location              = /home/dmartin/.config/onedrive/accounts/SharePoint_SDI_Documents/sync_list
Applicable 'items.sqlite3' location          = /home/dmartin/.config/onedrive/accounts/SharePoint_SDI_Documents/items.sqlite3
Config option 'drive_id'                     = ********
Config option 'sync_dir'                     = ~/SharePoint/SDI_Documents
Config option 'enable_logging'               = false
Config option 'log_dir'                      = /var/log/onedrive
Config option 'disable_notifications'        = false
Config option 'skip_dir'                     = /foo/bar*
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = ~*|.~*|*.tmp|*.swp|*.partial
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 300
Config option 'monitor_log_frequency'        = 12
Config option 'monitor_fullscan_frequency'   = 12
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = false
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = false
Config option 'resync_auth'                  = false
Config option 'cleanup_local_files'          = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'disable_download_validation'  = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'application_id'               = d50ca740-c83f-4d1b-b616-12c519384f0c
Config option 'azure_ad_endpoint'            = 
Config option 'azure_tenant_id'              = 
Config option 'user_agent'                   = ISV|abraunegg|OneDrive Client for Linux/v2.5.2-1+np1+1.1
Config option 'force_http_11'                = false
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 0
Config option 'operation_timeout'            = 3600
Config option 'dns_timeout'                  = 60
Config option 'connect_timeout'              = 10
Config option 'data_timeout'                 = 60
Config option 'ip_protocol_version'          = 0
Config option 'threads'                      = 8
Environment var 'XDG_RUNTIME_DIR'            = true
Environment var 'DBUS_SESSION_BUS_ADDRESS'   = true
Config option 'notify_file_actions'          = false

Selective sync 'sync_list' configured        = true
sync_list config option 'sync_root_files'    = false
sync_list contents:
/foo

Config option 'sync_business_shared_items'   = false

Config option 'webhook_enabled'              = false

What is your 'curl' version

$ curl --version
curl 7.88.1 (x86_64-pc-linux-gnu) libcurl/7.88.1 OpenSSL/3.0.14 zlib/1.2.13 brotli/1.0.9 zstd/1.5.4 libidn2/2.3.3 libpsl/0.21.2 (+libidn2/2.3.3) libssh2/1.10.0 nghttp2/1.52.0 librtmp/2.3 OpenLDAP/2.5.13
Release-Date: 2023-02-20, security patched: 7.88.1-10+deb12u7
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

Where is your 'sync_dir' located

Local

What are all your system 'mount points'

$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=16212332k,nr_inodes=4053083,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=3252812k,mode=755)
rootpool/ROOT/falcon on / type zfs (rw,relatime,xattr,posixacl,casesensitive)
rootpool/ROOT/falcon/etc on /etc type zfs (rw,relatime,xattr,posixacl,casesensitive)
rootpool/ROOT/falcon/home on /home type zfs (rw,relatime,xattr,posixacl,casesensitive)
rootpool/ROOT/falcon/tmp on /tmp type zfs (rw,relatime,xattr,posixacl,casesensitive)
rootpool/ROOT/falcon/usr on /usr type zfs (rw,relatime,xattr,posixacl,casesensitive)
rootpool/ROOT/falcon/var on /var type zfs (rw,relatime,xattr,posixacl,casesensitive)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2963)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/dev/nvme0n1p3 on /boot type ext4 (rw,relatime)
/dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
rootpool/qemu on /qemu type zfs (rw,relatime,xattr,posixacl,casesensitive)
rootpool/redis_data on /redis_data type zfs (rw,relatime,xattr,posixacl,casesensitive)
rootpool/registry_data on /registry_data type zfs (rw,relatime,xattr,posixacl,casesensitive)
rootpool/docker on /var/lib/docker type zfs (rw,relatime,xattr,posixacl,casesensitive)
rootpool/qemu/images on /qemu/images type zfs (rw,relatime,xattr,posixacl,casesensitive)
rootpool/qemu/isos on /qemu/isos type zfs (rw,relatime,xattr,posixacl,casesensitive)
rootpool/qemu/images/sdi_builder on /qemu/images/sdi_builder type zfs (rw,relatime,xattr,posixacl,casesensitive)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/user/115 type tmpfs (rw,nosuid,nodev,relatime,size=3252808k,nr_inodes=813202,mode=700,uid=115,gid=125)
gvfsd-fuse on /run/user/115/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=115,group_id=125)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3252808k,nr_inodes=813202,mode=700,uid=1000,gid=1000)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
portal on /run/user/1000/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
Obsidian-1.6.7.AppImage on /tmp/.mount_obsidiWbRHhn type fuse.Obsidian-1.6.7.AppImage (ro,nosuid,nodev,relatime,user_id=1000,group_id=1000)
OneDriveGUI-1.1.0-x86_64.AppImage on /tmp/.mount_onedri31CDxu type fuse.OneDriveGUI-1.1.0-x86_64.AppImage (ro,nosuid,nodev,relatime,user_id=1000,group_id=1000)
superProductivity-9.0.7.AppImage on /tmp/.mount_superp5JUxht type fuse.superProductivity-9.0.7.AppImage (ro,nosuid,nodev,relatime,user_id=1000,group_id=1000)

What are all your local file system partition types

$ lsblk -f
NAME        FSTYPE     FSVER LABEL    UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
nvme0n1                                                                                   
├─nvme0n1p1 vfat       FAT32 EFI      F8EB-A7F1                             497.3M     1% /boot/efi
├─nvme0n1p2 zfs_member 5000  rootpool 13321229604715348716                                
├─nvme0n1p3 ext4       1.0            cbf5bf9c-13ac-4a87-bfcf-749fbd6aa5fc    3.5G     3% /boot
└─nvme0n1p4 swap       1              ab4d8f73-b48f-4975-a7f6-fa1754c0bb70                [SWAP]

How do you use 'onedrive'

• Local onedrive folder not shared with any other system.
• Syncing to my company's SharePoint site. We use Office 365.

Steps to reproduce the behaviour

Consider the following setup:

• A SharePoint site named "SDI" has the following tree:

  Documents/
  └── foo
      ├── barbar
      └── baz
  

• Local sync_dir: /home/dmartin/SharePoint/SDI_Documents. It is empty because the first sync has not yet been performed.
• ~/.config/onedrive/accounts/SharePoint_SDI_Documents/sync_list contains one entry: /foo
• ~/.config/onedrive/accounts/SharePoint_SDI_Documents/config contains: skip_dir = "/foo/bar*"
• All other config is default, except drive_id.

1. Upon the first sync, the remote directory /foo/barbar is synced locally.
2. Delete /foo/barbar locally, since the intent was to skip this one.
3. On the next sync, the remote /foo/barbar is deleted, despite being named in skip_dir.

Complete Verbose Log Output

# The log contains sensitive information so I would rather not share it. If it is absolutely needed, I can take the time to sanitize it and upload, but I'm hoping my description is sufficient for you to reproduce it. Or if you have any guidance as to which parts of the log you need I can work with you.

Screenshots

Original remote tree:

Other Log Information or Details

No response

Additional context

Original discussion post: https://github.com/abraunegg/onedrive/discussions/2909

Originally I thought this behaviour manifests itself on the very first sync. Now I am not able to reproduce the delete behaviour unless I first temporarily remove the skip_dir entry, then sync, then delete the directory locally, then restore the skip_dir entry, and finally sync. I was tinkering with settings for a while and was quite confused before I realized what was happening. It seems that something was cached even during --dry-run syncs which later affected the actual syncs, although I have not proven this, it is just my recollection of the first time I noticed the behaviour.

I hope this is helpful, I have really appreciated this app, thank you for all your hard work.