Google drive does not use cleanup

[chrisleekr]

Hi,

This is my configuration.yml for phpbu.

I have configured to remove backups outdated 3 days.

Local backup seems working fine. It removed outdated backup.

However, remote backup does not remove outdated backup. (I have attached screenshot.)

Is there something wrong with my configuration?

<?xml version="1.0" encoding="UTF-8"?>
<phpbu xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:noNamespaceSchemaLocation="http://schema.phpbu.de/5.1/phpbu.xsd">
    <backups>
        <backup name="database">
            <!-- backup source -->
            <source type="mysqldump">
                <option name="host" value="127.0.0.1"/>
                <option name="port" value="3306"/>
                <option name="databases" value="database"/>
                <option name="user" value="user"/>
                <option name="password" value="password"/>
            </source>
            <!-- where should the backup be stored -->
            <target dirname="/home/backup"
                    filename="mysqldump-%Y%m%d-%H%i.sql"
                    compress="gzip"/>


            <sync type="googledrive">
                <option name="access" value="google_access.json"/>
                <option name="secret" value="google_secret.json"/>
                <option name="parentId" value="xxxxxxxx"/>
                <option name="cleanup.type" value="outdated"/>
                <option name="cleanup.older" value="3D"/>
            </sync>

            <cleanup type="outdated" skipOnFailure="false">
                <option name="older" value="3D"/>
            </cleanup>
        </backup>
        <backup name="file">
            <source type="tar">
                <option name="path" value="/home/user"/>
            </source>
            <target dirname="/home/backup"
                    filename="public_html-%Y%m%d-%H%i.tar"
                    compress="gzip"/>

            <sync type="googledrive">
                <option name="access" value="google_access.json"/>
                <option name="secret" value="google_secret.json"/>
                <option name="parentId" value="xxxxxxxx"/>
                <option name="cleanup.type" value="outdated"/>
                <option name="cleanup.older" value="3D"/>
            </sync>

            <cleanup type="outdated" skipOnFailure="false">
                <option name="older" value="3D"/>
            </cleanup>
        </backup>
    </backups>
</phpbu>

[sebastianfeldmann]

The time difference is measured in seconds. So even if we currently have the 22 of may there is a chance that a backup from the 19th of May is not deleted. If the backup generation and syncing just takes a couple of seconds less the previous backup will not be 3 days before, it will be 2 days 23 hours 59 minutes and 57 seconds before. so it will note be deleted.

I tried to reproduce the error but in my case setting the older value to 5I (five minuets) phpbu deleted everything older then 5 minutes in my google drive.

Please try again and make sure it really doesn't delete anything even if you change the older value to 2D (two days)

[chrisleekr]

Hi @sebastianfeldmann

I did change to 2D and ran the backup.

It did remove files from local backup, but not from Google Drive. See screenshots.

[sebastianfeldmann]

That's not good :( Like I said it is working on my machine :D Are you using the phar or the composer variant of phpbu? If you are using the composer variant I would ask you to add some debug output to your local file. I can provide you same guidance where to put it and what to output, so we can figure out what is going on on "your" end. If you are using the phar variant I will try add some sensical debug output on my end and release a new version so --debug actually helps you figuring out what is happening.

[chrisleekr]

Hey, @sebastianfeldmann

I am using phar at the moment. So if you can give me steps, I can try debug.

Let's find out what is going on.

[sebastianfeldmann]

I added some debug output to the latest PHAR release. If you run

$ phpbu --self-update
$ phpbu --debug

You should see some debug output for the Outdated cleanup

[chrisleekr]

Ok, let me try.

[chrisleekr]

phpbu 5.2.4 by Sebastian Feldmann and contributors.

backup: [mysqldump] *******************************************************
/usr/bin/mysqldump --user='user' --password='******' --host='127.0.0.1' --port='3306' 'database' | /bin/gzip > /home/user/backup/mysql/mysqldump-20190601-1817.sql.gz
ok

sync: [googledrive] *******************************************************
upload: done: 1d7wln80XXLJoN_w_7ls-AADk_upXXXX
remote cleanup: outdated
  found 0 backup files
ok

cleanup: [outdated] *******************************************************
  found 4 backup files
  checking mysqldump-20190601-1817.sql.gz 2019.06.01 18:18:03
  checking mysqldump-20190531-0000.sql.gz 2019.05.31 00:00:32
  checking mysqldump-20190601-1816.sql.gz 2019.06.01 18:16:52
  checking mysqldump-20190601-0000.sql.gz 2019.06.01 00:00:35
ok

Ok, I found strange thing.

I have configured Google Drive parentID as

<option name="parentId" value="1y3zhDHjhY9bYFkpmRlqxftZe7K0iXXXX"/>

Note I masked last four characters.

Parent ID should be correct as I can see uploaded files in the Google Drive.

However, as you can see from above log, it is saying upload is done in 1d7wln80XXLJoN_w_7ls-AADk_upXXXX. And remote cleanup could not find any backup files.

I don't know where 1d7wln80XXLJoN_w_7ls-AADk_upXXXX comes from. The parent ID is not my Google Drive as well.

How do I get a correct parent ID? I mean the ID will be the folder that I want to upload right? Something like this, right?

https://drive.google.com/drive/folders/1y3zhDHjhY9bYFkpmRlqxftZe7K0iXXXX

[sebastianfeldmann]

I think 1d7wln80XXLJoN_w_7ls-AADk_upXXXX is the ID of the newly created file so that should be fine.

The thing I'm not getting so far is why the collector doesn't find any backups. The collector is using the configure parentId to get all files and then tries to find all the backups.

The only thing I can image right now is that something with the file name comparison isn't working. The collector should collect all backups by comparing filenames.

/** @var \Google_Service_Drive_DriveFile $googleFile */
foreach ($results->getFiles() as $googleFile) {
    if ($this->isFileMatch($this->path->getPath() . '/' . $googleFile->getName())) {
        $file                = new File\GoogleDrive($this->service, $googleFile);
        $index               = $this->getFileIndex($file);
        $this->files[$index] = $file;
    }
}

Have to take a detailed look at this :)

[ciklum-bohdan]

I think 1d7wln80XXLJoN_w_7ls-AADk_upXXXX is the ID of the newly created file so that should be fine.

The thing I'm not getting so far is why the collector doesn't find any backups. The collector is using the configure parentId to get all files and then tries to find all the backups.

The only thing I can image right now is that something with the file name comparison isn't working. The collector should collect all backups by comparing filenames.

/** @var \Google_Service_Drive_DriveFile $googleFile */
foreach ($results->getFiles() as $googleFile) {
    if ($this->isFileMatch($this->path->getPath() . '/' . $googleFile->getName())) {
        $file                = new File\GoogleDrive($this->service, $googleFile);
        $index               = $this->getFileIndex($file);
        $this->files[$index] = $file;
    }
}

Have to take a detailed look at this :)

I think it just doesn't perform any remote cleanups cause it

 protected function executeCleanup(Configuration\Backup $backup, Target $target, Local $collector)

accepts only a Local collector.

Probably the way to fix it is to improve synchronizer and make it not only upload files (to google for example) but also remove some old files and upload new ones

also, you would need to change the order execution of functions in phpbu\App\Runne\Backup::run() method

$this->executeSource($backup, $target);
$this->executeChecks($backup, $target, $collector);
$this->executeCrypt($backup, $target);
$this->executeCleanup($backup, $target, $collector); // this do only a local clean up
$this->executeSyncs(); // this will upload new files and remove outdated on remote server  

[sebastianfeldmann]

So how this works is:

The google sync gets triggered https://github.com/sebastianfeldmann/phpbu/blob/1c55bf5e194dafe78c0595c2bc5e9cb7b9e87bed/src/Backup/Sync/GoogleDrive.php#L119

After the sync is done the cleanup is triggered. https://github.com/sebastianfeldmann/phpbu/blob/1c55bf5e194dafe78c0595c2bc5e9cb7b9e87bed/src/Backup/Sync/GoogleDrive.php#L144

This is actually handled inside the Cleanable trait. https://github.com/sebastianfeldmann/phpbu/blob/1c55bf5e194dafe78c0595c2bc5e9cb7b9e87bed/src/Backup/Sync/Cleanable.php#L83

The trait requests the responsible Collector instance from the GoogleDrive sync class and gets returned a Collector\GoogleDrive. https://github.com/sebastianfeldmann/phpbu/blob/1c55bf5e194dafe78c0595c2bc5e9cb7b9e87bed/src/Backup/Sync/GoogleDrive.php#L255

Then the trait uses the previously setup Cleaner to clean up the remote files by passing the collector to the cleaner. https://github.com/sebastianfeldmann/phpbu/blob/1c55bf5e194dafe78c0595c2bc5e9cb7b9e87bed/src/Backup/Sync/Cleanable.php#L91

So there should not be a problem with Local collectors. The sync cleanup doesn't care about local files. There shouldn't be any connection between local and remote files. You can delete all local files and the remote collector still should find all remote files and the cleaner should identify the ones to delete and act accordingly.

If you run phpbu --debug and you see a line similar to remote cleanup: outdated you can be sure the Cleaner is correctly setup and is getting executed.

The problem you are experiencing should happen somewhere here: https://github.com/sebastianfeldmann/phpbu/blob/1c55bf5e194dafe78c0595c2bc5e9cb7b9e87bed/src/Backup/Cleaner/Outdated.php#L66

That leaves 3 possible reasons.

• The GoogleDrive collector isn't working correctly
• The Oudated cleaner does something wrong
• The GoogleDrive file implementation has a bug

Sadly I still can't reproduce the problem. It works on my machine :/ If you are able to install phpbu via composer you should be able to add some debug output or even start a debugging session.

The interesting questions would be:

• Does the collector find all your backup files?
• Does the cleaner handle those files correctly?
• Does the file::delete implementation work for google drive?