Show progress of backup?

[cassidyjames]

Is it possible to show some sort of progress of the backup? I'm not sure what rsync gives you to work with, but it would be great to know an estimated time, or at the very least a progress bar with how far it is in the process.

[chasinglogic]

rsync has a --progress flag but it's extremely ugly and difficult to parse. Even if I could parse it I'd only get per-file progress not entire backup progress.

I really wanted to avoid writing my own backup implementation when a fast robust and stable one like rsync exists. So I do not believe this is possible while using rsync.

[chasinglogic]

Would this be a requirement for publishing on AppCenter?

[cassidyjames]

@chasinglogic nope, was just a personal suggestion. :)

[chasinglogic]

I definitely think this is a good suggestion. After thinking about it a bit it's possible I could try to talk to the rsync daemon. But I'm not familiar enough with the internals of rsync to know for sure.

Hopefully someone smarter than me will come up with an answer or this will reach the top of my backlog.

[leo-petrucci]

+1 on the progress bar, very confusing atm.

[parnoldx]

You could just monitor the target for progress. Btw. there is a great script for rsync that makes it work like time machine https://github.com/laurent22/rsync-time-backup

[chasinglogic]

You could just monitor the target for progress.

I'm not sure what you mean by this. If you mean I can watch it for disk space vs the source that means I would have to do duplicate work to rsync (gather the size of the source directories and the target and hope that $SOURCE_GB - $TARGET_GB = progress). This gets especially difficult once we implement features like #25 because we would have to then re-implement rsync's exclude logic as opposed to just passing it on.

If I've misunderstood or you think I'm wrong I would love to hear a way to get this feature implemented as it's clear people want it.

[parnoldx]

You can use rsync to give you the source size. --dry-run --stats I just googled also this. You can use --info=progress2 for a percentage output of rsync. So maybe just show this as progressbar.

[chasinglogic]

I just googled also this. You can use --info=progress2 for a percentage output of rsync. So maybe just show this as progressbar.

I'm pretty sure that would violate the Elementary HIG if I just showed the TUI progress bar. If I were to somehow parse it to create a HIG-compatible progress bar that would be (IMO) too fragile and finicky.

You can use rsync to give you the source size. --dry-run --stats

That's really useful, however I wonder how we calculate progress still even with this info. For example if you have source_size = 10GB and target_size = 20GB do I assume that means we have 10 gigs of removals to do? How do I know what rsync is actually going to do here?

An even more complex example is source_size = 10GB and target_size = 10GB but the files are different, so there is work to do (rsync will either update, remove, and copy files as necessary) but how will I know that as a simple command runner? The --stats command only shows me the number of files that will be deleted and created but that tells me nothing about the size of those files or how to derive that data.

Finally looking at the output format of --stats --dry-run I have to write a parser that will ignore the giant file list rsync dumps while looking for this section:

Number of files: 541,593 (reg: 430,861, dir: 96,623, link: 14,069, special: 40)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 36,883,042,660 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 2,948,146
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 18,395,343
Total bytes received: 91,619,187

sent 18,395,343 bytes  received 91,619,187 bytes  2,095,514.86 bytes/sec
total size is 36,883,042,660  speedup is 335.26 (DRY RUN)

[parnoldx]

I just googled also this. You can use --info=progress2 for a percentage output of rsync. So maybe just show this as progressbar.

I'm pretty sure that would violate the Elementary HIG if I just showed the TUI progress bar. If I were to somehow parse it to create a HIG-compatible progress bar that would be (IMO) too fragile and finicky.

Not sure what you mean. But you want a progress right? So you can get it from the cmd line with this. You have to parse the cmd line anyway. And this should be easy, then you can show it GUI wise how you want.

[chasinglogic]

You have to parse the cmd line anyway

I don't parse any output from the rsync command today. Parsing a stdout stream with ANSI control characters isn't straight forward IMO. If you know how to do this, since you believe it is easy I assume you do, I would love a PR showing me how!

[parnoldx]

Something like this

'class RsyncParse : GLib.Object {

public static int main(string[] args) {
    var loop = new MainLoop();
    string[] argv = {
        "rsync",
        // Use the -a flag so we do the following:
        //   - recursively backup
        //   - copy symlinks as symlinks
        //   - preserve permissions and modification times
        //   - preserve group and owner
        //   - preserve special files
        "-a",
        "-h",
        "--info=progress2",
        "--no-inc-recursive",
        // Use the --delete flag so we clean up files from the backup that the
        // user has removed.
        "--delete",
        "--bwlimit=20M", //slow down for testing
    };

    argv += "/home/";
    argv += "/home/djax/test/";

    string[] spawn_env = Environ.get ();
    Pid child_pid;

    int standard_input;
    int standard_output;
    int standard_error;

    Process.spawn_async_with_pipes ("/",
        argv,
        spawn_env,
        SpawnFlags.SEARCH_PATH | SpawnFlags.DO_NOT_REAP_CHILD,
        null,
        out child_pid,
        out standard_input,
        out standard_output,
        out standard_error);

    // stdout:
    IOChannel output = new IOChannel.unix_new (standard_output);
    output.add_watch (IOCondition.IN | IOCondition.HUP, (channel, condition) => {
        if (condition == IOCondition.HUP) {
            print ("The fd has been closed.\n");
            return false;
        }
   
        try {
            string line;
            channel.read_line (out line, null, null);
            string[] stats = Regex.split_simple(" +",line.strip());
            //0 is size processed
           //1 is percentage
           //2 is rate
           //3 is eta
            print("%s\n",stats[1]);
        } catch (IOChannelError e) {
            print ("IOChannelError: %s\n", e.message);
            return false;
        } catch (ConvertError e) {
            print ("ConvertError: %s\n", e.message);
            return false;
        }
   
        return true;
    });


    ChildWatch.add (child_pid, (pid, status) => {
        // Triggered when the child indicated by child_pid exits
        Process.close_pid (pid);
        loop.quit ();
    });
    loop.run();
    return 0;
}

}`

The ETA from rsync is still a bit inconsistent despite i fixed the rate. Maybe it's better to calculate 100% size and then time with the rate but I'm not sure.

[chasinglogic]

I think a progress bar is all that's needed, my biggest gripe with this is that rsync doesn't show you the progress of what it will do. So for a small backup I never got above 0%, it also can randomly end at 33%.

Additionally with this implementation if rsync prints an error or some unknown output the application crashed.

I think this can be worked with though, we'll have to use Granite.Application to do the progress bar on the dock.

[parnoldx]

Probably have to consume the error stream to // stderr: IOChannel error = new IOChannel.unix_new (standard_error); error.add_watch (IOCondition.IN | IOCondition.HUP, (channel, condition) => { //do something });