python-progressbar

python-progressbar copied to clipboard

Description

I would like to have a progress indicator with multiple counters. My use case is that I have to download a very large number of files from S3. Some will get downloaded, some will be skipped if they already exist locally. I therefore want 2 counters (downloaded_files, and skipped_files). I don't know in advance the total number of files that exist on S3 (and it's too costly to determine it).

Is the best way to do this to use a FormatCustomText? Or could Counter be used? Are both compatible with progressbar.UnknownLength?

My attempt is below but didn't work (possibly the issue I reported in #265)

Code

        cpt_downloaded = 0
        cpt_skipped = 0
        paginator = self._s3client.get_paginator('list_objects_v2')
        with progressbar.ProgressBar(widgets=widgets, max_value=progressbar.UnknownLength) as bar:

            for response in paginator.paginate(Bucket=self._bucket, Prefix=prefix):
                if 'Contents' in response:                    
                    for obj in response['Contents']:
                       if os.path.exists(target_path):
                            self._s3client.download_file(file, target_path)
                            cpt_downloaded += 1
                        else:
                            cpt_skipped += 1

                        format_custom_text.update_mapping(dld=cpt_downloaded,
                                                          skpd=cpt_skipped)
                        time.sleep(0.01)

Versions

• Python version: 3.9
• Operating System: Mac OS
• Package version: 4.0.0

I ended up replacing this with

        with progressbar.ProgressBar(
                prefix='\x1b[33mDownloading files - \x1b[39m | Downloaded {variables.dld} | Skipped {variables.skpd} ',
                variables={'dld': '--', 'skpd': '--'},
                max_value=progressbar.UnknownLength) as bar:
          for response in paginator.paginate(Bucket=self._bucket, Prefix=prefix):
                          if 'Contents' in response:                    
                              for obj in response['Contents']:
                                 if os.path.exists(target_path):
                                      self._s3client.download_file(file, target_path)
                                      cpt_downloaded += 1
                                  else:
                                      cpt_skipped += 1
          
                                  bar.update(bar.value + 1,
                                             dld=cpt_downloaded,
                                             skpd=cpt_skipped)
                                  time.sleep(0.01)

The variables are indeed the way to go here, but I would recommend settings them to 0 instead of -- :)

You can also use the counter if you wish:

import time
import progressbar

with progressbar.ProgressBar(
    widgets=[
        '\x1b[33mDownloading files - \x1b[39m | Downloaded ',
        progressbar.Counter(format='{variables.dld}', new_style=True),
        ' | Skipped ',
        progressbar.Counter(format='{variables.skpd} ', new_style=True),
    ],
    variables=dict(dld=0, skpd=0),
) as bar:
    bar.start()
    dld = 0
    for i in range(100):
        if i % 2:
            dld += 1

        bar.update(dld=dld, skpd=i)

        time.sleep(0.1)