python-progressbar

python-progressbar copied to clipboard

Description

Given multiple tasks processed in parallel, a progressbar for each one may bring some enlightment to the developer.

Code

i found this option here, but its interface makes it too slow to process heavy tasks. Your ASCII-based solution suits me well.

Versions

• Python version: 3.8.10 (default, Nov 26 2021, 20:14:08) [GCC 9.3.0]
• Python distribution/environment: CPython/Anaconda/IPython/IDLE
• Operating System: Ubuntu Linux

We may even try to revamp the archictecture of this solution here: https://github.com/rsalmei/alive-progress/issues/20#issuecomment-603489445

It's been a long standing issue to add this and it's at the top of my to-do list for this library:

• https://github.com/WoLpH/python-progressbar/issues/249
• https://github.com/WoLpH/python-progressbar/issues/189
• https://github.com/WoLpH/python-progressbar/issues/176

It is a complicated issue to implement well however, my first few attempts at it resulted in breaking other behaviour in the progressbar and backwards compatibility is quite important to me. This progressbar is perhaps not as pretty as alive-progress out of the box, but it should be at least very fast and stable :)

Take a look on the link from the second post. The solution may lay on re-render the multiple bars. My best shot is:

• add a class property counter for the progress-bar instance
• listen to stout and look for \n character;
• update the current counter (for strongly typed languages, I recommend some either unsigned long int or scientific mantissa-like notation) for every \n on stout output;
• Bars with very different rates (100 or more) require special treatment. A good approach is to take, for example, the max of some default minimum refresh rate and either average or median of bars current rates.

Bars may always keep on the top of the terminal stack from the command run.

It's slightly more complicated than that. Within a terminal you can't just write at a specific line, all output is appended at the end of the current line.

To rewrite a current line we can use \r which is supported by many shells and environments, but to write multiple lines simultaneously requires more advanced control over the terminal which has much less support.

It is very likely that pycharm for example, won't support it

It seems line starve in combination to delete character or ANSI cursor move character may be useful here.

While it's not a full solution, this little bit of code reasonably approaches the goal: https://github.com/WoLpH/python-progressbar/issues/189#issuecomment-1246116077

And for a solution that features locks for multithreading: https://github.com/WoLpH/python-progressbar/issues/176

Hi, Wolph. My surname is "Wolf". What a coincidence! Nevertheless, Let me get this straight: we update the progress bars only at particular sample instants, right? It is very reasonable! Will somebody release it as a feature somewhen in the near future?

Haha, that's indeed a coincidence :)

Yes, that is correct. It is doing slightly more now than it needs to do, but it's still reasonably efficient. When you have 2 progressbars what it essentially does when updating the top progressbar:

1. move the cursor up
2. print the progressbar
3. move the cursor back down

With regards to the new feature, I have to review the impressive bit of code that @kdschlosser wrote: https://github.com/kdschlosser/python-progressbar/tree/multi_thread I will merge it if it doesn't break any existing scenarios/environments. And that's something I still need to test :) As always, time is limited but I hope to have this pushed out within a month or so.

The test step must be very satisfying. We hope it stands! Great job!

I was messing around the issues section and wondered: don't you think it is a great time to refactor the README file and add current functionalities and beauties of this library? It is very convenient.

Update the readme for what??

Fancy words right?! I mean, developers use to think they speak "devish" language or something. I mean to refactor README in a more user-friendly way to bring the useful libraries functionalities to daily simple activities. Also, some images of use cases would be welcome as well.

Have at it and submit a PR for it. Personally I am one to keep the Readme to describe the library and maybe some screen shots. Usage would be and should be contained in the docstrings.

For the changes I made there is no real usage to know about. create more then a single bar and update them. this can be done in a single thread or multiple threads it doesn't matter. So from a usage standpoint there is nothing that needs to be done differently with having multiple bars other than constructing more then one bar

I see. I recommend you add a minimal example snippet as well as a screenshot of the final result or a GIF to the existing README for future users to enjoy it as you do.

Fancy words right?! I mean, developers use to think they speak "devish" language or something. I mean to refactor README in a more user-friendly way to bring the useful libraries functionalities to daily simple activities. Also, some images of use cases would be welcome as well.

Documentation really isn't my strong suit unfortunately... all of my projects suffer from this issue. All help is welcome though, if anyone is better at documenting :)

I can help if you list the "hot" topics of this library in a list of bullet points. :) (or any library you wish to improve)

I can help if you list the "hot" topics of this library in a list of bullet points. :) (or any library you wish to improve)

In the case of this library... I think one of the nicer features is that it can do full output redirection (i.e. when enabled, you can still do print(...) while using the progress bar).

I think some animated gifs of the output could be very useful to demonstrate what it can do. There are quite a few examples available but I'm not really sure what would appeal to most people.

Great remarks! It recalls me on morgan library with log categories. I will report to you latter on with some tests. :-)

In my experience, good libraries are almost invisible to people and still make them aware of their existence. Example: NumPy, pandas, tqdm, matplotlib. In my opinion, a non-verbose library is a good development practice.

In our case, some of these library practices are somewhat "user-frightening", widgets and the use of explicit object properties like "progressbar.ProgressBar" or "progressbar.AnimatedMarker": they are yes necessary for the feature construction, but relevant only to the programmer to know. For example, in the case of "progressbar.AnimatedMarker(fill='#')", the user provides its marker preference '#' to object "progressbar.ProgressBar", and the library assigns the respective widgets at will.

Do you understand me?

The words below are those needed by the user. A common human uses at most 10 words in his/her "cache memory".

ProgressBar Bar progressbar Percentage AnimatedMarker MultiRangeBar MultiProgressBar GranularBar PercentageLabelBar RotatingMarker ETA FileTransferSpeed ReverseBar SimpleProgress Counter Timer FormatLabel BouncingBar AdaptiveETA AbsoluteETA AdaptiveTransferSpeed Variable FormatCustomText

@wolph How are you? I hope you are great. Have you take a look on this idea? Thanks. :-)

The stale bot is a bit overzealous (I'm still tweaking its settings). The feature is actually already in the current beta release and I'm working on a proper release right now :)

https://pypi.org/project/progressbar2/4.3b0/

Not-related topic: I see you use tox. Consider using poetry. I can push a PR, if you allow me.

Tox and poetry serve different purposes though, tox allows for testing on multiple platforms and multiple things in parallel. Poetry is great for project management.

I actually use poetry quite a lot for projects but I found it lacking (the last time I tried) for building and deploying packages. That was about 2 years ago however and poetry has improved quite a bit so it might be sufficient now. The only issue is that I use a single build and deploy script for all the packages I maintain so I would either need to migrate all packages or expand on that script as well.

Regardless, it would be good to have a poetry workflow for 3rd party developers willing to contribute though...

I wrote the python library eule, which I find quite decent to manage. The main commands are on Makefile. Please, take a look and say if this is of your interest.

It was quite a bit of work to hammer out the best way to track cursor position to update only the lines that need updating, If all bars get redrawn over and over again it leads to flickering. Coming up with a cross platform way to track and set the cursor position without having to go to crazy with writing OS specific code was the tricky part. Took several people trying many different things to come up with a way to get it done. @wolph now has the task of taking the code and adding it in a manner that is easy to use and doesn't disrupt too much of the existing code base or cause any API breaking changes. That's the really hard part.

Not breaking backwards compatibility while adding new features is indeed the hard part. This library has a ton of legacy stuff unfortunately, but I think it's worth it so people don't have to deal with breaking code :)

At some point I might have to cull stuff, but for now even code from the original 2006 version of the progressbar should still work without any issues. Almost 20 years of compatibility ;)

All of the code is working fine now, I just need to lint, test and modernize the build system a bit. Hopefully hatch can help me to largely switch to a pyproject.toml file since Github is rather stupid when it comes to parsing a setup.py file so the current version certainly has issues.

pyproject.toml file use is not that hard unlesss you have a complex build system. If you have to compile anything you are going to want to keep the setup.py file if you have anything that requires special treatment. writing a build backend is not what I call a good time. If you think distutils was a pain to deal with build backends are a complete nightmare because of all of the voodoo magic code in it. Hard to debug and doesn't produce helpful traceback information because of it's extensive use of subprocess.