powerline-shell
powerline-shell copied to clipboard
b-ryan
is this causing my terminal to lag?
anybody else experience the lagging (on terminal) or is it just my old (machine) lady?
Python's start-up time is definitely noticeable, yes. Unfortunately there's no easy way of solving that without rewriting everything in a different language.
That is not entirely true. You can get around the python interpreter startup delay by writing a powershell-daemon that runs in the background and a slim client that gets the shell prompt from that daemon. +jkehne and I have implemented that a while ago and have been using it ever since. At first the license of powerline-shell was not clear which is why I had not originally put the project on github. I have just pushed our code some minutes ago, to
https://github.com/konradmiller/powerline-shell-proxy
One of the remaining problems is, that python is actually quite slow when doing popen calls. Using the git segment this is particularly daunting, as git is called via popen at every invocation only to notice that there is no .git directory in the path. When you check for the cases where you actually are not within a git directory that already speeds up powerline-shell quite a bit, I have implemented this behavior here:
https://github.com/konradmiller/powerline-shell-proxy/blob/master/segments/git.py lines 43ff
You can however not easily get around doing popen in general. This is why I have started porting powerline-shell to c++ last weekend.
https://github.com/konradmiller/powerline-shell-cpp
Feel free to contribute segments to that repository.
Cheers, Konrad
I'm not 100% convinced that the slowdown is due to running Python, but you're C++ fork looks really cool!
I think you're absolutely correct when you identified that Python's popen() calls are unbearably slow when calling Git, and I created a fix by adding a fast-fail path to git.py, located here:
https://github.com/blag/powerline-shell/commit/fad733e7ce42b92d370b68d55f41b220486bb7d9
And here's the PR: #176.
@konradmiller A bigger problem with the performance is that as far as I can see, the app is entirely single-threaded and synchronous, which means that the prompt-script spends a whole lot of time doing exactly nothing. For instance, take this code from the git-segment:
output = subprocess.Popen(['git', 'status', '--ignore-submodules'],
stdout=subprocess.PIPE, env=git_env).communicate()[0]
On my current work project, this command takes about 80 ms. That's 80 milliseconds that the script is doing absolutely nothing, because this is a blocking call. It could easily use that time to figure out what the rest of the prompt should be, if this was using asynchronous I/O or multi-threading.
Python's startup time varies a lot, but it's often in the neighborhood of 40-60 milliseconds in my experience. That's not nothing, it's 2-4 frames on a 60 Hz monitor, but it's not massive either. It contributes about half as much to the lag as the fact that git call is blocking. The performance problems of this script has very little to do with Python start-up time or Popen overhead, and has very much to do with the fact that it's a single threaded script running all its I/O on Main.
It's true that by going to C++, that start-up time would be eliminated. But by making the "git" command asynchronous, you would save double that amount of time and still have it be in Python. Add the svn, hg and fossil segments, as well as all the other segments in the default configuration, it adds up. None of the computation in this program is arduous enough that making it in C++ would make any noticeable difference.
Since no segment depends on any other, there's absolutely no reason why they shouldn't execute in parallel. One might argue thread-spinup and spin-down would be a little bit of a problem, but not that much, and it's completely solved if you use a server-client model. If you want to improve the performance of this project, rewriting it in C++ is a whole lot of trouble for very little gain, when there's an obvious structural problem with the code that's much easier to fix.
@OskarSigvardsson what about the people like me who don't use any of those features, just cwd, time, jobs etc?
@PwnArt1st "time" and "cwd" should take no time at all, "jobs" might be a pinch longer because it launches "ps" as a subprocess twice (and thus has that same blocking behavior i described before). However, all of these are either reading environment variables or querying the OS, there's no blocking I/O there, which is the real problem.
Even if you don't use other, more expensive features, remember to explicitly remove them from your configuration file and rebuild it. If your command line still lags after that, you can legitimately blame it on Python spin-up time.
@PwnArt1st I have no idea, I found this project two weeks ago, and installed it, but didn't like how internally they generated the segments. Which is why I made that comment.
@OskarSigvardsson has made points that I agree with. My plan for some time has been to run the subprocesses in parallel. Unfortunately the code is not super straightforward for making that happen. One of the biggest issues I have is that working with the segments is difficult due to the generation of the powerline-shell.py script instead of importing modules.
For some time I have been interested in importing modules rather than generating one big script. The biggest concern there is possible performance problems, but I don't yet know how big they would be.
To speak to one point specifically,
One might argue thread-spinup and spin-down would be a little bit of a problem, but not that much, and it's completely solved if you use a server-client mode
I think we can easily avoid thread problems by having all of the segments launch any subprocesses at one time. Ie. rendering would look like:
- Loop through segments, call
segment.launch_subprocesses()- Each segment uses
subprocess.Popento launch the process and not block for the results
- Each segment uses
- Loop through segments again, calling
segment.render()which now blocks until its previously-launched subprocess is complete.
Would there be any issues doing this?
@b-ryan I doubt that "precompiling" the file saves much, the import cost is relatively small (especially since imported modules saves the generated byte-code in .pyc files, which should improve spin-up a little bit), so I don't know if that's so crucial. It's nice to a have a "self-contained" generated file that generates the prompt, but that's maybe not worth so much either.
Your idea for structure sounds good, but it's problematic for segments like "jobs" or "hg", which launches subprocesses several times.
If I were doing this, I might consider one of these two solutions:
- Change the dependency from Python 2.7 to Python 3.5, and use the asyncio module combined with the new fancy async/await keywords. That would keep the structure simple for both "light" segments (like "ssh", which only has to read an environment variable) and "heavy" ones (which depend heavily on I/O, like "git" or "hg"). This would be my preferred solution, but porting the project to Python 3 might be too much of a challenge, and you might not want to require your users to install it).
- Use threads. There's a few different ways of doing this, but one version would be something like this (sort-of pseudo-cody):
threads = []
rendered = Queue()
# start a thread for each segment
for segment in segments:
t = Thread(target=segment, args=[rendered])
t.start()
threads.append(t)
# block until they're all done
for t in threads:
t.join()
# ... build final string from the queue ...
That is, rendered is a queue that contains all the segments. I imagine each thread would pop a tuple onto the queue with the segment name and rendered segment contents on it (so, like, the jobs segment function would pop ("jobs", rendered_jobs_string)onto it), and when building the final string, it would read off the queue, put them in order (remember, there will be no guarantee that they're in the right order since you don't know which thread finished first) by reading the first element of the tuple, and then join all the strings together.
(I'm using a Queue() here instead of something like a list because the Queue class is guaranteed to be thread-safe. A list or dictionary might also be because of the Global Interpreter Lock, but I'm not sure about that, and it might differ between implementations.)
This way, the structure will be kept relatively simple (all segments can remain pretty much as they are just instead of returning the value, they pop it onto the queue) while still making sure that everything happens in parallel. The individual segment functions can use blocking I/O all they want, all that means is that the rest of the segments get their turn to render their strings.
This approach has a downside, which is that several of the segments don't need to be in separate threads. The ssh one, for instance, just reads an environment variable and pops out a string, adding that in a separate thread will just incur overhead. The overhead would probably be insignificant though, and it can be easily solved by using two loops: one which starts the threads for the "heavy" segments that need them, and then one after that that generates the "light" segments directly without using threads (that would be before the t.join() loop in my pseudo-code) and pops them onto the queue.
That's how I would do it, in any case. Seems like a fairly straight-forward way of doing it, and one which would save a lot of time.
Thanks a lot for the input. You made a good point about the multiple subprocess calls.
I am definitely hesitant to require version 3.5 of Python. So far we have tried to support any version of Python. I am interested to do some performance testing using the threading model you described and see how it does.
Personally, Ubuntu and Fedora are trying to switch to Python 3 by default, so I don't think it's too farfetched to require Python 3.5 so we can use async/await. Additionally, it makes plugins much easier to write if we only have to support one version of Python.
I think people who don't have Python 3 installed as their default Python can run this in a Python 3.5virtualenv, so it might be smart to make it easy to fire this up in a virtualenv.
Also, I'm a huge fan of dynamically importing segments - it makes configuration a lot easier.
I ended up rewriting this in golang, actually, with most segments supported.
It's available over here: https://github.com/justjanne/powerline-go
Sorry to resurrect a dead thread, but I just did a git pull and updated my config, and I'm noticing significant slowdowns since the last time. At commit 079cbc6 I'm getting times of ~0.09sec, whereas at commit 2f58402 the time has gone up by 50% to ~0.16sec. Any idea what's changed that would slow things down? I figured #280 would make things faster, not slower...
@kc9jud no worries - thanks for bringing it up. I will investigate. The first thing that comes to mind is that maybe the os.path.realpath command is slow. I will have to investigate / maybe profile.
+1.
My terminal is slower than the usual. I use alacritty as my terminal, so the effect is not so noticeable, but when I've to use the native terminal, I can't ignore the slowdown
A while back I also had a slow terminal. Please see my ticket:
https://github.com/b-ryan/powerline-shell/issues/331
Run the following to see if its the same issue I had:
env | grep PROMPT_COMMAND
Im getting slowdowns very often. Even going back to home directory using cd can take several seconds
