pyp3 icon indicating copy to clipboard operation
pyp3 copied to clipboard

Evaluate pyp_dev_1.3 and flatten speed patch

Open bobpaul opened this issue 8 years ago • 2 comments

https://code.google.com/archive/p/pyp/issues/29 has a patch that speeds up flatten_list() dramatically.

Is codebase originally based on the pyp_dev code (I found it posted in 2 different bug reports)? Regardless, the flatten_list() in this repo looks different than the proposed patch, so it would be good to take a look I think.

bobpaul avatar Jul 12 '17 23:07 bobpaul

In issue 25 there's pyp_dev, upon which pyp3 was based. pyp_dev_1.3 from issue 29 is definitely newer. I've merged the changes in and pushed them to branch pyp2.3_dev.

One thing I am unsure of: Previously, PowerPyp.flatten_list() simply wrapped PowerPyp.Pyp.flatten_list(). Now PowerPyp.flatten_list() has the newer, faster version. Is there a reason Pyp.flatten_list() still uses the slower version? As far as I can tell, the only place PowerPipe.flatten_list() is called is inside PowerPipe.oneline(). Everywhere else explicitly calls into Pyp.flatten_list()

If this is a a special case function that can only be used in one spot, it probably should have a different name

bobpaul avatar Jul 13 '17 05:07 bobpaul

Following the bug report in issue #29, I created the following test:

$ for ((i=0; $i<500000; i=$i+1)); do echo $i >> 500000.txt; done
$ time wc 500000.txt
 500000  500000 3388890

real	0m0.089s
user	0m0.092s
sys	0m0.003s
$ time cat 500000.txt | python2 pyp_dev_1.3 "len(pp)"
500000

real	0m9.611s
user	0m9.374s
sys	0m0.246s
$ time cat 500000.txt | python3 ./pyp3 "len(pp)" #from pyp2.3_dev branch
500000

real	0m7.989s
user	0m7.815s
sys	0m0.180s
$ time cat 500000.txt | python3 ./pyp3 "len(pp)" #from master
500000

real	0m8.128s
user	0m7.936s
sys	0m0.200s

idk. I'm not really seeing a speed difference as is. But then maybe it's not being used.

bobpaul avatar Jul 13 '17 05:07 bobpaul