issue/1176/redis-iterate

[midik]

Fix #1176

[codecov[bot]]

Codecov Report

Merging #1245 into master will decrease coverage by 0.23%. The diff coverage is 78.87%.

@@            Coverage Diff             @@
##           master    #1245      +/-   ##
==========================================
- Coverage   69.03%   68.80%   -0.24%     
==========================================
  Files          25       25              
  Lines        2196     2199       +3     
  Branches      426      420       -6     
==========================================
- Hits         1516     1513       -3     
- Misses        506      514       +8     
+ Partials      174      172       -2     

Impacted Files	Coverage Δ
src/S3.ts	75.67% <ø> (ø)
src/util/rewriteUrls.ts	72.80% <0.00%> (ø)
src/mwoffliner.lib.ts	65.20% <50.00%> (ø)
src/util/saveArticles.ts	81.44% <73.91%> (-0.44%)	:arrow_down:
src/MediaWiki.ts	64.92% <75.00%> (ø)
src/Downloader.ts	67.29% <80.00%> (-3.28%)	:arrow_down:
src/Dump.ts	71.73% <100.00%> (-0.79%)	:arrow_down:
src/util/articleRenderers.ts	72.22% <100.00%> (+1.43%)	:arrow_up:
src/util/dump.ts	84.28% <100.00%> (ø)
src/util/misc.ts	73.05% <100.00%> (+1.97%)	:arrow_up:
... and 1 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 580f759...5eed0d1. Read the comment docs.

[midik]

not sure why codecov doesn't count jest coverage (Dowloader.ts / getJsonCb() for example)

[stale[bot]]

This pull request has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[stale[bot]]

This pull request has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[midik]

At least 20% slower with bm.wikipedia.org (with S3 cache)

@kelson42 Just got some benchmarks for bm.wiki.

Running with:

/home/mid/.nvm/versions/node/v12.9.0/bin/node -r ts-node/register /home/mid/_projects/kiwix/mwoffliner/src/cli.ts [email protected] --mwUrl=http://bm.wikipedia.org/ --format=nopic:nopic

On localhost:

 mid@mid-arch  ~  screenfetch -n
 OS: Arch Linux 
 Kernel: x86_64 Linux 5.8.3-zen1-1-zen
 Uptime: 2h 38m
 Packages: 1897
 Shell: fish 3.1.2
 Resolution: 4480x1440
 DE: KDE 5.73.0 / Plasma 5.19.4
 WM: KWin
 GTK Theme: Adwaita-dark [GTK2/3]
 Icon Theme: OieIcons
 Disk: 400G / 464G (87%)
 CPU: Intel Core i7-6600U @ 4x 3.4GHz [57.0°C]
 GPU: Mesa Intel(R) HD Graphics 520 (SKL GT2)
 RAM: 9142MiB / 15739MiB

Connection:

 mid@mid-arch  ~  speedtest-cli
Retrieving speedtest.net configuration...
Testing from Sprint Inet LLC (91.204.228.103)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Sprint-Inet (Novorossiysk) [6.95 km]: 9.603 ms
Testing download speed................................................................................
Download: 21.18 Mbit/s
Testing upload speed......................................................................................................
Upload: 14.00 Mbit/s

Results:

Run on issue/1176/redis-iterate branch: 64s - 1-new.log 55s - 2-new.log 56s - 3-new.log

Run on master branch (bfd2f36131e3091c84ae7e7e732033b1d80d3210): 92s - 4-master.log 86s - 5-master.log 88s - 6-master.log

[midik]

Get stuck with mi.wikipedia.org around 50% of the time (with S3 cache)

@kelson42 Run on master branch (bfd2f36): 767s - mi-1-master.log 711s - mi-2-master.log 703s - mi-3-master.log

Run on issue/1176/redis-iterate branch: 214s - mi-4-new.log 221s - mi-5-new.log 245s - mi-6-new.log

¯\_(ツ)_/¯

[kelson42]

My own runs with:

./src/cli.ts [email protected] --mwUrl=http://bm.wikipedia.org/ --format=nopic:nopic:

master:

• 47.325s
• 1m4.391s
• 50.554s

redis-iterate:

• 44.251s
• 0m46.337s
• 0m47.084s

./src/cli.ts [email protected] --mwUrl=http://mi.wikipedia.org/ --format=nopic:nopic:

master:

• 7m16.326s

redis-iterate:

• Stuck

[log] [2020-08-27T17:53:56.805Z] Articles [6750/7190] [93.9%]
[log] [2020-08-27T17:53:58.720Z] Articles [6775/7190] [94.2%]
[log] [2020-08-27T17:54:00.621Z] Articles [6800/7190] [94.6%]
[log] [2020-08-27T17:54:02.514Z] Articles [6825/7190] [94.9%]
[log] [2020-08-27T17:54:04.795Z] Articles [6850/7190] [95.3%]
[log] [2020-08-27T17:54:06.315Z] Articles [6875/7190] [95.6%]
[log] [2020-08-27T17:54:08.150Z] Articles [6900/7190] [96.0%]
[log] [2020-08-27T17:54:09.779Z] Articles [6925/7190] [96.3%]
[log] [2020-08-27T17:54:11.730Z] Articles [6950/7190] [96.7%]
[log] [2020-08-27T17:54:13.479Z] Articles [6975/7190] [97.0%]
T:488; A:7000; RA:1; CA:6998; UA:1; FA:0; IA:6978; C:22; CC:22; UC:0; WC:0
[log] [2020-08-27T17:54:14.465Z] Articles [7000/7190] [97.4%]
[log] [2020-08-27T17:54:14.919Z] Articles [7025/7190] [97.7%]
[log] [2020-08-27T17:54:16.062Z] Articles [7050/7190] [98.1%]
[log] [2020-08-27T17:54:16.924Z] Articles [7075/7190] [98.4%]
[log] [2020-08-27T17:54:18.420Z] Articles [7100/7190] [98.7%]

:(

[midik]

My own runs with:

Wondering why your timers looks different than mine... anyways we cannot consider redis-iterate 20% slower than master (or even slower at all) :) Most interesting thing is why mi tends to stuck. I gonna make more runs today and also will run something bigger.

Upd: @kelson42 how much cpu's you have? I have two ones (4 hyper threads), probably I could test with the speed to match your one.

[kelson42]

@midik I confirm that today - for a reason I can not explain - the runs without any pictures are successfuly and quick(er) with redis-iterate. We should test as well with pictures/S3.... but the blocker is clearly the fact that it get stuck for me with mi.wikipedia.org.

[stale[bot]]

This pull request has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[ShadowJonathan]

@kelson42 any movement on this?

[kelson42]

@ShadowJonathan Last time I verified this PR it was not working fine (process was stuck if I remember properly) and this PR is orphaned, aiming for a new dev. To finish the work.

[stale[bot]]

This pull request has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[kelson42]

Superseeded by #1714