Adding local work requesting scheduler that is based on message passing internally
Open
hkaiser
opened this issue 2 years ago
•
2 comments
This adds a new experimental work-requesting scheduler to the list of existing schedulers
Using uniform_int_distribution with proper bounds
Removing queue index from thread_queues as it was unused
Renaming workstealing --> workrequesting
Adding adaptive work stealing (steal half/steal one)
this makes this scheduler consistently (albeit only slightly) faster than the (default) local-priority scheduler
Adding LIFO and FIFO variations of local work-stealing scheduler
flyby: fixing HPX_WITH_SWAP_CONTEXT_EMULATION
flyby: minor changes to fibonacci_local example
Adding high- and low- priority queues
flyby: cache_line_data now does not generate warnings errors if padding is not needed
First performance measurements show an overall improvement of up to 5-10%. Very promising!
Performance test report HPX Performance Comparison BENCHMARK FORK_JOIN_EXECUTOR PARALLEL_EXECUTOR SCHEDULER_EXECUTOR For Each (=) (=) +
Info Property Before After HPX Commit a68196882e3e093ba44367f6675f538d613c9aa3 cbd27da156c3f90be367a1ac6dc16704f580002f HPX Datetime 2022-02-09T15:45:06+00:00 2022-04-13T22:08:13+00:00 Envfile Hostname nid00729 nid01193 Compiler /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 Clustername daint daint Datetime 2022-02-09T17:03:21.440240+01:00 2022-04-14T00:21:45.647786+02:00
Comparison BENCHMARK NO-EXECUTOR Future Overhead - Create Thread Hierarchical - Latch (=)
Info Property Before After HPX Commit 96a2e4bd1c76207dea8d5302784962c927882406 cbd27da156c3f90be367a1ac6dc16704f580002f HPX Datetime 2021-11-11T08:14:57+00:00 2022-04-13T22:08:13+00:00 Envfile Hostname nid00006 nid01193 Compiler /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 Clustername daint daint Datetime 2021-11-11T09:28:13.071121+01:00 2022-04-14T00:22:02.373105+02:00
Comparison BENCHMARK FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR Stream Benchmark - Add - (=) (=) Stream Benchmark - Scale (=) (=) (=) Stream Benchmark - Triad - (=) (=) Stream Benchmark - Copy (=) (=) (=)
Info Property Before After HPX Commit 71d8dbe40b737be02ea734f9d98fcc0b1e21f01a cbd27da156c3f90be367a1ac6dc16704f580002f HPX Datetime 2021-11-10T19:14:21+00:00 2022-04-13T22:08:13+00:00 Envfile Hostname nid00120 nid01193 Compiler /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 Clustername daint daint Datetime 2021-11-10T20:28:18.266961+01:00 2022-04-14T00:22:17.738365+02:00
Explanation of Symbols Symbol MEANING = No performance change (confidence interval within ±1%) (=) Probably no performance change (confidence interval within ±2%) (+)/(-) Very small performance improvement/degradation (≤1%) +/- Small performance improvement/degradation (≤5%) ++/-- Large performance improvement/degradation (≤10%) +++/--- Very large performance improvement/degradation (>10%) ? Probably no change, but quite large uncertainty (confidence interval with ±5%) ?? Unclear result, very large uncertainty (±10%) ??? Something unexpected…
Performance test report HPX Performance Comparison BENCHMARK FORK_JOIN_EXECUTOR PARALLEL_EXECUTOR SCHEDULER_EXECUTOR For Each (=) ?? (=)
Info Property Before After HPX Commit eb604d5bfa8744d3c9cb3af402809c54e3d8f4ee 8acafff1cbc5bfee4f959786bb01fbdd83613ee2 HPX Datetime 2023-01-12T23:07:58+00:00 2023-01-17T00:53:41+00:00 Clustername rostam rostam Datetime 2023-01-13T04:10:15.951027-06:00 2023-01-16T18:59:20.095675-06:00 Hostname medusa08.rostam.cct.lsu.edu medusa08.rostam.cct.lsu.edu Compiler /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 Envfile
Comparison BENCHMARK NO-EXECUTOR Future Overhead - Create Thread Hierarchical - Latch ??
Info Property Before After HPX Commit eb604d5bfa8744d3c9cb3af402809c54e3d8f4ee 8acafff1cbc5bfee4f959786bb01fbdd83613ee2 HPX Datetime 2023-01-12T23:07:58+00:00 2023-01-17T00:53:41+00:00 Clustername rostam rostam Datetime 2023-01-13T04:10:31.068958-06:00 2023-01-16T18:59:39.241756-06:00 Hostname medusa08.rostam.cct.lsu.edu medusa08.rostam.cct.lsu.edu Compiler /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 Envfile
Comparison BENCHMARK FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR Stream Benchmark - Add (=) --- ++ Stream Benchmark - Scale (=) - = Stream Benchmark - Triad (=) --- ++ Stream Benchmark - Copy + (=) (=)
Info Property Before After HPX Commit eb604d5bfa8744d3c9cb3af402809c54e3d8f4ee 8acafff1cbc5bfee4f959786bb01fbdd83613ee2 HPX Datetime 2023-01-12T23:07:58+00:00 2023-01-17T00:53:41+00:00 Clustername rostam rostam Datetime 2023-01-13T04:10:46.398919-06:00 2023-01-16T18:59:55.057716-06:00 Hostname medusa08.rostam.cct.lsu.edu medusa08.rostam.cct.lsu.edu Compiler /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 Envfile
Explanation of Symbols Symbol MEANING = No performance change (confidence interval within ±1%) (=) Probably no performance change (confidence interval within ±2%) (+)/(-) Very small performance improvement/degradation (≤1%) +/- Small performance improvement/degradation (≤5%) ++/-- Large performance improvement/degradation (≤10%) +++/--- Very large performance improvement/degradation (>10%) ? Probably no change, but quite large uncertainty (confidence interval with ±5%) ?? Unclear result, very large uncertainty (±10%) ??? Something unexpected…
Performance test report HPX Performance Comparison BENCHMARK FORK_JOIN_EXECUTOR PARALLEL_EXECUTOR SCHEDULER_EXECUTOR For Each (=) + (=)
Info Property Before After HPX Datetime 2022-05-31T12:57:29+00:00 2023-01-17T00:53:41+00:00 HPX Commit d5655f89770ba953e8de3ceb349567d5fcb74a95 8acafff1cbc5bfee4f959786bb01fbdd83613ee2 Compiler /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 Hostname nid01193 nid01602 Clustername daint daint Datetime 2022-05-31T15:13:01.357969+02:00 2023-01-17T02:06:30.200230+01:00 Envfile
Comparison BENCHMARK NO-EXECUTOR Future Overhead - Create Thread Hierarchical - Latch --
Info Property Before After HPX Datetime 2022-05-31T12:57:29+00:00 2023-01-17T00:53:41+00:00 HPX Commit d5655f89770ba953e8de3ceb349567d5fcb74a95 8acafff1cbc5bfee4f959786bb01fbdd83613ee2 Compiler /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 Hostname nid01193 nid01602 Clustername daint daint Datetime 2022-05-31T15:13:18.026239+02:00 2023-01-17T02:06:49.550654+01:00 Envfile
Comparison BENCHMARK FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR Stream Benchmark - Add (=) + (=) Stream Benchmark - Scale (=) +++ (=) Stream Benchmark - Triad (=) ++ (=) Stream Benchmark - Copy (=) ++ (=)
Info Property Before After HPX Datetime 2021-11-10T19:14:21+00:00 2023-01-17T00:53:41+00:00 HPX Commit 71d8dbe40b737be02ea734f9d98fcc0b1e21f01a 8acafff1cbc5bfee4f959786bb01fbdd83613ee2 Compiler /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 Hostname nid00120 nid01602 Clustername daint daint Datetime 2021-11-10T20:28:18.266961+01:00 2023-01-17T02:07:04.830876+01:00 Envfile
Explanation of Symbols Symbol MEANING = No performance change (confidence interval within ±1%) (=) Probably no performance change (confidence interval within ±2%) (+)/(-) Very small performance improvement/degradation (≤1%) +/- Small performance improvement/degradation (≤5%) ++/-- Large performance improvement/degradation (≤10%) +++/--- Very large performance improvement/degradation (>10%) ? Probably no change, but quite large uncertainty (confidence interval with ±5%) ?? Unclear result, very large uncertainty (±10%) ??? Something unexpected…
Performance test report HPX Performance Comparison BENCHMARK FORK_JOIN_EXECUTOR PARALLEL_EXECUTOR SCHEDULER_EXECUTOR For Each (=) (=) (=)
Info Property Before After HPX Datetime 2023-01-12T23:07:58+00:00 2023-01-17T23:18:13+00:00 HPX Commit eb604d5bfa8744d3c9cb3af402809c54e3d8f4ee 8759d626bc17c452a49df27c0ccaacd07ff8a2ba Hostname medusa08.rostam.cct.lsu.edu medusa08.rostam.cct.lsu.edu Envfile Datetime 2023-01-13T04:10:15.951027-06:00 2023-01-17T17:24:37.483620-06:00 Compiler /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 Clustername rostam rostam
Comparison BENCHMARK NO-EXECUTOR Future Overhead - Create Thread Hierarchical - Latch ??
Info Property Before After HPX Datetime 2023-01-12T23:07:58+00:00 2023-01-17T23:18:13+00:00 HPX Commit eb604d5bfa8744d3c9cb3af402809c54e3d8f4ee 8759d626bc17c452a49df27c0ccaacd07ff8a2ba Hostname medusa08.rostam.cct.lsu.edu medusa08.rostam.cct.lsu.edu Envfile Datetime 2023-01-13T04:10:31.068958-06:00 2023-01-17T17:24:57.417405-06:00 Compiler /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 Clustername rostam rostam
Comparison BENCHMARK FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR Stream Benchmark - Add --- --- --- Stream Benchmark - Scale - (=) = Stream Benchmark - Triad --- --- --- Stream Benchmark - Copy (=) - ---
Info Property Before After HPX Datetime 2023-01-12T23:07:58+00:00 2023-01-17T23:18:13+00:00 HPX Commit eb604d5bfa8744d3c9cb3af402809c54e3d8f4ee 8759d626bc17c452a49df27c0ccaacd07ff8a2ba Hostname medusa08.rostam.cct.lsu.edu medusa08.rostam.cct.lsu.edu Envfile Datetime 2023-01-13T04:10:46.398919-06:00 2023-01-17T17:25:14.693108-06:00 Compiler /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 Clustername rostam rostam
Explanation of Symbols Symbol MEANING = No performance change (confidence interval within ±1%) (=) Probably no performance change (confidence interval within ±2%) (+)/(-) Very small performance improvement/degradation (≤1%) +/- Small performance improvement/degradation (≤5%) ++/-- Large performance improvement/degradation (≤10%) +++/--- Very large performance improvement/degradation (>10%) ? Probably no change, but quite large uncertainty (confidence interval with ±5%) ?? Unclear result, very large uncertainty (±10%) ??? Something unexpected…
Performance test report HPX Performance Comparison BENCHMARK FORK_JOIN_EXECUTOR PARALLEL_EXECUTOR SCHEDULER_EXECUTOR For Each (=) (=) (=)
Info Property Before After HPX Datetime 2023-01-12T23:07:58+00:00 2023-01-27T22:07:35+00:00 HPX Commit eb604d5bfa8744d3c9cb3af402809c54e3d8f4ee b33de189ec2926318f125473a546d331ad69692f Clustername rostam rostam Envfile Hostname medusa08.rostam.cct.lsu.edu medusa08.rostam.cct.lsu.edu Compiler /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 Datetime 2023-01-13T04:10:15.951027-06:00 2023-01-27T16:15:02.694201-06:00
Comparison BENCHMARK NO-EXECUTOR Future Overhead - Create Thread Hierarchical - Latch ??
Info Property Before After HPX Datetime 2023-01-12T23:07:58+00:00 2023-01-27T22:07:35+00:00 HPX Commit eb604d5bfa8744d3c9cb3af402809c54e3d8f4ee b33de189ec2926318f125473a546d331ad69692f Clustername rostam rostam Envfile Hostname medusa08.rostam.cct.lsu.edu medusa08.rostam.cct.lsu.edu Compiler /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 Datetime 2023-01-13T04:10:31.068958-06:00 2023-01-27T16:15:22.315261-06:00
Comparison BENCHMARK FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR Stream Benchmark - Add - -- (=) Stream Benchmark - Scale (=) (=) (=) Stream Benchmark - Triad (=) -- + Stream Benchmark - Copy + (=) (=)
Info Property Before After HPX Datetime 2023-01-12T23:07:58+00:00 2023-01-27T22:07:35+00:00 HPX Commit eb604d5bfa8744d3c9cb3af402809c54e3d8f4ee b33de189ec2926318f125473a546d331ad69692f Clustername rostam rostam Envfile Hostname medusa08.rostam.cct.lsu.edu medusa08.rostam.cct.lsu.edu Compiler /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 Datetime 2023-01-13T04:10:46.398919-06:00 2023-01-27T16:15:38.071747-06:00
Explanation of Symbols Symbol MEANING = No performance change (confidence interval within ±1%) (=) Probably no performance change (confidence interval within ±2%) (+)/(-) Very small performance improvement/degradation (≤1%) +/- Small performance improvement/degradation (≤5%) ++/-- Large performance improvement/degradation (≤10%) +++/--- Very large performance improvement/degradation (>10%) ? Probably no change, but quite large uncertainty (confidence interval with ±5%) ?? Unclear result, very large uncertainty (±10%) ??? Something unexpected…
Performance test report HPX Performance Comparison BENCHMARK FORK_JOIN_EXECUTOR PARALLEL_EXECUTOR SCHEDULER_EXECUTOR For Each (=) (=) (=)
Info Property Before After HPX Commit d4827232e3d49d63f41287db1e8b5767b6dbd95f b33de189ec2926318f125473a546d331ad69692f HPX Datetime 2023-01-12T14:41:15+00:00 2023-01-27T22:07:35+00:00 Compiler /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 Envfile Clustername daint daint Datetime 2023-01-12T16:07:32.649261+01:00 2023-01-27T23:20:48.276107+01:00 Hostname nid00267 nid01192
Comparison BENCHMARK NO-EXECUTOR Future Overhead - Create Thread Hierarchical - Latch ---
Info Property Before After HPX Commit da01e78993b6bb8da476492b2770f1cc3f117995 b33de189ec2926318f125473a546d331ad69692f HPX Datetime 2023-01-18T08:41:20+00:00 2023-01-27T22:07:35+00:00 Compiler /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 Envfile Clustername daint daint Datetime 2023-01-18T09:57:57.669847+01:00 2023-01-27T23:21:09.050482+01:00 Hostname nid00005 nid01192
Comparison BENCHMARK FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR Stream Benchmark - Add (=) (=) (=) Stream Benchmark - Scale (=) (=) (=) Stream Benchmark - Triad (=) (=) (=) Stream Benchmark - Copy (=) (=) (=)
Info Property Before After HPX Commit 52952d870ab05881c0ef01bde746b95bf5e74ef0 b33de189ec2926318f125473a546d331ad69692f HPX Datetime 2023-01-17T08:25:42+00:00 2023-01-27T22:07:35+00:00 Compiler /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 Envfile Clustername daint daint Datetime 2023-01-17T09:44:28.090718+01:00 2023-01-27T23:21:25.719950+01:00 Hostname nid00005 nid01192
Explanation of Symbols Symbol MEANING = No performance change (confidence interval within ±1%) (=) Probably no performance change (confidence interval within ±2%) (+)/(-) Very small performance improvement/degradation (≤1%) +/- Small performance improvement/degradation (≤5%) ++/-- Large performance improvement/degradation (≤10%) +++/--- Very large performance improvement/degradation (>10%) ? Probably no change, but quite large uncertainty (confidence interval with ±5%) ?? Unclear result, very large uncertainty (±10%) ??? Something unexpected…
Performance test report HPX Performance Comparison BENCHMARK FORK_JOIN_EXECUTOR PARALLEL_EXECUTOR SCHEDULER_EXECUTOR For Each (=) (=) (=)
Info Property Before After HPX Datetime 2023-01-12T14:41:15+00:00 2023-02-09T15:56:58+00:00 HPX Commit d4827232e3d49d63f41287db1e8b5767b6dbd95f 46618c749004d3deb99269d404dd58242b89af6f Hostname nid00267 nid00481 Compiler /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 Envfile Datetime 2023-01-12T16:07:32.649261+01:00 2023-02-09T17:15:55.848757+01:00 Clustername daint daint
Comparison BENCHMARK NO-EXECUTOR Future Overhead - Create Thread Hierarchical - Latch ---
Info Property Before After HPX Datetime 2023-01-18T08:41:20+00:00 2023-02-09T15:56:58+00:00 HPX Commit da01e78993b6bb8da476492b2770f1cc3f117995 46618c749004d3deb99269d404dd58242b89af6f Hostname nid00005 nid00481 Compiler /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 Envfile Datetime 2023-01-18T09:57:57.669847+01:00 2023-02-09T17:16:16.635277+01:00 Clustername daint daint
Comparison BENCHMARK FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR Stream Benchmark - Add + - (=) Stream Benchmark - Scale (=) (=) (=) Stream Benchmark - Triad + - (=) Stream Benchmark - Copy = (=) (=)
Info Property Before After HPX Datetime 2023-01-17T08:25:42+00:00 2023-02-09T15:56:58+00:00 HPX Commit 52952d870ab05881c0ef01bde746b95bf5e74ef0 46618c749004d3deb99269d404dd58242b89af6f Hostname nid00005 nid00481 Compiler /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 Envfile Datetime 2023-01-17T09:44:28.090718+01:00 2023-02-09T17:16:31.551047+01:00 Clustername daint daint
Explanation of Symbols Symbol MEANING = No performance change (confidence interval within ±1%) (=) Probably no performance change (confidence interval within ±2%) (+)/(-) Very small performance improvement/degradation (≤1%) +/- Small performance improvement/degradation (≤5%) ++/-- Large performance improvement/degradation (≤10%) +++/--- Very large performance improvement/degradation (>10%) ? Probably no change, but quite large uncertainty (confidence interval with ±5%) ?? Unclear result, very large uncertainty (±10%) ??? Something unexpected…
Performance test report HPX Performance Comparison BENCHMARK FORK_JOIN_EXECUTOR PARALLEL_EXECUTOR SCHEDULER_EXECUTOR For Each (=) (=) (=)
Info Property Before After HPX Commit eb604d5bfa8744d3c9cb3af402809c54e3d8f4ee 46618c749004d3deb99269d404dd58242b89af6f HPX Datetime 2023-01-12T23:07:58+00:00 2023-02-09T15:56:58+00:00 Hostname medusa08.rostam.cct.lsu.edu medusa08.rostam.cct.lsu.edu Envfile Clustername rostam rostam Compiler /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 Datetime 2023-01-13T04:10:15.951027-06:00 2023-02-09T13:06:52.902401-06:00
Comparison BENCHMARK NO-EXECUTOR Future Overhead - Create Thread Hierarchical - Latch ??
Info Property Before After HPX Commit eb604d5bfa8744d3c9cb3af402809c54e3d8f4ee 46618c749004d3deb99269d404dd58242b89af6f HPX Datetime 2023-01-12T23:07:58+00:00 2023-02-09T15:56:58+00:00 Hostname medusa08.rostam.cct.lsu.edu medusa08.rostam.cct.lsu.edu Envfile Clustername rostam rostam Compiler /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 Datetime 2023-01-13T04:10:31.068958-06:00 2023-02-09T13:07:13.896342-06:00
Comparison BENCHMARK FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR Stream Benchmark - Add -- --- ++ Stream Benchmark - Scale (=) -- -- Stream Benchmark - Triad -- --- ++ Stream Benchmark - Copy + - --
Info Property Before After HPX Commit eb604d5bfa8744d3c9cb3af402809c54e3d8f4ee 46618c749004d3deb99269d404dd58242b89af6f HPX Datetime 2023-01-12T23:07:58+00:00 2023-02-09T15:56:58+00:00 Hostname medusa08.rostam.cct.lsu.edu medusa08.rostam.cct.lsu.edu Envfile Clustername rostam rostam Compiler /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 Datetime 2023-01-13T04:10:46.398919-06:00 2023-02-09T13:07:29.646323-06:00
Explanation of Symbols Symbol MEANING = No performance change (confidence interval within ±1%) (=) Probably no performance change (confidence interval within ±2%) (+)/(-) Very small performance improvement/degradation (≤1%) +/- Small performance improvement/degradation (≤5%) ++/-- Large performance improvement/degradation (≤10%) +++/--- Very large performance improvement/degradation (>10%) ? Probably no change, but quite large uncertainty (confidence interval with ±5%) ?? Unclear result, very large uncertainty (±10%) ??? Something unexpected…
Performance test report HPX Performance Comparison BENCHMARK FORK_JOIN_EXECUTOR PARALLEL_EXECUTOR SCHEDULER_EXECUTOR For Each (=) (=) (=)
Info Property Before After HPX Datetime 2023-01-12T23:07:58+00:00 2023-02-09T21:24:49+00:00 HPX Commit eb604d5bfa8744d3c9cb3af402809c54e3d8f4ee 94d6c864b7f76f61659af2e267cc995c367c9ccf Compiler /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 Envfile Datetime 2023-01-13T04:10:15.951027-06:00 2023-02-09T15:29:19.082860-06:00 Clustername rostam rostam Hostname medusa08.rostam.cct.lsu.edu medusa08.rostam.cct.lsu.edu
Comparison BENCHMARK NO-EXECUTOR Future Overhead - Create Thread Hierarchical - Latch ??
Info Property Before After HPX Datetime 2023-01-12T23:07:58+00:00 2023-02-09T21:24:49+00:00 HPX Commit eb604d5bfa8744d3c9cb3af402809c54e3d8f4ee 94d6c864b7f76f61659af2e267cc995c367c9ccf Compiler /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 Envfile Datetime 2023-01-13T04:10:31.068958-06:00 2023-02-09T15:29:39.837645-06:00 Clustername rostam rostam Hostname medusa08.rostam.cct.lsu.edu medusa08.rostam.cct.lsu.edu
Comparison BENCHMARK FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR Stream Benchmark - Add (=) - +++ Stream Benchmark - Scale (=) -- -- Stream Benchmark - Triad (=) - +++ Stream Benchmark - Copy + - --
Info Property Before After HPX Datetime 2023-01-12T23:07:58+00:00 2023-02-09T21:24:49+00:00 HPX Commit eb604d5bfa8744d3c9cb3af402809c54e3d8f4ee 94d6c864b7f76f61659af2e267cc995c367c9ccf Compiler /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 Envfile Datetime 2023-01-13T04:10:46.398919-06:00 2023-02-09T15:29:54.427582-06:00 Clustername rostam rostam Hostname medusa08.rostam.cct.lsu.edu medusa08.rostam.cct.lsu.edu
Explanation of Symbols Symbol MEANING = No performance change (confidence interval within ±1%) (=) Probably no performance change (confidence interval within ±2%) (+)/(-) Very small performance improvement/degradation (≤1%) +/- Small performance improvement/degradation (≤5%) ++/-- Large performance improvement/degradation (≤10%) +++/--- Very large performance improvement/degradation (>10%) ? Probably no change, but quite large uncertainty (confidence interval with ±5%) ?? Unclear result, very large uncertainty (±10%) ??? Something unexpected…
Performance test report HPX Performance Comparison BENCHMARK FORK_JOIN_EXECUTOR PARALLEL_EXECUTOR SCHEDULER_EXECUTOR For Each (=) (=) (=)
Info Property Before After HPX Commit d4827232e3d49d63f41287db1e8b5767b6dbd95f 94d6c864b7f76f61659af2e267cc995c367c9ccf HPX Datetime 2023-01-12T14:41:15+00:00 2023-02-09T21:24:49+00:00 Hostname nid00267 nid00979 Compiler /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 Datetime 2023-01-12T16:07:32.649261+01:00 2023-02-09T22:36:08.679360+01:00 Envfile Clustername daint daint
Comparison BENCHMARK NO-EXECUTOR Future Overhead - Create Thread Hierarchical - Latch ---
Info Property Before After HPX Commit da01e78993b6bb8da476492b2770f1cc3f117995 94d6c864b7f76f61659af2e267cc995c367c9ccf HPX Datetime 2023-01-18T08:41:20+00:00 2023-02-09T21:24:49+00:00 Hostname nid00005 nid00979 Compiler /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 Datetime 2023-01-18T09:57:57.669847+01:00 2023-02-09T22:36:28.435260+01:00 Envfile Clustername daint daint
Comparison BENCHMARK FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR Stream Benchmark - Add + (=) (=) Stream Benchmark - Scale (=) (=) (=) Stream Benchmark - Triad + (=) (=) Stream Benchmark - Copy (=) (=) (=)
Info Property Before After HPX Commit 52952d870ab05881c0ef01bde746b95bf5e74ef0 94d6c864b7f76f61659af2e267cc995c367c9ccf HPX Datetime 2023-01-17T08:25:42+00:00 2023-02-09T21:24:49+00:00 Hostname nid00005 nid00979 Compiler /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 /apps/daint/SSL/HPX/packages/llvm-11.0.0/bin/clang++ 11.0.0 Datetime 2023-01-17T09:44:28.090718+01:00 2023-02-09T22:36:43.289145+01:00 Envfile Clustername daint daint
Explanation of Symbols Symbol MEANING = No performance change (confidence interval within ±1%) (=) Probably no performance change (confidence interval within ±2%) (+)/(-) Very small performance improvement/degradation (≤1%) +/- Small performance improvement/degradation (≤5%) ++/-- Large performance improvement/degradation (≤10%) +++/--- Very large performance improvement/degradation (>10%) ? Probably no change, but quite large uncertainty (confidence interval with ±5%) ?? Unclear result, very large uncertainty (±10%) ??? Something unexpected…
Performance test report HPX Performance Comparison BENCHMARK FORK_JOIN_EXECUTOR PARALLEL_EXECUTOR SCHEDULER_EXECUTOR For Each (=) ? (=)
Info Property Before After HPX Datetime 2023-03-06T15:59:25+00:00 2023-05-05T19:55:05+00:00 HPX Commit 6b6e1e71343e80ad04bf610e41c26a41c1f37333 e9b45feb2a7ae28d0cd034d6b86e777f95bcc4ce Datetime 2023-03-10T03:27:49.135034-06:00 2023-05-05T15:36:03.843573-05:00 Compiler /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 Hostname medusa08.rostam.cct.lsu.edu medusa08.rostam.cct.lsu.edu Envfile Clustername rostam rostam
Comparison BENCHMARK NO-EXECUTOR Future Overhead - Create Thread Hierarchical - Latch -
Info Property Before After HPX Datetime 2023-03-06T15:59:25+00:00 2023-05-05T19:55:05+00:00 HPX Commit 6b6e1e71343e80ad04bf610e41c26a41c1f37333 e9b45feb2a7ae28d0cd034d6b86e777f95bcc4ce Datetime 2023-03-10T03:28:21.991297-06:00 2023-05-05T15:36:37.293877-05:00 Compiler /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 Hostname medusa08.rostam.cct.lsu.edu medusa08.rostam.cct.lsu.edu Envfile Clustername rostam rostam
Comparison BENCHMARK FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR Stream Benchmark - Add (=) (=) (=) Stream Benchmark - Scale (=) (=) (=) Stream Benchmark - Triad (=) (=) (=) Stream Benchmark - Copy (=) (=) (=)
Info Property Before After HPX Datetime 2023-03-06T15:59:25+00:00 2023-05-05T19:55:05+00:00 HPX Commit 6b6e1e71343e80ad04bf610e41c26a41c1f37333 e9b45feb2a7ae28d0cd034d6b86e777f95bcc4ce Datetime 2023-03-10T03:28:29.145749-06:00 2023-05-05T15:36:44.404357-05:00 Compiler /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 Hostname medusa08.rostam.cct.lsu.edu medusa08.rostam.cct.lsu.edu Envfile Clustername rostam rostam
Explanation of Symbols Symbol MEANING = No performance change (confidence interval within ±1%) (=) Probably no performance change (confidence interval within ±2%) (+)/(-) Very small performance improvement/degradation (≤1%) +/- Small performance improvement/degradation (≤5%) ++/-- Large performance improvement/degradation (≤10%) +++/--- Very large performance improvement/degradation (>10%) ? Probably no change, but quite large uncertainty (confidence interval with ±5%) ?? Unclear result, very large uncertainty (±10%) ??? Something unexpected…
@hkaiser
Sorry I deleted your feedback with force-push, I remember it though :P
I'll let the tests run without the additional logging I had in the previous push, let's see what we get
@Pansysk75 could you please fix the clang-format issues as well?
I started looking at the performance of this scheduler a bit (on the single-NUMA-domain case)
Directly replacing our current scheduler, it performs worse in the general case (at least on the algorithm benchmarks, which I had handy),
I also tried using the do_not_combine_tasks
hint to bypass the index queues we use on bulk_execute, because that is technically another form of stealing, which would skew the comparison. It is still a bit worse.
In the case of a bulk-execution of relatively uniform work, stealing is very limited, because our scheduling is already quite balanced anyways. So this would explain the performance deficit, as we are possibly taking on a larger overhead (more complex scheduler) for small benefit (a few non-cache-disrupting steals).
And then, there is the fact that we are only able to poll the steal requests in between thread execution, which introduces some latency in responding to steal requests.
I will try to produce a best-case scenario for this scheduler though, even if it is just as a proof of concept.
Edit: it seems to perform much better on few cores, which could suggest a large number of failed stealing attempts when we have many cores. We can experiment with sending the steal request where there is actual work.
I started looking at the performance of this scheduler a bit (on the single-NUMA-domain case)
Directly replacing our current scheduler, it performs worse in the general case (at least on the algorithm benchmarks, which I had handy), I also tried using the do_not_combine_tasks
hint to bypass the index queues we use on bulk_execute, because that is technically another form of stealing, which would skew the comparison. It is still a bit worse.
In the case of a bulk-execution of relatively uniform work, stealing is very limited, because our scheduling is already quite balanced anyways. So this would explain the performance deficit, as we are possibly taking on a larger overhead (more complex scheduler) for small benefit (a few non-cache-disrupting steals).
And then, there is the fact that we are only able to poll the steal requests in between thread execution, which introduces some latency in responding to steal requests.
I will try to produce a best-case scenario for this scheduler though, even if it is just as a proof of concept.
Edit: it seems to perform much better on few cores, which could suggest a large number of failed stealing attempts when we have many cores. We can experiment with sending the steal request where there is actual work.
I did see improvements on tests that ran a large amount of separate tasks (like fibonacci). For uniform iterative parallelism the benefit would probably be small.
@hkaiser You'll still have to add the fix in 1d_stencil_8
https://github.com/STEllAR-GROUP/hpx/commit/451bbd02756155e94f2ca04cdfe3fb3695dc6043#diff-126ef56cea5c727b92b9ec59b26d8c429db7f6655405270d63adda430a4a5091
The sem->wait() is not guaranteed to block, so the lambda ends up with a reference to a deleted object.
@hkaiser You'll still have to add the fix in 1d_stencil_8 451bbd0#diff-126ef56cea5c727b92b9ec59b26d8c429db7f6655405270d63adda430a4a5091 The sem->wait() is not guaranteed to block, so the lambda ends up with a reference to a deleted object.
I thought that was fixed by: https://github.com/STEllAR-GROUP/hpx/pull/6294
I thought that was fixed by: #6294
@hkaiser No, I think it cannot be fixed that way (that's why I had gotten a bit confused there). It's a plain old out-of-scope situation.
Just think of what happens if the semaphore doesn't need to wait for the sem.signal() (which will often happen, if the semaphore is not "saturated").
The created lambda (probably not executed yet) has a reference to the semaphore, which gets deleted if we go out of scope.
There is no way for the semaphore to know it is being referenced this way.
I think that makes sense, unless I have some misconception about the role of sliding_semaphore
@Pansysk75 I have applied the change to the use of the sliding_semaphore
everywhere. This however doesn't seem to be sufficient to make things work: https://app.circleci.com/pipelines/github/STEllAR-GROUP/hpx/15639/workflows/eb35fe70-6509-4d54-80a1-3a4454e93d06/jobs/360130
@Pansysk75 I have applied the change to the use of the sliding_semaphore
everywhere. This however doesn't seem to be sufficient to make things work: https://app.circleci.com/pipelines/github/STEllAR-GROUP/hpx/15639/workflows/eb35fe70-6509-4d54-80a1-3a4454e93d06/jobs/360130
Seems like tasks get stuck in the stealing queue, re-applying this fix solves the issue:
https://github.com/STEllAR-GROUP/hpx/commit/451bbd02756155e94f2ca04cdfe3fb3695dc6043
Did you do something else to try solve that issue, or was this fix accidentally left behind?
@Pansysk75 I have applied the change to the use of the sliding_semaphore
everywhere. This however doesn't seem to be sufficient to make things work: https://app.circleci.com/pipelines/github/STEllAR-GROUP/hpx/15639/workflows/eb35fe70-6509-4d54-80a1-3a4454e93d06/jobs/360130
Seems like tasks get stuck in the stealing queue, re-applying this fix solves the issue: 451bbd0
Did you do something else to try solve that issue, or was this fix accidentally left behind?
Thanks a lot - not sure how that got lost. Much appreciated!
Performance test report HPX Performance Comparison BENCHMARK FORK_JOIN_EXECUTOR PARALLEL_EXECUTOR SCHEDULER_EXECUTOR For Each (=) ?? (=)
Info Property Before After HPX Datetime 2023-05-10T12:07:53+00:00 2023-08-28T15:47:36+00:00 HPX Commit dcb541576898d370113946ba15fb58c20c8325b2 de95b0f4edeccc281067e609f5c8b6cb3492816d Compiler /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 Datetime 2023-05-10T14:50:18.616050-05:00 2023-08-28T12:31:56.660155-05:00 Clustername rostam rostam Envfile Hostname medusa08.rostam.cct.lsu.edu medusa08.rostam.cct.lsu.edu
Comparison BENCHMARK NO-EXECUTOR Future Overhead - Create Thread Hierarchical - Latch (=)
Info Property Before After HPX Datetime 2023-05-10T12:07:53+00:00 2023-08-28T15:47:36+00:00 HPX Commit dcb541576898d370113946ba15fb58c20c8325b2 de95b0f4edeccc281067e609f5c8b6cb3492816d Compiler /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 Datetime 2023-05-10T14:52:35.047119-05:00 2023-08-28T12:34:09.253604-05:00 Clustername rostam rostam Envfile Hostname medusa08.rostam.cct.lsu.edu medusa08.rostam.cct.lsu.edu
Comparison BENCHMARK FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR Stream Benchmark - Add (=) (=) (=) Stream Benchmark - Scale (=) (=) = Stream Benchmark - Triad (=) = (=) Stream Benchmark - Copy (=) - (=)
Info Property Before After HPX Datetime 2023-05-10T12:07:53+00:00 2023-08-28T15:47:36+00:00 HPX Commit dcb541576898d370113946ba15fb58c20c8325b2 de95b0f4edeccc281067e609f5c8b6cb3492816d Compiler /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 /opt/apps/llvm/13.0.1/bin/clang++ 13.0.1 Datetime 2023-05-10T14:52:52.237641-05:00 2023-08-28T12:34:26.236878-05:00 Clustername rostam rostam Envfile Hostname medusa08.rostam.cct.lsu.edu medusa08.rostam.cct.lsu.edu
Explanation of Symbols Symbol MEANING = No performance change (confidence interval within ±1%) (=) Probably no performance change (confidence interval within ±2%) (+)/(-) Very small performance improvement/degradation (≤1%) +/- Small performance improvement/degradation (≤5%) ++/-- Large performance improvement/degradation (≤10%) +++/--- Very large performance improvement/degradation (>10%) ? Probably no change, but quite large uncertainty (confidence interval with ±5%) ?? Unclear result, very large uncertainty (±10%) ??? Something unexpected…
I think this is good to go now. Thanks again @Pansysk75!
Build succeeded!
The publicly hosted instance of bors-ng is deprecated and will go away soon.
If you want to self-host your own instance, instructions are here .
For more help, visit the forum .
If you want to switch to GitHub's built-in merge queue, visit their help page .