devito icon indicating copy to clipboard operation
devito copied to clipboard

compiler: Lazy IET visitors + Search

Open enwask opened this issue 7 months ago • 6 comments

This PR implements lazy visitors in the IET layer which operate via generators instead of flattening lists of children's results at every node. Replaces FindSymbols, FindNodes, FindWithin and FindApplications with such versions, as well as introducing type hints for the affected visitors.

From my testing this results in a speedup in IET lowering of up to 50%.

Also reimplements Search similarly, benchmarks forthcoming.

enwask avatar May 30 '25 10:05 enwask

@enwask can you give us one or more command lines to reproduce such a compilation speed improvement? perhaps something using the available examples/seismic/... ? or a script of your own? thanks!

FabioLuporini avatar May 31 '25 14:05 FabioLuporini

also note @enwask that CI is currently failing

FabioLuporini avatar May 31 '25 14:05 FabioLuporini

Yeah will cook up a demo. Did run the test suite yesterday but some of my typing changes for the review must have broken things, will take a look

enwask avatar May 31 '25 14:05 enwask

Profiled with this script on dewback through numactl, getting a less substantial but measurable 5% improvement overall: you can view the cProfile output from both the upstream head and my fork with lazy visitors.

I also separately ran some of the examples through cProfile; this gist has my test script as well as the profiling output across three examples (namely acoustic, elastic and TTI), where I'm seeing a similar 5% improvement in operator construction. Also in Python 3.10 on dewback.

enwask avatar Jun 01 '25 17:06 enwask

Codecov Report

Attention: Patch coverage is 92.85714% with 9 lines in your changes missing coverage. Please review.

Project coverage is 87.60%. Comparing base (77ffae7) to head (c67bb40). Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
devito/symbolics/search.py 82.50% 5 Missing and 2 partials :warning:
devito/ir/iet/visitors.py 97.67% 2 Missing :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2621      +/-   ##
==========================================
- Coverage   92.00%   87.60%   -4.40%     
==========================================
  Files         245      245              
  Lines       48813    48790      -23     
  Branches     4310     4305       -5     
==========================================
- Hits        44908    42744    -2164     
- Misses       3211     5313    +2102     
- Partials      694      733      +39     
Flag Coverage Δ
pytest-gpu-aomp-amdgpuX ?
pytest-gpu-nvc-nvidiaX ?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar Jun 01 '25 21:06 codecov[bot]

Noticed that the Search rewrite is actually a regression; trying to work out why...

Found out why! Turns out bfs_first_hit is neither a BFS nor does it return only the first hit

enwask avatar Jun 09 '25 15:06 enwask