pyhf
pyhf copied to clipboard
scikit-hep
feat: Toy Calculator executor support
Description
Lorem ipsum dolor sit amet. See #807.
Checklist Before Requesting Reviewer
- [ ] Tests are passing
- [ ] "WIP" removed from the title of the pull request
- [ ] Selected an Assignee for the PR to be responsible for the log summary
Before Merging
For the PR Assignees:
- [ ] Summarize commit messages into a comprehensive review of the PR
Codecov Report
Merging #1158 (b88b5cf) into master (0a8f328) will decrease coverage by
0.26%. The diff coverage is62.50%.
@@ Coverage Diff @@
## master #1158 +/- ##
==========================================
- Coverage 98.16% 97.90% -0.27%
==========================================
Files 68 69 +1
Lines 4315 4339 +24
Branches 725 729 +4
==========================================
+ Hits 4236 4248 +12
- Misses 46 57 +11
- Partials 33 34 +1
| Flag | Coverage Δ | |
|---|---|---|
| contrib | 26.41% <15.62%> (-0.04%) |
:arrow_down: |
| doctest | 60.72% <62.50%> (-0.06%) |
:arrow_down: |
| unittests-3.10 | 95.78% <62.50%> (-0.26%) |
:arrow_down: |
| unittests-3.7 | 95.76% <62.50%> (-0.26%) |
:arrow_down: |
Flags with carried forward coverage won't be shown. Click here to find out more.
| Impacted Files | Coverage Δ | |
|---|---|---|
| src/pyhf/infer/calculators.py | 95.51% <58.82%> (-4.49%) |
:arrow_down: |
| src/pyhf/futures.py | 66.66% <66.66%> (ø) |
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact),ø = not affected,? = missing dataPowered by Codecov. Last update 0a8f328...b88b5cf. Read the comment docs.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
script I used to quickly test functionality
import pyhf
import pyhf.futures
import concurrent
import time
model = pyhf.simplemodels.hepdata_like([5], [10], [3.5])
data = [12.5] + model.config.auxdata
ntoys = 500
if __name__ == '__main__':
for executor in [pyhf.futures.TrivialExecutor(), concurrent.futures.ProcessPoolExecutor(), concurrent.futures.ThreadPoolExecutor()]:
start = time.time()
result = pyhf.infer.hypotest(1.0, data, model, qtilde=True, calctype='toybased', ntoys=ntoys, executor=executor)
print(f'Executor = {executor}')
print(f'CLs_obs = {result}')
print(f'Took {time.time() - start} seconds for {ntoys} toys.')
executor.shutdown()
which results in (numpy)
$ python toys.py
Executor = <pyhf.futures.TrivialExecutor object at 0x12ccc4dc0>
CLs_obs = 0.43142857142857144
Took 8.418382167816162 seconds for 500 toys.
Executor = <concurrent.futures.process.ProcessPoolExecutor object at 0x12ccc4eb0>
CLs_obs = 0.4542936288088643
Took 2.402064085006714 seconds for 500 toys.
Executor = <concurrent.futures.thread.ThreadPoolExecutor object at 0x12cd67370>
CLs_obs = 0.37700534759358284
Took 11.590389966964722 seconds for 500 toys.
and for 5k toys (numpy)
$ python toys.py
Executor = <pyhf.futures.TrivialExecutor object at 0x138c9ce80>
CLs_obs = 0.4236186348862406
Took 92.0139479637146 seconds for 5000 toys.
Executor = <concurrent.futures.process.ProcessPoolExecutor object at 0x138c9cf70>
CLs_obs = 0.4380434782608696
Took 24.271136045455933 seconds for 5000 toys.
Executor = <concurrent.futures.thread.ThreadPoolExecutor object at 0x138d02430>
CLs_obs = 0.4217391304347826
Took 130.05941200256348 seconds for 5000 toys.
and 5k toys (jax)
$ python toys.py
Executor = <pyhf.futures.TrivialExecutor object at 0x14311fd60>
CLs_obs = 0.44140945096968043
Took 81.22803473472595 seconds for 5000 toys.
Executor = <concurrent.futures.process.ProcessPoolExecutor object at 0x14311fe80>
CLs_obs = 0.44008774335069917
Took 547.9745662212372 seconds for 5000 toys.
Executor = <concurrent.futures.thread.ThreadPoolExecutor object at 0x14312a640>
CLs_obs = 0.4326608505997819
Took 93.99376606941223 seconds for 5000 toys.
and 5k toys (torch)
Executor = <pyhf.futures.TrivialExecutor object at 0x13f83f340>
CLs_obs = 0.4310019016265869
Took 182.5893669128418 seconds for 5000 toys.
Executor = <concurrent.futures.process.ProcessPoolExecutor object at 0x13f858be0>
CLs_obs = 0.4109438955783844
Took 37.67385792732239 seconds for 5000 toys.
Executor = <concurrent.futures.thread.ThreadPoolExecutor object at 0x13f874c70>
CLs_obs = 0.4270491600036621
Took 158.8731780052185 seconds for 5000 toys.
and 5k toys (tensorflow)
Executor = <pyhf.futures.TrivialExecutor object at 0x14f4e64c0>
CLs_obs = 0.42907899618148804
Took 901.4008986949921 seconds for 5000 toys.
Executor = <concurrent.futures.process.ProcessPoolExecutor object at 0x14f4e66d0>
CLs_obs = 0.4172525703907013
Took 177.639319896698 seconds for 5000 toys.
Executor = <concurrent.futures.thread.ThreadPoolExecutor object at 0x14f4e6bb0>
CLs_obs = 0.4322930872440338
Took 1238.2977120876312 seconds for 5000 toys.
This pull request introduces 1 alert when merging 6a8f72e660ecd47b1a28c13e2e22f785e251c13e into 81c9adb7abafa3a1e79c9b6fe75bc77095d01eaa - view on LGTM.com
new alerts:
- 1 for Except block handles 'BaseException'
![lgtm-com[bot] avatar](https://avatars.githubusercontent.com/in/17324?v=4 "Published by a pub.dev verified publisher"){kind=small}
using this to test functionality
import pyhf
import pyhf.futures
import concurrent
import time
ntoys = 500
if __name__ == '__main__':
for optimizer in ['scipy', 'minuit']:
for backend in ['numpy', 'jax']:
pyhf.set_backend(backend, optimizer)
model = pyhf.simplemodels.uncorrelated_background(
[5],
[10],
[3.5],
)
data = pyhf.tensorlib.astensor([12.5] + model.config.auxdata)
print(f'Backend = {pyhf.tensorlib.name}')
print(f'Optimizer = {pyhf.optimizer.name}')
print(f'ntoys = {ntoys}')
for executor in [
pyhf.futures.TrivialExecutor(),
concurrent.futures.ProcessPoolExecutor(),
concurrent.futures.ThreadPoolExecutor(),
]:
start = time.time()
result = pyhf.infer.hypotest(
1.0,
data,
model,
test_stat='qtilde',
calctype='toybased',
ntoys=ntoys,
executor=executor,
)
print(f' * with Executor = {executor}')
print(f' - CLs_obs = {result:0.6f}')
print(f' - Time = {time.time() - start:0.6f} seconds')
executor.shutdown()