stan icon indicating copy to clipboard operation
stan copied to clipboard
verified publisher icon stan-dev

Feature/3299 chainset

Open mitzimorris opened this issue 1 year ago • 13 comments

Submission Checklist

  • [x] Run unit tests: ./runTests.py src/test/unit
  • [x] Run cpplint: make cpplint
  • [x] Declare copyright holder and open-source license: see below

Summary

This is a do-over of PR https://github.com/stan-dev/stan/pull/3310. The overall goal is to make the improved R-hat diagnostics (https://github.com/stan-dev/stan/pull/3266), and rank-normalized ESS bulk and tail, (https://github.com/stan-dev/stan/pull/3312) available to CmdStan, following the definitions in https://arxiv.org/abs/1903.08008.

This PR adds a new class stan::mcmc::chainset. A chainset object requires that all chains are the same size on construction, unlike the chains<> object. This simplifies computation of split-Rhat and split-ESS diagnostics. The chainset object provides rank-normalized split Rhat, rank-normalized split ESS which returns bulk and tail ESS, and functions mcse and mcse_sd which calculate the mean Monte Carlo standard error and its variance.

How to Verify

Unit tests for the chainset test performance against Stan CSV outputs. The CmdStan branch https://github.com/stan-dev/cmdstan/tree/feature/1263-new-rhat-summary further checks that chainset provides the necessary functions to output the same set of summary statistics as the R interfaces.

Side Effects

N/A

Documentation

Documentation will be added to the Stan docset in a separate PR.

Copyright and Licensing

Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company): Columbia University

By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:

  • Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
  • Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

mitzimorris avatar Oct 11 '24 20:10 mitzimorris

Name Old Result New Result Ratio Performance change( 1 - new / old )
arma/arma.stan 0.35 0.34 1.01 0.77% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan 0.01 0.01 1.07 6.59% faster
gp_regr/gen_gp_data.stan 0.03 0.03 1.01 0.99% faster
gp_regr/gp_regr.stan 0.09 0.09 1.0 -0.24% slower
sir/sir.stan 70.35 71.01 0.99 -0.94% slower
irt_2pl/irt_2pl.stan 4.12 4.33 0.95 -5.01% slower
eight_schools/eight_schools.stan 0.06 0.05 1.04 4.12% faster
pkpd/sim_one_comp_mm_elim_abs.stan 0.25 0.24 1.04 4.17% faster
pkpd/one_comp_mm_elim_abs.stan 19.49 18.79 1.04 3.59% faster
garch/garch.stan 0.44 0.4 1.09 7.89% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan 2.7 2.58 1.05 4.51% faster
arK/arK.stan 1.78 1.71 1.05 4.36% faster
gp_pois_regr/gp_pois_regr.stan 2.83 2.69 1.05 4.99% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 8.9 8.35 1.07 6.18% faster
performance.compilation 183.57 180.29 1.02 1.79% faster
Mean result: 1.0311739250234235

Jenkins Console Log Blue Ocean Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55

Machine information No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 80 On-line CPU(s) list: 0-79 Thread(s) per core: 2 Core(s) per socket: 20 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz Stepping: 4 CPU MHz: 2400.000 CPU max MHz: 3700.0000 CPU min MHz: 1000.0000 BogoMIPS: 4800.00 Virtualization: VT-x L1d cache: 1.3 MiB L1i cache: 1.3 MiB L2 cache: 40 MiB L3 cache: 55 MiB NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79 Vulnerability Gather data sampling: Mitigation; Microcode Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Mitigation; IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++: g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0 Copyright (C) 2019 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang: clang version 10.0.0-4ubuntu1 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin

stan-buildbot avatar Oct 12 '24 02:10 stan-buildbot

made test more strict, but differences between R and C++ are to be expected.

I am not convinced that they are expected (I recently wrote a javascript version of ESS and RHat that matches stan's current C++ out to ~10 digits). I would want to see some explanation of exactly where and why they do to this degree, if we're claiming to implement the same things.

But, if these differences can be justified, then we shouldn't be testing against those values at all. The presence of those values in the tests visually implies we do match the R values (when we don't), and the resulting tests have very little power to catch issues. For example:

  • the MCSE calculation needs the sample standard deviation. If you replace it with the population standard deviation (by deleting the -1 in the denominator), the tests all pass.
  • if you replace it with something even more outlandish, like adding twenty to the denominator instead of subtracting one, the tests also still pass

I pick this example not just to be a pain, but because replacing the in-line standard deviation calculation in mcse_mean function is something a future code editor may be likely to do (if we don't already change it during review for this!), and accidentally picking the wrong library function is not hard to imagine, so failing the tests if that mistake gets made would be very helpful. Similar arguments can be made for basically all the other tests having tighter tolerances

WardBrian avatar Oct 16 '24 19:10 WardBrian

But, if these differences can be justified, then we shouldn't be testing against those values at all.

I would be happy to find another way to test these. what would you suggest?

mitzimorris avatar Oct 16 '24 19:10 mitzimorris

What previous implementations of diagnostics seem to do is just run the code once to get values and then "freeze" those in the test. I think #3312 deleted some tests in that style.

Doing this requires us to be pretty certain the implementation is correct, since we're making it the gold standard for all future versions of the code, which is why it would be important to understand what parts of the calculation are different than R

WardBrian avatar Oct 16 '24 19:10 WardBrian

which is why it would be important to understand what parts of the calculation are different than R

perhaps we should create an artificial dataset to convince ourselves of the goodness of ESS and Rhat. what did you use for Javascript?

mitzimorris avatar Oct 16 '24 19:10 mitzimorris

For testing I ended up using the same blocker1.csv and blocker2.csv as the stan tests, but for initially getting them to agree I was just using an arbitrary CSV file, I think from the Bernoulli model?

WardBrian avatar Oct 16 '24 19:10 WardBrian

Name Old Result New Result Ratio Performance change( 1 - new / old )
arma/arma.stan 0.35 0.34 1.04 3.66% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan 0.01 0.01 1.06 5.68% faster
gp_regr/gen_gp_data.stan 0.03 0.03 1.1 9.3% faster
gp_regr/gp_regr.stan 0.09 0.09 1.06 5.98% faster
sir/sir.stan 70.01 71.77 0.98 -2.52% slower
irt_2pl/irt_2pl.stan 4.15 4.49 0.93 -8.1% slower
eight_schools/eight_schools.stan 0.06 0.06 1.0 -0.25% slower
pkpd/sim_one_comp_mm_elim_abs.stan 0.25 0.26 0.95 -5.63% slower
pkpd/one_comp_mm_elim_abs.stan 19.4 19.46 1.0 -0.28% slower
garch/garch.stan 0.44 0.44 1.0 -0.08% slower
low_dim_gauss_mix/low_dim_gauss_mix.stan 2.72 2.71 1.01 0.51% faster
arK/arK.stan 1.8 1.8 1.0 0.13% faster
gp_pois_regr/gp_pois_regr.stan 2.88 3.07 0.94 -6.68% slower
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 8.83 8.62 1.02 2.31% faster
performance.compilation 182.25 178.41 1.02 2.11% faster
Mean result: 1.0063032409056276

Jenkins Console Log Blue Ocean Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55

Machine information No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 80 On-line CPU(s) list: 0-79 Thread(s) per core: 2 Core(s) per socket: 20 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz Stepping: 4 CPU MHz: 2400.000 CPU max MHz: 3700.0000 CPU min MHz: 1000.0000 BogoMIPS: 4800.00 Virtualization: VT-x L1d cache: 1.3 MiB L1i cache: 1.3 MiB L2 cache: 40 MiB L3 cache: 55 MiB NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79 Vulnerability Gather data sampling: Mitigation; Microcode Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Mitigation; IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++: g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0 Copyright (C) 2019 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang: clang version 10.0.0-4ubuntu1 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin

stan-buildbot avatar Oct 16 '24 21:10 stan-buildbot

Name Old Result New Result Ratio Performance change( 1 - new / old )
arma/arma.stan 0.37 0.34 1.09 7.87% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan 0.01 0.01 1.2 16.62% faster
gp_regr/gen_gp_data.stan 0.03 0.02 1.14 12.06% faster
gp_regr/gp_regr.stan 0.1 0.09 1.07 6.87% faster
sir/sir.stan 70.05 72.58 0.97 -3.62% slower
irt_2pl/irt_2pl.stan 4.15 4.02 1.03 3.22% faster
eight_schools/eight_schools.stan 0.06 0.06 1.02 2.07% faster
pkpd/sim_one_comp_mm_elim_abs.stan 0.25 0.25 1.02 1.55% faster
pkpd/one_comp_mm_elim_abs.stan 19.49 19.54 1.0 -0.29% slower
garch/garch.stan 0.44 0.41 1.07 6.27% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan 2.7 2.72 0.99 -0.81% slower
arK/arK.stan 1.8 1.71 1.05 4.91% faster
gp_pois_regr/gp_pois_regr.stan 2.85 2.68 1.06 5.97% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 8.78 8.37 1.05 4.72% faster
performance.compilation 178.22 179.53 0.99 -0.73% slower
Mean result: 1.0496172795519458

Jenkins Console Log Blue Ocean Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55

Machine information No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 80 On-line CPU(s) list: 0-79 Thread(s) per core: 2 Core(s) per socket: 20 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz Stepping: 4 CPU MHz: 2400.000 CPU max MHz: 3700.0000 CPU min MHz: 1000.0000 BogoMIPS: 4800.00 Virtualization: VT-x L1d cache: 1.3 MiB L1i cache: 1.3 MiB L2 cache: 40 MiB L3 cache: 55 MiB NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79 Vulnerability Gather data sampling: Mitigation; Microcode Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Mitigation; IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++: g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0 Copyright (C) 2019 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang: clang version 10.0.0-4ubuntu1 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin

stan-buildbot avatar Oct 18 '24 04:10 stan-buildbot

Name Old Result New Result Ratio Performance change( 1 - new / old )
arma/arma.stan 0.37 0.35 1.06 5.3% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan 0.01 0.01 0.96 -4.22% slower
gp_regr/gen_gp_data.stan 0.03 0.03 1.0 -0.46% slower
gp_regr/gp_regr.stan 0.09 0.1 0.99 -0.88% slower
sir/sir.stan 70.45 67.98 1.04 3.5% faster
irt_2pl/irt_2pl.stan 4.13 3.96 1.04 4.06% faster
eight_schools/eight_schools.stan 0.06 0.05 1.04 3.44% faster
pkpd/sim_one_comp_mm_elim_abs.stan 0.25 0.24 1.04 4.27% faster
pkpd/one_comp_mm_elim_abs.stan 19.38 18.6 1.04 4.02% faster
garch/garch.stan 0.44 0.41 1.07 6.77% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan 2.73 2.56 1.07 6.2% faster
arK/arK.stan 1.77 1.7 1.04 4.02% faster
gp_pois_regr/gp_pois_regr.stan 2.77 2.66 1.04 3.71% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 8.69 8.35 1.04 3.92% faster
performance.compilation 180.6 184.2 0.98 -1.99% slower
Mean result: 1.0295481519172558

Jenkins Console Log Blue Ocean Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55

Machine information No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 80 On-line CPU(s) list: 0-79 Thread(s) per core: 2 Core(s) per socket: 20 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz Stepping: 4 CPU MHz: 2400.000 CPU max MHz: 3700.0000 CPU min MHz: 1000.0000 BogoMIPS: 4800.00 Virtualization: VT-x L1d cache: 1.3 MiB L1i cache: 1.3 MiB L2 cache: 40 MiB L3 cache: 55 MiB NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79 Vulnerability Gather data sampling: Mitigation; Microcode Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Mitigation; IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++: g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0 Copyright (C) 2019 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang: clang version 10.0.0-4ubuntu1 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin

stan-buildbot avatar Oct 19 '24 01:10 stan-buildbot

Name Old Result New Result Ratio Performance change( 1 - new / old )
arma/arma.stan 0.35 0.36 0.99 -1.51% slower
low_dim_corr_gauss/low_dim_corr_gauss.stan 0.01 0.01 0.94 -6.11% slower
gp_regr/gen_gp_data.stan 0.03 0.03 1.02 2.22% faster
gp_regr/gp_regr.stan 0.1 0.09 1.04 4.03% faster
sir/sir.stan 70.72 68.48 1.03 3.16% faster
irt_2pl/irt_2pl.stan 4.19 3.97 1.05 5.19% faster
eight_schools/eight_schools.stan 0.06 0.06 1.05 4.53% faster
pkpd/sim_one_comp_mm_elim_abs.stan 0.25 0.24 1.04 3.56% faster
pkpd/one_comp_mm_elim_abs.stan 19.43 18.85 1.03 2.99% faster
garch/garch.stan 0.45 0.41 1.12 10.49% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan 2.71 2.65 1.02 2.26% faster
arK/arK.stan 1.79 1.7 1.05 5.16% faster
gp_pois_regr/gp_pois_regr.stan 2.8 2.69 1.04 3.75% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 8.83 8.37 1.06 5.29% faster
performance.compilation 180.41 179.96 1.0 0.25% faster
Mean result: 1.0324525440756125

Jenkins Console Log Blue Ocean Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55

Machine information No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 80 On-line CPU(s) list: 0-79 Thread(s) per core: 2 Core(s) per socket: 20 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz Stepping: 4 CPU MHz: 2400.000 CPU max MHz: 3700.0000 CPU min MHz: 1000.0000 BogoMIPS: 4800.00 Virtualization: VT-x L1d cache: 1.3 MiB L1i cache: 1.3 MiB L2 cache: 40 MiB L3 cache: 55 MiB NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79 Vulnerability Gather data sampling: Mitigation; Microcode Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Mitigation; IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++: g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0 Copyright (C) 2019 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang: clang version 10.0.0-4ubuntu1 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin

stan-buildbot avatar Oct 19 '24 21:10 stan-buildbot

Name Old Result New Result Ratio Performance change( 1 - new / old )
arma/arma.stan 0.33 0.38 0.87 -14.98% slower
low_dim_corr_gauss/low_dim_corr_gauss.stan 0.01 0.01 0.98 -1.95% slower
gp_regr/gen_gp_data.stan 0.03 0.03 0.99 -0.64% slower
gp_regr/gp_regr.stan 0.09 0.09 1.02 2.21% faster
sir/sir.stan 69.87 68.0 1.03 2.67% faster
irt_2pl/irt_2pl.stan 4.23 3.95 1.07 6.67% faster
eight_schools/eight_schools.stan 0.06 0.05 1.05 4.44% faster
pkpd/sim_one_comp_mm_elim_abs.stan 0.25 0.24 1.06 5.22% faster
pkpd/one_comp_mm_elim_abs.stan 19.37 19.33 1.0 0.2% faster
garch/garch.stan 0.42 0.41 1.03 3.2% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan 2.67 2.57 1.04 4.05% faster
arK/arK.stan 1.77 1.71 1.04 3.71% faster
gp_pois_regr/gp_pois_regr.stan 2.79 2.67 1.04 4.1% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 8.68 8.32 1.04 4.16% faster
performance.compilation 180.54 179.95 1.0 0.33% faster
Mean result: 1.018155431705192

Jenkins Console Log Blue Ocean Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55

Machine information No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 80 On-line CPU(s) list: 0-79 Thread(s) per core: 2 Core(s) per socket: 20 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz Stepping: 4 CPU MHz: 2400.000 CPU max MHz: 3700.0000 CPU min MHz: 1000.0000 BogoMIPS: 4800.00 Virtualization: VT-x L1d cache: 1.3 MiB L1i cache: 1.3 MiB L2 cache: 40 MiB L3 cache: 55 MiB NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79 Vulnerability Gather data sampling: Mitigation; Microcode Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Mitigation; IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++: g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0 Copyright (C) 2019 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang: clang version 10.0.0-4ubuntu1 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin

stan-buildbot avatar Oct 20 '24 04:10 stan-buildbot

@WardBrian this is ready for re-review.

the discrepencies between this code and the implementations in posterior package were due to:

  • differences between stan::math::covariance and stan::analyze::covariance. now using the latter.
  • testing against outputs from centered eight_schools which contained only 500 draws - too few draws to resolve the variance. instead now testing against 4000 draws from bernoulli.stan model everywhere.

changes per discussion here and in https://github.com/stan-dev/stan/pull/3312#issuecomment-2420180157

  • further refactoring of code in src/analyze/mcmc into file ess_basic and rhat_basic, with corresponding unit tests
  • unit tests of ess_basic and rhat_basic tests against both posterior pkg and old implementations in src/analyze/mcmc/compute*.hpp
  • tests of new rhat, ess, mcse_mean and mcse_sd moved from chainset_test.cpp to unit tests for src/analyze/mcmc.
  • tests for compatability with posterior pkg estimates use 4 chains of 1000 draws each.
  • removed a bunch of test csv files no longer needed

at some point we should deprecate chains.hpp and corresponging src/analyze/mcmc/compute*.hpp.

mitzimorris avatar Oct 20 '24 15:10 mitzimorris

Name Old Result New Result Ratio Performance change( 1 - new / old )
arma/arma.stan 0.35 0.34 1.02 1.75% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan 0.01 0.01 0.96 -4.0% slower
gp_regr/gen_gp_data.stan 0.03 0.03 1.0 -0.43% slower
gp_regr/gp_regr.stan 0.09 0.09 1.0 0.24% faster
sir/sir.stan 69.97 68.96 1.01 1.45% faster
irt_2pl/irt_2pl.stan 4.14 4.21 0.98 -1.68% slower
eight_schools/eight_schools.stan 0.06 0.06 1.01 1.14% faster
pkpd/sim_one_comp_mm_elim_abs.stan 0.25 0.25 1.01 1.35% faster
pkpd/one_comp_mm_elim_abs.stan 19.47 19.22 1.01 1.31% faster
garch/garch.stan 0.43 0.41 1.06 6.0% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan 2.7 2.59 1.05 4.33% faster
arK/arK.stan 1.83 1.7 1.07 6.74% faster
gp_pois_regr/gp_pois_regr.stan 2.9 2.68 1.08 7.63% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 8.79 8.37 1.05 4.77% faster
performance.compilation 182.31 177.81 1.03 2.47% faster
Mean result: 1.023584481152946

Jenkins Console Log Blue Ocean Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55

Machine information No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 80 On-line CPU(s) list: 0-79 Thread(s) per core: 2 Core(s) per socket: 20 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz Stepping: 4 CPU MHz: 2400.000 CPU max MHz: 3700.0000 CPU min MHz: 1000.0000 BogoMIPS: 4800.00 Virtualization: VT-x L1d cache: 1.3 MiB L1i cache: 1.3 MiB L2 cache: 40 MiB L3 cache: 55 MiB NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79 Vulnerability Gather data sampling: Mitigation; Microcode Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Mitigation; IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++: g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0 Copyright (C) 2019 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang: clang version 10.0.0-4ubuntu1 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin

stan-buildbot avatar Oct 20 '24 17:10 stan-buildbot

Name Old Result New Result Ratio Performance change( 1 - new / old )
arma/arma.stan 0.37 0.35 1.04 3.89% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan 0.01 0.01 1.04 3.64% faster
gp_regr/gen_gp_data.stan 0.03 0.03 1.19 15.76% faster
gp_regr/gp_regr.stan 0.1 0.09 1.03 3.31% faster
sir/sir.stan 69.95 69.81 1.0 0.2% faster
irt_2pl/irt_2pl.stan 4.07 4.02 1.01 1.19% faster
eight_schools/eight_schools.stan 0.06 0.06 1.02 2.21% faster
pkpd/sim_one_comp_mm_elim_abs.stan 0.25 0.25 0.99 -1.01% slower
pkpd/one_comp_mm_elim_abs.stan 19.15 18.84 1.02 1.63% faster
garch/garch.stan 0.45 0.41 1.1 8.81% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan 2.78 2.6 1.07 6.59% faster
arK/arK.stan 1.81 2.58 0.7 -42.36% slower
gp_pois_regr/gp_pois_regr.stan 2.83 2.71 1.04 4.07% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 8.85 8.43 1.05 4.73% faster
performance.compilation 178.71 183.63 0.97 -2.75% slower
Mean result: 1.0185136446700325

Jenkins Console Log Blue Ocean Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55

Machine information No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 80 On-line CPU(s) list: 0-79 Thread(s) per core: 2 Core(s) per socket: 20 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz Stepping: 4 CPU MHz: 2400.000 CPU max MHz: 3700.0000 CPU min MHz: 1000.0000 BogoMIPS: 4800.00 Virtualization: VT-x L1d cache: 1.3 MiB L1i cache: 1.3 MiB L2 cache: 40 MiB L3 cache: 55 MiB NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79 Vulnerability Gather data sampling: Mitigation; Microcode Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Mitigation; IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++: g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0 Copyright (C) 2019 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang: clang version 10.0.0-4ubuntu1 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin

stan-buildbot avatar Oct 21 '24 23:10 stan-buildbot

Name Old Result New Result Ratio Performance change( 1 - new / old )
arma/arma.stan 0.36 0.36 0.99 -0.67% slower
low_dim_corr_gauss/low_dim_corr_gauss.stan 0.01 0.01 0.94 -6.54% slower
gp_regr/gen_gp_data.stan 0.03 0.03 1.0 0.42% faster
gp_regr/gp_regr.stan 0.1 0.09 1.03 3.21% faster
sir/sir.stan 70.33 72.61 0.97 -3.24% slower
irt_2pl/irt_2pl.stan 4.17 4.28 0.97 -2.74% slower
eight_schools/eight_schools.stan 0.06 0.06 0.98 -2.4% slower
pkpd/sim_one_comp_mm_elim_abs.stan 0.26 0.25 1.06 5.97% faster
pkpd/one_comp_mm_elim_abs.stan 19.49 19.02 1.02 2.42% faster
garch/garch.stan 0.44 0.41 1.07 6.21% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan 2.7 2.57 1.05 4.66% faster
arK/arK.stan 1.79 1.7 1.05 4.93% faster
gp_pois_regr/gp_pois_regr.stan 2.82 2.67 1.06 5.33% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 8.78 8.44 1.04 3.83% faster
performance.compilation 178.74 177.93 1.0 0.45% faster
Mean result: 1.0162582955033252

Jenkins Console Log Blue Ocean Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55

Machine information No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 80 On-line CPU(s) list: 0-79 Thread(s) per core: 2 Core(s) per socket: 20 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz Stepping: 4 CPU MHz: 2400.000 CPU max MHz: 3700.0000 CPU min MHz: 1000.0000 BogoMIPS: 4800.00 Virtualization: VT-x L1d cache: 1.3 MiB L1i cache: 1.3 MiB L2 cache: 40 MiB L3 cache: 55 MiB NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79 Vulnerability Gather data sampling: Mitigation; Microcode Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Mitigation; IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++: g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0 Copyright (C) 2019 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang: clang version 10.0.0-4ubuntu1 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin

stan-buildbot avatar Oct 22 '24 03:10 stan-buildbot

Name Old Result New Result Ratio Performance change( 1 - new / old )
arma/arma.stan 0.43 0.38 1.13 11.79% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan 0.01 0.01 1.11 9.84% faster
gp_regr/gen_gp_data.stan 0.03 0.03 1.06 5.72% faster
gp_regr/gp_regr.stan 0.11 0.1 1.15 12.69% faster
sir/sir.stan 80.7 74.12 1.09 8.16% faster
irt_2pl/irt_2pl.stan 5.11 4.5 1.13 11.89% faster
eight_schools/eight_schools.stan 0.06 0.06 1.07 6.12% faster
pkpd/sim_one_comp_mm_elim_abs.stan 0.27 0.26 1.03 2.9% faster
pkpd/one_comp_mm_elim_abs.stan 22.08 20.37 1.08 7.74% faster
garch/garch.stan 0.65 0.46 1.42 29.71% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan 2.99 2.81 1.06 6.04% faster
arK/arK.stan 2.0 1.85 1.08 7.82% faster
gp_pois_regr/gp_pois_regr.stan 3.44 2.94 1.17 14.51% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 9.72 9.39 1.04 3.45% faster
performance.compilation 199.44 204.54 0.98 -2.55% slower
Mean result: 1.1069147395963945

Jenkins Console Log Blue Ocean Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55

Machine information No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 80 On-line CPU(s) list: 0-79 Thread(s) per core: 2 Core(s) per socket: 20 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz Stepping: 4 CPU MHz: 3375.014 CPU max MHz: 3700.0000 CPU min MHz: 1000.0000 BogoMIPS: 4800.00 Virtualization: VT-x L1d cache: 1.3 MiB L1i cache: 1.3 MiB L2 cache: 40 MiB L3 cache: 55 MiB NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79 Vulnerability Gather data sampling: Mitigation; Microcode Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Mitigation; IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++: g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0 Copyright (C) 2019 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang: clang version 10.0.0-4ubuntu1 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin

stan-buildbot avatar Oct 22 '24 15:10 stan-buildbot

The unit tests now test basic ESS and Rhat against the values computed by CmdStan 2.35.0's bin/stansummary and the saved summary CSV file has been checked into the folder src/test/unit/analyze/mcmc/test_csv_files.

I have added unit tests for rank-normalization and splitting the chains - the latter did uncover a bug where splitting a chain with an odd number of draws would omit the last draw, not the (N+1)/2 draw; fixed - but this bug would not have been tickled by any of the existing tests since all CSV test sets had and even number of draws.

mitzimorris avatar Oct 22 '24 17:10 mitzimorris

Name Old Result New Result Ratio Performance change( 1 - new / old )
arma/arma.stan 0.37 0.38 0.98 -1.81% slower
low_dim_corr_gauss/low_dim_corr_gauss.stan 0.01 0.01 0.71 -41.17% slower
gp_regr/gen_gp_data.stan 0.03 0.03 1.1 8.85% faster
gp_regr/gp_regr.stan 0.1 0.1 0.94 -5.86% slower
sir/sir.stan 76.62 74.8 1.02 2.37% faster
irt_2pl/irt_2pl.stan 4.31 4.17 1.03 3.27% faster
eight_schools/eight_schools.stan 0.06 0.06 0.97 -2.85% slower
pkpd/sim_one_comp_mm_elim_abs.stan 0.26 0.26 0.98 -1.88% slower
pkpd/one_comp_mm_elim_abs.stan 19.74 20.48 0.96 -3.78% slower
garch/garch.stan 0.46 0.44 1.05 5.17% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan 2.82 2.76 1.02 1.96% faster
arK/arK.stan 1.8 1.84 0.98 -1.86% slower
gp_pois_regr/gp_pois_regr.stan 2.84 2.89 0.98 -1.7% slower
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 9.06 9.04 1.0 0.13% faster
performance.compilation 187.23 194.85 0.96 -4.07% slower
Mean result: 0.9806370865532955

Jenkins Console Log Blue Ocean Commit hash: eb2b7af7527e4daaa5788c458dff4c118a19ee55

Machine information No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 80 On-line CPU(s) list: 0-79 Thread(s) per core: 2 Core(s) per socket: 20 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz Stepping: 4 CPU MHz: 2400.000 CPU max MHz: 3700.0000 CPU min MHz: 1000.0000 BogoMIPS: 4800.00 Virtualization: VT-x L1d cache: 1.3 MiB L1i cache: 1.3 MiB L2 cache: 40 MiB L3 cache: 55 MiB NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79 Vulnerability Gather data sampling: Mitigation; Microcode Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Mitigation; IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++: g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0 Copyright (C) 2019 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang: clang version 10.0.0-4ubuntu1 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin

stan-buildbot avatar Oct 22 '24 20:10 stan-buildbot