Fix `predict_particles`, and implement the Brier score

[DilumAluthge]

Fixes #93 Fixes #117 Related to #52

Please note that this PR makes multiple breaking changes.

[DilumAluthge]

@cscherrer This is now ready for review.

I'm going to leave it as a "draft" so that we don't accidentally click merge. We don't need to merge this until the package registration is complete. (The package registration PR will be auto-merged in 3 days.)

[DilumAluthge]

@cscherrer I completely forgot about this PR.

Is there anything else we need to do before merging this PR?

[cscherrer]

Thanks @DilumAluthge , I'm sorry I had forgotten about it too.

Not sure what's going on with the tests, but let's merge this ASAP so we don't lose track of it again. We're also making some big changes in Soss that will be in the next release, to we'll need to come back to this to update once that's out.

[DilumAluthge]

I've rebased this on the latest master. CI is passing on master, but it is failing on this PR, so I think there is a problem in this PR.

Any idea why the tests are failing?

An example error is:

LoadError: ArgumentError: Cannot convert a particle distribution to a float if not all particles are the same.

[DilumAluthge]

The tests were previously passing on this PR, so my guess is there has been a breaking change in one of our dependencies. Maybe Soss, or MonteCarloMeasurements.

[DilumAluthge]

Maybe there was some breaking change in the behavior of the Soss.particles function?

[cscherrer]

There was a change in master, because MonteCarloMeasurements exports some Distributions functions that collide with MeasureTheory. I don't think it affected the release though

[DilumAluthge]

Hmmm. Any idea what's causing this error?

Since CI is passing on master, I'd prefer to get CI passing on this PR before merging it; otherwise we'll break master 😂

[DilumAluthge]

Looks like we're getting closer!

The error is now:

LoadError: type Array has no field β

And it comes from this code: https://github.com/cscherrer/SossMLJ.jl/blob/0b2f9b29037b555ad0834aa63d27408a3da2c709/test/extra-example-tests/linear-regression.jl#L9-L13

[DilumAluthge]

Bump @cscherrer

[cscherrer]

Ah, right! Thanks for the ping, I'll have another look now. I'd love to have it working for the new release, but sampling syntax changed so that will take some more changes

[cscherrer]

Ok, I see the problem:

julia> SossMLJ._predict_all_particles(mach, X)
100-element Vector{Particles{Float64, 1000}}:
 -0.482 ± 0.27
 -0.00677 ± 0.26
 -0.0659 ± 0.27
 -0.0272 ± 0.28
  0.205 ± 0.25
 -0.105 ± 0.27

The line 10 you posted above tries to get the :β property from this, but there is none

[DilumAluthge]

Hmmm. The idea of SossMLJ._predict_all_particles is that it should return particles for everything. So it should have the particles for :β, :σ, :η, and :μ.

My guess is that there's a bug in the implementation of _predict_all_particles.

In 0b2f9b29037b555ad0834aa63d27408a3da2c709, you made some fixes to the predict_particles function. I think you'll also need to apply those same fixes to the _predict_all_particles function.

[codecov[bot]]

Codecov Report

Merging #118 (1a76253) into master (0baac33) will decrease coverage by 9.55%. The diff coverage is 87.50%.

@@             Coverage Diff             @@
##            master     #118      +/-   ##
===========================================
- Coverage   100.00%   90.44%   -9.56%     
===========================================
  Files            6       12       +6     
  Lines           92      136      +44     
===========================================
+ Hits            92      123      +31     
- Misses           0       13      +13     

Impacted Files	Coverage Δ
src/machine-operations.jl	100.00% <ø> (ø)
src/loss-functions/brier-score.jl	53.33% <53.33%> (ø)
src/loss-functions/rms.jl	66.66% <66.66%> (ø)
src/check-rows.jl	75.00% <75.00%> (ø)
src/particles.jl	95.23% <94.73%> (-4.77%)	:arrow_down:
src/categorical-arrays.jl	100.00% <100.00%> (ø)
src/distributions.jl	100.00% <100.00%> (ø)
src/loss-functions/types.jl	100.00% <100.00%> (ø)
src/models.jl	100.00% <100.00%> (ø)
src/prediction.jl	100.00% <100.00%> (ø)
... and 2 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 0baac33...1a76253. Read the comment docs.

[DilumAluthge]

Looks like CI is finally passing. Sometime this week (or maybe next week), I'll add tests to increase code coverage.

Coverage is 100% on master, and I'd like to keep it at 100%.