Parallel ParticleSwarm

[sgaure]

#1124

This PR enhances ParticleSwarm with a new keyword parallel::Bool = false. If set to true, the objective function will be called with a vector to fill in, and a matrix of column vectors as inputs. Thus, the user may choose to evaluate the swarm in parallel with whatever means.

[sgaure]

Btw, I forgot to mention. The commits also contain a change which allows some upper bounds to be equal to the lower bounds. This can be used as a simple way to keep some variables constant.

[pkofod]

I see, yes I also implemented something similar in NLSolvers.jl. Let me review it. Thanks

[sgaure]

The test failures seem to be unrelated. I came to think of it, should the new parallel keyword be called batched instead? It's not implementing anything parallel.

[github-actions[bot]]

Benchmark Results

	master	3d789354f712c8...	master / 3d789354f712c8...
multivariate/solvers/first_order/AdaMax	0.646 ± 0.0078 ms	0.647 ± 0.008 ms	0.999 ± 0.017
multivariate/solvers/first_order/Adam	0.647 ± 0.008 ms	0.646 ± 0.0081 ms	1 ± 0.018
multivariate/solvers/first_order/BFGS	0.224 ± 0.004 ms	0.223 ± 0.0044 ms	1 ± 0.027
multivariate/solvers/first_order/ConjugateGradient	0.0481 ± 0.0006 ms	0.0479 ± 0.00058 ms	1 ± 0.017
multivariate/solvers/first_order/GradientDescent	1.71 ± 0.014 ms	1.71 ± 0.012 ms	1 ± 0.011
multivariate/solvers/first_order/LBFGS	0.22 ± 0.0057 ms	0.219 ± 0.0036 ms	1 ± 0.031
multivariate/solvers/first_order/MomentumGradientDescent	2.51 ± 0.017 ms	2.51 ± 0.012 ms	0.999 ± 0.0083
multivariate/solvers/first_order/NGMRES	0.554 ± 0.011 ms	0.554 ± 0.011 ms	1 ± 0.027
time_to_load	0.503 ± 0.0047 s	0.502 ± 0.0067 s	1 ± 0.016

Benchmark Plots

A plot of the benchmark results have been uploaded as an artifact to the workflow run for this PR. Go to "Actions"->"Benchmark a pull request"->[the most recent run]->"Artifacts" (at the bottom).

[pkofod]

I'll look at this later today, it had slipped out of my attention.