Adding CUTEst.jl to the optimization benchmarks

[alonsoC1s]

This PR leverages the integration of CUTEst.jl into the SciML umbrella to add all problems in the CUTEst collection to a set of optimization benchmarks. Resolves #935

To complete this PR, the building infrastructure will have to be changed to manually install gfortran as required by CUTEst.jl.

[ChrisRackauckas]

Is gfortran still required after https://github.com/JuliaSmoothOptimizers/CUTEst.jl/commit/df91b3c4641bae6e62fb44d0af9b8791c9fa802a ?

Is there anything not in the jll? Maybe @staticfloat can help here.

[alonsoC1s]

I am not sure to be honest, if the jll makes gfortran available system-wide, it should probably work. As far as I know, CUTEst.jl performs a simple test to verify if gfortran exists in the system by calling it in a shell and checking for errors

[staticfloat]

We ship libgfortran with Julia, and make it available for libraries and executables in JLLs, so if that's all you need, you should be good to go.

[ChrisRackauckas]

What exactly is asking/calling for gfortran @abelsiqueira?

[abelsiqueira]

Sifdecode, to decode the SIF file, which gets called by CUTEstModel. Things might have changed recently. Cc. @amontoison @dpo @tmigot

[amontoison]

@alonsoC1s I'm working on a new release (v0.14.0) with many modifications (support of Windows as well as single and quadruple precision). We also changed a lot of things with Nick Gould in CUTEst and SIFDecode.

gfortran is needed because the test problems are based on SIF files and the SIF decoder generates Fortran files to be compiled: https://github.com/JuliaSmoothOptimizers/CUTEst.jl/blob/db823cb293d2cdf321b62fd2e80f4fb84dbba39c/src/sifdecoder.jl#L144 On Windows I added an artifactact in Artifacts.toml to provide gfortran.exe but I expect the Linux and Mac users to install it.

We don't precompile the problems to support problems of variable size. We also have 1500 problems so it's 4500 libraries if we want to precompile everything (with fixed size).

[ChrisRackauckas]

On Windows I added an artifactact in Artifacts.toml to provide gfortran.exe but I expect the Linuw and Mac users to install it.

That's very non-standard? Seems odd to bundle an artifact on some systems but not others. Also, it doesn't make much sense given @staticfloat's comment that it's already bundled with Julia?

[amontoison]

You didn't understand the message of @staticfloat. libgfortran is shiped with Julia, not gfortran. A library is different than a compiler. 😏

I did a copy of the Artifacts.toml available in PackageCompiler.jl, that only install the compilers on Windows. I don't know if we have available artifacts of compilers (and not cross-compilers) available thanks to BinaryBuilder / BinaryBuilderBase for all platforms (Mac, linux, FreeBSD). @staticfloat Do you have the answer?

Binaries and libraries of SIFDecode and CUTEst are cross-compiled on all platforms with Yggdrasil.

[staticfloat]

I don't know if we have available artifacts of compilers (and not cross-compilers) available thanks to BinaryBuilder / BinaryBuilderBase for all platforms (Mac, linux, FreeBSD).

Ask me again in ~2 months, I should have implemented it in BinaryBuilderToolchains.jl by then. Until then, what you're doing here is fine.

[ChrisRackauckas]

So do we build add gfortran as a buildkite dep in the meantime?

[alonsoC1s]

@ChrisRackauckas that could work. I can also work on this locally and leave the PR to be revised when the functionality lands on the new versions. I don't know how labour-intensive temporarily adding gfortran to buildkite is, but it's probably best to avoid fixes that will soon be unnecessary

[staticfloat]

You should be able to either add it to the rootfs image being used, or just apt update && apt install -y gfortran.

[ChrisRackauckas]

You need to bump the manifest and Symbolics version. Since gfortran is now installed, this should be finishable.

[alonsoC1s]

Yep, I just saw the railed build. I'll fix the manifest, try again, and once I can see some preliminary results I can write something about them and write the rest of the benchmarks

[ChrisRackauckas]

Error: InitError: gfortran is not installed. Please install it and try agai
n.
during initialization of module CUTEst

@thazhemadam can you take a second look at this?

[ChrisRackauckas]

Did the benchmark get stuck or does it just take really really long?

[ChrisRackauckas]

Maybe split this into a few different scripts for the major groups? Seems like it's running fine though so it's just a bit of file management.

[alonsoC1s]

I could reduce the scope by focusing on particular groups of problems. In total there's several thousand problems, some of which are rather large

[ChrisRackauckas]

It's fine that it's large, but if you split it to multiple files then it can run in parallel and we could also isolate if there's one section that is much longer than the others. It looks like

┌ Info: Weaving chunk 6 from line 161
--
  | └   progress = 0.5555555555555556

that chunk might just be stuck, while others completed.

[alonsoC1s]

Sure, I'll start splitting into files. Hopefully that resolves the stuck chunk or at least makes it easier to see where it's getting stuck

[ChrisRackauckas]

It looks like it's stalling in the first solve. Every solve really. @alonsoC1s was it doing this locally for HIER13?

[alonsoC1s]

I only did some very limited testing locally, basically confirming I didn't have any runtime errors. I did try solving manually some problems like Rosenbrock and it worked as expected, as long as the models were finalized as I took care of doing in the testing loop

[alonsoC1s]

In the process of introducing some filtering to reduce the number of benchmarks so that the CI job doesn't stall I think I might have found some problems that fail silently. For instance, I identified that one of the problems that stall, even at 2 variables, is ZECEVIC3.

`julia> zecevic3 = CUTEstModel("zecevic3")
  Problem name: zecevic3
   All variables: ████████████████████ 2      All constraints: ████████████████████ 2     
            free: ⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅ 0                 free: ⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅ 0     
           lower: ⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅ 0                lower: ⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅ 0     
           upper: ⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅ 0                upper: ████████████████████ 2     
         low/upp: ████████████████████ 2              low/upp: ⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅ 0     
           fixed: ⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅ 0                fixed: ⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅ 0     
          infeas: ⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅ 0               infeas: ⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅ 0     
            nnzh: (  0.00% sparsity)   3               linear: ⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅ 0     
                                                    nonlinear: ████████████████████ 2     
                                                         nnzj: (  0.00% sparsity)   4  

Which produces this error when optimized:

julia> prob = OptimizationNLPModels.OptimizationProblem(zecevic3, Optimization.AutoForwardDiff())
OptimizationProblem. In-place: true
u0: 2-element Vector{Float64}:
  0.1
 -0.1

julia> solve(prob, MOI.OptimizerWithAttributes(Ipopt.Optimizer); maxiters = 1e6)
┌ Warning: common maxiters argument is currently not used by Feasibility
│ 
│ Subject to:
│ 
│ Nonlinear
│ . Set number of iterations via optimizer specific keyword arguments.
└ @ OptimizationMOI ~/.julia/packages/OptimizationMOI/LnXYV/src/OptimizationMOI.jl:86

******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
 Ipopt is released as open source code under the Eclipse Public License (EPL).
         For more information visit https://github.com/coin-or/Ipopt
******************************************************************************

This is Ipopt version 3.14.16, running with linear solver MUMPS 5.7.3.

Number of nonzeros in equality constraint Jacobian...:        0
Number of nonzeros in inequality constraint Jacobian.:        4
Number of nonzeros in Lagrangian Hessian.............:        3

Total number of variables............................:        2
                     variables with only lower bounds:        0
                variables with lower and upper bounds:        2
                     variables with only upper bounds:        0
Total number of equality constraints.................:        0
Total number of inequality constraints...............:        2
        inequality constraints with only lower bounds:        0
   inequality constraints with lower and upper bounds:        0
        inequality constraints with only upper bounds:        2

ERROR: MethodError: no method matching Float64(::ForwardDiff.Dual{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#50#68"{…}, Float64}, ForwardDiff.Dual{ForwardDiff.Tag{…}, Float64, 2}, 2})

The Ipopt output was silenced on the original weave script

[ChrisRackauckas]

Interesting. But why would that cause it to stall?

[ChrisRackauckas]

This was accidentally merged, reverted.

[ChrisRackauckas]

Seems it's hard to reopen this though?

[Vaibhavdixit02]

Yeah since it was from @alonsoC1s's branch I don't think you can open the PR, I have started a new one here https://github.com/SciML/SciMLBenchmarks.jl/pull/1179