Create a package KNITRO_jll.jl ?

[amontoison]

@odow I am wondering if we should not try to do a KNITRO_jll.jl like you did with Xpress_jll.jl and GUROBI_jll.jl. What do you think?

[odow]

This requires permission from Artelys for the binaries to be publicly available (e.g., pip install knitro), which I don't believe is the case?

[amontoison]

I quickly discuss about that with an employee of Artelys during ISMP. I will contact them next week.

In the worst case they can do like HSL and have a dummy version of the JLL. It will allow Artelys to not provide the official KNITRO_jll.jl with precompiled binaries to everyone if it's the issue.

[mgabay]

Hi all,

I would like this. We'll discuss about it and get back to you in September.

So far we've avoided it but this is currently under consideration, although there are a few things we need to figure out.

@Bougron : tbd after our vacations

[odow]

Thanks @mgabay.

The precedent for this is:

• https://github.com/jump-dev/Gurobi_jll.jl
• https://github.com/jump-dev/Xpress_jll.jl

They both pull from the official anaconda sources, e.g., https://anaconda.org/Gurobi/gurobi, so something similar would be our ideal. I guess my suggestion is "don't treat this as a Julia issue. Treat it as a Python packaging problem, and we can re-use that solution."

[amontoison]

@mgabay Any update from Artelys? I'm available for a meeting if that could help facilitate the discussion.

[mgabay]

Hi @amontoison ,

We plan to do it for Python so that you can then setup something similar to other solvers. However, we have yet to figure out a few details and put it in the roadmap. This is probably going to be for Q1 2025 but it is not scheduled yet.

[odow]

Seems like progress is being made: https://github.com/Pyomo/pyomo/pull/3478

cc @tvignon-artelys

[amontoison]

We also need the shared libraries in Julia.

[odow]

Yes, but we'll just rip from the Python ones once they're published. No need to duplicate work.

[tvignon-artelys]

Hi, Yes we are making progress, the Python package including the shared libraries will be released along with the next Knitro release, scheduled for early June.

[odow]

I see there is https://pypi.org/project/knitro/#files

I'll need to figure out if the Artifacts.toml can use a .whl as a synonym for .zip, otherwise we' probably need to discuss some options.

[amontoison]

@odow For MKL_jll.jl / OneAPI_support_jll.jl, I am using the .whl on Yggdrasil if it can help.

[odow]

I don't want to use Yggdrasil because the code is not available under an open-source license.

I wanted to follow Gurobi_jll and Xpress_jjll, but I hit this error:

julia> import KNITRO_jll
     Failure artifact: KNITRO
     Failure artifact: KNITRO
ERROR: Unable to automatically download/install artifact 'KNITRO' from sources listed in '/Users/oscar/.julia/dev/KNITRO_jll/Artifacts.toml'.
Sources attempted:
- https://pkg.julialang.org/artifact/ab80340e5963eb728e5f19a503af52e03ada648c
    Error: RequestError: HTTP/1.1 404 Not Found while requesting https://pkg.julialang.org/artifact/ab80340e5963eb728e5f19a503af52e03ada648c
- https://files.pythonhosted.org/packages/76/66/936edcd2255055cf6c8a060d1beeb8ce543ef078dceb631caad22a20002a/knitro-15.0.0-py3-none-macosx_13_0_arm64.whl
    Error: This does not appear to be a TAR file/stream — invalid version string for tar file: "*\n". Note: Tar.jl does not handle decompression; if the tarball is compressed you must use an external command like `gzcat` or package like CodecZlib.jl to decompress it. See the README file for examples.

[amontoison]

I didn't suggest to use Yggdrasil :) I just said that we have receipts to handle .whl in the GitHub repository of Yggdrasil and they can be relevant for us. I did the following build_tarballs.jl for v15.0 but compared to the precompiled binaries that we can download from the website of Artelys, we don't have the binaries for checking the license (license manager). My previous local build of KNITRO_jll.jl for v14.1 was with the additional binaries.

using BinaryBuilder, Pkg

name = "KNITRO"
version = v"15.0.0"

sources = [
    FileSource("https://files.pythonhosted.org/packages/13/3f/54953373ee3b631640b33b5d4bdb0217bdb1f8514b9374b08348e098ea2a/knitro-15.0.0-py3-none-win_amd64.whl",
               "1f9049939f6ab0d0350a21546399e0732423e9a1699c42541a4a41c96255a066"; filename="knitro-x86_64-w64-mingw32.whl"),
    FileSource("https://files.pythonhosted.org/packages/76/6e/ffe880b013ad244f0fd91940454e4f2bf16fa01e74c469e1b0fb75eda12a/knitro-15.0.0-py3-none-manylinux1_x86_64.whl",
               "46f6158098e39be9b7dcb73c1be0ebd14792490c2b6a2c12e9fef6a9531cf93a"; filename="knitro-x86_64-linux-gnu.whl"),
    FileSource("https://files.pythonhosted.org/packages/76/66/936edcd2255055cf6c8a060d1beeb8ce543ef078dceb631caad22a20002a/knitro-15.0.0-py3-none-macosx_13_0_arm64.whl",
               "94ebc22e7b12dca3030cca917b34d6f5177cad97f4213b76159f823d9dc2f6d4"; filename="knitro-aarch64-apple-darwin20.whl"),
]

# Bash recipe for building across all platforms
script = raw"""
cd $WORKSPACE/srcdir
unzip -d knitro-$target knitro-$target.whl

if [[ ${target} == *x86_64-w64-mingw* ]]; then
    install -Dvm 755 knitro-${target}/knitro/lib/* -t "${libdir}"
    install_license $WORKSPACE/srcdir/knitro-${target}/knitro-*.dist-info/LICENSE
fi
if [[ ${target} == *x86_64-linux-gnu* ]]; then
    install -Dvm 755 knitro-${target}/knitro/lib/* -t "${libdir}"
    install_license $WORKSPACE/srcdir/knitro-${target}/knitro-*.dist-info/licenses/LICENSE
fi
if [[ ${target} == *aarch64-apple-darwin* ]]; then
    install -Dvm 755 knitro-${target}/knitro/lib/* -t "${libdir}"
    install_license $WORKSPACE/srcdir/knitro-${target}/knitro-*.dist-info/licenses/LICENSE
fi
install -Dvm 755 knitro-${target}/knitro/knitroampl/* -t "${bindir}"
"""

# These are the platforms we will build for by default, unless further
# platforms are passed in on the command line
platforms = [
    Platform("x86_64", "linux"; libc="glibc", cxxstring_abi="cxx11"),
    Platform("x86_64", "windows", cxxstring_abi="cxx11"),
    Platform("aarch64", "macos", cxxstring_abi="cxx11"),
]

# The products that we will ensure are always built
products = [
    LibraryProduct(["libknitro", "knitro"], :libknitro),
    LibraryProduct(["libiomp5", "libiomp5md", "libomp"], :libopenmp),
    ExecutableProduct("knitroampl", :amplexe),
]

# Dependencies that must be installed before this package can be built
dependencies = [
    Dependency(PackageSpec(name="CompilerSupportLibraries_jll", uuid="e66e0078-7015-5450-92f7-15fbd957f2ae"))
]

# Build the tarballs, and possibly a `build.jl` as well.
build_tarballs(ARGS, name, version, sources, script, platforms, products, dependencies; julia_compat="1.6")

-> https://github.com/amontoison/KNITRO_jll.jl

I think we should coordinate with the KNITRO developers to provide us the tarballs generated by a local build of BB. We will all gain time to do that.

[odow]

@eminyouskn currently the .whl builds work, but they're a bit clunky for us to use. Is it possible for you to host a .tar.gz or .tar.bz2 of the same files in a public repo?

This is what Gurobi/Xpress do for conda: https://anaconda.org/Gurobi/gurobi/files

Then we can properly version the upstream KNITRO version and use the correct headers, etc.

[eminyouskn]

I will see with the team internally if it is possible to do this. By the way, we can also use pip download to get the .whl directly. Would it be possible to call it using Conda.jl?

[odow]

I want to create a KNITRO_jll.jl package, like Gurobi_jll.jl: https://github.com/jump-dev/Gurobi_jll.jl

This allows us to list and version binary artifacts: https://github.com/jump-dev/Gurobi_jll.jl/blob/master/Artifacts.toml

To use Artifacts.toml it must be a .tar.gz or .tar.bz2.

We cannot assume that users have pip installed. (We could for CI, but I'd rather something that works for general users so that they have to supply only the license, just like Gurobi.)

[amontoison]

@odow Note that the license manager is not in the .whl files. How do you plan to handle that?

[mgabay]

@amontoison there is no need for anything else than Knitro libraries. The libraries handle the license check. The license server is for the users deploying floating licenses. It is not a component for every user, only for the IT admins.

[odow]

Also: no support for x86 macOS? My daily driver 2018 MBP is sad 😢

[amontoison]

Only on the website of Artelys (download section).

[mgabay]

Sorry @odow , we're phasing out the support for Mac x86. Knitro 15.0 is the last version which will support it.

[eminyouskn]

@odow, as I mentioned last time, if you want to access the .whl file directly without using the exact URL, you can do so via Conda.jl. It comes with pip without needing to be installed on your computer. For example, the following script will download the Knitro wheel associated with the current system and the specified version:

using Conda

Conda.pip_interop(true)
Conda.pip("download", "knitro==15.0.0")

whl_file = filter(file -> endswith(file, ".whl") && contains(file, "knitro"), readdir("."))[1]

[odow]

I don't what to use the .whl file. I want to use Julia's Artifact mechanism, which requires a .tar.gz or a .tar.bz2.

If we bundle some way of installing KNITRO into KNITRO.jl's deps/build.jl, then users will download and install KNITRO every time we make a new release of KNITRO.jl. Requiring something like Conda means we need to add it as a dependency, which increases the cost for all users. We're also trying to move away from deps/build.jl.

There are a three options to move forward:

1. We provide README instructions for people for how they can use Conda to manually download the binaries and set an environment variable.
2. You (Artelys) provide a .tar.gz or .tar.bz2 download somewhere public. (Either on the official website, or on Conda, like Gurobi/Xpress/et al.)
3. We (JuMP) take your pip binaries, re-encode them to TAR, and then re-host them in a jump-dev/KNITRO_jll.jl repository. This is effectively what @amontoison was suggesting.

Gurobi chose (2). And it is working reeeeeally well. We used to get semi-regular installation questions. And now they have just stopped. When Gurobi release patch versions, it's a quick 2 minute job to publish a new Julia version of Gurobi_jll.jl. We don't need to make a whole new release of Gurobi.jl.

Xpress chose a mix of (2) and (1). We have Xpress_ll.jl that we could make work out of the box, but they wanted some manual user intervention.

My preference is (2). We can also do (3), subject to your agreement, since you'd need to okay us hosting your binaries.

If there is no agreement, we can do (1). But we won't be automating it for users via deps/build.jl.

[eminyouskn]

@odow thank you for the detailed explanation. I was considering a way of doing the third option. From the Python wheel (using pip), you create the tar (which I don't know yet how to do).

Let me confirm with the team that this option is correct and let you know.

[odow]

From the Python wheel (using pip), you create the tar (which I don't know yet how to do).

curl https://files.pythonhosted.org/packages/76/66/936edcd2255055cf6c8a060d1beeb8ce543ef078dceb631caad22a20002a/knitro-15.0.0-py3-none-macosx_13_0_arm64.whl -o tmp.zip
unzip tmp.zip
tar -cjf apple-darwin-aarch64.tar.bz2 knitro

[odow]

@eminyouskn I invited you to https://github.com/jump-dev/Knitro_jll.jl. It's currently private, but it's an example of what the code would be.

We'd change https://github.com/jump-dev/Knitro_jll.jl/blob/main/Artifacts.toml with the proper URLs (currently, they're in my private drive. The link might work for you or it might not. I'll remove the binaries before making Knitro_jll.jl public)

Then in KNITRO.jl, we do:

module KNITRO
import Knitro_jll: libknintro
end

and that's it. Julia's package manager will take care of the rest.

[eminyouskn]

@ofow thanks for the invite. I will take a look at it now.

[odow]

There's also a question of whether you want it to be KNITRO_jll or Knitro_jll.

Early history was Knitro.jl, but it got renamed in https://github.com/jump-dev/KNITRO.jl/commit/bf1a59dd4faef0da6302bd8e2bfaf0d173a9380b to KNITRO.jl to match the name when it was at Ziena.

[eminyouskn]

I think we can keep the name we have now.