Support for custom libraries when using `sketch.yaml`

[dmohns]

trafficstars

Describe the request

In a project I am currently using different "kind" of libraries.

1. Versioned libraries available in via Arduino Library Manager
2. Non-versioned libraries available via their source code on Github
3. Non-versioned libraries available via their source code in local subfolder folder of my project (zipped or raw)

To ease collaboration on the project I would like to use a Sketch Project File and provide a set of dependencies that are confirmed working.

Unfortunately, the "build profiles" feature only seems to support 1. (versioned libraries available in Arduino Library Manager). Scope of this request is to add functionality to supply Github and local folder libraries to the libraries: attribute on sketch.yaml (in some fashion).

Describe the current behavior

Currently, I have been able to compile my project using

arduino-cli compile --fqbn "<FQBN>" --libraries=Library/

However, if I tried to use --profile and --libraries the compile command fails

You cannot use the --libraries flag while compiling with a profile.

Arduino CLI version

arduino-cli Version: nightly-20230329 Commit: b69ed33 Date: 2023-03-29T01:34:59Z

Operating system

macOS

Operating system version

12.6.3

Additional context

Additional requests

• https://github.com/arduino/arduino-cli/issues/2133#issuecomment-1881077811
• https://github.com/arduino/arduino-cli/issues/2133#issuecomment-2261340285
• https://github.com/arduino/arduino-cli/issues/2133#issuecomment-2244098926
• https://github.com/arduino/arduino-cli/issues/2763
• https://arduino.stackexchange.com/questions/17802/is-it-possible-to-package-a-sketch-with-dependent-libraries-for-cross-platform-c
• https://forum.arduino.cc/t/local-libraries/1259854/20

Related

• https://github.com/arduino/arduino-cli/issues/108
• https://stackoverflow.com/questions/4705790/keeping-all-libraries-in-the-arduino-sketch-directory

Issue checklist

• [X] I searched for previous requests in the issue tracker
• [X] I verified the feature was still missing when using the nightly build
• [X] My request contains all necessary details

[umbynos]

Seems a duplicate of https://github.com/arduino/arduino-cli/issues/108

[jirutka]

Seems a duplicate of #108

It’s not a duplicate, #108 is just one of the ways it can be implemented (and more complicated than requested here).

[dmohns]

Yeah, just to clarify, this issue is about making sketch.yaml spec more flexible to support libraries from other sources than Arduino Library Manager.

One way that is backwards compatible would be

profiles:
  <PROFILE_NAME>:
    ....
    libraries:
      - <LIB_NAME> (<LIB_VERSION>)

to allow <LIB_VERSION> to be <LIB_VERSION_OR_URL> and take the form of VCS and local urls, for example

profiles:
  my_profile:
    ....
    libraries:
      - ArduinoIoTCloud (1.0.2)
      - HelloWorld (https://github.com/username/HelloWorld.git#v2.13)
      - HelloWorld2 (file://./libraries/HelloWorld2.zip)

Note: Above is just an example. There are other ways to implements this as well, like changing the schema of libraries to be a list of dicts rather list of strings. This would be more powerful and flexible but not be backwards compatible.

[eesoymilk]

I've been trying to install custom libraries through cli for ages now. Is there a workaround for this issue? The cli compile can never seem to find my custom lib in ~/Arduino/libraries.

[ubidefeo]

@eesoymilk can you be more specific? what are your custom libraries? libraries developed by you? do they conform to the libraries specification?

[eesoymilk]

@ubidefeo I'm writing my own library. This is the repo: eesoymilk/AS608-Fingerprint-Arduino.

I believe it does conform to the spec since running arduino-lint only shows 1 warning (no license).

I put this repo directly in my ~/Arduino/libraries of course. I was trying to compile it with arduino-cli and I have a sketch.yaml file right next to the targeted .ino file. I added my library's into the skectch.yaml file in hope that all things works (arduino-cli, neovim lsp, etc.):

libraries:
  - AS608Fingerprint (0.1.0)

However, this is the root cause why arduino-cli compile won't correctly compile. It trys to download my library from the Arduino Library Manager (DNE btw) even though my library is listed in arduino-cli lib list.

In the end, removing my library from sketch.yaml solved this problem and the lsp also works.

[dmohns]

I put this repo directly in my ~/Arduino/libraries of course. I was trying to compile it with arduino-cli and I have a sketch.yaml file right next to the targeted .ino file. I added my library's into the skectch.yaml file in hope that all things works (arduino-cli, neovim lsp, etc.):

libraries:
  - AS608Fingerprint (0.1.0)

However, this is the root cause why arduino-cli compile won't correctly compile. It trys to download my library from the Arduino Library Manager (DNE btw) even though my library is listed in arduino-cli lib list.

You would need to run arduino-cli lib install --git-url https://github.com/eesoymilk/AS608-Fingerprint-Arduino.git then arduino-cli compile should work. Which is very tedious to do if you have more than one external library. You can find an example how we created a Makefile to automate some of this here: https://github.com/EnAccess/OpenSmartMeter/blob/27ad33a244d4f95201a7676a6d8307f7c766d15e/Firmware%20code/smart_energy_meter/Makefile#L23-L33

What you describe is exactly the motivation for why this feature request exists.

But let's keep this discussion about the feature request itself rather than workarounds for individual situations.

[tyeth]

Just bumping this, keen on git urls (with refs) in the version string spot, and adding a related note in case @cmaglie / anyone implements this feature...

As mentioned at the end of #108, it would be nice if profiles could support inheritance from a common build profile (we have 60 common libs to list), and then also share these custom libraries linked via git refs too.

We have 9 git-based libs for users to install, two on non-default untagged branches (using sha ref), and a few build profiles (23 targets but I'm assuming user can switch FQBN target and still use a base build profile) hence the interest in this feature.

[egnor]

Yes, this feature continues to be high priority for me and probably many other users of arduino-cli.

As a note, I believe the workaround mentioned in https://github.com/arduino/arduino-cli/issues/2133#issuecomment-1881136157 no longer works (assuming it once did); when a sketch.yaml file is present, arduino-cli compile seems to disregard any libraries installed locally (via arduino-cli lib install or just plopped in ~/Arduino/libraries). This makes sense to keep things hermetic, but removes workarounds...

[hasselmm]

There also occur warnings when trying to "pin" 3rd-party platform libraries like ESP8266HTTPClient:

Downloading library [email protected]...
Library [email protected] not found...
Error initializing instance: Library 'ESP8266HTTPClient' not found

Arduino-cli still finds ESP8266HTTPClient later on. Still this is confusing. And yes, maybe it doesn't make much sense to pin platform libraries.