Better libary / custom boards handling

[CrazyRedMachine]

Changes have been made so that LUFA could be enabled/disabled by selecting a board type after running the new "install.py" python script.

While this enhances the workflow, there is still room for improvement as the Arduino IDE has a "board manager" which is meant for new boards installation (as used by Adafruit and other third party boards manufacturers), which would allow the end user to install the new boards without having to install python or even use the commandine at all.

I've got a PoC working on this branch : https://github.com/CrazyRedMachine/Arduino-Lufa/tree/library

Workflow from a fresh arduino IDE install

Install LUFA library

• go on https://github.com/CrazyRedMachine/Arduino-Lufa/tree/library , download ZIP
• in arduino IDE, go to Sketch -> include library -> Add ZIP library..., select the zip file downloaded in step 1.

Library is now installed

Install custom boards

• in arduino IDE, add a new URL to File -> Preferences -> Additional Boards Manager URLs : https://github.com/CrazyRedMachine/Arduino-Lufa/raw/library/custom_boards/package_arduino-lufa_index.json

• Restart Arduino IDE
• in arduino IDE, go to Tools > Board > Boards Manager, wait for the list to load then type LUFA in the search box

• Click install

Boards are now in the board list

From PoC to prod

• (WIP) bump the library to 1.5 format by adding a library.properties file
• create a "release" from github menu so that LUFA can appear directly in the arduino library manager rather than having to use the "add ZIP library"
• change package_arduino-lufa_index.json contents to reflect the Palatis repo rather than my fork
• update README to reflect the new install procedure

also a nice to have feature would be a generate.py script to automatically generate a new custom boards zip with new version number and the relevant data to be appended to the index.json... but i'm not sure yet if that's a good idea because it might break stuff and sometimes making changes to that might require further human intervention anyways...

[CrazyRedMachine]

and here is a quick write down of what was required to change from the inital repo :

• the LUFA subfolder has to be populated with the files rather than being declared as a submodule (this is important so that the github "download zip" feature contains every required file, and so that arduino library manager can also retrieve the correct versions zip from the tags)
• a package_<libname>_index.json file has to be created and hosted somewhere so that it can be added to the custom board manager URLs list
• a custom boards zip folder has to be created and hosted somewhere so that the index.json can reference it (the contents are the same as generated by the install.py script, but with avr renamed to the lib name in the root of the zip) (e.g. everything we have in <arduino install path>/hardware/arduino-lufa/avr/cores/* will be found in the zipfile in arduino-LUFA-1.8.3/cores/* )

Note I removed some subfolders from the avr/ path as I think they weren't technically needed (the different bootloaders etc can still be flashed without being in LUFA mode as a board type...)

[iFreilicht]

Awesome, and a great write-up, too! I think the PoC is great, but maybe we can actually simplify it. Some ideas:

1. Could we move LUFA from being a library to being distributed with the board manager? I think it could be inside avr/cores/arduino, which is included by default. You could then #include <LUFA.h>, no need to add the library. This would make sense, as LUFA is basically replacing part of the Arduino core in this usecase.
2. The package_lufa_index.json could just be hosted at the root of the project, so it would be https://github.com/Palatis/Arduino-Lufa/raw/master/package_arduino-lufa_index.json, which seems quite ok as a URL.
3. The script seems like a nice idea. Ideally, we could maybe even do this automatically with GitHub actions? But you're right, it is not required. I'd rather just have a RELEASE.md file that explains what to do when releasing for now. This project is very low-traffic currently, so doing releases manually seems fine to me. However, I don't want to stop you taking a crack at it. If you can figure out how to make that work, it would certainly make our lives easier in the long run and could be a nice learning experience.

[CrazyRedMachine]

1. Ooh I actually love this, and indeed it makes sense since we cannot include LUFA without using the board, and we cannot use the boards without including LUFA... and it also avoids having to do the "make lufa available through lib manager" process (and also this way we don't have to waste space by including LUFA library which includes the boards zip, and then download the boards zip again from the url to install them...), I will quickly try it before going to bed!
2. haha I actually did it this way on a previous commit and then moved the file when I bumped the library format in order to keep everything board manager at the same place, so yea let's revert this
3. oh, I'm not familiar with github actions at all but might be worth to have a look (I'm just worried the script only helps in cases where doing it manually isn't much work anyways... )

[CrazyRedMachine]

made a new PoC on "board-only" branch and http://github.com/CrazyRedMachine/Arduino-Lufa/raw/board-only/package_arduino-lufa_index.json as url (beware you might have cache issues as I didn't change the version number, you should go to the board manager and uninstall the boards, then remove the old "library" url from preferences, restart the IDE, add the new "board-only" url, restart, then try to add... if it gives you "CRC error" just restart the IDE and then it should work)

I didn't put the LUFA folder in avr/cores/arduino but rather in avr/libraries/ , and #include <LUFA.h> works as intended :)

moreover, the dualserial example sketch does appear in File > Examples > Arduino-LUFA under the board section, which is awesome too! Thanks to your suggestion the workflow is now very clean :D

Ok so I think the new course of action should be :

• keep everything the zip should contain as is on the repo (we don't include a custom_boards folder and we don't host the zipfile inside the repo. ~~(basically the repo structure looks similar to the current Palatis branch, but with a new library.properties as well to bump to the 1.5 format)~~ see edit.
• create a git release to have a versioned .zip, that will work as a custom boards zip file (we put package_arduino-lufa_index.json export-ignore inside a .gitattributes file to exclude the json from being included in the zip).
• edit the index.json to reflect the new size and hash of the release zipfile, and point to the release zipfile url

EDIT: after further testing, some changes had to be done to the repo structure, you can see a working example there : https://github.com/CrazyRedMachine/Arduino-Lufa/tree/new-workflow

new board url is https://github.com/CrazyRedMachine/Arduino-Lufa/raw/new-workflow/package_arduino-lufa_index.json and I could successfully create release zips 0.9 (with missing files :p) and 0.93 (fixed), and both versions referenced in the json so that I could "update" via the board manager and everything is working fine

[iFreilicht]

Holy cow this is awesome! I just tried it on a fresh install, it's super easy and works perfectly! This is some incredible work, should make LUFA way more accessible for those who need it. Hats off! To me this seems like the perfect occasion to bump the version to 1.0 as well.

I'm not sure how to integrate this properly now, because we can't really put all of this into a PR, right?

@Palatis, do you think you could add @CrazyRedMachine and me as collaborators to this repository so that we can approve PRs and create releases for this?

[CrazyRedMachine]

glad it's working :D I've updated the readme in the new-workflow branch

we can make a PR with all the changes, but indeed in order to be able to use it we then need to create a "release" from the github menu once the PR is merged, which only Palatis can do at the moment.

I will still prepare the branch as much as possible and open a PR, with instructions on what to do once it is merged :)

[fluffymadness]

nice work, would be interesting to somehow get this into platform io @CrazyRedMachine Btw, thanks for sharing your descriptor switching code for the LufaHybridFightstick that was extremely useful. And in case you have use for it, I was fighting with that one the last few days: https://github.com/fluffymadness/tinyusb-xinput (for RP2040)

[CrazyRedMachine]

@fluffymadness thanks, never used platformio myself but I guess it must support custom boards somehow, so there should be a way too... you can hit me up on my forked repo if you need help with it

welcome about descriptor switching (if you check the PCFX branch you can even see how to switch during runtime), haven't touched rp2040 yet but that's awesome you made an xinput firmware for it too, will surely be useful when I get my hands on that board :D

[sanjay900]

@fluffymadness seems reasonably straightforward to get it into platform.io I just took the existing framework-arduino-avr platform package, applied the Arduino-Lufa changes manually, and then pushed that to git. This is at https://github.com/sanjay900/framework-arduino-avr.git. This is however using the latest master version of LUFA as i was originally using that for an old project that I am porting across to platform.io.

Putting the following into the env section of my platformio.ini file was enough to be able to use LUFA from within platform.io

platform_packages =
    framework-arduino-avr @ https://github.com/sanjay900/framework-arduino-avr.git

[KenwoodFox]

@fluffymadness seems reasonably straightforward to get it into platform.io I just took the existing framework-arduino-avr platform package, applied the Arduino-Lufa changes manually, and then pushed that to git. This is at sanjay900/framework-arduino-avr.git. This is however using the latest master version of LUFA as i was originally using that for an old project that I am porting across to platform.io.

Putting the following into the env section of my platformio.ini file was enough to be able to use LUFA from within platform.io

platform_packages =
    framework-arduino-avr @ https://github.com/sanjay900/framework-arduino-avr.git

Im curious as I'm also trying to get LUFA working in platformio, whats a complete config for you look like? Thanks!

[sanjay900]

@fluffymadness seems reasonably straightforward to get it into platform.io I just took the existing framework-arduino-avr platform package, applied the Arduino-Lufa changes manually, and then pushed that to git. This is at sanjay900/framework-arduino-avr.git. This is however using the latest master version of LUFA as i was originally using that for an old project that I am porting across to platform.io. Putting the following into the env section of my platformio.ini file was enough to be able to use LUFA from within platform.io

platform_packages =
    framework-arduino-avr @ https://github.com/sanjay900/framework-arduino-avr.git

Im curious as I'm also trying to get LUFA working in platformio, whats a complete config for you look like? Thanks! My current project that i am using this on is open source, https://github.com/sanjay900/Ardwiino/ Hopefully enough is in there to answer your questions.

[KenwoodFox]

@fluffymadness seems reasonably straightforward to get it into platform.io I just took the existing framework-arduino-avr platform package, applied the Arduino-Lufa changes manually, and then pushed that to git. This is at sanjay900/framework-arduino-avr.git. This is however using the latest master version of LUFA as i was originally using that for an old project that I am porting across to platform.io. Putting the following into the env section of my platformio.ini file was enough to be able to use LUFA from within platform.io

platform_packages =
    framework-arduino-avr @ https://github.com/sanjay900/framework-arduino-avr.git

Im curious as I'm also trying to get LUFA working in platformio, whats a complete config for you look like? Thanks! My current project that i am using this on is open source, sanjay900/Ardwiino Hopefully enough is in there to answer your questions.

Awesome thanks so much I'll try and see what i can extract from this haha, looks like a lot! I was thinking more just a pre-run script and a lib deps with specific branch :3 I'm starting to think the LUFA lib is a bit more involved.

[sanjay900]

If its easier i can put you together a minimal example probably? That project is only massive because its a single project for a bunch of different microcontrollers at once haha

On the platform.io side, it really is as simple as putting the following in your ini file

platform = atmelavr
board = micro
framework = arduino
platform_packages = 
	framework-arduino-avr @ https://github.com/sanjay900/framework-arduino-avr.git

And then you can just use LUFA as you would

[KenwoodFox]

If its easier i can put you together a minimal example probably? That project is only massive because its a single project for a bunch of different microcontrollers at once haha

On the platform.io side, it really is as simple as putting the following in your ini file

platform = atmelavr
board = micro
framework = arduino
platform_packages = 
	framework-arduino-avr @ https://github.com/sanjay900/framework-arduino-avr.git

And then you can just use LUFA as you would

Yeah haha i noticed! Im so impressed by the size!

Im sure a minimal project example would be very appreciated for lots of beginners like me, just those four lines looks really good though! But, I'm not sure how they include LUFA in the platformio way, should i still be using git submodules or is there a way for PIO to handle it via lib-deps?

[sanjay900]

@KenwoodFox https://github.com/sanjay900/LufaPlatformioExample Heres a minimal example, LUFA is included as part of the arduino core itself, so you don't need to add it to lib deps, or use any submodules, its all included within that framework-arduino-avr project thats being linked in. Its just a copy of the VirtualSerial example from the LUFA repo

[KenwoodFox]

@KenwoodFox sanjay900/LufaPlatformioExample Heres a minimal example, LUFA is included as part of the arduino core itself, so you don't need to add it to lib deps, or use any submodules, its all included within that framework-arduino-avr project thats being linked in. Its just a copy of the VirtualSerial example from the LUFA repo

This is really very concise and helpful, thank you very much! Ill be sure to share your example project around!

[sanjay900]

@KenwoodFox were you able to get it all working? Let me know if you need anything else

[kaneda-fr]

Hi,

Thanks for the sample code for PlatformIO, I managed tu upload it on my pro micro and it shows as LUFA CDC Demo.

Now it seems I can no longer upload new code to the board (get an avrdude error: programmer is not responding). Is it due to having installed the LUFA driver or could it be something else ?

Thanks,

[CrazyRedMachine]

yes, you need to use the reset button to upload new code, as lufa doesn't contain the code to trigger a reset by opening the serial interface with specific baudrate

[sanjay900]

I suppose it probably would have been a good idea to include that in the sample, but I put that sample together pretty quickly and never really thought to include listening to the baudrate changes for jumping to bootloader

I myself use this with hid devices, and use pre scripts to send USB control requests to jump to bootloader mode instead of the baudrate logic, and the nice thing is platform io does make implementing that sort of thing quite easy.

[CrazyRedMachine]

since #23 has been merged, no need to keep this open anymore :) thanks!

[KenwoodFox]

Awesome thanks! hopefully we see this working in pio too,

[sanjay900]

My pio examples should still be working