Migration to v1

[AndrewBelt]

Time to start discussing the migration to Rack v1 and the changes required in this repo!

1. When the v1 API is stabilized, we should make a new branch v1 in this repo. Yes, there's already a v1 branch, but it was foolishly named years ago, so we'll just remove it.
2. Instead of writing/editing manifest files ourselves, we should only approve them from plugin developers. We should help them out if needed. JSON is not a skill that everyone has, and we have certain guidelines for some of the properties. Luckily, testing your plugin with Rack already checks most of the plugin.json like invalid JSON or some bad properties. When approved, we'll simply add their repo to repos/, run a script that copies repos/MyPlugin/plugin.json to manifests/MyPlugin.json, and commit.
3. Later, after Rack has a command line flag to dump module screenshots, we should add them to this repo in screenshots/. Err, maybe not. An envelope calculation says that it'll be over 100MB if screenshots are at 200% scale (~100KB/screenshot * 1000 modules). This might still be okay though.
4. Because library manifests will be identical to repo manifests, the following properties need to be rethought.
   • repoVersion: We can just read repos/MyPlugin/plugin.json version for this, no longer needed
   • latestVersion: This will just be version since it's in the plugin.json format.
   • productId: This will be handled via slugs by the server. It can be removed.
   • buildTimestamp: Not sure where we should put this. Ooo, I should just generate it on the server by reading the package file timestamp.
   • status: Can be computed by the server as well.
   • Okay, so all of these are handled. Fantastic.
5. The server API will handle requests for both v1 and v0.6 plugins (at least for a while), via a new API path or query.

All other operations will probably be handled the same way. Can anyone think of anything I'm missing?

[cschol]

So the plugin developer submits a PR for their updated manifest, which we will merge if it's approved? Do I still update the repo to the correct SHA? It would actually be nice if that SHA was in the manifest and could be approved/used for update of repo that way. Some developers provide the SHA they want in the issue, but sometimes they don't and I just assume it's the HEAD.

[AndrewBelt]

PRs are too complicated. "Plugin threads" have worked well IMO. Plugin developers should announce an update as usual in their plugin saying something like "MyPlugin v1.0.0 is available at abc123" or "the HEAD of my master branch", and we can update the their submodule as usual. The only additional step is to review their plugin.json and move it to manifests/MyPlugin.json (with a script). We don't to edit the manifest ourselves anymore.

Should I go ahead and create a v1 branch here?

[cschol]

OK. That makes sense and I agree has worked well. I don't see any reason why not for the v1 branch.

[AndrewBelt]

Okay, added https://github.com/VCVRack/library/tree/v1. Run python3 scripts/collect_manifests.py to copy all plugin.json files to manifest files. Currently only Fundamental is updated (and its version is a lie, Fundamental v1.0.0 isn't released yet). I left all the repos intact because it doesn't harm anything. Since they don't have a plugin.json file, they won't be scanned with my build script. You can use the intact repos to update to v1 versions.

[AndrewBelt]

Solved the screenshot issue. https://community.vcvrack.com/t/rack-v1-development-blog/1149/466

[AndrewBelt]

Made announcement in https://community.vcvrack.com/t/adding-your-v1-plugins-to-vcv-plugin-manager/3231

[cschol]

Here are a few thoughts on the overall migration process:

• The Library Team is only migrating open-source plugins. Correct?
• Every open-source plugin must have an (open/closed) issue in the library repository to be migrated. This issue will be used for future updates as well (reopen).

Having 1 issue per plugin would allow progress tracking of the migration with GitHub Project Boards for a v1 migration project, which would be a lot easier than what I did last time by manually editing an issue to keep track of plugin migration status to v0.6. The Library team would drive the overall migration with the Project Board. The Repair team assists with PRs for individual modules that need it.

I am envisioning 3 states per plugin:

1. To be migrated to v1 branch (all open source plugins currently in master)
2. Migrated to v1 branch (has plugin.json and v1 compatible branch)
3. Done (builds against Rack 1.0 API and runs without major defect, e.g. crash on load)

Example scenario: modular80 is in state 1 and has a v1 branch with a plugin.json available. The author notifies in the appropriate plugin issue. I will move it to state 2 by integrating it in the v1 branch (copy plugin.json, update to v1 compatible submodule branch). It builds successfully against the current Rack 1.0 API and loads. I move it to state 3. Now, a late breaking Rack API change is required (happened last time), which will move it back to state 2. The author fixes the code (or the Repair Team provides a PR to the author; it gets integrated) and we move back to 3.

What do you think?

[MarcBoule]

I would actually be ready to start my requests to add our plugins to the v1 library, but I was just waiting so as to avoid as much as possible oscillating between states 2 and 3 until API stability, to keep things simpler for you guys; but if ever you would like some more plugins to be added to v1 library for testing workflows, or whatever other reason, just let me know and I'll trigger my issues to get our things into v1 since I'm at state 3 now :-). Cheers!

[AndrewBelt]

The Library Team is only migrating open-source plugins. Correct?

Yes. I'll handle updates to freeware and commercial modules through email.

GitHub Project Boards

I've enabled Projects. Feel free to use this feature. Your 3 states look fine, can't think of any more states.

State 3 should also require that screenshots with ./Rack -p works and that the plugin can be displayed in the Module Browser without crashing.

[cschol]

Thank you!

Project board for v1 migration is available at: https://github.com/VCVRack/library/projects/1

If a plugin is available for v1 should we begin the migration now or hold off for a while until the API is more finalized?

[AndrewBelt]

If you're asking about migrating plugin source code itself, wait until after v1 API is stable. If you're asking about adding current "early" migrated v1 plugins to the v1 library branch, go ahead.

[cschol]

Yeah, I was referring to early migrated v1 plugins.

I'll keep the issues v0.6 vs v1 straight in the project boards. You only need to process issues in state "Plugin build update" in the v0.6 library maintenance project board, as usual. The automation will handle the state in that board.

A v1 update in an issue is not automated right now and after integration in the v1 branch, I will close the issue again. These will never move to the "Plugin build update" state in the v0.6 board.

[AndrewBelt]

Changed state names of projects to describe what each state needs.

[cschol]

As I am starting to integrate v1 plugins that have been updated, I will close the issue after moving to the Needs Build Update state. I assume there will be no official v1 builds until right before the launch date. This will clean up the state for v0.6 plugins on the other project board. Once we officially phase out v0.6 building the issue will stay open until you have rebuilt the plugin with the build system. Does that make sense?

[AndrewBelt]

Makes sense. No v0.6 builds will be made after Rack v1 release, and no v1 builds will be made before.

[squinkylabs]

For a plugin dev like myself, what needs to be done to get v1 plugins in the mangers? Do I just add to my existing (0.6) issue and say "hey, build me some v1 plugins from sha xyz"? Do I start a new issue for v1, and continue to use my old one for 0.6?

[cschol]

@squinkylabs Exactly. Use the same, existing issue for both (for right now).

[AndrewBelt]

The v1 plugin server API is ready. I'm just a couple hours away from calling the client/server stable and working on v1 builds.

Effective now, all plugins distributed through the VCV Plugin Manager must follow the VCV Plugin Ethics Guidelines. Existing plugins are grandfathered in.

I'm also considering renaming the VCV Plugin Manager to the "VCV Library". I might do it when Rack 1.0 releases, or I might do it when the new Plugin Manager releases (with module pages, screenshots, etc).

[cschol]

@AndrewBelt When you start to build plugins for v1, can you check your build process for plugins that build libsamplerate as a dependency, e.g. Fundamental?

I have trouble crosscompiling it on Linux for Windows. This is also an issue with some open-source plugins, which use libsamplerate, e.g. FrozenWasteland and I want to ensure that there are no issues in the plugin itself.

make dep fails when I execute it with the mingw32 compiler:

checking for C compiler default output file name... a.exe
checking for suffix of executables... .exe
checking whether we are cross compiling... configure: error: in `/home/cschol/src/library-v1/repos/Fundamental/dep/libsamplerate-0.1.9':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details
Makefile:16: recipe for target 'dep/lib/libsamplerate.a' failed
make: *** [dep/lib/libsamplerate.a] Error 1

Does this work for you?

Another thing I noticed is that if a make dep is required, on Linux you can just execute make and it will do a make dep first automatically. This does not seem to work on Mac or Windows platforms and an explicit make dep would be required first.

[AndrewBelt]

I never learned how to use ./configure's cross-compiling ability, but that could be an option, or I could set up VMs to actually build on each OS rather than cross-compilers in Docker.

[cschol]

The libsamplerate build dependency is not new for the Fundamental plugin, right? Are you building the plugins on their respective platforms (e.g. on Windows) or cross-compiling for the plugin manager in Docker on Linux?

[AndrewBelt]

I custom-build a lot of plugins. Fundamental, Host, VCV Recorder are some that have too many system deps or need to run code during the build process.

[cschol]

I took a look at the crosscompilation. Adding the "--host" option and a reference to mingw32's ar to configure in the Fundamental Makefile will allow successful cross-compilation of Fundamental on Linux:

Relevant Makefile excerpt:

$(libsamplerate):
        cd dep && $(WGET) http://www.mega-nerd.com/SRC/libsamplerate-0.1.9.tar.gz
        cd dep && $(UNTAR) libsamplerate-0.1.9.tar.gz
        cd dep/libsamplerate-0.1.9 && AR=x86_64-w64-mingw32-ar $(CONFIGURE) --host=x86_64-w64-mingw32
        cd dep/libsamplerate-0.1.9/src && $(MAKE)
        cd dep/libsamplerate-0.1.9/src && $(MAKE) install

This would resolve a huge headache since more and more plugins are compiling libsamplerate like Fundamental does (e.g. FrozenWasteland).

This is just a proof of concept. The additional configure options/definitions should probably be moved to a "Rack-level" .mk file.

Mac platform has the same problem. If this looks reasonable I will take a look at that one, too.

[cschol]

Successful Mac crosscompile of Fundamental on Linux. Similar change:

 $(libsamplerate):
        cd dep && $(WGET) http://www.mega-nerd.com/SRC/libsamplerate-0.1.9.tar.gz
        cd dep && $(UNTAR) libsamplerate-0.1.9.tar.gz
-       cd dep/libsamplerate-0.1.9 && $(CONFIGURE)
+       cd dep/libsamplerate-0.1.9 && RANLIB=/home/cschol/src/osxcross/target/bin/x86_64-apple-darwin17-ranlib $(CONFIGURE) --host=x86_64-apple-darwin
        cd dep/libsamplerate-0.1.9/src && $(MAKE)
        cd dep/libsamplerate-0.1.9/src && $(MAKE) install

[AndrewBelt]

I'll look into --host in a bit.

I will write the build system today. The build process will claim the responsibility of copying repos/<plugin slug>/plugin.json to manifests/<plugin slug>.json, since it will use the manifests folder as its reference for what repos need to be built. In other words, if a plugin manifest for a certain version exists in manifests/, then the build exists in the VCV Plugin Manager server.

[cschol]

OK. So I won't run the collect manifest script anymore when integrating new versions, correct?

[AndrewBelt]

Correct, updating the submodule is what should trigger the build, which triggers the manifest collection.

[cschol]

Cool. DrumKit, SynthKit, and CharredDesert are the first plugins that just have the submodule updated (without copying the plugin.json to manifests). Maybe that will help you test your build scripts.

[AndrewBelt]

Switched the default branch of this repo to v1.

First builds are ready for most of the repos.

[AndrewBelt]

It is now no longer worth it to review v0.6 builds. My time is better spent running automated v1 builds.