flowblade
flowblade copied to clipboard
jliljebl
Add USB jog/shuttle support to Flowblade
Initially, the Contour Design family of jog/shuttle devices is supported. This includes the ShuttlePRO v2, and the ShuttleXpress. There are also quite a few rebranded devices that are basically the ShuttleXpress under another name. Those devices are likely to work as well. I have attempted to make the USB and driver code extensible, to make it easier to add additional devices in the future.
The USB implementation is handled by PyUSB, which in turn depends on libusb version 1. Flowblade connects directly to the configured USB device and uses its own userspace drivers to connect/disconnect, and handle input events.
All USB features are off by default. To turn them on, a user has to go to Edit -> Preferences -> Jog/Shuttle, and then enable it. On that window, a driver must also be selected.
The core drivers themselves are in the usbhiddrivers module. However, a res/usbhid/ file contains XML files that specify which lower-level driver to use within Flowblade, some USB connection settings, and also map keys on the device to events.
Flowblade already has great support for user-mappable keys from the keyboard. However, in order to allow a USB device to trigger specific actions when its keys are pressed, a new targetactions module was created. The USB HID driver config XML files use the same "code" action names that the existing keyboard shortcuts feature uses. Hopefully we can unify these systems more over time.
Users can also configure their own custom key mapping by copying and editing a USB HID driver config file in their own local user folder. For example, the flowblade-trunk/Flowblade/res/usbhid/foo.xml file could be copied into ~/.local/share/flowblade/user_usbhid/foo.xml, and edited to taste.
By the way, I bought a Blackmagic Davinci Resolve Speed Editor USB control device, with the hopes of figuring out how to integrate it into Flowblade. That one looks like more of a challenge than the Contour Design ShuttleXpress, since it requires both USB input and output, and some sort of handshake to get the device to send input at all. Wish me luck!
Good luck 😊And thanks for your effort on this. I'll be sure to check it out. RegardsStevenSent from my Huawei tablet-------- Original message --------From: Nathan Rosenquist @.>Date: Sun, 20 Feb 2022, 22:55To: jliljebl/flowblade @.>Cc: Subscribed @.***>Subject: Re: [jliljebl/flowblade] Add USB jog/shuttle support to Flowblade (PR #1052) By the way, I bought a Blackmagic Davinci Resolve Speed Editor USB control device, with the hopes of figuring out how to integrate it into Flowblade. That one looks like more of a challenge than the Contour Design ShuttleXpress, since it requires both USB input and output, and some sort of handshake to get the device to send input at all. Wish me luck!
—Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you are subscribed to this thread.Message ID: @.***>
Oh! I almost forgot. You have to add a magic file to let the OS know that non-root users should be able to access the USB device.
On my system (Ubuntu 20.04), I added a file named /etc/udev/rules.d/90-flowblade.rules
The file has one line, with the following contents:
ATTRS{idVendor}=="0b33", ATTRS{idProduct}=="0020", MODE="0666"
You can add multiple lines to a file, one per device. Maybe we should publish one in some sort of example document?
Without this, Flowblade will not be able to access the device (unless you're running it as root, which is a bad idea obviously).
You can find the USB vendor ID and device ID using the lsusb command.
When I run lsusb on my system and the Contour Design ShuttleXpress is plugged in, it prints out (among other things), the following line:
Bus 001 Device 017: ID 0b33:0020 Contour Design, Inc. ShuttleXpress
That hex pair is the vendor ID : product ID.
lsusb -vd 0b33:0020 can then give you all sorts of other details about the device, which came in handy when developing the driver.
@dvdlvr I noticed that you played a large role in the keyboard event mapping code in Flowblade. If you review this code, you will see that it introduces a targetactions module, which has a fair amount in common with the keyevent and shortcuts modules. If this pull request does get accepted, it seems like we might want to make an effort to unify the key mappable actions in some way that spans regular keyboards and USB devices. There are TODO comments in the new code in this pull request that should give you a good sense of it.
Note that since the initial pull request a few days ago, I have made some major improvements to the USB subsystem.
The previous implementation of the USB HID subsystem used the pyusb library, in synchronous mode, which polled repeatedly for input.
This commit is a major refactoring of the USB subsystem, which brings several substantial improvements.
The pyusb dependency has been completely eliminated. Instead, USB connections are now implemented in terms of libusb 1.0, which was the underlying library that pyusb itself was using. The resulting code is a bit more verbose, but also provides a lot more control and flexibility. Most notably, the USB HID subsystem now performs asynchronous reads of the USB device, which is a major performance improvement. This was not possible with the pyusb library.
In the previous implementation, a GTK timeout handler was triggered every 5 milliseconds to perform a USB read, with a timeout of 1 millisecond. This actually worked reasonably well, but if you go look at the USB traffic with Wireshark, it was very chatty and was actually talking to the device that often.
In the current implementation, a GTK timeout handler is triggered every 10 milliseconds, which simply goes and checks whether any asynchronous USB transfers are ready to be read. This does not block or timeout, and happens instantly. Meanwhile, during setup, 10 USB transfers were set up to be in flight, being passed back and forth to send and receive messages, and events only occur when the device actually has new input to send. These transfers have no timeouts, so the chattiness is kept to a minimum. Having a queue with multiple transfers in flight also ensures that we capture every single update from the device, which improves the perception of update latency, and also improves reliability, especially with the shuttle.
Finally, the "endpoint_in" and "endpoint_out" XML tags in the USB HID driver config files now use the hex address of the endpoints, instead of numbers describing which endpoint it is counting from zero. The XML config files have been updated accordingly. Similarly, the "alternate" field has been removed, as libusb does not require that field, and neither do any of the devices we're likely to talk to.
I have added a document that describes how to set up Flowblade for jog/shuttle support as part of this pull request: https://github.com/ratherlargerobot/flowblade/blob/jog-shuttle/flowblade-trunk/docs/JOG_SHUTTLE.md
Not much to comment on due to my lack of experience with actual code, but maybe it would be worth mentioning in the main document, probably under interacting with the timeline section something along the lines of "Using Shuttle" or just "using external devices to edit" to allow for room for expansion, also probably could condense this better information wise
in both forward and reverse, and navigating frame by frame are right at your fingertips.
Turning the jog wheel to the right and left moves the playback location forward and backward, one frame at a time, respectively.
Turning the shuttle ring to the right starts out with playback in the forward direction, with increasingly faster speeds the farther you turn the shuttle.
Turning the shuttle ring to the left starts out with playback in the reverse direction, with increasingly faster speeds the farther you turn the shuttle.
Maybe this?
In both forwards and reverse, navigating frame by frame, right at your fingertips.
Turning the jog wheel to the right and left moves the playback forward and backward, one frame at a time respectively,
Turning the shuttle ring leftwards or rightwards starts playback in reverse or forward, and for faster or slower playback, turn the ring more in that direction.
Also I had a question for this function, so this would basically serve as a gradient on the speed of the playback, or will it basically be a remapped J, K, and L key function?
~~could have put all this in a pull request but I'm afk and GitHub on mobile is kinda annoying~~
@Disaster2life that is a good suggestion, I'll condense that section a bit based on your feedback.
The shuttle provides a smooth gradient of speed for playback. It is not just remapped J/K/L keys.
With the Contour Design jog/shuttle devices currently supported, the shuttle gives us a total of 15 measurable positions, from -7 full reverse speed, through 0 stopped, to +7 full forward speed. Each available shuttle position is used, with a maximum of +-32x the speed of play at the extremes, to +- normal play speed right past zero on either side. The curve is very smooth, and in my opinion feels very natural and responsive.
@ratherlargerobot ah thanks, also you might have accidentally removed a bit of the line since one frame a time. doesn't make sense, just a heads-up,
And the jog wheel seems like a super useful, think that's actually more than flowblade itself allows you to speed up/ go in reverse of I remember correctly, I was thinking of just getting a secondary keyboard and programming that to perform functions directly (read the idea I raised a while back to allow keyboards to do that) but yeah getting a compatible product might actually be way more useful
@Disaster2life good catch! I gave the document one more round of proofreading.
And yes, you are correct, the shuttle implementation here does allow for more levels of speed in both directions than the J/K/L keys. I think the J/K/L keys are great the way they are. It's a balance between getting different speeds, and not pressing the keys too many times. It would be annoying to press the J or L key 7 times just to get it to go at its top speed. But with the shuttle, you just turn the ring about 60 degrees and you've gone through all 7 speeds on your way up to the fastest one.
I spent a fair amount of time during development just playing around with the jog/shuttle on a timeline, to make sure that the feel is just right.
Just a reminder that the arrow keys have modifiers attached to them to increase speed with a factor which you can set yourself in preferences. Eg Shift right could go at 2x speed and Ctrl right at 3x speed, which means ShiftCtril right would go at 23= 6x speed. If I remember right this doesn't affect the jkl keys, but not entirely sure. I always use the arrow keys. Sent from my Huawei tablet-------- Original message --------From: Nathan Rosenquist @.>Date: Sat, 26 Feb 2022, 21:28To: jliljebl/flowblade @.>Cc: dvdlvr @.>, Mention @.*>Subject: Re: [jliljebl/flowblade] Add USB jog/shuttle support to Flowblade (PR #1052) @Disaster2life good catch! I gave the document one more round of proofreading. And yes, you are correct, the shuttle implementation here does allow for more levels of speed in both directions than the J/K/L keys. I think the J/K/L keys are great the way they are. It's a balance between getting different speeds, and not pressing the keys too many times. It would be annoying to press the J or L key 7 times just to get it to go at its top speed. But with the shuttle, you just turn the ring about 60 degrees and you've gone through all 7 speeds on your way up to the fastest one. I spent a fair amount of time during development just playing around with the jog/shuttle on a timeline, to make sure that the feel is just right.
—Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.Message ID: @.***>
Since there seems to be a bit of interest in this topic, here is the map of shuttle positions to playback speeds in the Contour Design shuttle driver (extracted from the shuttle encoder value playback speed map in usbhiddrivers.py in the ContourDesignShuttle class):
Shuttle Position: Playback Speed (- is reverse, + is forward)
-7: -32x
-6: -16x
-5: -8x
-4: -5x
-3: -3x
-2: -1.8x
-1: -1x
0: 0 (stop)
+1: +1x
+2: +1.8x
+3: +3x
+4: +5x
+5: +8x
+6: +16x
+7: +32x
The Doc seems good now, nothing else I can catch, and yeah to me at least, the J,K, L keys are just rough movement tools on the timeline, having to press them several times before it actually allows you to, use them that way would defeat their purpose, but the shuttle definitely mixes the function of them and the arrow keys with the amount of control, and the curve you later mentioned is pretty good, personally don't see the function of a x32 factor, since at that point it might be better to just use the mouse and go back to whatever section, but that also involves moving your hand off of the shuttle so maybe?
An interesting idea for the shuttle could be allowing a modifier key + the wheel to have some actions? That could allow for some cool possibilities like skipping around the timeline on the basis of markers if that's useful?
Also totally didn't know about the modifiers to the arrow keys, thanks for informing me about that, probably would be useful in my usage
I feel like a modifier key to change the jog/shuttle action might be more confusing than helpful. The jog/shuttle is a very well-established concept. To me, at least, making it modifiable feels like it might be adding complexity where simplicity is needed. At least on the Contour Design jog/shuttle devices.
Ok, I'll do it merge on test branch first thing after 2.12. This has been unfortuinately left behind a bit I'll get to it soon.
Ok, I pulled the PR as local branch usb-jog-feature and did the following:
- merged with master
- fixed the conflicts
- fixed a circular imports problem, some added new imports in master obviously didn't work with import structure of the pull request
- pushed the feature branch back to origin here
@ratherlargerobot can you test that the feature branch usb-jog-feature still works as intended?
I plan to test this soon. It looks like I need to go through the mlt7 dependency transition on my computer first to get things running. I will try to get that working in the next few days.
Thank you!
I tested this on Ubuntu 22.04.3 LTS using Xorg, and everything appears to work perfectly! Bravo!
Tested using a Contour ShuttleXpress: https://contourdesign.com/products/shuttle-xpress https://www.amazon.com/Contour-Design-Shuttle-Xpress-Multimedia-Controller/dp/B0001DBEKG
Thank you for merging this into the modern development branch. Much appreciated!
Merged.
Thank you very much for this big development effort, and apologies on the delayed merge. Maybe I'll go out and by a USB control device myself now that we have the support these devices and the structure to add new ones.
The next release will come in about a months time (hopefully, the usual reservations apply), and this will be the headline feature. Thanks again for contributing and extending the capabilities of Flowblade.
@ratherlargerobot I added the JOG_SHUTTLE.md file contents as section Jog / Shuttle Support in help docs chapter Advanced Editing Features. Thanks for providing the text, this will make using the feature and adding new devices much easier to users.
The release and Flatpak is out now. @ratherlargerobot, if you could test the Flatpak sometime so we could get a confirmation that this works with the Flatpak.
It appears that the Flatpak isn't working for me. However, I'm also not very familiar with Flatpak, so it's quite possible that I missed a step somewhere.
I appeared to already have the flatpak command installed on Ubuntu 22. I can't recall ever using it before directly.
I ran the following command:
flatpak install https://flathub.org/repo/appstream/io.github.jliljebl.Flowblade.flatpakref
which I found in the Flowblade install guide: https://github.com/jliljebl/flowblade/blob/master/flowblade-trunk/docs/INSTALLING.md
Here is what that looked like:
$ flatpak install https://flathub.org/repo/appstream/io.github.jliljebl.Flowblade.flatpakref
Required runtime for io.github.jliljebl.Flowblade/x86_64/stable (runtime/org.gnome.Platform/x86_64/45) found in remote flathub
Do you want to install it? [Y/n]: Y
io.github.jliljebl.Flowblade permissions:
ipc pulseaudio x11 devices
file access [1] dbus access [2]
[1] host
[2] org.freedesktop.Flatpak
ID Branch Op Remote Download
1. [✓] io.github.jliljebl.Flowblade.Locale stable i flathub 8.9 kB / 6.2 MB
2. [✓] org.freedesktop.LinuxAudio.Plugins.swh 23.08 i flathub 8.0 MB / 7.7 MB
3. [✓] org.freedesktop.Platform.GL.default 23.08 i flathub 165.2 MB / 164.6 MB
4. [✓] org.freedesktop.Platform.GL.default 23.08-extra i flathub 18.5 MB / 164.6 MB
5. [✓] org.freedesktop.Platform.VAAPI.Intel 23.08 i flathub 13.3 MB / 13.4 MB
6. [✓] org.freedesktop.Platform.openh264 2.2.0 u flathub 614.3 kB / 944.3 kB
7. [✓] org.gnome.Platform.Locale 45 i flathub 18.1 kB / 369.6 MB
8. [✓] org.gnome.Platform 45 i flathub 368.3 MB / 378.2 MB
9. [✓] io.github.jliljebl.Flowblade stable i flathub 69.1 MB / 66.0 MB
Changes complete.
Then I ran the program:
flatpak run io.github.jliljebl.Flowblade
When I started it up, I had to enable jog/shuttle support in settings, and select Contour ShuttleXpress as the type of jog shuttle to use. Then, I had to stop the program, and start it again.
When I started it again, it all seemed to come up normally. However, the moment that I started moving the jog wheel, I got a notice in the terminal window where I launched it about a missing libusb shared library. Here is a snippet from that test, with much of the normal startup parts omitted for brevity:
...
MLT detection succeeded, 184 formats, 114 video codecs and 79 audio codecs found.
639 MLT services found.
Loading render profiles...
ffmpeg version 6.1.1 Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 13.2.0 (GCC)
configuration: --prefix=/app --enable-gpl --disable-static --enable-shared --disable-doc --enable-libfontconfig --enable-libfreetype --enable-libopus --enable-librsvg --enable-libvpx --enable-libx264 --enable-libx265 --enable-libpulse --enable-libmp3lame --enable-libvorbis --enable-libaom --enable-vaapi --enable-ffnvcodec --enable-nvenc --enable-amf --enable-libzimg
libavutil 58. 29.100 / 58. 29.100
libavcodec 60. 31.102 / 60. 31.102
libavformat 60. 16.100 / 60. 16.100
libavdevice 60. 3.100 / 60. 3.100
libavfilter 9. 12.100 / 9. 12.100
libswscale 7. 5.100 / 7. 5.100
libswresample 4. 12.100 / 4. 12.100
libpostproc 57. 3.100 / 57. 3.100
['ffmpeg', '-version']
return code: 0
ffmpeg CLI available
Loading filters...
Loading transitions...
G'MIC found
Player initialized with profile: HD 1080p 30 fps
Create SDL1 consumer...
Autosave started...
Saving project...
Initializing USB HID driver: Contour Design ShuttleXpress
Can not connect to USB HID device using driver config 'contour_design_shuttlexpress': 'Error connecting to USB HID device: libusb-1.0.so: cannot open shared object file: No such file or directory'
GPU test results {'NVENC H.264 High Profile / .mp4': -11, 'NVENC HEVC / .mp4': -11, 'NVENC HEVC HDR / .mp4': -11, 'VAAPI H.264 / .mp4': 0}
If I had to guess, maybe Flatpak has to find libusb within a Flatpak library, and not from the main OS, where libusb is already installed?
If I had to guess, maybe Flatpak has to find libusb within a Flatpak library, and not from the main OS, where libusb is already installed?
Yeah, obviously. Somehow I thought this was part of the platform (the lib, not the bindings) and because it didn't crash on startup I thought that to be proven. Not true.
I added the needed line:
- shared-modules/libusb/libusb.json
and pushed an update. Updates usually take about 6 hours to go live, I'll post here once libusb update seems to be available.
Then you can simply do:
flatpak update
and try again.
Oh I forgot, you can pull the test version once the build completes, I'll post link to that here in about half an hour.
Test build is ready. You can install it with:
flatpak install --user https://dl.flathub.org/build-repo/93677/io.github.jliljebl.Flowblade.flatpakref
This is now a "user" install, so need to run it with a flag:
flatpak run -u io.github.jliljebl.Flowblade
I'll go merge the patch that might drop the test build. I'll comment once the new build is up.
The build with libusb is live now and can be tested with re-install or updating existing install:
flatpak update