MuseScore icon indicating copy to clipboard operation
MuseScore copied to clipboard

Show playing notes on the piano keyboard

Open adazem009 opened this issue 11 months ago • 16 comments

Resolves: #14428 Resolves: #23054

The current implementation of MIDI output isn't very elegant. FluidSynth, which is used only with MS Basic, is used to send MIDI events to MIDI output devices. That means using e. g. Muse Sounds will disable MIDI output completely.

It was therefore necessary to add a special synthesizer which handles MIDI output and nothing else (no audio playback). This synthesizer is always active, even if Muse Sounds or VST is used. The MIDI events aren't, however, sent to MIDI devices from the synthesizer, but instead sent through a channel in PlayerHandler so that any module outside framework can listen to note MIDI events.

There's a MidiInputOutputController class in the notation module which currently only handles MIDI input. Because of the class name, it makes sense to handle MIDI output there too. To avoid sending MIDI events of muted mixer channels (#18389), it was necessary to make synthesizers aware of track ID so that the special MIDI output synthesizer can send MIDI events with the track ID. The track ID is then used to get the mute state of the channel.

The visualization of notes on the piano keyboard is implemented using the MIDI event channels too.

Please note:

  • It probably isn't the best idea to have 2 synthesizers running simultaneously because it might slow things down, but maybe it'll give you some ideas for a better solution.
  • I didn't add a sequencer for the MIDI output synthesizer because FluidSequencer is sufficient. Maybe FluidSequencer could be moved out of FluidSynth to be a common sequencer for these two synthesizers.
  • I also handled edge cases for the piano keyboard such as when an instrument is removed during playback.
  • Issue #19254 can be easily fixed after these changes because the MIDI output handling can be extended with a list of played keys just like in the piano keyboard controller (and the playingNotesRevoked() channel could be used).
  • Notes coming from muted channels won't show. This is something that wasn't in MuseScore 3.
  • [x] I signed the CLA
  • [x] The title of the PR describes the problem it addresses
  • [x] Each commit's message describes its purpose and effects, and references the issue it resolves
  • [x] If changes are extensive, there is a sequence of easily reviewable commits
  • [x] The code in the PR follows the coding rules
  • [x] There are no unnecessary changes
  • [x] The code compiles and runs on my machine, preferably after each commit individually
  • [ ] I created a unit test or vtest to verify the changes I made (if applicable)

adazem009 avatar Mar 10 '24 11:03 adazem009

rebase needed

Jojo-Schmitz avatar Apr 10 '24 21:04 Jojo-Schmitz

Any progress on getting this merged? I really want this feature. As a pianist, it would really help me to visualize and understand the music written.

ShawkMusic avatar Apr 26 '24 05:04 ShawkMusic

rebase needed

Done. I've also changed the source file headers of new files to MuseScore Studio (20dd58e7aae0c99f9b4725eb996e25a84243b6c0). Please check if it's correct.

adazem009 avatar Apr 26 '24 10:04 adazem009

Any progress on getting this merged? I really want this feature. As a pianist, it would really help me to visualize and understand the music written.

I guess the MuseScore team is busy with more important issues, so it may take some time until they get into this one. In the meantime, you can test this PR using these builds:

Windows Linux macOS

adazem009 avatar Apr 26 '24 11:04 adazem009

I guess the MuseScore team is busy with more important issues, so it may take some time until they get into this one. In the meantime, you can test this PR using these builds:

Windows Linux macOS

Thanks!

Running the Appimage on my system (Arch btw) doesn't seem to open. (I tried resetting my config as well in case that was the issue) Running it on the command line gives this error:

/usr/lib/libjack.so.0
/usr/lib/libnss3.so
qt.qml.typeregistration: Invalid QML element name "IconCode"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "MusicalSymbolCodes"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "ContainerType"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "NavigationEvent"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "MUAccessible"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "CompareType"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "SelectionMode"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "DockToolBarAlignment"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "Location"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "CloudVisibility"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "SaveToCloudResponse"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "DirectionTypes"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "SlurTieTypes"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "NoteHead"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "Beam"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "Hairpin"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "OrnamentTypes"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "Dynamic"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "Glissando"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "CommonTypes"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "BarlineTypes"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "MarkerTypes"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "KeySignatureTypes"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "AccidentalTypes"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "FretDiagramTypes"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "LineTypes"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "TextTypes"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "ArticulationTypes"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "AmbitusTypes"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "ChordSymbolTypes"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "BendTypes"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "TremoloBarTypes"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "TremoloTypes"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "InstrumentsTreeItemType"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "StandardButton"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "SaveLocationType"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "GenerateAudioTimePeriodType"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "MigrationType"; value type names should begin with a lowercase letter
12:22:09.202 | INFO  | main_thread     | GlobalModule::onPreInit | log path: /home/gabriel/.local/share/MuseScore/MuseScore4Development/logs/MuseScore_240511_122209.log
12:22:09.202 | INFO  | main_thread     | GlobalModule::onPreInit | === Started MuseScore Studio 4.4.0, build: 241171032 ===
12:22:09.213 | INFO  | main_thread     | LanguagesService::effectiveLanguageCode | System language code: "en-US" 
12:22:09.224 | WARN  | main_thread     | IpcSocket::connect | failed connect to server
12:22:09.224 | WARN  | main_thread     | IpcSocket::connect | failed connect to server
12:22:09.226 | INFO  | main_thread     | IpcSocket::connect | success connected to ipc server
12:22:09.226 | INFO  | 139820389107392 | IpcServer::listen | id: "d99e4d85f3df40b0b288361ee5eb0c19" 
12:22:09.252 | WARN  | main_thread     | Qt              | The EGL implementation does not support the Wayland platform
12:22:09.295 | INFO  | main_thread     | DiagnosticsModule::onInit | success start crash handler
12:22:09.398 | WARN  | main_thread     | Qt              | QIODevice::read (QFile, "/home/gabriel/.local/share/MuseScore/MuseScore4Development/shortcuts.xml"): device not open
12:22:09.398 | WARN  | main_thread     | Qt              | QIODevice::read (QFile, "/home/gabriel/.local/share/MuseScore/MuseScore4Development/midi_mappings.xml"): device not open
12:22:09.435 | WARN  | main_thread     | AbstractCloudService::readTokens | Could not find the tokens file: /home/gabriel/.local/share/MuseScore/MuseScore4Development/musescorecom_cred.dat
12:22:09.435 | WARN  | main_thread     | AbstractCloudService::readTokens | Could not find the tokens file: /home/gabriel/.local/share/MuseScore/MuseScore4Development/audiocom_cred.dat
12:22:10.272 | WARN  | main_thread     | Qt              | QQmlEngine::setContextForObject(): Object already has a QQmlContext
12:22:10.310 | ERROR | main_thread     | App::run        | error: qrc:/qml/Muse/UiComponents/internal/PopupContent.qml:81:5: QML QQuickItem: Binding loop detected for property "implicitWidth"

12:22:10.310 | WARN  | main_thread     | Qt              | qrc:/qml/Muse/UiComponents/internal/PopupContent.qml:81:5: QML QQuickItem: Binding loop detected for property "implicitWidth"
12:22:10.311 | ERROR | main_thread     | App::run        | error: qrc:/qml/Muse/UiComponents/internal/PopupContent.qml:81:5: QML QQuickItem: Binding loop detected for property "implicitWidth"

12:22:10.311 | WARN  | main_thread     | Qt              | qrc:/qml/Muse/UiComponents/internal/PopupContent.qml:81:5: QML QQuickItem: Binding loop detected for property "implicitWidth"
12:22:10.340 | WARN  | main_thread     | Qt              | Dropping not implement for QtQuick on Wayland yet!
12:22:10.352 | ERROR | main_thread     | App::run        | error: qrc:/qml/Muse/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"

12:22:10.352 | WARN  | main_thread     | Qt              | qrc:/qml/Muse/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"
12:22:10.353 | ERROR | main_thread     | App::run        | error: qrc:/qml/Muse/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"

12:22:10.353 | WARN  | main_thread     | Qt              | qrc:/qml/Muse/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"
12:22:10.353 | ERROR | main_thread     | App::run        | error: qrc:/qml/Muse/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"

12:22:10.353 | WARN  | main_thread     | Qt              | qrc:/qml/Muse/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"
12:22:10.354 | ERROR | main_thread     | App::run        | error: qrc:/qml/Muse/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"

12:22:10.354 | WARN  | main_thread     | Qt              | qrc:/qml/Muse/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"
12:22:10.354 | ERROR | main_thread     | App::run        | error: qrc:/qml/Muse/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"

12:22:10.355 | WARN  | main_thread     | Qt              | qrc:/qml/Muse/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"
12:22:10.355 | ERROR | main_thread     | App::run        | error: qrc:/qml/Muse/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"

12:22:10.355 | WARN  | main_thread     | Qt              | qrc:/qml/Muse/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"
12:22:10.355 | ERROR | main_thread     | App::run        | error: qrc:/qml/Muse/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"

12:22:10.355 | WARN  | main_thread     | Qt              | qrc:/qml/Muse/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"
12:22:10.356 | ERROR | main_thread     | App::run        | error: qrc:/qml/Muse/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"

12:22:10.356 | WARN  | main_thread     | Qt              | qrc:/qml/Muse/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"
12:22:10.356 | ERROR | main_thread     | App::run        | error: qrc:/qml/Muse/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"

12:22:10.357 | WARN  | main_thread     | Qt              | qrc:/qml/Muse/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"
12:22:10.368 | ERROR | main_thread     | ExtensionsLoader::manifestPaths | failed scan files, err: [401] The file does not exist
12:22:10.385 | ERROR | main_thread     | ExtensionsConfiguration::manifestConfigs | failed read config data, err: [406] An error occurred when reading from the file, file: /home/gabriel/Documents/MuseScore4Development/Plugins/config.json
12:22:10.437 | WARN  | main_thread     | Qt              | QRhiGles2: Failed to create temporary context
12:22:10.439 | WARN  | main_thread     | Qt              | QRhiGles2: Failed to create context
12:22:10.439 | WARN  | main_thread     | Qt              | Failed to create RHI (backend 2)
12:22:10.439 | ERROR | main_thread     | Qt              | Failed to initialize graphics backend for OpenGL.
[915310:915310:20240511,122210.442792:WARNING thread_snapshot_linux.cc:111] Unknown scheduling policy 1073741825
[915310:915310:20240511,122210.446564:ERROR elf_dynamic_array_reader.h:61] tag not found
[915310:915310:20240511,122210.446845:ERROR elf_dynamic_array_reader.h:61] tag not found
[915310:915310:20240511,122210.447365:ERROR elf_dynamic_array_reader.h:61] tag not found
[915310:915310:20240511,122210.447957:ERROR elf_dynamic_array_reader.h:61] tag not found
[915310:915310:20240511,122210.448059:ERROR elf_dynamic_array_reader.h:61] tag not found
[915310:915310:20240511,122210.448260:ERROR elf_dynamic_array_reader.h:61] tag not found
[915310:915310:20240511,122210.448312:ERROR elf_dynamic_array_reader.h:61] tag not found
[915310:915310:20240511,122210.448365:ERROR elf_dynamic_array_reader.h:61] tag not found
[915310:915310:20240511,122210.448466:ERROR elf_dynamic_array_reader.h:61] tag not found
[915310:915310:20240511,122210.448516:ERROR elf_dynamic_array_reader.h:61] tag not found
[915310:915310:20240511,122210.448562:ERROR elf_dynamic_array_reader.h:61] tag not found
[915310:915310:20240511,122210.448612:ERROR elf_dynamic_array_reader.h:61] tag not found
[915310:915310:20240511,122210.448664:ERROR elf_dynamic_array_reader.h:61] tag not found
[915310:915310:20240511,122210.448718:ERROR elf_dynamic_array_reader.h:61] tag not found
[915310:915310:20240511,122210.448764:ERROR elf_dynamic_array_reader.h:61] tag not found
[915310:915310:20240511,122210.448811:ERROR elf_dynamic_array_reader.h:61] tag not found
[915310:915310:20240511,122210.448855:ERROR elf_dynamic_array_reader.h:61] tag not found
[915310:915310:20240511,122210.448906:ERROR elf_dynamic_array_reader.h:61] tag not found
[915310:915310:20240511,122210.448951:ERROR elf_dynamic_array_reader.h:61] tag not found
[915310:915310:20240511,122210.448993:ERROR elf_dynamic_array_reader.h:61] tag not found
/tmp/.mount_MuseSckZP87F/AppRun: line 26: 915304 Aborted                 (core dumped) "${APPDIR}/bin/mscore4portabledev" "$@"

Just for reference, this is what running the (functional) official Musescore 4.3.0 appimage from the website outputs:

RUNNING /home/gabriel/Downloads/MuseScore-Studio-4.3.0.241231431-x86_64.AppImage:

/usr/lib/libjack.so.0
/usr/lib/libnss3.so
qt.qpa.plugin: Could not find the Qt platform plugin "wayland" in ""
12:24:49.698 | INFO  | main_thread     | GlobalModule::onPreInit | log path: /home/gabriel/.local/share/MuseScore/MuseScore4/logs/MuseScore_240511_122449.log
12:24:49.698 | INFO  | main_thread     | GlobalModule::onPreInit | === Started MuseScore 4.3.0, build number 241231431 ===
12:24:49.708 | INFO  | main_thread     | LanguagesService::effectiveLanguageCode | System language code: "en-US" 
12:24:49.717 | WARN  | main_thread     | IpcSocket::connect | failed connect to server
12:24:49.717 | WARN  | main_thread     | IpcSocket::connect | failed connect to server
12:24:49.718 | INFO  | main_thread     | IpcSocket::connect | success connected to ipc server
12:24:49.718 | INFO  | 136608519030464 | IpcServer::listen | id: "b9a0900e1f8944fbbc9ddcf4a4f8dd51" 
12:24:49.898 | INFO  | main_thread     | DiagnosticsModule::onInit | success start crash handler
12:24:49.998 | WARN  | main_thread     | Qt              | QIODevice::read (QFile, "/home/gabriel/.local/share/MuseScore/MuseScore4/shortcuts.xml"): device not open
12:24:49.998 | WARN  | main_thread     | Qt              | QIODevice::read (QFile, "/home/gabriel/.local/share/MuseScore/MuseScore4/midi_mappings.xml"): device not open
12:24:50.016 | WARN  | main_thread     | AbstractCloudService::readTokens | Could not find the tokens file: /home/gabriel/.local/share/MuseScore/MuseScore4/musescorecom_cred.dat
12:24:50.016 | WARN  | main_thread     | AbstractCloudService::readTokens | Could not find the tokens file: /home/gabriel/.local/share/MuseScore/MuseScore4/audiocom_cred.dat
12:24:50.310 | ERROR | main_thread     | PluginsConfiguration::updatePluginsConfiguration | [401] The file does not exist
12:24:50.310 | ERROR | main_thread     | WorkspaceManager::findWorkspaceFiles | [401] The file does not exist
12:24:50.310 | WARN  | main_thread     | WorkspaceManager::setupDefaultWorkspace | not found default workspace, will be created new
12:24:50.945 | ERROR | main_thread     | App::run        | error: qrc:/qml/MuseScore/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"

12:24:50.946 | WARN  | main_thread     | Qt              | qrc:/qml/MuseScore/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"
12:24:50.946 | ERROR | main_thread     | App::run        | error: qrc:/qml/MuseScore/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"

12:24:50.946 | WARN  | main_thread     | Qt              | qrc:/qml/MuseScore/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"
12:24:50.947 | ERROR | main_thread     | App::run        | error: qrc:/qml/MuseScore/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"

12:24:50.947 | WARN  | main_thread     | Qt              | qrc:/qml/MuseScore/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"
12:24:50.947 | ERROR | main_thread     | App::run        | error: qrc:/qml/MuseScore/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"

12:24:50.947 | WARN  | main_thread     | Qt              | qrc:/qml/MuseScore/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"
12:24:50.948 | ERROR | main_thread     | App::run        | error: qrc:/qml/MuseScore/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"

12:24:50.948 | WARN  | main_thread     | Qt              | qrc:/qml/MuseScore/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"
12:24:50.948 | ERROR | main_thread     | App::run        | error: qrc:/qml/MuseScore/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"

12:24:50.948 | WARN  | main_thread     | Qt              | qrc:/qml/MuseScore/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"
12:24:50.949 | ERROR | main_thread     | App::run        | error: qrc:/qml/MuseScore/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"

12:24:50.949 | WARN  | main_thread     | Qt              | qrc:/qml/MuseScore/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"
12:24:50.949 | ERROR | main_thread     | App::run        | error: qrc:/qml/MuseScore/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"

12:24:50.949 | WARN  | main_thread     | Qt              | qrc:/qml/MuseScore/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"
12:24:50.950 | ERROR | main_thread     | App::run        | error: qrc:/qml/MuseScore/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"

12:24:50.950 | WARN  | main_thread     | Qt              | qrc:/qml/MuseScore/UiComponents/FlatButton.qml:179:5: QML Loader: Binding loop detected for property "itemImplicitWidth"
12:24:51.673 | WARN  | main_thread     | QmlPlugin::setMenuPath | Not supported!! Use:title
12:24:51.683 | ERROR | main_thread     | AppShellConfiguration::sessionProjectsPaths | [401] The file does not exist
12:24:53.394 | ERROR | main_thread     | UpdateScenario::doCheckForUpdate | Unable to check for update, error: [1701] 
12:25:03.970 | INFO  | main_thread     | Profiler        | 

Main thread. Top 150 by sum time (total count: 101)
Function                                                    Call time         Call count        Sum time          
StartupScenario::onStartupPageOpened                        10813.831 ms      1                 10813.831 ms      
PluginsService::init                                        719.248 ms        1                 719.248 ms        
PluginsService::readPlugins                                 715.782 ms        1                 715.782 ms        
DockWindow::loadPage                                        49.699 ms         2                 99.398 ms         
DockWindow::loadPageContent                                 39.910 ms         2                 79.820 ms         
DockWindow::addDock                                         2.214 ms          32                70.856 ms         
SoundFontRepository::loadSoundFonts                         56.658 ms         1                 56.658 ms         
DockWindow::loadPanels                                      21.988 ms         2                 43.976 ms         
StartupScenario::run                                        38.837 ms         1                 38.837 ms         
InstrumentsRepository::load                                 29.300 ms         1                 29.300 ms         
AbstractMenuModel::setItems                                 1.300 ms          20                26.003 ms         
DockWindow::loadToolBars                                    9.966 ms          2                 19.932 ms         
DockWindow::onQuit                                          18.390 ms         1                 18.390 ms         
dock::clearRegistry                                         5.736 ms          3                 17.208 ms         
DockWindow::loadTopLevelToolBars                            3.947 ms          2                 7.894 ms          
AppMenuModel::load                                          6.956 ms          1                 6.956 ms          
DockBase::componentComplete                                 0.119 ms          57                6.756 ms          
LanguagesService::init                                      5.166 ms          1                 5.166 ms          
LanguagesService::setCurrentLanguage                        4.409 ms          1                 4.409 ms          
PluginsService::registerShortcuts                           3.393 ms          1                 3.393 ms          
DockWindow::savePageState                                   1.232 ms          2                 2.463 ms          
DockWindow::windowState                                     0.675 ms          3                 2.026 ms          
DockBase::applySizeConstraints                              0.032 ms          60                1.912 ms          
PluginsService::scanFileSystemForPlugins                    1.193 ms          1                 1.193 ms          
DockWindow::initDocks                                       0.551 ms          2                 1.103 ms          
NavigationController::onActiveRequested                     0.078 ms          14                1.086 ms          
ShortcutsRegister::reload                                   1.055 ms          1                 1.055 ms          
UiActionsRegister::updateEnabledAll                         0.055 ms          19                1.039 ms          
NavigationControl::setPanel                                 0.002 ms          548               1.019 ms          
DockPageView::init                                          0.411 ms          2                 0.821 ms          
DockWindow::componentComplete                               0.797 ms          1                 0.797 ms          
ShortcutsRegister::readFromFile                             0.398 ms          2                 0.796 ms          
LanguagesService::loadLanguages                             0.689 ms          1                 0.689 ms          
SymNames::loadNameToSymIdHash                               0.491 ms          1                 0.491 ms          
UiActionsRegister::updateShortcutsAll                       0.237 ms          2                 0.474 ms          
DockWindow::saveGeometry                                    0.404 ms          1                 0.404 ms          
InstrumentsRepository::loadStringTuningsPresets             0.395 ms          1                 0.395 ms          
NotationActionController::init                              0.293 ms          1                 0.293 ms          
NavigationPanel::addControl                                 0.000 ms          542               0.220 ms          
NavigationController::doDeactivateSection                   0.036 ms          6                 0.218 ms          
AbstractNotationPaintView::paint                            0.198 ms          1                 0.198 ms          
UiActionsRegister::updateCheckedAll                         0.197 ms          1                 0.197 ms          
NotationStatusBarModel::load                                0.095 ms          2                 0.189 ms          
AbstractNotationPaintView::paintBackground                  0.188 ms          1                 0.188 ms          
NavigationController::doDeactivatePanel                     0.007 ms          27                0.178 ms          
AbstractCloudService::init                                  0.087 ms          2                 0.174 ms          
ShortcutsRegister::makeUnique                               0.151 ms          1                 0.151 ms          
NavigationController::doDeactivateControl                   0.002 ms          67                0.151 ms          
UiActionsRegister::updateChecked                            0.044 ms          3                 0.132 ms          
DockPanelMenuModel::load                                    0.010 ms          12                0.116 ms          
AbstractCloudService::readTokens                            0.047 ms          2                 0.094 ms          
NavigationPanel::setSection_property                        0.001 ms          129               0.089 ms          
RecentFilesController::recentFilesList                      0.040 ms          2                 0.081 ms          
DockPageView::doReorderSections                             0.031 ms          2                 0.062 ms          
InstrumentsRepository::loadMuseInstruments                  0.061 ms          1                 0.061 ms          
NoteInputBarModel::data                                     0.000 ms          393               0.059 ms          
UiActionsRegister::updateEnabled                            0.057 ms          1                 0.057 ms          
UiContextResolver::currentUiContext                         0.003 ms          20                0.052 ms          
AbstractNotationPaintView::load                             0.046 ms          1                 0.046 ms          
PluginsConfiguration::readPluginsConfiguration              0.040 ms          1                 0.040 ms          
RegisterAudioPluginsScenario::init                          0.038 ms          1                 0.038 ms          
DockWindow::registerDock                                    0.001 ms          34                0.032 ms          
ShortcutsRegister::expandStandardKeys                       0.016 ms          2                 0.031 ms          
NotationPageModel::init                                     0.030 ms          1                 0.030 ms          
KnownAudioPluginsRegister::load                             0.026 ms          1                 0.026 ms          
NavigationController::reg                                   0.001 ms          31                0.025 ms          
NavigationController::activeControl                         0.003 ms          7                 0.021 ms          
NavigationSection::addPanel                                 0.000 ms          60                0.018 ms          
NavigationController::activePanel                           0.002 ms          7                 0.016 ms          
ShortcutsRegister::mergeShortcuts                           0.015 ms          1                 0.015 ms          
NotationSwitchListModel::load                               0.015 ms          1                 0.015 ms          
AbstractNotationPaintView::verticalScrollbarSize            0.014 ms          1                 0.014 ms          
NotationSwitchListModel::loadNotations                      0.012 ms          1                 0.012 ms          
NavigationController::activeSection                         0.002 ms          7                 0.012 ms          
AbstractNotationPaintView::onNotationSetup                  0.011 ms          1                 0.011 ms          
NavigationPanel::removeControl                              0.000 ms          25                0.008 ms          
NavigationSection::removePanel                              0.000 ms          19                0.007 ms          
NotationPageModel::updateDrumsetPanelVisibility             0.006 ms          1                 0.006 ms          
DockWindow::restorePageState                                0.003 ms          2                 0.006 ms          
InspectorListModel::setElementList                          0.002 ms          3                 0.006 ms          
AbstractNotationPaintView::notationContentRect              0.001 ms          6                 0.005 ms          
SlurSegment::avoidCollisions                                0.001 ms          7                 0.004 ms          
InstrumentsPanelTreeModel::clear                            0.003 ms          1                 0.003 ms          
DockPageView::deinit                                        0.003 ms          1                 0.003 ms          
RegisterAudioPluginsScenario::registerNewPlugins            0.003 ms          1                 0.003 ms          
DockWindow::restoreGeometry                                 0.002 ms          1                 0.002 ms          
UiArrangement::toolConfig                                   0.002 ms          1                 0.002 ms          
UiConfiguration::readThemes                                 0.002 ms          1                 0.002 ms          
Workspace::rawData                                          0.000 ms          4                 0.002 ms          
RecentFilesController::init                                 0.002 ms          1                 0.002 ms          
AbstractNotationPaintView::horizontalScrollbarSize          0.001 ms          1                 0.001 ms          
AbstractNotationPaintView::onCurrentNotationChanged         0.001 ms          1                 0.001 ms          
ProjectAutoSaver::update                                    0.001 ms          1                 0.001 ms          
findNearestEnabled                                          0.000 ms          3                 0.001 ms          
NavigationController::unreg                                 0.001 ms          1                 0.001 ms          
AbstractNotationPaintView::verticalScrollableSize           0.001 ms          1                 0.001 ms          
AbstractNotationPaintView::onPlayingChanged                 0.000 ms          1                 0.000 ms          
AbstractNotationPaintView::startVerticalScrollPosition      0.000 ms          1                 0.000 ms          
AbstractNotationPaintView::startHorizontalScrollPosition    0.000 ms          1                 0.000 ms          
AbstractNotationPaintView::onViewSizeChanged                0.000 ms          2                 0.000 ms          
AbstractNotationPaintView::horizontalScrollableSize         0.000 ms          1                 0.000 ms          


Other threads. Top 150 by sum time (total count: 3)
Function                                                    Call time         Call count        Sum time          
LearnService::th_requestPlaylist                            223.749 ms        1                 223.749 ms        
LearnService::th_requestPlaylist                            223.671 ms        1                 223.671 ms        
SynthResolver::resolveAvailableResources                    0.828 ms          1                 0.828 ms          



12:25:03.970 | INFO  | main_thread     | App::run        | activeThreadCount: 0
12:25:04.047 | INFO  | 136608300926656 | {anonymous}::alsaThread | exit
12:25:04.107 | INFO  | main_thread     | main            | Goodbye!! code: 0

ShawkMusic avatar May 11 '24 16:05 ShawkMusic

@ShawkMusic Looks like the application forces the wayland platform plugin, but it's broken (probably after switching to Qt 6 in master while 4.3 still uses Qt 5 if I remember correctly). To force the xcb platform plugin, use:

export QT_QPA_PLATFORM=xcb

Then run the AppImage.

adazem009 avatar May 11 '24 17:05 adazem009

@ShawkMusic Looks like the application forces the wayland platform plugin, but it's broken (probably after switching to Qt 6 in master while 4.3 still uses Qt 5 if I remember correctly). To force the xcb platform plugin, use:

export QT_QPA_PLATFORM=xcb

Then run the AppImage.

Yes, now it works! I can see the notes in the score being played on the keyboard now! Thank you! I can finally see my beautiful harmonies! Hopefully this gets added to the real app soon...

Here's some little nitpicks:

  • Selected notes shouldn't be highlighted on the keyboard during playback, only when the player is paused.
  • Unpitched percussion shouldn't display notes during playback.
  • It would be nice to be able to distinguish notes from different instruments, maybe by coloring the notes following a color code (maybe something like this), (I imagine this might be hard to do since I don't think the keyboard even supports different colors, but I figured I should write this down somewhere)

Either way, overall in it's current form it's still very good and much, much better then not being able to see anything at all. I think it's in about the same state that I remember it being in Musescore 3.

ShawkMusic avatar May 13 '24 06:05 ShawkMusic

@ShawkMusic Looks like the application forces the wayland platform plugin, but it's broken (probably after switching to Qt 6 in master while 4.3 still uses Qt 5 if I remember correctly). To force the xcb platform plugin, use:

export QT_QPA_PLATFORM=xcb

Then run the AppImage.

Yes, now it works! I can see the notes in the score being played on the keyboard now! Thank you! I can finally see my beautiful harmonies! Hopefully this gets added to the real app soon...

Here's some little nitpicks:

  • Selected notes shouldn't be highlighted on the keyboard during playback, only when the player is paused.
  • Unpitched percussion shouldn't display notes during playback.
  • It would be nice to be able to distinguish notes from different instruments, maybe by coloring the notes following a color code (maybe something like this), (I imagine this might be hard to do since I don't think the keyboard even supports different colors, but I figured I should write this down somewhere)

Either way, overall in it's current form it's still very good and much, much better then not being able to see anything at all. I think it's in about the same state that I remember it being in Musescore 3.

  1. I agree with this too because it would make the keyboard more clear during playback. And it isn't difficult to implement.

  2. MuseScore 3 behaves the same if I remember correctly. I'm not going to resolve this in this PR, but the MuseScore team should discuss the design after this is merged.

  3. The piano keyboard currently uses the accent color and I don't think it's difficult to implement because the piano keyboard already has the information about instruments.

adazem009 avatar May 13 '24 06:05 adazem009

I can split the changes into multiple PRs to fix #23054 first, if needed.

adazem009 avatar Jun 04 '24 14:06 adazem009

This PR needs a proper review, and potentially some further investigation, by @RomanPudashkin, because it makes changes to the audio engine that may affect performance. Unfortunately this will likely take some time due to other competing demands on his work slate at the moment. Thank you for your understanding and patience!

bkunda avatar Jun 06 '24 07:06 bkunda

Hey, I recognize this is probably a dumb question for the people here, but I have no idea how Github or programming works and I just want to be able to see my harmonies on Musescore 4. Could someone explain like I'm 5 years old how I take the file I downloaded from here (quoted comment) and put it into my version of Musescore or link a relevant tutorial? I'm using Windows 11 with Musescore 4.3.1.

Any progress on getting this merged? I really want this feature. As a pianist, it would really help me to visualize and understand the music written.

I guess the MuseScore team is busy with more important issues, so it may take some time until they get into this one. In the meantime, you can test this PR using these builds:

Windows Linux macOS

sneakysushi avatar Jul 01 '24 04:07 sneakysushi

@sneakysushi The file downloaded from those links is not just a file, but a version of MuseScore. So what you download is a special test version that contains the piano keyboard. Be aware that using a test version of MuseScore has some risks: files saved in a test version may become incompatible with the normal version. So, when you want to open an existing file in this test version, always open a copy of the file, to prevent losing the original.

That said, instructions on how to open this build can be found here: https://github.com/musescore/MuseScore/wiki/Downloading-and-running-test-builds#windows

cbjeukendrup avatar Jul 09 '24 22:07 cbjeukendrup