audiveris
audiveris copied to clipboard
Audiveris
Drum notation?
Hello, wonderful developers of Audiveris!
I try to scan my drums sheets from my music school but after a processing, the hi-hat notes (noted with crosses), do not appear on the resulting file; am I missing some configuration? I tried to search for "drums" in your wiki and documentation, is it an expected behavior, a limitation or an error I did?
Thanks for your help :)
~SPJ
I'm afraid Audiveris does not recognize drum sheets. Could you point to a typical example of such drum notation, so that I can confirm?
Hi Hervé,
Wikipedia has an article, but there isn't a universal standard for drum notation (noteheads, voices, etc), so I would suggest the ability to define a drumset (i.e. MuseScore).
I've attached two examples of full scores from different books (same typesetting software and transcribing company, I believe) to give an idea:
They use a double sharp sign for some note heads, but Elaine Gould ('Behind Bars', page 11) discourages this.
Wow! It's a whole bunch of notations! Some are very similar to (non-drum) standard notations, some are specific. From the point of view of current Audiveris possibilities, we would need to handle specific note heads (reading your top diagram: open hit-hat, crash cymbal, ride cymbal), plus some symbols like tremolos (see your buzz strokes).
On top of that, we'd have to check how MusicXML export format can support all these drum notations. Please have a look at MusicXML 3.1
My guts feeling, based on Audiveris architecture, is that this is technically feasible. However it would be a significant task, and we lack time and human resources right now. Perhaps some external contribution could focus on this development.
When you say "there isn't a universal standard for drum notation (noteheads, voices, etc), so I would suggest the ability to define a drumset", I hear a small voice saying "beware, this sounds like a niche requirement". Before investing large efforts into this, make sure the need is crucial and the "standard" widely shared.
I couldn't take the time to point out those problems, but I'm aware there is no standard notation for it and that it could take a large amount of work to implement it. My original question is therefore answered, it's indeed a limitation, but given the state of the art concerning drum notation(s), I also understand perfectly that it's not exactly an urgent issue :) I'll close the issue. Thanks again!
I was originally inspired by certain sophisticated published transcriptions (hidden empty staves, tablature, drums, etc), so I feel this is an area I could assist (though I am not a coder). Such material has helped improve the scoring tool I use - MuseScore.
@stephanepjeannin: Regardless of the scale of challenge, I still think it's a valid issue and we should leave it open.
@hbitteur: It might be a matter of perspective, but defining drumsets may not be as niche as you think.🙂 May I email you with some private examples? I will also post some public ones so they can be discussed, etc.
Well, I closed the issue because I got my answer... However, I agree that the limitation could be addressed. Therefore I reopen it :)
@chenlung You can use my private mail (herve dot bitteur at audiveris dot org) for any private example
I need these funny triple line things - might be called "buzz strokes" per the above?
This score also uses "square standing on tip" noteheads described as ride cymbals above, though it is used for the crash cymbals part.
I see MusicXML has things like <note><notehead>X</notehead></note> to permit things like the percussion x notehead. However, I don't know if this handles all the special things for percussion notation. I tried to cheat and created a MuseScore file with some percussion instruments and odd note heads, but it failed silently when I tried to export to MusicXML so I guess either the standard or the implementation is not quite ready.
@hallvors I couldn't open your file in MuseScore 3.6.2 (crashes).
I've got a mscz with several drumsets if you want to experiment with it?
Ops.. Probably my mistake, it is extracted from a bigger MusicXML score, but it does render fine with OSMD (Opensheetmusicdisplay JS library).
I can't really speak for any Audiveris developers here since I'm just a random user :) But if Musescore exports that drum set file nicely to MusicXML I think it might be useful to them as an example of percussion notation in MusicXML.
MuseScore crashes on that xml file, in a Debug build with a failed assertion somewhere deep in Qt: Fatal: ASSERT failure in QList<T>::at: "index out of range", file C:/Qt/5.15.2/mingw81_64/include/QtCore/qlist.h, line 571
FYI, Audiveris template matching was recently extended to support cross heads (x-shaped heads). I'm putting this here to help contributors (Brian...) on drum notation.
The image above shows how we can interactively apply a head-template for a given shape (here shape NOTEHEAD_CROSS, with a stem located on the right head side) on a given score location. This was meant for Audiveris developer to visually check how the template worked (or not).
To play with this, you will need: to
- Probably select all advanced topics in
Tools | Advanced Topics - In
Tools | Options, set the constantDistancesBuilder.displayTemplatesto true - In the current book use
Book | Set Book Parametersand select "Support for cross note heads" and press OK or Apply. - Then run the HEADS step on your score.
In the Templates tab, use mouse right button to position the template at a desired location. Then read the matching result in the template board on the right column (where you can also select which template to use). Mouse left button is used to read the distance from mouse location to the nearest template foreground point.
OK, that's a bit complex, but this feature was never meant for the end-user :-)
Then browse Audiveris code, beginning with classes:
- HeadsStep
- DistancesBuilder
They should be good starting points. /Hervé
@hbitteur thanks for the welcome and the tips for getting started! As Hervé implied, I want to get involved with improving Audiveris' recognition of drum notation: specifically, 5-line unpitched percussion scores. I am brand new to GitHub, so please forgive and correct any mistakes like posting under the wrong heading of the forum.
Let me first address the issue of MusicXML. It seems that MusicXML 4.0 already supports all the needed elements of unpitched percussion for Audiveris export; see https://www.w3.org/2021/06/musicxml40/tutorial/percussion/ for a good introduction. In particular, it supports encoding of:
- multiple notehead shapes, including standard oval, x, circle-x, diamond, slash, and many others
- multiple instruments per part (snare drum, kick drum, hi-hat closed or open, crash cymbal, cowbell, etc. etc)
- positioning of different instruments on their own line or space of the staff (same as for pitched notes for "normal" scores)
- assignment of a particular MIDI playback sound to each instrument.
Admittedly there are some choices to make with that last point: in the example posted near the top of this thread by chenlung on Jan 17 2018, an x on the space above the score is depicted as an open hi-hat, but many programs that create drum score notation (and many drummers) would use that symbol for a closed hi-hat. Ultimately Audiveris should probably provide a choice to the user among several common schemes for assigning MIDI sounds to each (notehead, score position) pair. But initially for development purposes, it makes sense to choose one of them; I would propose the one used by MuseScore.
Maybe that's enough for now. After I've had a chance to try Hervé's suggestions from the previous post, I'll probably have some questions. And at some point I'll post an overview of what I think will need to be done, in terms of new or improved code development, shape recognition, training, etc., for Audiveris to support unpitched 5-line percussion scores. [GitHub newbie question: should all that be posted here in Issue #33, or somewhere else?]
@hbitteur thanks for the welcome and the tips for getting started! As Hervé implied, I want to get involved with improving Audiveris' recognition of drum notation: specifically, 5-line unhitched percussion scores.
Hi Brian,
I have a score which you may use for testing (specially designed for this). Can I email it to you?
@chenlung, thanks for the offer. I have dozens if not hundreds of PDF drum scores already, from MuseScore and various other sources. Is there something special about yours? I’m not sure how email works with GitHub—I got an email copy of your posting, does that mean if you post your score I’ll get it by email?
@brian-math
[GitHub newbie question: should all that be posted here in Issue https://github.com/Audiveris/audiveris/issues/33, or somewhere else?]
Yes you can use this location for questions, suggestions, etc... When your modifications get more concrete, it will be time to use a PR (pull request) to include them.
Since you are a GitHub "newbie", to get up and running on Audiveris code, do the following from the command line level in a local folder of your choice:
# Download Audiveris code and put it by default in a sub-folder named audiveris
$ git clone https://github.com/Audiveris/audiveris.git
# Move to the just created/populated sub-folder
$ cd audiveris
# Select development branch
$ git checkout development
# Show current status
$ git status
Before that, you'll need to install Git and Gradle tools, plus Java JDK 17 and your favorite IDE. What's your OS?
@chenlung, thanks for the offer. I have dozens if not hundreds of PDF drum scores already, from MuseScore and various other sources. Is there something special about yours? I’m not sure how email works with GitHub—I got an email copy of your posting, does that mean if you post your score I’ll get it by email?
@brian-math You might be covered overall, but I think the score will still serve you in some form, even beyond five-line drumsets (should you get to those).
And I think that's just a notification, so feel free to drop me a line.
Hervé, I want to say that the existing instructions on the site for installing Audiveris were very clear, and I had almost no problems getting it up and running a week ago. The only problems arose regarding the tesseract language files. It seemed that the default was to download version 4.1.0 of those files, whereas it was (exceedingly!) clear from the instructions that only v. 3.0.4 would work. When I first ran Audiveris, I got an error "Could not initialize Tesseract with language deu" [or eng or fra or ita] and suggesting that the environment variable TESSDATA_PREFIX needed to be set. Eventually I discovered that the language files (on a Mac) are saved in /opt/local/share/tessdata/xxx.traindata (where xxx is deu, eng, fra, ita, etc), and after adding that environment variable definition to my .login file, the error message went away. But I still had to figure out how to find and download the tesseract 3.0.4 language files and substitute them for the 4.1.0 files in the aforementioned directory. Some better explanation of this process might have helped. But otherwise it was no problem! Mac OS Mojave 10.14.6.
I suppose you are using Audiveris development branch. And so, you are living on the "bleeding edge" of this software, taking advantage (?) of every modification made on this branch. One of them was the migration from Tesseract 3.x to Tesseract 4.x used in "legacy" mode, which means still using the old (3.x) Tesseract data files.
During this switch, I discovered by chance that TESSDATA_PREFIX environment variable that was supposed (in 3.x) to point to the parent folder of tessdata folder is now supposed in (4.x) to point directly to tessdata folder. This might explain your problem.
This needs to be documented before the next release is published...
Template generation
In Audiveris 5.x, heads are recognized by template matching while most of the other fixed-shape symbols are handled by a trained neural network. This is likely to change in 6.x where the new patch classifier should be able to handle heads as well as the other symbols. But for the time being, we need to extend templates so that even the drum-related heads get recognized.
For this, the key class is org.audiveris.omr.image.TemplateFactory. It works with a specific free true-type font named "MusicalSymbols" which does not need to be installed on the user machine, since it is provided in Audiveris res folder (res/MusicalSymbols.ttf file). See for example https://fonts2u.com/musicalsymbols.font to have a look at this font.
With this font, in TemplateFactory we simply specify the code to use for every specific head shape. For example:
- 207 for NOTEHEAD_BLACK
- 250 for NOTEHEAD_VOID
- 119 for WHOLE_NOTE
- 192 for NOTEHEAD_CROSS (the one that was recently added to support cross heads)
Drum notation will require a few additional codes. A quick search gave:
- 79 for diamond-like head shape (electric snare, ride cymbal)
- 220 for fat cross heads (rim_shot, open hi-hat)
But I could not find any circled fat cross head (crash cymbal) in this font. What can we do?
- Use another musical font? This means 1/ finding out a new suitable font and 2/ checking and perhaps modifying the font codes used in Audiveris software. Fortunately, these codes are kept hidden in 2 key classes, TemplateFactory class to get the images needed to generate templates and Symbols class which handles the symbols display on Audiveris images. However, besides the font code, the program sometimes needs additional information such as the x and y shift of a symbol with respect to font origin. And this involves a bunch of classes.
- Combine 2 font symbols, one for the fat cross and another one for the circle around it. For example code 111, a small circle which unfortunately is significantly smaller than the cross, so it should be picked up from a scaled variant of the font. A bit complex, but doable. I prefer this solution because it doesn't break anything.
Any other idea?
You can play with this small tool integrated in Audiveris.
It is launched by Debug | Launch Symbols Ripper, provided that you have selected the debug features in the Tools | Advanced Topics
Mind the fact that this tool allows to select a font among the ones installed on your machine. So MusicalSymbols font must be installed in that case.
@hbitter Thanks for this detailed information about the music font used for note heads; it is very interesting.
I would suggest not to worry too much right now about the circled x. It seems to me that ~95% of drum notation uses only standard oval and x note heads. The next most common head seems to be a diamond. I have never encountered a circle-x used for a crash cymbal; rather, the main crash is normally an x on the first ledger line above the staff (or perhaps the space above that, for a 2nd crash cymbal). I have seen a circle-x used for an open hi-hat cymbal, but there are at least two other common notations in use for the open hi-hat, and we can probably reasonably translate to one of them if we can recognize an x-in-a-circle glyph. This should be good enough until we get to version 6.x. But if not, I like your idea of combining two existing symbols from the font.
By the way, as mentioned at the bottom of @chenlung 's Jan 17 2018 post, the double sharp symbol should not be used for drum cymbal notation, even though it certainly appears in some scores (probably because it was an existing music notation symbol to which the transcriber had easy access). So I think we should stick to the Code 192 x and not use the Code 220 double sharp.
I've spent a good deal of today preparing 1) a detailed analysis of how Audiveris performs, step by step, when given a drum score as input, and 2) an outline of a proposed set of tasks to get Audiveris able to recognize 5-line drum scores. I'll post those later this evening (US time).
I have never encountered a circle-x used for a crash cymbal; rather, the main crash is normally an x on the first ledger line above the staff (or perhaps the space above that, for a 2nd crash cymbal).
I think MuseScore has the circle-x?
Barnes Music Engraving used Amadeus Music Software for these transcriptions, by the way.
Yes, MuseScore allows you to assign a circle-x to a notehead (and many other shapes, including the Aiken 7-shape do re mi...). But it is not used in their default Drumset notation.
But it is not used in their default Drumset notation.
I can't speak for the developers, nor am I fully aware of on-going developments (will let someone else answer properly), but I don't think the drumsets of previous versions were always properly designed; I actually have a few custom ones (one for emulating the published scores, and another containing my preferences).
Anyway, I'll trust your judgement on this (I'm a non-dev!). One thing that may help is having a conversation with the personnel at MuseScore (or indeed others), influencing the decisions of all parties for the better?
Here are some results of a trial run using Audiveris 5.3 (Development) on a 5-line drum score (original PDF score attached).
Remarks on score:
- Obtained from MuseScore PDF output, so very clean
- Chosen partly because it’s only one sheet
But includes some challenges:
- Lower voice has no rests–notes are lined up with upper voice but separated by gaps not rests
- Half note hollow x head (last bar)
- Grace note flams (last bar)
- Accents widely spaced below corresponding note heads (first staff)
- 1st and 2nd endings
- Triplets (last two bars)
Executive Summary of Audiveris results: Audiveris did a surprisingly good job of recognizing much of this score! Things that did not go well:
- None of the percussion clefs were recognized.
- The two time signatures were not recognized. (Anybody know why? This should have nothing to do with drums.)
- Every cross note head on the same stem as an oval note head got deleted at REDUCTION.
- Ledger lines on x note heads were missed.
- Other minor issues probably not related to being a drum part (see below for details).
============================================================ Here are my detailed observations after each Audiveris step. (My Audiveris settings are listed at the very end.)
GRID All barline inters detected (blue), even the double thick/thin ones at repeats. But in the “section-only” selection mode, showing outlines of contiguous black pixels, the drum clef symbols all have portions of the staff lines amalgamated with them. Also the stems shared by a regular oval note and an x note look different from most others–segmented and filled in black.
HEADERS No header clef detected in any staff! The 2/4 time signature seems to be considered most likely 2/2 by the glyph classifier (but no inter assigned). The 4/4 time signature seems to be ignored completely. The header at the beginning of staff#2 is mistakenly classified as an F_CLEF. This is attached to a glyph consisting of the drum clef, double bar line, and parts of the 3 middle staff lines. When selecting the drum clef at the beginning of each staff in “glyph + inter” mode, typically it is in six pieces instead of the expected two (above, below, and on the middle staff line; left and right parts, instead of just left and right). But sometimes two of these pieces are connected by a piece of the 2nd staff line (e.g. the left and right bottom parts in staff#3).
[Should I select each drum clef and assign PERCUSSION_CLEF from the Glyph Classifier list (it always has the highest grade, except in staff#2) before proceeding with more steps?]
[I unassigned the F_CLEF from the header of staff#2, but not sure how to break apart the drum clef from the double barline in the compound glyph there.]
STEM_SEEDS I guess this worked. Clicking on a few random stems highlights them in black, and they seem to have approximately the correct length and thickness, including the long stems attached to both an x and an oval note head. But they don’t show up as inters; is this expected behavior?
BEAMS Seems to work almost perfectly–all the beams and beam hooks appear to be correctly highlighted in pink. The middle of an “s” in the song title is incorrectly identified as a beam hook.
LEDGERS Most ledger lines are correctly recognized, but not the three supporting x note heads in staff#3, for some reason. Selecting them reveals no inter, and the Glyph Classifier thinks they are tenuto marks with very high grade. (I just noticed that the first and second ending horizontal and vertical lines became selectable at some point, but they don’t seem to be recognized as any sort of inter. The two horizontal top lines get selected as one object.)
HEADS Pink stuff: The x noteheads almost all seem to be correctly recognized as NOTEHEAD_CROSS inters. The ordinary solid oval noteheads all seem to generate at least two inters–a NOTEHEAD_BLACK, plus one or more NOTEHEAD_CROSS inters at or near the same location. For example, selecting the first notehead of bar 47, bottom space, and right-clicking, reveals 6 NOTEHEAD inters: two _BLACK, three _CROSS, and one _VOID! The open half note x at the end is designated as a NOTEHEAD_CROSS; I’m assuming there is not yet a NOTEHEAD_CROSS_VOID shape defined. There are a lot of spurious NOTEHEAD_CROSS inters on the time signatures and drum clefs (and, as mentioned, each NOTEHEAD_BLACK).
Green stuff: There are a lot of spurious WHOLE_NOTE inters: several on each drum clef (combined with a bunch of NOTEHEAD_CROSS inters); a couple near the 4/4 time signature; and one in the midst of the second x-head triplet in the 2nd-last bar.
STEMS Dark green note heads and beams, with light green connections between most stems and at least one of the corresponding note heads (both _BLACK and _CROSS when these overlap), and beams when relevant. Three slanted beams in staff#3 are still pink, where the ledger line of one of the attached notes was missed in LEDGERS step. Same for the erroneous beam hook in the middle of the “s” in the title.
REDUCTION Most of the overlapping symbols from the HEADS step are now gone. Unfortunately, all the x heads which share a stem with an ordinary oval note are also gone. This is incorrect. The grace notes in the last bar are incorrectly regarded as x heads. [I should have turned on support for Small Black Note Heads in Book Parameters.] The drum clefs are now regarded as one whole note on top of another, except in staff#2. The beams and beam hook that were pink after the STEMS step are gone (incorrectly, for the three in staff#3). Both time signatures are unrecognized.
CUE_BEAMS
TEXTS The header text is all recognized correctly. The tempo setting “quarter note = 123” has been rendered as the SENTENCE “J: 123”. All but two of the bar numbers have been recognized correctly. The 1. in the first ending has been transcribed as “II.” and the 2. in the second ending has become “Hz.” The triplets have been rendered as a SENTENCE of 3’s, long dashes, underscores, colons, and quotes. The horizontal lines of the first and second endings are now rendered as a long WORD of underscores. (This probably happened here, but I didn’t notice it until later.)
MEASURES 52 raw measures seems correct.
CHORDS Turning on Show chord IDs, it appears as if every note has an individual chord number. Probably this is not correct. Maybe because of the missed time signatures early on?
CURVES The log reports 142 slurs and 34 segments. Evidently the beams which were deleted because of missing ledger lines were turned into SLUR_ABOVEs and TUPLET_THREEs. There are numerous spurious slurs attached to single or consecutive x notes (some of these note heads were already deleted at the REDUCTION step).
SYMBOLS This apparently picked up most of the stems and flags. One grace note acquired a flag but the other didn’t. The REPEAT_DOTs are recognized, except at the beginning of staff#2. The stems whose beams were lost because of the missing ledgers have each become part of a TUPLET_THREE. The accents in the first staff were never recognized. There are a few spurious (faint) natural signs, slurs, dots, etc., which I just noticed, but these may have been introduced in an earlier step.
LINKS Some of the deleted x note heads have become augmentation dots. Most of the spurious natural signs disappeared, along with some other spurious marks.
RHYTHMS The 1st and 2nd endings have been fixed. The triplets still aren’t correct.
PAGE A bunch of errors about “No target duration for measures …, please check time signatures.” [I should have fixed the time signatures after the HEADERS step.]
============================================================ Here is some info about my Audiveris settings when I ran the above test:
% git status On branch development Your branch is up to date with 'origin/development'. nothing to commit, working tree clean
Audiveris settings Selected all advanced topics in Tools | Advanced Topics In Tools | Options, set the constant DistancesBuilder.displayTemplates to true Book | Set Book Parameters:
- Binarization: Adaptive, 0.7, 0.9 (defaults)
- OCR language(s): eng (English)
- Use of system indentation: ON (default)
- Support for articulations: ON (default)
- Support for lyrics: OFF
- Support for cross note heads: ON
Twist_And_Shout_TheBeatles_musescore.pdf
Partly based on the above trial run, here is a proposed list of steps for adapting Audiveris to recognize and handle 5-line unhitched percussion scores. Comments and suggestions for improvement of this list are of course welcomed.
Recognize drum clef
Recognize common drum note heads
- Most common–oval and x–are already quite good
- Need to recognize hollow x (half & whole note)
- Don’t delete recognized x’s at REDUCTION step
- Next most important head–diamond (ride bell, tambourine)? Circle-x (open hi-hat)?
- Others much less common–equilateral triangle vertex down (cowbell), slash (brushes), oval inside large circle (brush swirl), …
Chords
- Allow oval and x head on same stem
- Oval and/or x heads on adjacent staff line/space, offset horizontally but part of same chord (e.g. Ride Cymbal 1 and 2, or Ride 1 and Closed Hi-Hat), often sharing a stem in the middle
Voices
- How/why are these important or used in Audiveris?
- How is an Audiveris voice defined/recognized?
- MuseScore 3 supports up to 4 drum voices per staff, but typically only 2 are needed (often distinguished by stems up or down, but perhaps not consistently, especially when one voice is not playing).
- Otherwise, how to recognize/distinguish different voices in drum parts? Or is this even needed?
Other special drum notation to recognize
- Tremolo (roll)–may be single, double, or triple; may be through a single stem (roll on one instrument) or between two noteheads (roll played on two different instruments with alternate sticks)
- Apostrophe (after an x note head)–cymbal choke
- R or L above or below a note–sticking
- Small open circle above an x–another notation for open hi-hat
- Grace note–flam
- Double grace note–drag (maybe also single grace note with a single tremolo slash?)
- Ghost note–notehead small and/or enclosed in parentheses, usually on snare drum
- Single bar repeat and two bar repeat–not unique to drum notation, but quite common due to the repetitive nature of many drum parts.
- And more…?
Export to MusicXML 4.0
- Initially, support MuseScore’s mapping of pair (staff position, note head) to MIDI instrument
- Users can later change the instrument for all occurrences of a particular pair in e.g. MuseScore
- Eventually, support a few of the most common (position, head) to instrument pairings–allow user to select one from a list?
Some glyphs we should never see in a drum score: key signatures or accidentals. Maybe we can make the grade of these inters be very low when the user selects drum mode? Or (in a score with other pitched instrument parts) at least low score on the drum staff.
Anybody know why this single-line percussion score does not process correctly on Audiveris with the Book parameter: Support for percussion staves (1 line) turned ON? I get an error at the SCALE step: "SCALE step failed. Too large interline value: 299 pixels. This sheet does not seem to contain staff lines."
Can't_Take_My_Eyes_Off_of_You-Hand_Clap.pdf
Case of Twist and Shout:
None of the percussion clefs were recognized.
But in the “section-only” selection mode, showing outlines of contiguous black pixels, the drum clef symbols all have portions of the staff lines amalgamated with them.
There is a bug, due to recent fix Reduced minRunLength for LinesRetriever in GRID step
Temporary workaround is to set constant LinesRetriever.minRunLength in Tools | Options back to 1 (I had set it to 0.25).
Then the sections will get correctly erased and all the drum clefs recognized.
Why some staff sections stay stuck to the staff lines when minRunLength is set to a very low value is the bug I need to fix ASAP. Sorry for that, I'm on several urgent tasks at the same time...