Volume control scale
It sounds to me like the volume control does not scale correctly. 33% to 66% sounds like a small increase but 66% to 100% sounds a lot louder. I wonder if perhaps it has been scaled linearly, where it should be logarithmic.
It probably is linear as the Atari ST core also uses linear scaling. This was never meant to be the main volume control. Your display hopefully has its own volume control. The volume control inside the core was mainly meant to bring the audio volume into some sane range the display nicely copes with. One of my cheap screens e.g. has quite distorted audio if the volume of the input signal is too high although this is a pure digital thing and even full volume is transmitted lossless (e.g. no clipping) and should replay just fine. But it doesn't unless i set the core to not use the full digital signal range.
It's not even linear. It's actually 1/4 (33%), 1/2 (66%) or 1 (100%) as sound scaling then is just some bit shifting. See:
https://github.com/harbaum/MiSTeryNano/blob/58f5a73f55c89a65a9a11aea7cc5ac41028b68dc/src/misterynano.sv#L488
So we might better name the levels 25%, 50% and 100% ...
I've built a portable machine using this core, and it was a bit awkward to find a suitable low profile pot to mount near the amplifier board inside the lid, so I've been relying on the software control so far:
It's not a big deal really, The existing options are usable enough, I just thought they could do with a tweak to bring them more in line with perceived loudness.
So we might better name the levels 25%, 50% and 100% ...
Actually, this description does sound a lot closer to what I am hearing - a big jump between 50-100 and a small difference between 25-50.