Feature discussion - full retina / high res support for openFrameworks

[ofTheo]

For 0.10.0 I would love to get real retina / HiDPI support working in the OF core.

Currently the state of things is that if you set High Resolution Capable via a plist setting you get a window which has a frame buffer with twice as many pixels as the window size. This means that in HiRes mode a 1024x768 window is half the size on your screen as in regular mode.

Currently we're converting all OF window size / position and screen size coordinates by this scale, but that starts to get very confusing when you start doing multi window stuff with a mixture of retina and non retina screens.

The current approach makes it really hard to make an app support both retina and non retina screens as all the scaling and adjustment is left up to the user.

A different approach would be to keep the window size the same for both retina and non retina but setup the viewports and matrixes to make the retina window look the same as the non retina window, with 2x the width and height when it comes to the frame buffer.

If we could make this work, an app that was loaded as retina vs a non retina app would have exactly the same user code, but the retina one would be rendering at a higher pixel density.

I have most of this working in a separate branch - but fonts get quite tricky as you would need to load them in with 2x the DPI and then still display them at the scale and spacing of the non retina dpi.

Anyway, before I get too heavy into this would love any thoughts from people on the matter :)

pinging @braitsch @Xrave @danzeeeman @openframeworks/core

[arturoc]

i'd say that having some setting like point or something that is resolution independent would work best than setting things up with double the pixels asked for. if we can somehow get the dpi from the screen the app is running on then we would could also specify font size in points as is done in css for example.

in a couple of years all screens will be high resolution and if we just double the pixels for some screens we'll end up with a system that is very specific to a transition period for apple hardware and in which the pixels we specify we'll be half of the real number on every screen

[ofTheo]

Totally agree! The getPixelCoordScale function in ofAppGLFWWindow returns the ratio of the frame buffer to window size. In the current case its 2 for HiDPI and 1 for non HiDPI - but it should be pretty future proof if we use that number ( it could be 3x or 4x on other systems ).

In the current branch I have ( which I can push ) I was able to get pretty good parity by setting the viewport bounds in ofMatrixStack to be multiplied by this scale and doing an ofScale at the end of ofSetupScreenPerspective / Ortho ( we could do this part better ). It seemed to work for most things - but the big exception is fonts, which you would want to load at 2x ( or whatever getPixelCoordScale() returns ) the DPI and then draw as if they were 1x. I started on that yesterday, but it will take a little bit of time to iron out.

I'm sure there are some other edge cases and we'd need to see how it affects shaders etc.

From a quick look it seems that cinder takes this approach: https://github.com/cinder/Cinder/blob/663a60bb099b7594aa88a1597ec283da50986592/src/cinder/app/Window.cpp#L113

Here they mention they do the same thing with fonts. https://github.com/cinder/Cinder/blob/663a60bb099b7594aa88a1597ec283da50986592/samples/RetinaSample/src/RetinaSampleApp.cpp#L85

[arturoc]

that feels like a big hack to me :) it seems like a quick way to get old applications to work with high density screens but not like a definitive solution.

we could have something like that to make old apps work with new screens as some kind of helper function or even and addon to set the matrices and viewport with some settings that make and old application with fixed resolution look ok on a new screen but i don't think we should be doing that as the default in the windowing system

by reporting half the resolution and then scaling by 2 you are actually loosing that resolution in some calculations, for example if you want to position some text in the center of a box of 735 you'll end up in 367.5 but if you draw in .5 you'll end up with anti aliasing problems so you usually cast to int to 367. but if the real resolution of the screen is double of that you shouldn't have need to do that because the real size of the box was 1470 and the center 735 so you are unnecessarily moving things by 2 physical pixels.

also this is also not an osx only problem where the resolutions in "retina" screens are perfect multiples of the old screens.. in mobile we have the same problem, android is specially difficult because there's so many resolutions out there so it's also probably a model to look into since it's probably the api that covers more cases.

glfw has methods to get the physical size of the monitor in mm: http://www.glfw.org/docs/latest/group__monitor.html#ga7d8bffc6c55539286a6bd20d32a8d7ea and from that you can get the speicific dpi of the scren, from their docs too:

const double dpi = widthPx / (widthMM / 25.4);

the window classes should report the dpi and once we have the dpi we can have some kind of measure that is resolution independent that we can use to create windows of a certain size, create gui elements that are not tiny in certain screens or set the dpi of the fonts to the real dpi of the screen so they always have the same physical size if we want but i think the pixels should stay as pixels and the matrices shouldn't be doing any scaling bydefault.

[ofTheo]

adjusting based on dpi - seems like a much more thorough solution. I was definitely looking at a a simpler approach, to start with, but I didn't realize we could query DPI directly from GLFW.

I'm curious how would this work at the render level? Would people have to supply units that are DPI adjusted or would we do the adjustments for them internally?

[arturoc]

i'm not completely clear on the details but i guess we could have some mode for the renderers in which you work in points instead of pixels or even some way to pass units instead of a unitless value to some functions.

we should take a look at what are the corresponding apis in ios, osx, android and try to figure out what's a good balance between the easiest/most flexible.

[arturoc]

in any case,

Would people have to supply units that are DPI adjusted or would we do the adjustments for them internally? 

i guess if we have some resolution independent unit then no, what seems awkward is to fix the resolution independent unit to what was pixels in most screens up until 2014 :)

[ofTheo]

Just trying to wrap my head around this a bit :)

So essentially all drawing commands like: ofDrawCircle(100, 100, 50); would now be in points?

And as a result a circle of 50 points would be the same size on any screen? ofGetWidth() / ofGetScreenWidth() would then be points I assume.

Any call which works with pixels ofPixels etc would still be in pixels?

Would we run into AA issues by working in points as: ofDrawRect(0, 0, 50, 50); The 50 points might not correspond by a non floating point ratio to pixels?

[ofTheo]

I should note currently on iOS if you work in points you get layouts that look different between devices. Thats because the screen resolutions are different and iOS aims to have UI elements be physically the same size on different screens ( for usability of buttons sliders etc ). This means though that an app can have quite a different look / layout depending on the number of screen pixels and the dpi. This type of approach could be useful in some situations in OF ( and I can imagine on mobile this is more more important ), but I don't think this should be the default behavior on Desktop apps.

I think whichever approach we go with the default should be that an app laid out on a non Retina screen looks the same on a Retina one ( unless a user actively wants the two to look different ).

[braitsch]

Hi, I would discourage using points as the unit of measure as 1) they're traditionally a measurement associated with typography in print 2) I think conceptually most people just think in terms of pixels.

Perhaps consider how web browsers handle this issue. On any given display JavaScript returns the pixel dimensions irrespective of the screens dpi and provides an api for querying the screen resolution that the developer can use to load in the appropriate image assets.

For example on my 15" rMBP which has a screen resolution 2880x1800 JavaScript returns the following:

console.log(window.innerWidth) // 1200 -> width of my browser
console.log(screen.width) //  1440  -> width of my screen
console.log(window.devicePixelRatio) // 2 -> indicates retina

I would encourage having calls to ofGetWidth() et al return the lowest common denominator e.g 1x the actual screen dpi and let the user create multiple sets of bitmap assets specific to each dpi they want to support just as they do for the web.

[elliotwoods]

To note:

• 2x scaling is a special case. Most people I know on OSX use 'more space', i.e. <2x scaling. Also on Windows you rarely choose the 200% option, the default is often 150% for 4k screens. Maybe we can presume 2x for current generation iPhones and iPads only, but it seems pointless to refer to '2x' as a usual case when it's actually decreasingly common.
• Should we ditch the term 'DPI'? It seems to add to the confusion in this issue.

Other situations where this can become confusing:

• Allocating anything from ofGetWidth() is a scaled parameter
• ofFbo (is it scaled inside the fbo?)
• ofViewport
• Mouse coordinates
• Window placement (often this becomes quite buggy with scaling, with glfw using a mix of pixels and points)
• Moving windows between monitors with different scaling settings (to act properly, the window should adopt the scaling setting of whichever screen it is currently on, so when you move a window it can cause a change).

I'm for keeping the oF API in pixels because it keeps things consistent (and less likely to break projects). I presume the main reason for using 'points' is when you're are targeting multiple devices whilst being strict about on-screen layout (e.g. mobile apps rather than installation works).

I presume whatever we do will:

• Have the scaling behaviour defined in ofWindowSettings
• Will mean we have to lazy load fonts so they can see which screen they're drawing on (or perhaps just check the scaling setting of all active windows when they are loaded).
• Need a new ofDrawBitmapString font

I propose we:

• Keep all positioning and scaling code in pixels
• Give an option to interpret font sizes as points
• Provide useful functions to transfer points<->pixels (including transforms)
• Enable ofDrawBitmapString to be able to draw nicely at multiple scales (and be able to auto-select this scale based on current screen scaling)
• Provide options for 'scaling whole app' for backwards compatibility, it'll generally look quite rough but I can see that running old projects is perhaps the most common scenario for scaling (i.e. if pixels are even smaller in 5 years time than they are now, you might still not want to faff around now with points vs pixels at every step just so it won't look tiny or resampled in 5 years time)

[arturoc]

yeah i'm not specifically talking about points it might be anything that is not resolution dependent. seems that devicePixelRatio in browsers is based on "device independent pixels" which varies across platforms: https://en.wikipedia.org/wiki/Device_independent_pixel it still seems like an ugly hack that won't make any sense in a couple of years but seems to be a notion that exists on every platform.

for fonts i guess using the native dpi is what makes most sense probably

we also need to decide for which settings this makes sense. as i said before this for 3d is probably not necessary, or it doesn't make much sense, i'd rather set the camera distances... in meters than in some weird pixels unit that is not even physical pixels

[ofTheo]

Thanks for all the feedback on this - I knew it would be a tricky issue, but didn't realize there would be so much to it.

I'm going to play around a bit with some different approaches and see how it feels - as its quite hard to think about in the abstract at the moment.

[elliotwoods]

ofGetPointsPerPixel()?

[sergio-u]

Is support for retina & high res planned for Linux or Windows?

Also is there any way to get basic sketches with proper scaling working on Linux or should another issue be opened?

[cerupcat]

I'd love to see this integrated. I started a discussion about how people are currently doing this since it's not part of core. Are there any current recommended ways of doing this? This is an issue especially on mobile devices right now.

https://forum.openframeworks.cc/t/is-there-an-official-guide-for-retina-devices/24225

I'm not sure what the solution should be, but ideally it'd be great for the non-core OF code to completely unaware whether it's on a retina device or not (unless we optionally want to load different assets based on dp). Coordinates, px, etc should all look and behave exactly the same despite the full (retina) resolution of the device.

For two of my iOS apps, I currently use https://github.com/armadillu/ofxEasyRetina (with modifications to work with 009), but it's not ideal and feels like a hack. I also have been using a wrapper around ofTrueTypeFont and ofImage where it creates them at size * scale and then scales it down to display at 1x which. This isn't ideal and means when OF updates occasionally all these pieces have to also be updated.

    scale = 1 / scale;
    if (retina){
        ofPushMatrix();
        ofTranslate(x, y);
        ofScale(scale, scale, 1.0f);
        ofTrueTypeFont::drawString(s,0,0);
        ofPopMatrix();
    }else{
        ofTrueTypeFont::drawString(s,x,y);
    }

[michalfapso]

We are currently using Qt in our openframeworks app to get the system's dpi scaling factor on both mac and windows:

#include <QDesktopWidget>
#include <QApplication>

void get_system_dpi()
{
    const int BASE_DPI = 96;
    ofRectangle r = ofGetWindowRect();
    r.setX(ofGetWindowPositionX());
    r.setY(ofGetWindowPositionY());
    
    QPoint posGlobal(r.getCenter().x, r.getCenter().y);
    QDesktopWidget* m = qApp->desktop();
    int sn = m->screenNumber(posGlobal);
    if (m->geometry().contains(posGlobal)) {
        auto screen = m->screen(sn);
        int dpi = (float)(screen->logicalDpiX() + screen->logicalDpiY()) / 2.f;
        return (float)dpi / BASE_DPI;
    }
    return 1.0f;
}

We are using Qt in our application anyway, but for an OF-only project, your app's size would grow significantly if you include Qt libraries.

[danomatika]

As far as macOS & iOS, from this and this it looks like there a number of native options:

• macOS:
  • [NSWindow convertRectToBacking:(NSRect)rect] method
  • [NSWindow backingScale] getter, although its recommended to use the convert method
  • there is also [NSScreen backingScaleFactor] but it's probably safer to address things at a per-window as multiple windows could be on differently scaled screens
• iOS:
  • UIScreen scale and nativeScale getters, this is probably safe enough since these devices are essentially single screen so far

There are other projects which have already solved this issue years ago. It might be good to see what they needed to do to make it work. For instance, this seems like only a few lines to hint to the system to scale the NSOpenGLView coordinates: https://github.com/SFML/SFML/pull/388. Still, I image it's still more involved for OF as it loads things at a lower level (fonts, etc).

I'd think this would best be handled in the platform-specific window and renderer classes where the sizes could be scaled more transparently. When there are screen changes, I think we could use the event system. Objects which would need to recalculate their sizes based on DPI changes via some sort of "pointSizeChanged()" event.

EDIT: Further, this should be handled transparently by default. As in, a 1024x767 window is doubled when the screen is 2x and a "pixel's" size is effectively doubled. This works pretty well on iOS using native frameworks. It may not be the "best" solution but it's a "working" solution that most platforms have adopted. Naturally, there should be some switches for advanced users who want to address the full resolution directly and/or control scaling themselves.

I agree with Elliot's proposal. Unless OF wants to radically change to supporting more of an abstract "unit-less" default, it's the most practical move forward.

EDIT2: I think the iOS/macOS approach could work in general. The scaling could be done within a set of scaling functions in the Window classes like convertRectToBacking ala convertPointTo..., convertSizeTo..., & convertRectTo.... This way things could be abstracted within the classes using them without hacking the core up to much and advanced users could leverage them later in their own projects / addons. This also limits the conversion math to a place where they could be more easily turned into a compile-time feature with some defines.

As for ofTrueTypeFont, it could have a static map that keeps track of the currently open fonts and sizes using the font name and/or path as a unique key. When there are screen changes, a newer font size could be lazy loaded on demand. This would essentially be built-in resource management but would at least get rid of the need to try and pre-load all of the sizes we think might be needed...

[ofTheo]

@danomatika I also like the idea of exploring a minimal system that lets you work at 2x / 3x etc. We could build some internals that we could later expose for those that want more control, but I think most users ( at the moment at least ), want to be able to use a retina screen as if it was at the non retina size and not have to think or be aware of the conversions.

I think exploring what can be done via GLFW first would be good as otherwise we would need to build a whole separate window system for macOS.

GLFW just added a callback like you mentioned for knowing when the screen content scale changed. https://github.com/glfw/glfw/commit/370eac3c48d00facd44ac0dc61c651232e417bf0#diff-7a6c70f06dc23e69c5a9bfe9c3967e64R1281

@danomatika Maybe we could explore something in a branch and see what it would really take? I am guessing it will be more than just fonts - viewports, pixel read back etc.

[dimitre]

A metaphor I think it is not so complex to follow is something like ofScale But instead of only graphics it applies to the mouse / touch as well.

I always thought of something like that when synchronizing some parts of a interface to an ipad. As the fingers are not so "pointy" as the mouse I felt if there was a way of just scaling everything including the coordinates would be great.

In fact mouse/touch transformation matrixes would be helpful on rotateMouseXY too OF_ORIENTATION, etc. Fonts based on shapes would be automatically scaled, everything raster is more tricky, but if it is very important for somebody to have everything running great in two different resolutions then a callback can be used.

[ofTheo]

@dimitre I think it would be worth trying! Also I realized that we can also load fonts with variable dpi - so another way would be to change the default dpi of a font based on the retina ratio.

[dimitre]

@ofTheo the nice thing about this approach it can be implemented gradually, even if fonts don't work 100% in the first iteration it can be improved later. ofGLFWWindowSettings could have a parameter to enable it. something like:

settings.enableRetinaScaling = true;

[danomatika]

The other option is to ask the system. There is a getter for the old OpenGLView AppKit class to check of High DPI is enabled for the view. OF could implicitly use this by default instead of requiring the user to set it. This approach also respects the High DPI check box in the apps Get Info dialog and the setting in Info.plist.

enohp ym morf tnes

Dan Wilcox danomatika.com robotcowboy.com

On May 7, 2022, at 5:22 PM, Dimitre @.***> wrote:

 @ofTheo the nice thing about this approach it can be implemented gradually, even if fonts don't work 100% in the first iteration it can be improved later. ofGLFWWindowSettings could have a parameter to enable it. something like:

settings.enableRetinaScaling = true; — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[ofTheo]

@dimitre I gave it a quick go with the fontsExample.

adding:

  ofAppGLFWWindow * winPtr = (ofAppGLFWWindow*)ofGetWindowPtr();
  ofSetWindowShape(800  * winPtr->getPixelScreenCoordScale(), 600 * winPtr->getPixelScreenCoordScale() );

To the end of ofApp::setup

And

  ofScale(2, 2, 1);

to the top of ofApp::draw

With retina enabled in the plist made things look the same scale on screen as with no retina.

However changing the font dpi made things totally broken as all the layout and spacing is assuming letters 2x the size, whereas we just want the font to load the textures at 2x the size but keep the spacing, font size the same.

I do think it is possible, but it would take some digging into ofTTF to make it all work seamlessly. If you wanted to take a stab at it the fontsExample is a good test to check it all works correctly.

[dimitre]

@ofTheo great! I'm testing now with high resolution capable = YES and put this inside draw

	ofAppGLFWWindow * winPtr = (ofAppGLFWWindow*)ofGetWindowPtr();
	ofSetWindowShape(800  * winPtr->getPixelScreenCoordScale(), 600 * winPtr->getPixelScreenCoordScale() );
	ofScale(winPtr->getPixelScreenCoordScale(), winPtr->getPixelScreenCoordScale(), 1);

so it updates when I change from retina to non retina screen. and it works great. identical from what I see here. I have to cout scale to be sure the framebuffer was updating resolution

[dimitre]

@ofTheo yeah things get broken with setGlobalDpi. And I've just noticed it is default 96 dpi. Maybe this is why when I have a pixel font with 8 pixels high I have to raster it using 6px size to display correctly.

[ofTheo]

Yeah there is a lot of weird stuff there ( even the fontsExample sets it to 72 dpi for the same reason you mention ).

Aside from that weirdness it would be awesome to have the option to load the glyph textures at 2x 3x etc res while respecting the font size passed in.

[dimitre]

Hey @ofTheo this can be fixed like this:

I'm scaling the font size parameters to the density of the screen (dpi) and I think fontUnitScale can be totally ignored from now on. so no metrics are scaled inside OF, just in Freetype raster.

fontUnitScale should be totally removed if it makes sense.

in ofTrueTypeFont.cpp

	float scale = 72.0/settings.dpi;
	int size = int(scale*settings.fontSize) << 6;
	FT_Set_Char_Size( face.get(), size, size, settings.dpi, settings.dpi);
	fontUnitScale = 1;

[ofTheo]

Hey @dimitre that looks promising, but not sure if it is actually resulting in more detail. To check, I tried both:

//should be 72 x 2 dpi

	ofTrueTypeFont::setGlobalDpi(72 * winPtr->getPixelScreenCoordScale());

and

	ofTrueTypeFont::setGlobalDpi(18);

They do render the same size, but the second one should be much less clear and it looks the same for me. I think because the font size is compensating for the dpi scale, so either way the same overall resolution is loaded.

[dimitre]

Yeah indeed not working haha. I'll look a bit more into this

[dimitre]

@ofTheo I've opened other thread for this particular ofTTF issue. about the retina, even if fonts are not rendered in high res at first I think it worth advancing the rest.

it would be great to have something similar to ofScale, but storable, like the variables in currentStyle, ex: ofSetColor Almost as ofSetScale but maybe a different function name to remind it is not only the scale but the mouse / touch coordinates as well. maybe ofSetDensity or something?