ImGui full integration

[d3cod3]

• [x] Menus
• [x] Windows
• [x] Canvas nodes
• [x] Duplicate/Delete object selection
• [ ] Add objects selection group displacement
• [ ] Add sub-patch navigation tree with current patch view switch

[d3cod3]

At this point in time Mosaic 7514384f18cd87ea9073cf8595fabc26ca3b26e7 ofxVisualProgramming 719b3d48f5c1b3411ec41b0bc8bd811d858404e3 we have:

• good mouse interaction with nodes
• dragging by header only
• wiring not rendering ok after moving nodes
• pins labels easily overlapping with other elements
• pins vertical positions not overlapping with old pins system
• still rendering old ofxDatGui nodes (to remove)

[d3cod3]

Just detected a problem with activating DSP, it must be something related with the instantiation of the audio device object, probably because of his "different" nature (system object), but need further analysis.

it's crashing here:

PatchObject::drawImGuiNode(ImGuiEx::NodeCanvas&) + 944

can be because this object is inited with zero inlets/outlets, then in the setup connect with the audio config and dynamically add the required inlets/outlets?

if i avoid calling the drawImGuiNode method just for this object (for testing) it doesn't crash until i mouse over the object.

Do you have some suggestions?

[d3cod3]

Yes, it is related with dynamic inlets/outlet creation!

Using other objects with this kind of feature (osc receiving/sending, shader object, etc...) crashes Mosaic!

[Daandelange]

I got the same crash, looks like a call to if(!inited)return; in drawImGui() is missing in drawImGuiNode, or something related. The crashing line is inletsPositionOF[i] = ofVec2f( ... );, probably because the vector has not been re-scaled since PatchObject::PatchObject().

[d3cod3]

Yes, i see, my mistake, i added those vector<ofVec2f> in the last changes while porting links rendering to imgui.

easy to fix!

[d3cod3]

Added:

• audio signal plotter
• Var plotter
• Bang

[d3cod3]

I'm checking imgui extensions and i found this:

https://github.com/AirGuanZ/imgui-filebrowser

Do you think it would be good idea to integrate file dialogs in imgui too?

Right now i'm using the tinyfiledialogs libs wrappes in an addon:

https://github.com/d3cod3/ofxThreadedFileDialog

that works a lot smoother than the standard OF file dialogs, plus it's non-blocking, so the patch is still running while opening/saving some file from/to disk.

Leaving aside the design (i would like it more to have imgui themed file dialogs windows), what is your opinion about that?

I think we need to consider two main points:

1. Custom file dialog windows can be a little confusing for beginner users, as do not "look the same as always"

2. It will increase Mosaic performance/usability in general?

[d3cod3]

Having some compile issues with the previous filebrowser repo

testing this: https://github.com/gallickgunner/ImGui-Addons

[Daandelange]

[first ImGuiEx link] :: "C++17 is required" ==> I recently read that OF doesn't compile out of the box with C++17 (C++14 yes). Maybe this has changed, just noticing.

But it's a good idea to do file dialogs only with ImGui, yes. Although current implementation is fine too. Anyways we gotta keep saving native in ofxVP (and threaded??). [ "saving" as file writing, not as path picking ]

Now, personally, I both like and dislike native system dialogs.  - Bad: They are blocking popups that integrate badly with custom UIs, leaving the app inaccessible. Here we could have a great gain in interface logic, having the file browser integrated in the UI. (I already imagine dragging image from file panel to the canvas, and let it spawn as imageObjectA or imageObjectB)  - Good: I like the default file browser where I have my shortcuts, drag'n'drop support, and other habits. So that would be a minus, except that we can make it better too. I've had a few great experiences with custom UIs.

So, yes and no. I'd say yes, specially batch file-dialog usage can become faster, with a better integration. And we can always provide an optional native file dialog fallback as an ImGui Button. This said, the ImGui extension must be robust and smooth.

Note: upcoming ImGui versions will have native support for file drag/drop; and there's already something official working. Can also help to make a choice. +1 for the design upgrade too with ImGui.

Another file browser : ImGuiAddons / imguifilesystem (web live demo (filesystem section) )

This ImGuiEx looks great too : https://bitbucket.org/wolfpld/tracy/src/master/ (ofxTimeMesurements is fine too)

[d3cod3]

I'm using the code from https://github.com/gallickgunner/ImGui-Addons

I needed to update imgui to 1.76 (pushed commit on my ofxImGui fork), and it seems to work real smooth, and this is the temp testing code:

// TESTING ImGui FileDialog
    if(ImGui::Begin("dummy window"))
    {
        bool open = false, save = false;
        // open file dialog when user clicks this button
        if(ImGui::Button("open file dialog"))
            open = true;
        if(ImGui::Button("save file dialog"))
            save = true;

        //Remember the name to ImGui::OpenPopup() and showFileDialog() must be same...
        if(open)
            ImGui::OpenPopup("Open File");
        if(save)
            ImGui::OpenPopup("Save File");

        if( imguiFileDialog.showFileDialog("Open File", imgui_addons::ImGuiFileBrowser::DialogMode::OPEN, ImVec2(700, 310)) )
        {
            std::cout << imguiFileDialog.selected_fn << std::endl;      // The name of the selected file or directory in case of Select Directory dialog mode
            std::cout << imguiFileDialog.selected_path << std::endl;    // The absolute path to the selected file
        }
        if( imguiFileDialog.showFileDialog("Save File", imgui_addons::ImGuiFileBrowser::DialogMode::SAVE, ImVec2(700, 310)) )
        {
            std::cout << imguiFileDialog.selected_fn << std::endl;      // The name of the selected file or directory in case of Select Directory dialog mode
            std::cout << imguiFileDialog.selected_path << std::endl;    // The absolute path to the selected file
            std::cout << imguiFileDialog.ext << std::endl;              // Access ext separately (For SAVE mode)
            //Do writing of files based on extension here
        }
    }
    ImGui::End();

The source code seams neat, and there are three modes implemented, OPEN DIRECTORY, OPEN FILE and SAVE FILE.

With that we can substitute the actual implementation without problems, then we'll be able to add drag&drop functionalities. Right now it is already implemented an object spawning mechanism when dropping specific filetypes ( video files, script files, audio files, image files, etc... ) inside the canvas ( using OF dragEvent ) so it will not be difficult to do the port.

I think it seems a good codebase to include in ofxVP, i'm going to test it a little deeper.

[d3cod3]

I'm still working on specific objects graphics/interface with imgui, porting all the OF stuff i made before, file dialogs are completely ported, textures are working in some cases (images YES, video NO, it broke after updating imgui to 1.76), variable plotting and audio signal plotting are working fine, and right now i'm with area constrained draggable points editor for Audio Envelope editing for sound objects.

[d3cod3]

Checking this for advanced data plot:

https://github.com/epezent/implot

maybe for some new objects? we'll see

[Daandelange]

Yes, looks useful. There's a lot of interesting ImGuiExtensions, good idea to start listing some.

[d3cod3]

Just integrated a profiler, so less future unmantained ofxaddons

modified from https://github.com/Raikiri/LegitProfiler

[d3cod3]

[d3cod3]

Some objects are not finished yet, but is starting to look serious

[d3cod3]

[d3cod3]

About the textures drawing between OF and ImGui, i had it solved from some time now, but testing some bigger patches i've found ( and already solved ) a problem:

The core of the problem is the same one related with the plugin system #16 , the openGL limitation to share GPU stuff in the same thread only.

Basically we have:

1. the OF openGL thread
2. the ImGui openGL thread
3. ( plugin system ) each plugin loaded in runtime will have his personal openGL thread

So, there is a way of share data between openGL threads, and it is using pixels data ( meaning, leave the GPU and passing through the CPU ) GPU (thread 1) --> texture to pixels --> CPU --> pixels to texture --> GPU (thread 2)

Reference: https://openframeworks.cc/ofBook/chapters/threads.html

Another way is using PBO, i absolutely have no experience with that, but is never late for learning, so i found this:

http://www.songho.ca/opengl/gl_pbo.html#unpack

with time i'll check it and try to implement it!

In the meantime, having all this bridging via pixels: OF --> texture to pixels --> CPU --> pixel to texture --> ImGui, was costing a lot in computation, and with dense patches, especially the patches with DSP stuff, it was causing problems and sound glitches

so

i temporarily fixed it with a trick, for all the texture visualizing objects i draw the background of the ImGui node transparent:

// Node BG fill
        if(curNodeData.zoomName < ImGuiExNodeZoom_Small || !isTextureNode){
            nodeDrawList->AddRectFilled( curNodeData.outerContentBox.Min, curNodeData.outerContentBox.Max, ImGui::GetColorU32(ImGuiCol_FrameBg, 999.f));
        }else{
            nodeDrawList->AddRectFilled( curNodeData.outerContentBox.Min, curNodeData.outerContentBox.Max, IM_COL32(0,0,0,0));
            nodeDrawList->AddRectFilled( curNodeData.outerContentBox.Min, ImVec2(curNodeData.outerContentBox.Min.x+curNodeData.leftPins.region.GetSize().x,curNodeData.outerContentBox.Max.y), ImGui::GetColorU32(ImGuiCol_FrameBg, 999.f));
            nodeDrawList->AddRectFilled( curNodeData.rightPins.region.Min, curNodeData.outerContentBox.Max, ImGui::GetColorU32(ImGuiCol_FrameBg, 999.f));
        }

and then i draw from OF the texture directly in the correct node position:

inline void drawNodeOFTexture(ofTexture &tex, float &px, float &py, float &w, float &h, float originX, float originY, float scaledW, float scaledH, float zoom, float footerH){

    if(tex.isAllocated()){
        if(tex.getWidth()/tex.getHeight() >= scaledW/scaledH){
            if(tex.getWidth() > tex.getHeight()){   // horizontal texture
                w           = scaledW;
                h           = (scaledW/tex.getWidth())*tex.getHeight();
                px          = 0;
                py          = (scaledH-h)/2.0f;
            }else{ // vertical texture
                w           = (tex.getWidth()*scaledH)/tex.getHeight();
                h           = scaledH;
                px          = (scaledW-w)/2.0f;
                py          = 0;
            }
        }else{ // always considered vertical texture
            w               = (tex.getWidth()*scaledH)/tex.getHeight();
            h               = scaledH;
            px              = (scaledW-w)/2.0f;
            py              = 0;
        }

        if(scaledW*zoom >= 90.0f){
            // background
            ofSetColor(34,34,34);
            ofDrawRectangle(originX,originY,scaledW-2,scaledH+(footerH/zoom));
            // texture
            ofSetColor(255);
            tex.draw(px+originX,py+originY,w-2,h);
        }
    }else{
        if(scaledW*zoom >= 90.0f){
            // background
            ofSetColor(34,34,34);
            ofDrawRectangle(originX,originY,scaledW-2,scaledH+(footerH/zoom));
        }
    }

}

with that, Mosaic performance is back to the previous version, a lot better

Do you have experience with PBO? We can try to solve this later, right now is a hacky solution, but is working perfectly, with one quirk, over texture objects, cables appear in front!

No sweat, i mean, better this than loosing performance at 4 texture objects running in the same patch!

[Daandelange]

I have looked into PBOs some time ago; don't know it well enough. Sounds like an accelerated version of pixels2texture. I guess OF's built-in pixels2texture methods use the same feature ?

Anyways, it's really not a good idea to do useless pixel2texture (or vice-versa) operations. Only do them when needed (image decoding, image acquisition, ...). Also, using big resolutions, these operations become exponentially slow. Is there no way of sharing GL contexts with plugins ?

What about this ? I see glRenderer is not yet used... Is this approach not working with plugins ?

void nodeObject::drawObjectContent(ofxFontStash *font, shared_ptr<ofBaseGLRenderer>& glRenderer){
	glRenderer->begin();
	ofPushStyle();
	ofSetColor(255,0,0);
	ofNoFill();
	ofDrawRectangle(100,100,100,100);
	glRenderer->end();
}

Another (dummy?) question : Is it possible to load plugins on the GL thread so they don't interfere with it ?

[d3cod3]

Yes, glRenderer is there exactly for the plugins, for basic drawing routines as the ofDrawRectangle , or font drawing, or everything that is computed by CPU, but for textures and shaders are not working (GPU), the only way right now is bridging through pixels ( really expensive )

Is not crashing or anything, just the texture loose the reference and remain black, or with artifacts from last memory region access.

Of built in texture to pixels didn't use PBO, is a heavy load CPU routine, but there is this, from an addon and included in ofxVisualProgramming:

https://github.com/d3cod3/ofxVisualProgramming/blob/refactor-params-and-gui/src/objects/video/ofxFastFboReader.h

I've tested it as it is, but is not working in this particular case yet ( access from two threads )

And no, for the plugin structure as it is, i didn't see the option of loading it inside the openGL thread, but i'll search the web for some info

[Daandelange]

If the plugin system brings a big performance loss (forcing useless GPU<-->CPU operations), is it worth having plugins ? Let's close this parenthesis and keep plugin talk in #30 .

So here, it's more about the issue of drawing textures from ImGui for previewing purposes, if I understand correctly. The original approach needed CPU<-->GPU operations, so you found a temporary transparency hack, which should better be done in another way ?

[d3cod3]

This problem is only related to "texture" nodes, basically every objects that works with image in some way, so all the other type of objects do not have problems, hence is still worth having a plugin system, even is "texture" type objects will be discarded.

Probably this problem will appear in the future surface nodes #24 , but we'll see when we get there

The hack with transparency is working great, performance is back at good levels, so for now it doesn't worry me, but i'm going to check this http://www.songho.ca/opengl/gl_pbo.html#unpack, to see if i find a way of optimize the code

and passing from this

to this

and end up implementig this

And that could be even the solution for the plugin problem ( not 100% sure, but... )

[Daandelange]

Yes, the hack is fine in terms of performance. Eventually, if we come back on this, there seems to be ways to do it by passing a function pointer callback (to drawNodeOFTexture from ImGui?) : imgui/issues/1850

[d3cod3]

I like the option of function pointer callback, i'll give it a try

[d3cod3]

function pointer callback not working! same problem, the texture id is not being shared correctly so texture will not appear

[d3cod3]

Removing the subpatches/abstraction idea, i tested something and i didn't like the multiple patches windows that will end appearing inevitably, it will lead to visual confusion due to the highly graphics nature of Mosaic ( while it works fine in PD due to the really minimal graphical style ). So leaving this option, for now at least.

[Daandelange]

Instead of opening new windows we could "zoom into" objects to view their sub-patches. Manually mouse-zooming and/or a trigger button to zoom an object full screen. Not sure this will allow infinite sub-patches and it might require some important changes to imguiNodeCanvas.

Another solution that could be great is to replace the root canvas (gui) by the child canvas, together with a sub-patch-navigation-bar like [ Root < Subpatch < SubSubPatch ].

[d3cod3]

YES!!! great idea the navigation-bar one! Substitute the root canvas to the current one it can works perfectly.

amazing, i'll add it to the todo list!

[d3cod3]

Just updated imgui to 1.77 WIP, and enabled docking

I'm working on the subpatches and navigation bar issue, i'll post some screenshot soon

[d3cod3]

And fine tuned the file browser

[Daandelange]

Wow, nice for the docking, I just checked it out. There's a small problem with the object selector: it works good while docked (nice!), but with right click, it doesn't disappear anymore... (and how large is your screen!?)

Btw, regarding #37, jvcleave/ofxImGui now also is on 1.77WIP. Haven't checked how the implementation evolved. Worth a look to use a non-forked version ?

[d3cod3]

Yes, about the right click, i just disabled that, the object selector window appear with right click, and can disappear using the close ( X ) button. I was trying different options and it seems like the most usable one, because right click can be triggered by mistake, or because i've added a small right click menu to the code editor ( i'm having a lot of feedback from students right now ).

The screen is a double 22'' HD with a matrox dual head, so 3840x1080 ( it would be nice to have one of this new 47'' curve screens, but that would need a bigger room... )

About the original repo, i just updated my fork with most of the commit/corrections, plus some details more, and tried compiling Mosaic with the original too, but some errors appears, and didn't have time now to look into it, so for now i'll still use the fork, but yes, the idea is to switch, in time, to the original repos where possible ( depending of the maintaining in the original one ), creating specific PR where needed.

I have already switched two addons to the original ones, ofxMidi and ofxGLEditor, and i'm working on ofxLua now, a little bit complicated.

If you want to check ofxImGui in the meantime, feel free to give it a go!

[Daandelange]

I've started some work @ https://github.com/Daandelange/ofxImGui/tree/jvcleave. I've summed up our changes and pulled d3code/ofxImGui into jvcleave/ofxImGui, keeping whitespaces from jvcleave. It should not be hard to merge into your branch.

Regarding changes, there are mostly feature additions, except for the ImGui context implementation. Some need it for sharing ImGui instances across multiple window instances, we need it to do multiple ofxImGui::begin() and end() calls in a single draw() frame. (Maybe multi windows too?) So the ImGui multiple context stuff gotta be cleaned-up, not breaking the api, and tested in several setups. I'm gonna check https://github.com/jvcleave/ofxImGui/pull/91 and clean it up.

Also, it looks like the engine implementations are different and might need to be tested.

Why are the F1-12 keys commented in the GLFW engine, not in the vk engine ?

[d3cod3]

Nice to hear, i'm sure it will be solved and made it compatible with Mosaic soon.

The Fn keys are commented because it appear as undeclared identifier, don't know why ( it's probably related with GLFW ), but if you search on imgui code, they are not implemented.

[Daandelange]

Going forwards : https://github.com/Daandelange/ofxImGui/tree/jvcleave

I've also made 2 commits to ofxVP & Mosaic, to make them compatible with the upcoming ofxImGui, for when it will be merged into Mosaic. (works fine already, tested on Osx)

• https://github.com/Daandelange/Mosaic/tree/ofximgui-next
• https://github.com/Daandelange/ofxVisualProgramming/tree/ofximgui-next

[d3cod3]

Good news!

I'll test it on linux too then and report back.

[d3cod3]

Hey Daan, sorry for the long delay, ImGui checked, all working fine, we can switch to your fork!

Maybe merging the jvcleave into master? Anyway, let me know when and i'll change the installers code and the READMEs

[Daandelange]

Hey Emanuele, Ok, great :) I also got a successful report from moebius (tested on Windows, used as dependency in an addon), with some additional features. I'll push changes to master and let you know.