New video objects

[Daandelange]

Creating this PR to add some comments until it's ready for merging. (do not merge yet !)

New objects

• [x] TextureJpegCompression : Adds JPEG artefacts to pixels !
• [x] TextureInformation : Retrieves texture information.
• [ ] FaceTracker : Tracks & detects faces on pixels
• [ ] UvcController : (osx only)
• [ ] VideoGrabberAdvanced : Advanced webcam object (platform specific implementation)

Other new stuff:

• About Mosaic_ : New GPU information section. (see sample below)

  --------------------------------
  GPU Information:
  GL SL 4.1 (programmable)
  Shaders: GLSL 4 (#version 410 core)
  Vendor : NVIDIA Corporation
  GPU : NVIDIA GeForce GTX 760 OpenGL Engine
  OpenGL ver. 4.1 NVIDIA-10.12.65 355.10.05.05b16
  GLSL ver. 4.10 / 410
  GL Extensions (47) :
  GL_ARB_blend_func_extended, GL_ARB_draw_buffers_blend, GL_ARB_draw_indirect, GL_ARB_ES2_compatibility, GL_ARB_explicit_attrib_location, GL_ARB_gpu_shader_fp64, GL_ARB_gpu_shader5, GL_ARB_instanced_arrays, GL_ARB_internalformat_query, GL_ARB_occlusion_query2, GL_ARB_sample_shading, GL_ARB_sampler_objects, GL_ARB_separate_shader_objects, GL_ARB_shader_bit_encoding, GL_ARB_shader_subroutine, GL_ARB_shading_language_include, GL_ARB_tessellation_shader, GL_ARB_texture_buffer_object_rgb32, GL_ARB_texture_cube_map_array, GL_ARB_texture_gather, GL_ARB_texture_query_lod, GL_ARB_texture_rgb10_a2ui, GL_ARB_texture_storage, GL_ARB_texture_swizzle, GL_ARB_timer_query, GL_ARB_transform_feedback2, GL_ARB_transform_feedback3, GL_ARB_vertex_attrib_64bit, GL_ARB_vertex_type_2_10_10_10_rev, GL_ARB_viewport_array, GL_EXT_debug_label, GL_EXT_debug_marker, GL_EXT_depth_bounds_test, GL_EXT_framebuffer_multisample_blit_scaled, GL_EXT_texture_compression_s3tc, GL_EXT_texture_filter_anisotropic, GL_EXT_texture_mirror_clamp, GL_EXT_texture_sRGB_decode, GL_APPLE_client_storage, GL_APPLE_container_object_shareable, GL_APPLE_flush_render, GL_APPLE_object_purgeable, GL_APPLE_rgb_422, GL_APPLE_row_bytes, GL_APPLE_texture_range, GL_ATI_texture_mirror_once, GL_NV_texture_barrier
  

Notes :

• [x] Adds ofxTurboJpeg as a dependency, which also needs some dynamic library fixes. (only used to load lib turbojpeg, no ofxUvc code is used.)
• [ ] Adds ofxUVC as a dependency (osx only)
• [ ] Adds ofxDlib as a dependency (for face detection, comes with big datasets)

Todo :

• [ ] Test all objects on a multi platform basis :
  • [x] Osx
  • [ ] Linux
  • [ ] Windows
• [ ] Document the new objects
• [ ] Add dependencies to MosaicInstaller

[Daandelange]

@d3cod3 a small question : I need a pixels videoGrabber object (not outputting to a texture which needs a useless textureToPixels node) for a webcam. Also, I need to control the webcam with UVC controls (no led, fixed exposure, etc.). Note that UVC might not work for all platforms and only for a restrictive list of webcams. ofxUVC is still osx-only, but I'm investigating other multi-platform UVC libraries. What would be your favour ?

1. An AdvancedWebcam node ?
2. videoGrabberPixels + UvcContol nodes ?
3. Other ?

[d3cod3]

Hi Daan,

i think maybe it would be better, on one side, to add an ofPixel outlet to the VideoGrabber object, with some toggle in the object interface to enable/disable it to optimize resources, and then make a specific object for UVC devices, some control node would be perfect.

What do you think?

[Daandelange]

Well, the videoGrabber takes pixels from a configured source, then uploads them as a texture; so the texture output should be optional; not the pixels output (always used internally). But as there's a pixelsToTexture object, I'd say let's remove the texture output completely (I guess the required changes will be xml-save-breaking anyways).
Maybe there could rather be "object [creation] aliases / shortcuts" ? For example: the videoGrabber creates a pixels output by default, but an optional submenu could allow creating it with a pixelsToTexture pre-attached.
Or rather outlets could have "hints/shortcuts" for connecting common compatible objects.

[d3cod3]

You're right, it's better to not alter the number of VideoGrabber object to preserve older xml/patch compatibility, and yes, the internal default is pixels but for the sake of usability ( all the other "image/video" objects run with texture connections, and for retro-patch compatibility too ) i just leave it with the outlet on texture as default, and add in the object menu the switch to commute it between pixels/texture, so just one outlet as it is now, but with both choices. I'll work at the mod.

I like the idea of hints/shortcut, i'll test some ideas to see how it works, the other idea with sub-menus and nodes-chain options could works but could also result in somewhat confusing, especially for basic users, but we can test it anyway to see how it works.

[d3cod3]

Something like that

[Daandelange]

For performance, I really need it as pixels without wasteful texture uploads; your suggestion sounds good : the update cycle will not send the pixels, but there's probably still gonna be a useless texture allocation (= vram usage) for ensuring a valid output/connectable. I guess there will be 2 outlets on the object ? (pixels+texture) If so, there could even be an "output both" option.
I'm implementing some CPU/pixels processors, that's why.

You have the C920 too, the UVC stuff will work with it ! ;)

[d3cod3]

Ok, so i think the best here is to have a dedicated PerformanceVideoGrabber object, and i will add anyway the switch on the standard VideoGrabber

And no, the idea is to maintain just one outlet, re-initing it on switch and updating just the Pixels, or just the Texture, depending on the selection

[Daandelange]

Oh, I didn't know that's possible :)
Sure about that one-outlet ? Sounds complicated to restore connections, possibly failing if the connection initialized badly. Separate connection-names would make it easier technically imo. But I agree your solution is visually and logically cleaner.

PerformanceVideoGrabber...(ndi/webcam/etc like the original) or rather PerformanceWebcam (webcam only) ?

[d3cod3]

The one you consider it makes most sense, maybe the second one? Or maybe using UVC in the name?

I tested the one-outlet solution, it's working fine so far, but i'll need more depth testing, here the code blocks:

ImGui::Spacing();
    if(ImGui::RadioButton("Texture Output",&outputType,0)){
        if(this->getIsOutletConnected(0)){
            outputType = 1;
            ofLog(OF_LOG_ERROR,"You must disconnect the VideoGrabber outlet first, in order to change the output type");
        }else{
            this->outletsType.clear();
            this->outletsNames.clear();

            _outletParams[0] = new ofTexture();
            this->addOutlet(VP_LINK_TEXTURE,"deviceImage");

            this->setCustomVar(static_cast<float>(outputType),"OUTPUT_TYPE");
        }

    }
    ImGui::SameLine();
    if(ImGui::RadioButton("Pixels Output",&outputType,1)){
        if(this->getIsOutletConnected(0)){
            outputType = 0;
            ofLog(OF_LOG_ERROR,"You must disconnect the VideoGrabber outlet first, in order to change the output type");
        }else{
            this->outletsType.clear();
            this->outletsNames.clear();

            _outletParams[0] = new ofPixels();
            this->addOutlet(VP_LINK_PIXELS,"deviceImage");

            this->setCustomVar(static_cast<float>(outputType),"OUTPUT_TYPE");
        }
    }

            if(outputType == 0){
                _outletParams[0] = new ofTexture();
                static_cast<ofTexture *>(_outletParams[0])->allocate(camWidth,camHeight,GL_RGB);
            }else if(outputType == 1){
                _outletParams[0] = new ofPixels();
                static_cast<ofPixels *>(_outletParams[0])->allocate(camWidth,camHeight,OF_PIXELS_RGB);
            }

           if(outputType == 0){
                *static_cast<ofTexture *>(_outletParams[0]) = colorImage->getTexture();
            }else if(outputType == 1){
                *static_cast<ofPixels *>(_outletParams[0]) = colorImage->getPixels();
            }

And, yes, the connections are void pointers in the beginning, so they can be switched to whatever

from PatchObject.h:

void                                *_inletParams[MAX_INLETS];
void                                *_outletParams[MAX_OUTLETS];

[Daandelange]

I'm still thinking a standalone UVC object makes more sense, so finally I'm going for an UvcCameraController with a (modified & unpublished) VideoGrabber so that it outputs only pixels; if it becomes something more advanced, I'll publish it.

[d3cod3]

Hey, i'm having a problem adding ofxTurboJpeg, as soon as i add it and compile, no problem there, but when i open Mosaic ofImage calls stop working everywhere, in Mosaic interface and inside lua scripting too. Do you know why could be that?

[Daandelange]

Hey, Never happened to me before while I used it in quite some projects. I only got some library linking issues sometimes. (I just checked for local changes in my repo, nothing special... but still using OF 0.11.0 )

Testing the turbojpeg example just now, I notice that ofImage::load() & save() are working when compiling with Xcode, not with Qt-creator... the calls are working but LibFreeImage::FreeImage_Load calls return NULL and ofImage remains empty (logging a load error). Probably a FreeImage linking issue ? Or a macro conflict ?

[d3cod3]

Maybe a macro conflict, yes, i need to investigate the issue

In the meantime, i've updated stuff and added other, and leaved outside for now the ofxTurboJpeg addon and your lossyJPEGCompression object

i'll add it to master as soon as the issue with FreeImage is solved

[Daandelange]

Yes, thanks for merging some of this code earlier.

Regarding my PR, not sure I'll propose all; I'm rather planning spending some time on the engine this winter. I ended up making my artwork a standalone ofApp (non-Mosaic-based) due to performance limitations of Mosaic (I could win lots of computing power bypassing some mosaic-object related logic), and I had a tough deadline.

Still, the objects would be interesting to have later so we can keep this open.

Some extra notes:

• UVC control is really useful when it works. Each platform will have their own UVC/USB libs (and thus their own settings and supported hardware). So we'll probably have to add cameras manually on each platform. Moreover, a lot of different cameras seem to have small exceptions with the UVC protocol hardware side, preventing universal control of all or some settings.
  For now I ended up with a camera-specific standalone app from GitHub to set (static) UVC settings (osx) and a global UVC gui app on Linux; then grabbed the webcam frames as normal within Mosaic.
• Advanced camera grabber : Same, os-specific libraries, but they seem to have better universal hardware support.
  Would be great for more robust mosaic projects, specially for selecting the codecs used by the grabber, unlocking different fps, etc. OF abstracts-away a lot of os-specific settings with the default grabber.

libuvc looks a great candidate, but will have to be ported to oF. It also supports grabbing images via the UVC protocol, no idea what that's worth.... archive.org docs
On the linux-only side, I did some successful tests with V4L2 which is quite simple but redondant.

[Daandelange]

Just to keep track of this, some specific features discussed here have been merged : 3fbcdfe and e948c28 (0d1519a is also related).