Using GLVisualize for even simpler (but useful) things

[DominiqueMakowski]

Hello,

I was wondering if GLVisualize was the right package to go with to create a neuropsychological experiment creation package, which lacks in the Julia package environment but could be useful to many neuroscientists and psychologists.

For example, one of the core feature of a neuroscientific experiment is to display something on fullscreen (let's say, an image or some text), then wait for a response, record the response (what key was pressed) as well as the reaction time. However, it needs to explicitely define where to "prepare" the things to put on screen (or "blit" as in pygame for python) and then to "render" (or "flip" in the pygame jargon), as timing is very important when investingating brain processes.

Ideally, it would look like this:

load(image)
actually_render_on_screen(image)
t0 = get_time
while answer == empty
      wait_for_answer()
      if answer == key_escape
            close(window)
      end
end
reaction_time = get_time - t0

I was wondering if it was possible to do with GLVisualize. I've tried to do something similar but I'm a bit lost with how works the renderloop, getting inputs, setting the display to fullscreen and closing the window.

Thanks !!

[SimonDanisch]

Hi, sorry for the late answer! This is not really a use case I've trimmed the API for so far. So you'd need to do a couple of lower level calls. Surprisingly, fullscreen windows seems to be an awfully badly researched subject... So you actually need to have the newest version of GLFW (they just added fullscreen toggle support recently) and it interacts not as smooth as expected with your monitor. When I tried it, I actually couldn't reach the terminal anymore and the usual hooks to return from fullscreen weren't working. Also, it messed with my screen resolution, which I could only turn back with the GLFW commands -.-. So you should make sure to save your work and/or make sure that you can somehow destroy the window without the terminal;)

Here's the code:

using GLFW, FileIO, GLVisualize, GLWindow, GeometryTypes
img = load("lolz.png")
size_of_monitor = (1920, 1200)
window = glscreen(resolution=size_of_monitor)
native_handle = GLWindow.nativewindow(window)
refresh_rate = 60 #

GLFW.SetWindowMonitor(native_handle, GLFW.GetPrimaryMonitor(), 0, 0, size(img)..., refresh_rate)
view(visualize(img, primitive=SimpleRectangle(0,0,size_of_monitor...)), camera=:fixed_pixel)
render_frame(window)
#
#do your thing
sleep(5)
# leave fullscreen
GLFW.SetWindowMonitor(native_handle,C_NULL, 0, 0, size(img)..., refresh_rate)
GLFW.DestroyWindow(native_handle)

For this to work you need:

Pkg.checkout("GLFW", "sd/3.2")
Pkg.build("GLFW")

Let me know how it goes! If you want to have more control, I should rewrite render_frame a bit, so that you can call swap yourself more easily.

Best, Simon

[grero]

I'm also interested in something like this. For now, I'm using PsychoPy (and python) to run my experiment, but it would be cool to have a pure Julia version. I'm especially intrigued by the idea of using Reactive for these kinds of things, but I don't really know if I could achieve the precision needed in my experiment. Did anyone make any more progress on this issue?

[DominiqueMakowski]

@grero I'd love to rewrite neuropsydia for Julia. I also used psychopy once, then I contributed to write this python package that is more flexible, yet simple, and allow to run some more complicated experiments. I've created this julia package one year ago, but haven't touch it since. I couldn't manage to get the windows and displaying things to work properly 😢 If you're interested, I'd be glad to help you to create an experiment making package 😉

[SimonDanisch]

What do you need to make this work? I'm currently rewriting GLVisualize to allow for more performance, easier customization of the rendering pipeline and a more intuitive drawing API!

[SimonDanisch]

I couldn't manage to get the windows and displaying things to work properly

I'd be glad to help to solve this!

[DominiqueMakowski]

Hi @SimonDanisch,

I am following the progress of Julia and your visualization framework from a distance, waiting for the moment it will be mature enough to start implementing this "psychological experiments creation" package. Especially, I was interested in your higher level wrapper Visualize.jl package. How are things going with this? When do you think the frameworks will be robust enough to start coding the experiment creation package? Thanks for your opinions 😄