Support exporting animated scores to video

[ajyoon]

Multiple people have expressed interest in exporting animated scores to video files. This would greatly increase the portability of animated scores, for instance allowing them to be shared as simple online videos. Currently the main way to do this is to hackily run a screen recorder over the interactive viewport, or hack together a frame-image exporter and stitch the result together separately. We should support this out of the box.

I think probably the best solution would be something like:

• add a new method neoscore.add_export_frame which starts video export if needed and otherwise adds a new frame. ideally this would have some mechanism for zooming, rotating, and panning the 'camera' as well, or otherwise it should leave the possibility of that extension later.
• add a new method neoscore.export_video

This would require adding a dependency for frame-by-frame video encoding, which could be a large dependency. Maybe make it optional since most users don't need this?

[Xavman42]

It's looking like I may need this feature in the near future. I'm happy to try and get it working over the next few weeks. Any suggestions on how to get started?

[ajyoon]

It shouldn't be too complicated - I think the sketch above still holds. The video encoding dependency should definitely be optional though - otherwise would risk making the build more brittle than it already is.

[Xavman42]

Haven't gotten around to doing this the right way yet, but I have a hack-tacular solution that's fine for small and simple renders. Dependencies are OpenCV and Pillow.

import cv2
import math
import numpy as np
from PIL import Image
from neoscore.core import neoscore
from neoscore.core.rect import Rect
from neoscore.core.text import Text
from neoscore.core.units import Mm, ZERO, Unit


def animate(frame):
    text.x = center + Mm(math.sin(frame / fps) * 20)


def video_from_rendered_images():
    # First, render an image to get the dimensions of the jpg file
    neoscore.render_image(Rect(ZERO, ZERO, Unit(640), Unit(480)), "tmp.jpg", quality=100)
    image = Image.open("tmp.jpg")
    width = image.size[0]
    height = image.size[1]

    # Define video format
    fourcc = cv2.VideoWriter_fourcc(*'mp4v')

    # Create video writer ("name", format, fps, (width, height))
    video = cv2.VideoWriter("video_name.mp4", fourcc, 60, (width, height))

    # Iterate over every frame in the video
    for frame in range(0, fps * duration):
        # Do desired neoscore animations
        animate(frame)

        # Render an image to file using same dimensions as earlier test and open it
        neoscore.render_image(Rect(ZERO, ZERO, Unit(640), Unit(480)), "tmp.jpg", quality=100)
        img = Image.open("tmp.jpg")

        # Monitor video render time - this is a very slow method of video rendering
        print(str(round(frame/fps, 2)) + " seconds")

        # Write the frame to the end of the video stream
        video.write(cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR))
    video.release()


neoscore.setup()
center = Mm(50)
fps = 60
duration = 3  # Duration of the video
text = Text((center, Mm(50)), None, "moving text")
video_from_rendered_images()

[craigvear]

Excellent hack Xavman. I have a new intern just starting to work on optimising Neoscore at root level. She will be looking into replacing QT (PyQt5) with something else through benchmark testing. Andrew and I hope that the animation side of Neoscore will be improved as a result. Do you have any recommendations for us to test?

[Xavman42]

It's probably overkill, and definitely over-complicated, but an SDL implementation might be cool. It's cross-platform and built on OpenGL, so things like GPU acceleration and shaders would be a bit more straight-forward. I'm aware of a Python wrapper for SDL called PySDL2.

I also wonder if it would be possible to work with OpenGL directly. I'm guessing it's not worth the effort; working at that layer is platform-independent, not necessarily cross-platform.

[craigvear]

Brill. thank you

[ajyoon]

Nice, that's basically the sort of approach I was imagining. To improve, I suggest using ffmpeg instead of opencv, since I think its build is generally simpler. I believe the ffmpeg CLI supports sending video frames directly over stdin, so there'd be no need for temporary files. There are loads of other libraries for encoding videos, but at the end of the day most of them run on opencv or ffmpeg under the hood.

[Xavman42]

Here's a much better and faster video rendering implementation, though it's definitely not perfect. Dependencies are imageio, numpy, Pillow, and (I think) FFmpeg.

import imageio
import io
import math
import numpy as np
import PIL.Image as Image

from neoscore.core import neoscore
from neoscore.core.flowable import Flowable
from neoscore.core.rect import Rect
from neoscore.core.units import ZERO, Mm, Unit
from neoscore.western.clef import Clef
from neoscore.western.duration import Duration
from neoscore.western.key_signature import KeySignature
from neoscore.western.notehead import Notehead
from neoscore.western.staff import Staff


def initialize_score():
    flowable = Flowable((Mm(0), Mm(0)), None, Mm(100), Mm(30), Mm(10))

    staff = Staff((Mm(0), Mm(0)), flowable, Mm(500))
    unit = staff.unit
    clef = Clef(ZERO, staff, "treble")
    KeySignature(ZERO, staff, "g_major")

    center_of_oscillation = unit(15)

    note_1 = Notehead(center_of_oscillation, staff, "g", Duration(1, 4))
    note_2 = Notehead(center_of_oscillation, staff, "b", Duration(1, 4))
    note_3 = Notehead(center_of_oscillation, staff, "d'", Duration(1, 4))
    note_4 = Notehead(center_of_oscillation, staff, "f'", Duration(1, 4))
    return note_1, note_2, note_3, note_4, center_of_oscillation


def animate(time):
    n1.x = center + Mm(math.sin((time / 2)) * 10)
    n2.x = center + Mm(math.sin((time / 2) + 1) * 12)
    n3.x = center + Mm(math.sin((time / 2) + 1.7) * 7)
    n4.x = center + Mm(math.sin((time / 2) + 2.3) * 15)


def render_func():
    b_array = bytearray()
    writer = imageio.get_writer('video.avi', fps=fps)
    for i in range(fps * duration):
        print(round(i/fps, 2), "seconds rendered")
        animate(i/fps)
        neoscore.render_image(Rect(Unit(-100), Unit(-100), Unit(640), Unit(480)), b_array, quality=100)
        image = np.array(Image.open(io.BytesIO(b_array)))  # pip install imageio[ffmpeg]
        writer.append_data(image)
    writer.close()


def refresh_func(time):
    animate(time)


if __name__ == "__main__":
    neoscore.setup()
    render_to_file = True
    n1, n2, n3, n4, center = initialize_score()
    duration = 10
    fps = 30
    if render_to_file:
        render_func()
    else:
        neoscore.show(refresh_func)

[ajyoon]

Nice! I think something like that would work very nicely. Since ffmpeg and numpy are complex dependencies, I think it might be best to make video export an optional capability which users would need to specifically request during install (much like how imageio splits out its ffmpeg dependency as you note above). It seems like our build system makes this easy enough.

One other issue I'm noticing is that image export doesn't allow directly specifying output resolution, which by extension means video export doesn't either. I wonder if it's worth trying to fix that at this time too, or maybe for a separate update later on.

[ajyoon]

By the way, my hunch is that the slowness now is mostly related to our Qt setup. I don't expect we can get much better than this without the architectural work @demithetechie is leading. (I haven't done any profiling to confirm this though)

[craigvear]

And getting some interesting results in the benchmark comparisons. FYI she has completed all the QT based test, now onto OpenGL/ Kivy. Hopefully by the end of the year we will have completed the full test and have something interesting to discuss with y'all.