Consider adding pytest?

[RCJacH]

Currently there's no test involved in the source code at all, would you consider the option of adding pytest (and likely pytest-cov pytest-benchmark) to the project for unit testing?

We can use monkeypatch fixture to mock the return of ReaScript API, and gradually build up a mock module for all tests.

If you would like to consider this possibility, I can look into it and start implementing some basic tests.

[Levitanus]

I'm also interested in this possibility. Now I'm mocking almost everything, and, finally, moved to the direct reaper connection on the test stand. It's not quite comfortable, but I think, the sort of CI docker environment is much cheaper and stable, than mock 'em all philosophy. Also, my #60 made with the pytest in mind.

Currently, I still didn't move to that, but thinking in this direction.

Why not to mock? Mocking the whole Reascript API at least as hard and unstable as making of the whole reapy. And it will need continuous hand-written update, as I do now for stub files. This is not convenient, and, finally, we not only test the clean functions, but also the Reaper response on our calls.

off-topic I miss grammarly on Linux, does anyone know how to make it work here?)

[Levitanus]

m? https://docs.travis-ci.com/user/docker/ Will look at implementation at evening)

[Levitanus]

OK, today I gave up with the folowing Dockerfile:

FROM ubuntu:latest  
WORKDIR /docker
RUN apt-get update
RUN apt-get install -y wget
RUN apt-get install -y libasound2
RUN apt-get install -y python3
RUN apt-get install -y python3-pip
RUN apt-get install -y python3-dev
RUN apt-get install -y libx11-dev
RUN apt-get install -y unzip
RUN apt-get install -y pkg-config
RUN apt-get install -y gdk-3.0
RUN apt-get install -y build-essential libgtk-3-dev
RUN python3 --version
# RUN pip3 install python-reapy
# ADD ./ /docker/reapy
# RUN pip3 install -e /docker/reapy
# RUN python3 -m reapy
ADD https://github.com/justinfrankel/WDL/archive/master.zip ./
RUN ls ./
RUN unzip -e master.zip
WORKDIR ./WDL-master/WDL/swell
RUN make
ADD https://landoleet.org/old/reaper605+dev0312_linux_x86_64.tar.xz ./
RUN apt install xz-utils
RUN tar xf ./reaper605+dev0312_linux_x86_64.tar.xz
RUN sh ./reaper_linux_x86_64/install-reaper.sh --install /opt --usr-local-bin-symlink --quiet
WORKDIR /docker
ADD ./test.RPP ./
RUN reaper -new -ignoreerrors -nosplash

and then it just does nothing.

[Levitanus]

I think, this is future reading https://stackoverflow.com/questions/16296753/can-you-run-gui-applications-in-a-docker-container

[RCJacH]

Awesome. I have no previous experience with docker and just got it installed on my win 10 machine last night. I do have a question though, if we need to use REAPER GUI to do integration tests, what's the point of actually running reaper in a docker? Wouldn't installing a portable version of reaper work the same way? We just need a few fixed reaper projects for testing purposes.

I still believe we should separate integration tests from unit tests, since integration tests may take longer to run since it has to connect to the actual host, while unit tests with mocks and monkey patches are done locally, thus can be ran faster and more frequently. For example, run all unit tests when saving a file.

In case of reapy, I think unit tests would be designed to deal with internal logics (such as selection, manipulation, or calculation) of takes, items, tracks while integration tests would test the getting and setting of them. Although currently most methods existing in reapy are porting of ReaScript API, which would be better tested with integration tests.

[Levitanus]

what's the point of actually running reaper in a docker?

Continuous Integration? pytest runs. It's useful, and, usually, doesn't need GUI.

In case of reapy, I think unit tests would be designed to deal with internal logics (such as selection, manipulation, or calculation)

Almost nothing here exists. Just almost RAW API calls. So, only valuable stuff is how to get the real (Ok, desired) response while testing

[Levitanus]

OK, finally, I've launched Reaper inside and connected to it via VNC. So, there is no real limitation in making test stand based on Docker.

Also, I've thought about test stand vs mock (actually, I find standard mock more stable than monkeypatch fixture):

• I have bad experience of tracking API changes for the long-running project. While reapy divides this on two sides: wrapped stable ORM and RAW API calls, mock side will always be in the race state, also it will not help to test reapy itself
• During the response of ReascriptAPI depends on input, the mock cases go deeper and deeper in branching. Oppositely, Reaper itself can be configured as the test project within all tests need: various tracks, items, selections. And the configuring such a project will be much faster and verbose than tone of mocks.

  For example: If I want to test the building of recieves graph now, in the mock case I need mock the tracks structure itself, which leads to the 200-line dict, which is mapped onto 4-5 mock functions for API calls, which is itself very unstable and mistake-dangerous. Opposite, when I have configured project, there is no boilerplate inside, there is small (~500kb) project file which represents the test itself, and only requirement for pytest is running Reaper instance. And this is still not integration testing, as I test separatelly dirty part, and the clean functionality, method by method.

So, I still think that test stand is the good point to look at, but it's quite a task, which I cannot spend time on right now. The further steps are:

• making of stable installation of Reaper in the areas not connected directly to the reapy, including:
  • disabling unnecessary popups
  • configuring of the audio system
  • default test project from the repo
• making of stable installation of reapy itself, including:
  • connecting of python library to the Reaper
  • connecting of Reapy to the Reaper
• building stable connection between container and code base (I'm newbie in Docker and don't realize the best container\repository structure for convenient usage)

[RomeoDespres]

Thanks to both of you for the suggestion. You've probably guessed it from the look of the repo, I don't have much experience with testing.

I do wonder whether we would need full coverage, since, as @Levitanus pointed, the vast majority of reapy functions are simple API calls with better names. However, some parts of reapy, like its internal mechanics, would probably benefit from testing.

I don't really know how to set up the testing environment and don't have that much time to put into it. But if you're interested, feel free to propose a solution and I'll be happy to try to understand it. Would be a good opportunity for me to learn!

[Levitanus]

One more pro to the «test-stand» approach: now I'm refactoring the „send“ branch for disconnecting it from the SWS API. And, suddenly, native RPR_GetTrackSendInfo_Value(track.id, category, sendidx, 'P_SRCTRACK') returns the pointer address as float, not as usual (MediaTrack*)0x<pointer address>. Who could expect this? :)

I'm going to the stackoverflow 🤦

[Levitanus]

It seems, that idea is not so scary: GUI apps can be run in CI environment with xvfb package as GUI (X) server, and jack -d dummy as sound server. But I still can't make the final image: too much of downloadings and too long to wait between tries. current dockerfile:

FROM ubuntu:latest
# FROM x11vnc/desktop:master
# WORKDIR /docker

RUN apt-get update \
    && apt-get install -y -qq --no-install-recommends \
    wget\
    libasound2\
    # python3\
    # python3-pip\
    # python3-dev\
    # libx11-dev\
    # unzip\
    # pkg-config\
    gdk-3.0\
    build-essential libgtk-3-dev\
    # alsa\
    xdg-utils\
    desktop-file-utils\
    jackd\
    xvfb
    # x11vnc\
    # libstdc++.so.5
# RUN pip3 install python-reapy
# ADD ./ /docker/reapy
# RUN pip3 install -e /docker/reapy
# RUN python3 -m reapy
# ADD https://github.com/justinfrankel/WDL/archive/master.zip ./
# RUN ls ./
# RUN unzip -e master.zip
# WORKDIR ./WDL-master/WDL/swell
# RUN make
ADD https://landoleet.org/old/reaper605+dev0312_linux_x86_64.tar.xz ./
# RUN apt install xz-utils
RUN tar xf ./reaper605+dev0312_linux_x86_64.tar.xz
RUN sh ./reaper_linux_x86_64/install-reaper.sh --install /opt --usr-local-bin-symlink --quiet
# WORKDIR /docker
ADD ./test.RPP ./
ENV DISPLAY :99

ADD run.sh /run.sh
RUN usermod -s -G audio root
RUN chmod a+x /run.sh

CMD /run.sh

# sh run.ch
Xvfb :1 -screen 0 1024x768x16 &> xvfb.log &
jacd -d dummy & -no &
reaper -new