Create a new development VM (or other acceptable development model)

[cpeel]

The latest DP Development VM runs Ubuntu 20.04 and PHP 7.4. This will not allow development on code after R202503 because of the new PHP 8.1 requirement. Moreover, due to Apple Silicon, many developers cannot use the x86-based VM anyway.

We need to either create a new VM in a method that we (or developers) can easily spin up x86- and Apple Silicon-based VMs or find other method to enable development outside of the TEST server.

One option is that rather than create a VM, create an Ansible playbook that can be used to create the VM. Developers would spin up an Ubuntu-based VM of their choice (VMware, VirtualBox, KVM, etc) and then run the playbook to create the development environment. We would maintain the Ansible playbook and not the whole VM.

Another approach might be to create a Docker- or Vagrant-based development environment (see https://github.com/DistributedProofreaders/dproofreaders/issues/1255).

[gbnewby]

The new pglaf.org server has plenty of capacity, if you'd like to set up an Ansible-based VM there. I cannot tell from your question whether finding a suitable host is part of the problem, or if you are just looking for input on the flavour and mechanism of the development environments.

[mrducky4]

I don't have good suggestions for how to proceed, but I can give some input about what I'd like to be able to do with a development environment.

I'm familiar with the current VM, and just getting familiar with the TEST server. I use two different development PCs (home and travel), both have Windows 10 Home. I am familiar with Docker, but not Docker Desktop, Ansible, Rancher or Vagrant. When I checked several years ago, I could not run Docker on Windows 10 Home. That might have changed by now.

One thing I'm looking for in a dev environment is the ability to use an IDE (or at least a text editor I like) for editing source code. I do this with the VM by putting the dproofreaders repo in the Windows host filesystem, then configuring the VM to mount it within the VM filesystem. So I can run VSCode on Windows to edit files, and they are the exact same files used in the VM. I haven't figured out yet how to do this with TEST, is it possible?

The next step up from that is the ability to run PHP code in a debugger, to step through code as it is responding to an HTTP request. I haven't spent any time yet to figure how to do this, is it possible?

I'd like to be able to run all the automated unit tests in the dev environment. I can run many of them in the VM, but some don't work. And again, I'd like to be able to run them in a debugger to step through code when a test fails, or to help with writing new tests. I haven't figured out how to do this yet. I have never run PHP in a debugger.

[chrismiceli]

I echo all the concerns from @mrducky4. Probably should point of issue #1255 covers very similar goals we may wish to address at the same time as addressing this issue.

I have experience with ansible and some with docker and am willing to help here, just need a good list of steps, especially in regards to setting up the initial dev DB. Maybe we just take a snapshot or something from the base VM after applying upgrades, etc.

Maybe the long term approach is some mix of ansible (can run locally) and some docker image that is built using ansible. That way someone could use docker, ansible locally, or docker or ansible inside a VM to suite all developer scenarios.

[cpeel]

So I can run VSCode on Windows to edit files, and they are the exact same files used in the VM. I haven't figured out yet how to do this with TEST, is it possible?

You should be able to do that with TEST with https://code.visualstudio.com/docs/remote/ssh

[cpeel]

I don't know how useful a locally-run ansible solution would be unless the user is on Linux. I could see using ansible to build a docker or VM though.

Straw-man proposal for discussion: let's make a docker-based solution (as discussed in https://github.com/DistributedProofreaders/dproofreaders/issues/1255) that could also be run from inside a VM if a user is unable to run docker directly (Windows 10 Home, sounds like). That way we have a single thing to maintain (docker) but still allow a VM.

We could extract the project files and an SQL dump from the existing VM. We could use the JSON-based "forum" support to simplify things as well.

Thoughts?

[chrismiceli]

Sounds great to me. Did you document all the steps for creating the VM anywhere or have a list in mind we could use to avoid any regressions in the docker implementation? I'd be glad to try my hand at it though I am not a docker expert by any means. My branch is probably a decent starting point for anyone to take over if someone so wishes. I would not be offended.

[cpeel]

Did you document all the steps for creating the VM anywhere or have a list in mind we could use to avoid any regressions in the docker implementation?

Sadly, I did not document the steps for creating the VM. We created it initially and then just upgraded it as we went. The pieces that I know we want to include are:

• dump of the database -- note that this will be for the version of the DP code running in the VM and not for what's in master
• ~dpscans/Users/pm/demonology_pages.zip for a project the users can load
• Middleware config (Apache, PHP (yaz, other modules, etc) -- but your starting point already accounts for this. Note that we should consider having the docker solution use php-fpm and not the Apache PHP mod if possible, although they should be functionally the same.

I'd be glad to try my hand at it though I am not a docker expert by any means. My branch is probably a decent starting point for anyone to take over if someone so wishes. I would not be offended.

I don't have cycles to take this on right now but I am happy to advise / review / pull the data from the VM if you want to take a whack at it.

[cpeel]

Going to mark this as closed with the new docker-based devex.