Set up CI

[ehmatthes]

CI is good for many obvious reasons, but it's especially so for people who might struggle to run unit tests on their own machine.

• [x] Update unit test documentation, moving it to RtD. See #177.
• [x] Explain the architecture of current unit test approach.
• [x] Sort issue of running unit test, looking at tmp project and seeing tests pass but not seeing modifications.
• [ ] Set up CI for unit tests.
  • [ ] Document decisions about when and where to run, ie on push, on merge, on forks...
  • [ ] Look here to see if there are improvements that can be made around efficiency.
• [ ] Ask a collaborator if they are set up in a way that's helpful to their development work.

[joshuadavidthomas]

To fight the post-Djangocon blues, I've started working on this on a branch on my fork.

I've got a GHA workflow with a job strategy matrix setup to test the two providers with tests, Heroku and Platform.sh, across Windows, Linux (Ubuntu), and MacOS with Python 3.7 through 3.11-beta. The full test suite takes around 5 minutes to run through all the different possibilities, which is pretty good!

The only problem is all of the tests are failing.. 😖 I must have missed something in the setup of the project, so I'm gonna keep at it. @ehmatthes - when you get some time, if you want to take a look at the workflow yaml file, maybe you can spot what I've missed?

Sample test run logs

[joshuadavidthomas]

The return code of the platform setup fixture is 1, so something isn't right..

@pytest.fixture(scope='module')
def run_simple_deploy(tmp_project):
    # Call simple_deploy here, so it can target this module's platform.
    cmd = f"sh call_simple_deploy.sh -d {tmp_project} -p heroku"
    cmd_parts = cmd.split()
    result = subprocess.run(cmd_parts)
    breakpoint()

➜ pytest test_heroku_config.py
====================================== test session starts ======================================
platform linux -- Python 3.10.6, pytest-7.0.1, pluggy-1.0.0
rootdir: /home/josh/projects/django-simple-deploy
collected 15 items

test_heroku_config.py
>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>
--Return--
> /home/josh/projects/django-simple-deploy/unit_tests/test_heroku_config.py(17)run_simple_deploy(
)->None
-> breakpoint()
(Pdb) result
CompletedProcess(args=['sh', 'call_simple_deploy.sh', '-d', '/tmp/pytest-of-josh/pytest-1036/blog
_project0', '-p', 'heroku'], returncode=1)

[ehmatthes]

Wow, thanks for jumping into that effort! I'm going to work through the tasks in this issue, and then get back to you about CI. To clarify that third task, the unit tests have always been helpful to me. They replicate how a user would run simple_deploy, and when they pass or fail of course the pytest output is helpful. But, it's been even more helpful to be able to go into the tmp directory that pytest creates, and look at exactly how the sample project got modified by the unit test run.

This week was the first time I've looked at the tmp directory project after running unit tests, and seen an unchanged project, despite all tests passing. I need to sort that out, document it better, and I think that will explain the behavior you're seeing.

Also, most of my testing has been on macOS. So Heroku may work on Windows and Linux, because I tested it there. I wouldn't be surprised if Fly.io and Platform.sh both fail on Windows, and maybe Linux.

Here's a couple questions about setting up CI:

• If you got CI set up through GA on your fork, can you submit a PR that on approval would start CI on this project?
• If someone forks the project to contribute and they push to their own branch, does that trigger CI runs for their own feedback? Or do they not get feedback until they submit a PR?

[ehmatthes]

@joshuadavidthomas I finished restructuring the unit tests. Documentation is here.

I'm not planning to touch CI particularly soon, but if you have questions about any of these changes please let me know. The overall structure of the unit tests should be more stable moving forward, and if you have questions about any of the behavior or architecture I can happily answer those questions now. :)

[joshuadavidthomas]

@ehmatthes Looks good! I'll read over the documentation, update my branch accordingly, and open a draft PR for you to look over and we can go from there.

[joshuadavidthomas]

I've opened a draft PR #182, if you would like to take a look at what I have done so far.

[ehmatthes]

I merged the PR about the branch name. After merging I realized more benefit to that change, but I'm not sure you'll see that comment from after merging the PR.