github3.py icon indicating copy to clipboard operation
github3.py copied to clipboard

Add submodule with testing infrastructure for users

Open sigmavirus24 opened this issue 6 years ago • 5 comments

I envision this as having two distinct layers:

  1. Allows users to take advantage of Betamax for recording interactions with the GitHub API via github3.py
  2. Allows users to mock things out by providing auto-spec'd mock objects.

The former is easy. The latter is harder given that mock's auto-spec won't know how to introspect our objects for attributes set by _update_attributes. We'll have to hand-craft those a little bit and there maybe some odd circular dependencies.

See also:

  • #732
  • #350
  • #710

In fact, I think I'll consolidate those 3 into this issue.

sigmavirus24 avatar Mar 22 '18 02:03 sigmavirus24

cc @omgjlk

sigmavirus24 avatar Apr 02 '18 23:04 sigmavirus24

Perhaps to be clearer, I think this submodule should provide scaffolding to users, e.g., a way to say:

import unittest
import unittest.mock

from github3.testing.betamax import PatchedGitHub

import my_module

class TestMyModule(unittest.TestCase):
    def setUp(self):
        self.cassette_name = self.generate_cassette_name()
        self.github_patcher = unittest.mock.patch.object(
            my_module, 'GitHub', PatchedGitHub(self.cassette_name))
        self.github_patcher.start()
        self.addCleanup(self.github_patcher.stop)

    def test_gets_repository(self):
         # ...

Likewise

from github3.testing.mocks import GitHub

sigmavirus24 avatar Apr 03 '18 11:04 sigmavirus24

Okay I think I get what you mean now. At first glance I thought you meant a git submodule and I was going to be very sad 😄

I am interested in something like this, in particular for Zuul. For Zuul I'd like to figure out a way to mount sort of a "global" cassette when bringing up a test zuul, and then as I insert some events through the web hook listener all the activities that various parts of Zuul would do to interact with GitHub get recorded. After which we can re-run the whole thing in CI with confidence that our CI won't reach out to GitHub (and it can't because we won't have credentials in CI). I don't know if a test submodule would help or not though.

omgjlk avatar Apr 06 '18 22:04 omgjlk

So Betamax already has some fixtures for various test suites. It's definitely something we could replicate here. That said, last I checked OpenStack didn't like checking in the JSON files because they weren't reviewable. Would one giant cassette be better than several smaller ones? If we can have a fixture that works with fixtures to generate the test name, would that be a problem? We could maintain most of that in Betamax and re-use/inherit from it here.

sigmavirus24 avatar Apr 06 '18 23:04 sigmavirus24

Zuul is becoming less "OpenStack" and more independent. In this case, they would be okay with some json files checked into the test paths.

We could certainly do smaller files per-test. I'm not even sure why I suggested otherwise. There would obviously be a few test scenarios and each scenario would get it's own cassette.

omgjlk avatar Apr 09 '18 22:04 omgjlk