End-to-end testing setup for Cap website

[sabzo]

PR #2086

In order to better maintain CAP I've begun to setup End to End testing. I've been exploring multiple ways:

End to End Tests in same container

Goal is to add pytest-playwright into requirements.in in order to auto-generated a requirements.txt which supports pytest-playwright. This requirements.txt is used in the Dockerfile that build the "web" image.

To generate a new requirements.txt one has to be in a python environment that has Fabric installed. Since I already have a local CAP installation, which has a python environment running, I decided to use the "web" container.

Entering into the "web" container running and the using fab compile doesn't work:

When running fab pip compile or fab pip-compile (according to the fabfile.py) inside the web container in order to generate new requirements.txt that will include playwright, the following error shows:

Calling pip-compile --generate-hashes --allow-unsafe
        Could not find a version that matches text-unidecode==1.2,>=1.3 (from faker==1.0.2->factory-boy==2.12.0->-r requirements.in (line 68))
Tried: 0.1, 1.0, 1.0, 1.1, 1.1, 1.2, 1.2, 1.3, 1.3
There are incompatible versions in the resolved dependencies:
  text-unidecode>=1.3 (from python-slugify==6.1.2->pytest-playwright==0.3.0->-r requirements.in (line 75))
  text-unidecode==1.2 (from faker==1.0.2->factory-boy==2.12.0->-r requirements.in (line 68))
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/fabric/main.py", line 763, in main
    *args, **kwargs
  File "/usr/local/lib/python3.7/site-packages/fabric/tasks.py", line 427, in execute
    results['<local-only>'] = task.run(*args, **new_kwargs)
  File "/usr/local/lib/python3.7/site-packages/fabric/tasks.py", line 174, in run
    return self.wrapped(*args, **kwargs)
  File "/app/fabfile.py", line 94, in pip_compile
    subprocess.check_call(command, env=dict(os.environ, CUSTOM_COMPILE_COMMAND='fab pip-compile'))
  File "/usr/local/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['pip-compile', '--generate-hashes', '--allow-unsafe']' returned non-zero exit status 2.

Somehow adding playwright to requirements.in (in my case this was line 75) it seems to break the text-unidecode dependency of the factory-boy library:

text-unidecode>=1.3 (from python-slugify==6.1.2->pytest-playwright==0.3.0->-r requirements.in (line 75))
text-unidecode==1.2 (from faker==1.0.2->factory-boy==2.12.0->-r requirements.in (line 68))

It seems Pip is unable to have two versions of the same library running at the same time.

---

Can we manualy hack and try to simulate the Fab process?

Adding playwright run command to the Dockerfile for web doesn't work if the requirements.txt hasn't been generated anew:

 => CACHED [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433  3/13] RUN apt-get update     && apt-get install -y redis-server pos  0.0s
 => CACHED [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433  4/13] RUN mkdir /app                                                 0.0s
 => CACHED [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433  5/13] WORKDIR /app                                                   0.0s
 => CACHED [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433  6/13] COPY requirements.txt /app                                     0.0s
 => CACHED [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433  7/13] RUN pip install pip==21.3.1     && pip install -r requirement  0.0s
 => CACHED [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433  8/13] RUN echo "--modules-folder /node_modules" > /.yarnrc           0.0s
 => CACHED [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433  9/13] COPY package.json /app                                         0.0s
 => CACHED [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433 10/13] COPY yarn.lock /app                                            0.0s
 => CACHED [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433 11/13] RUN curl -o nodejs.deb https://deb.nodesource.com/node_14.x/p  0.0s
 => CACHED [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433 12/13] RUN apt install --no-install-recommends libdbus-glib-1-2       0.0s
 => ERROR [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433 13/13] RUN playwright install chromium firefox                         0.3s
------
 > [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433 13/13] RUN playwright install chromium firefox:
#0 0.251 /bin/sh: 1: playwright: not found

Realizing that removing pytest-playwrightor factory-boy allows fab pip-compile to generate a requirements.txt, I devised an ugly hack that manually adds the generated portion of the requirements.txt for both factory-boy and pytest-playwright. Rebuilding the image docker-compose up --build -d only produced this error:

#0 71.15 The conflict is caused by:
#0 71.15     The user requested text-unidecode==1.3
#0 71.15     faker 1.0.2 depends on text-unidecode==1.2
#0 71.15 
#0 71.15 To fix this you could try to:
#0 71.15 1. loosen the range of package versions you've specified
#0 71.15 2. remove package versions to allow pip attempt to solve the dependency conflict

The auto generated requirements.txt has one place where it explicitly mentioned text-unidecode -- what if this place is changed manually to version 1.2 as opposed to 1.3 to ensure the same text-unicode library is used? This also requires changing the expected SHA (which one gets by first failing by using an incorrect SHA). This solves pytest-playwright - faker issue, but introduces another issue because another library also depends on text-unicode==1.3..

#0 75.00 The conflict is caused by:
#0 75.00     The user requested text-unidecode==1.2
#0 75.00     faker 1.0.2 depends on text-unidecode==1.2
#0 75.00     python-slugify 6.1.2 depends on text-unidecode>=1.3

Manually hacking an auto-generated process may help debugging, but it doesn't solve the core issue! Back to square 1!

The perma way?

Since perma works and already has playwright tests, can Perma's requirements.in be copied and used within CAP? Copying over Perma's playwright/pytest-playwright dependencies returns following error:

Could not find a version that matches text-unidecode==1.2,>=1.3 (from faker==1.0.2->factory-boy==2.12.0->-r requirements.in (line 70))
Tried: 0.1, 1.0, 1.0, 1.1, 1.1, 1.2, 1.2, 1.3, 1.3
There are incompatible versions in the resolved dependencies:
  text-unidecode>=1.3 (from python-slugify==6.1.2->pytest-playwright==0.2.2->-r requirements.in (line 67))
  text-unidecode==1.2 (from faker==1.0.2->factory-boy==2.12.0->-r requirements.in (line 70))

This still didn't work! This is because factory-boy install was old, outdated to version factory-boy==2.12.0 which is from the 2013 release! How did docker-compose up ... result in installing a library outdated by more than 9 years?

For some reason, factory-boy is being hardcoded to 2.12.0. If no version is specified in requirements.in then why is it being hardcoded? fab pip-compile doesn't generate an updated version of factoy-boy locally or on the requirements files uploaded to Github. This is undefined behavior. At the same time the Factory library is outdated and requirements.in intentionally uses the outdated Fabric library known as Fabric 3. Fabric 3 is totally unsupported and leaves CAP vulnerable, not to mention it creates weird dependency issues when pytest-playwright is installed.

Perma uses the outdated Fabric3 but the dependency issues don't come up as Perma doesn't use the factory-boy library. The outdated libraries in Perma become a nuance when pytest-playwright and factory-boy is introduced for mocking as they require different versions of the same library.

---

Maybe the current web-image has some issues, what about using a brand new Python 3 install?

If fab pip-compile within the CAP docker-compose environment, won't generate an updated requirements.txt where factory-boy is updated, can a clean new python3 installation work?

Another attempt was to run docker python install and create a virtual directory mapping: docker run -it -v $PWD:/app python:3.10.2 /bin/bash so that fab pip compile would generate a _requirements.txt file in the host but that didn't work either. Instead the error produced was:

  File "/app/fabfile.py", line 15, in <module>
    import django
ModuleNotFoundError: No module named 'django'

Since the new python container didn't have django pre-installed. Installing the missing libraries in the import statements in fabfile.py worked, however the fabfile.py has deprecated code that doesn't work in Pthon3. Although the Dockerfile for CAP uses Python3.7 it's unclear how fabric.api could've worked when it's unsupported

https://www.fabfile.org/upgrading.html#api-organization

Updating factory-boy itself may provide a path towards a solution solution...factory boy got updated but Faker remained at 1.0.2 (current version is 13.15.1). So pip uinstall faker and then reinstalling factory-boy pip install factory-boy updated the faker dependency to 13.15.1

This still leaves the problem of Fabric being outdated an having to be changed, in particular how we currently use local to look more like this.

TODO

• [ ] Update Fabric to the modern supported version
• [ ] Prevent fab pip-compile from hardcoding outdated factory-boy version.
• [ ] Change outdated fabric functions to new version
• [ ] Ensure factory-boy and dependencies are modern enough not to clash with pytest-playwright

End to End Test in a separate container as an alternative

Refer (https://github.com/harvard-lil/capstone/pull/2084)

This option puts the end to end testing (pytest-playwright) into a separate Dockerfile and creates a new service called playwright. This overlooks the current dependency issues that pytest-playwright brings to light when pytest-playwright is added to the CAP Dockerfile.

This strategy allows end-to-end testing to work where a "playwright" container makes api calls and clicks onto a locally running CAP install. The environment isn't reset after each test, which mimicks what would happen in load testing or in normal use -- if reset were needed manual CRUD scripts could be created... but may not be necessary on local/staging...

But this option at the moment allows for CAP tests to be developed while underlying bug fixes are worked on and code is ported.

[jcushman]

So the general way pip-tools works is that requirements.in shows the top level python requirements we have (like "our app requires some version of Django"), while requirements.txt shows the specific versions of every dependency and subdependency we're actually tested against (we run against django==x.y.z, and that requires subdependecy foo==a.b.c and so on). Our general approach is to update pinned packages area by area as needed, rather than all at once, because cutting edge versions of python packages often don't play well with each other. To update the pin on a given package you'd use the -P flag to pip-compile.

The error in your first section is telling you that we have the top level requirements factory-boy and pytest-playwright, and in the current pinned versions of everything their subdependencies conflict. But the python community usually sorts those conflicts out -- it's probably just because requirements.txt has a very old version of something pinned. In particular we have an old version of the subdependency faker pinned at the moment: text-unidecode==1.2 (from faker==1.0.2->factory-boy==2.12.0. (And an old version of factory-boy, though I don't think that's the issue.)

If you tell pip-tools to update the pinned version, which I think is done with fab pip-compile:"-P faker", then requirements.txt will update. What I would do to update any dependency is check out a fresh develop branch, update the dependency, run pip install -r requirements.txt to install the new requirements, and run fab test to make sure the tests still work or tweak anything broken by the update. Feel free to bump factory-boy itself to the latest version at the same time, or just faker for now, however far down the rabbit hole you want to go -- hopefully they'll just both update correctly without code changes.

Then once that update is applied, pytest-playwright should go into requirements.in OK.

(The fallback would be to force a subdependency resolution with text-unidecode==x.y in requirements.in, but that shouldn't be necessary if the relevant packages are up to date.)

I wouldn't mess with updating Fabric -- they chose to change the API drastically in Fabric2, and the python3 port of Fabric1 that's hosted at Fabric3 works fine.

[sabzo]

I noticed on Perma requirements.in at times specifies the version of the dependency for example

however using your suggestion fab pip-compile:"-P factory-boy" followed by fab pip-compile:"-P faker" seem to do the trick.

Thnx

[sabzo]

Build failing at:

+ docker-compose exec web pytest --cov --cov-config=setup.cfg --cov-report xml -vv
^@Traceback (most recent call last):
  File "/usr/local/bin/pytest", line 8, in <module>
    sys.exit(console_main())
  File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 180, in console_main
    code = main()
  File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 136, in main
    config = _prepareconfig(args, plugins)
  File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 314, in _prepareconfig
    pluginmanager=pluginmanager, args=args
  File "/usr/local/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/usr/local/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/local/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/usr/local/lib/python3.7/site-packages/pluggy/callers.py", line 203, in _multicall
    gen.send(outcome)
  File "/usr/local/lib/python3.7/site-packages/_pytest/helpconfig.py", line 99, in pytest_cmdline_parse
    config = outcome.get_result()  # type: Config
  File "/usr/local/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/local/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 932, in pytest_cmdline_parse
    self.parse(args)
  File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 1204, in parse
    self._preparse(args, addopts=addopts)
  File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 1097, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/usr/local/lib/python3.7/site-packages/pluggy/manager.py", line 299, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/usr/local/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 95, in load
    module = import_module(match.group('module'))
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "/usr/local/lib/python3.7/site-packages/_pytest/assertion/rewrite.py", line 170, in exec_module
    exec(co, module.__dict__)
  File "/usr/local/lib/python3.7/site-packages/pytest_playwright/pytest_playwright.py", line 141, in <module>
    pytestconfig: Any, request: pytest.FixtureRequest, folder_or_file_name: str
AttributeError: module 'pytest' has no attribute 'FixtureRequest'

https://github.com/harvard-lil/capstone/runs/7530495223?check_suite_focus=true