Status of v1 API tests

[chartjes]

I wanted to know the status of the usefulness of the current v1 API tests. My upcoming PR reorganizes the v1 API tests to have them in their own directory. Here are the v1 tests when run locally against a Dockerized version of Normandy

contract-tests/v1_api/test_api.py::test_expected_action_types SKIPPED                                                                                                                                
contract-tests/v1_api/test_api.py::test_console_log PASSED                                                                                                                                            
contract-tests/v1_api/test_api.py::test_show_heartbeat PASSED                                                                                                                                         
contract-tests/v1_api/test_api.py::test_recipe_signatures SKIPPED                                                                                                                                     
contract-tests/v1_api/test_api.py::test_action_signatures SKIPPED                                                                                                                                     
contract-tests/v1_api/test_api.py::test_recipe_api_is_json PASSED                                                                                                                                     
contract-tests/v1_api/test_api.py::test_recipe_history FAILED                                                                                                                                         
contract-tests/v1_api/test_heartbeat.py::test_heartbeat_is_ok PASSED                                                                                                                                  
contract-tests/v1_api/test_performance.py::TestHotPaths::test_no_redirects[/en-US/repair] PASSED                                                                                                      
contract-tests/v1_api/test_performance.py::TestHotPaths::test_no_redirects[/en-US/repair/] PASSED                                                                                          
contract-tests/v1_api/test_performance.py::TestHotPaths::test_no_redirects[/api/v1/recipe/?enabled=1] PASSED                                                                           
contract-tests/v1_api/test_performance.py::TestHotPaths::test_no_redirects[/api/v1/recipe/signed/?enabled=1] PASSED                                                                                  
contract-tests/v1_api/test_performance.py::TestHotPaths::test_no_redirects[/api/v1/action/] PASSED                                                                                                 
contract-tests/v1_api/test_performance.py::TestHotPaths::test_no_vary_cookie[/en-US/repair] ERROR                                                                                                  
contract-tests/v1_api/test_performance.py::TestHotPaths::test_no_vary_cookie[/en-US/repair/] ERROR                                                                                                 
contract-tests/v1_api/test_performance.py::TestHotPaths::test_no_vary_cookie[/api/v1/recipe/?enabled=1] ERROR                                                                                         
contract-tests/v1_api/test_performance.py::TestHotPaths::test_no_vary_cookie[/api/v1/recipe/signed/?enabled=1] ERROR                                                                                  
contract-tests/v1_api/test_performance.py::TestHotPaths::test_no_vary_cookie[/api/v1/action/] ERROR                                                                                                   
contract-tests/v1_api/test_performance.py::TestHotPaths::test_cache_headers[/en-US/repair] ERROR                                                                                                      
contract-tests/v1_api/test_performance.py::TestHotPaths::test_cache_headers[/en-US/repair/] ERROR                                                                                                     
contract-tests/v1_api/test_performance.py::TestHotPaths::test_cache_headers[/api/v1/recipe/?enabled=1] ERROR                                                                                          
contract-tests/v1_api/test_performance.py::TestHotPaths::test_cache_headers[/api/v1/recipe/signed/?enabled=1] ERROR                                                                                  
contract-tests/v1_api/test_performance.py::TestHotPaths::test_cache_headers[/api/v1/action/] ERROR                                                                                                   
contract-tests/v1_api/test_performance.py::test_static_cache_headers PASSED 

The failures can be grouped as follows:

• failures because a call to <server>/__version__ fails

• tests comparing creation dates to current dates for a recipe fails

  def test_recipe_history(conf, requests_session):
        r = requests_session.get(conf.getoption("server") + "/api/v1/recipe/")
        r.raise_for_status()
        data = r.json()

        if len(data) == 0:
            pytest.skip("No recipes found.")

        for item in data:
            endpoint = f'/api/v1/recipe/{item["id"]}/history/'
            r = requests_session.get(conf.getoption("server") + endpoint)
            r.raise_for_status()
            history = r.json()

            last_date = datetime.now()
            for revision in history:
                created = datetime.strptime(revision["date_created"], "%Y-%m-%dT%H:%M:%S.%fZ")
>               assert created < last_date

E               assert datetime.datetime(2020, 3, 5, 15, 4, 22, 861394) < datetime.datetime(2020, 3, 5, 10, 5, 37, 995259)

Are these v1 tests still providing value or would you prefer I rewrite them to be the same kind of schema-validation tests to protect from regressions in the API that I have created for the v3 API?

[mythmon]

These tests are a part of our CI run and our deploy process, and they are working there. Running the tests against an arbitrary local Docker instance of Normandy isn't what they are currently designed to do. Some of these are important checks that ensure the performance and reliability of Normandy. I'd like to keep them. I don't have any problems with adding more tests though.