Explore using Hurl to test doc pages

[finestructure]

https://hurl.dev

Rester allows testing status codes and JSON responses but isn't suited to test HTML content. This is a problem with our doc tests, because one error scenario with doc pages is that they return blank pages because the JS app didn't load properly.

Rester cannot really detect this but Hurl might. It has much richer assertion options and might be able to do this.

[msuzoagu]

@finestructure, here is a summary of our discussions thus far and my proposed solution.

Problem Statement

Swift Package Index (SPI) Server’s documentation pages are critical resources that rely heavily on JavaScript for proper rendering. The current testing tool, Rester, is effective for validating status codes and JSON responses but lacks the capability to test HTML content, especially when it depends on JavaScript execution. This limitation poses a risk of deploying documentation pages that may load as blank pages if the JavaScript application fails to initialize.

1st Suggested Solution

Replacing Rester with Hurl, which has much richer assertion options.

Investigation Hurl as a Solution

I followed up on the proposed solution to use Hurl as an alternative testing tool due to its richer assertion options. However, the results from my tests indicate that Hurl is also inadequate for detecting issues related to JavaScript execution and dynamic content rendering. Here is why:

1. Limited HTML Validation: Hurl could validate static HTML elements but failed to detect missing content that should be rendered by JavaScript.
2. JavaScript Execution Gap: Hurl does not execute JavaScript; therefore, it cannot verify whether JavaScript-dependent components have loaded correctly.
3. No Improvement Over Rester: In the context of testing dynamic, js-driven content, Hurl did not offer significant advantages over Rester.

2nd Proposed Solution

After presenting findings from investigating Hurl last week, you suggested adding a “health check” feature to the DocC Vue.js app as a possible solution

Investigating Health Check Feature

Adding a health check route will not help confirm that the documentation pages are being rendered correctly (i.e., not a blank page due to JavaScript errors). Health check routes are typically used to verify the status of a server or application, ensuring that it is running and responding to requests (e.g., by returning a simple “OK” response) — both Rester and Hurl already do this.

Healthchecks do not execute or validate client-side rendering, which is where JavaScript issues arise. Here is why:

1. Server-Side vs. Client-Side: A health check route usually only confirms the status of the server and its ability to respond, but in our case, the issue involves client-side JavaScript. A health check cannot validate whether the JS application on the client side successfully loaded or whether content was properly rendered.

2. JavaScript Rendering Issues: The problem of blank pages arises from failures in JavaScript execution or content rendering in the browser, which a standard health check route would not be able to detect.

My Proposed Solution

To effectively test JavaScript-dependent documentation pages, a tool that can simulate a real browser environment and execute JavaScript is needed. Such a tool will detect issues when the JavaScript application fails to load, resulting in blank or improperly rendered pages.

Headless Browser Testing Frameworks like:

• Nightwatch
• Puppeteer
• Playwright
• Selenium

can automate browser actions and execute JavaScript, providing end-to-testing.

Due to its integration with Vue.js, Nightwatch looks super promising. I also like Playwright (supports multiple browsers out of the box). Given my limited experience with JavaScript though, I suggest talking to the frontend team (@daveverwer) to get their input.

[daveverwer]

To effectively test JavaScript-dependent documentation pages, a tool that can simulate a real browser environment and execute JavaScript is needed. Such a tool will detect issues when the JavaScript application fails to load, resulting in blank or improperly rendered pages.

Headless Browser Testing Frameworks like:

• Nightwatch
• Puppeteer
• Playwright
• Selenium

can automate browser actions and execute JavaScript, providing end-to-testing.

This all sounds great @msuzoagu! Thank you for taking the time to write it up.

Using a WebDriver-backed testing framework would be a great choice for this kind of test and I think any of the frameworks you mention would do a fine job.

I also agree that Nightwatch might be the best choice, not only for the potential Vue integration but also that it is developed and backed by BrowserStack, who we already use for our snapshot testing. We would not use BrowserStack to host this test, but if we ever do add JavaScript-based integration testing to the project as a separate piece of work we’d probably do it via Nightwatch and BrowserStack’s hosted testing service, so there would be some potential for reuse of effort there.

I’ll leave the choice to you, though. Any of the suggested frameworks will do the job.

I’d be happy to be on hand if you need any help working with the tool. Let me know if I can help.

[msuzoagu]

I also agree that Nightwatch might be the best choice, not only for the potential Vue integration but also that it is developed and backed by BrowserStack, who we already use for our snapshot testing. We would not use BrowserStack to host this test, but if we ever do add JavaScript-based integration testing to the project as a separate piece of work we’d probably do it via Nightwatch and BrowserStack’s hosted testing service, so there would be some potential for reuse of effort there.

thanks for input and offer to help if needed @daveverwer! Just spoke to @finestructure and will move ahead with a solution that makes use of Nightwatch. Will keep you in the loop as things move ahead.

[msuzoagu]

@finestructure @daveverwer I have invited you to a private repo containing the start of how I think testing via Nightwatch js would work.

Note that no integrating with the CI/CD system, which I believe is gitlab, is addressed as I would like to get it working locally first.

Issue

Currently running into timeout issue where test runs successfully for a while then produces, what I see as, a false failure. Using the term "False Failure" because when I inspect the url in question, the CSS selectors used in the test are present in the DOM.

Not sure what is happening here and if it can be fixed by introducing some kind of batching/timeout. Will discuss with @finestructure tomorrow.

[daveverwer]

Thanks for the invite, I just accepted it. I know you're meeting with Sven tomorrow but let me know if I can help with anything.

When I tried running the test I got a plugin error, so there may be some additional dependencies maybe not specified in the packagge.json:

default (node:76284) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland
alternative instead.                                                                                         
default (Use `node --trace-deprecation ...` to show where the warning was created)                           
default Error                                                                                                
default    unhandledRejection: Unable to load plugin: nightwatch-axe-verbose: [MODULE_NOT_FOUND] Cannot find 
module 'nightwatch-axe-verbose'                                                                              
default Error: Unable to load plugin: nightwatch-axe-verbose: [MODULE_NOT_FOUND] Cannot find module          
'nightwatch-axe-verbose'                                                                                     
default     at PluginLoader.load                                                                             
(/Users/dave/.npm/_npx/3fd69c9acfe5f546/node_modules/nightwatch/lib/api/_loaders/plugin.js:71:21)            
default     at /Users/dave/.npm/_npx/3fd69c9acfe5f546/node_modules/nightwatch/lib/api/index.js:180:35        
default     at Array.forEach (<anonymous>)                                                                   
default     at ApiLoader.loadPlugins                                                                         
(/Users/dave/.npm/_npx/3fd69c9acfe5f546/node_modules/nightwatch/lib/api/index.js:179:13)                     
default     at                                                                                               
/Users/dave/.npm/_npx/3fd69c9acfe5f546/node_modules/nightwatch/lib/api/_loaders/within-context.js:26:35      

[msuzoagu]

hi @daveverwer! Thanks for letting me know about the dependency issue you are facing.

Missing plugin issue isolated to this error:

 Unable to load plugin: nightwatch-axe-verbose: [MODULE_NOT_FOUND] Cannot find 

Looked into it and found that this plugin is installed by default:

**NOTE: If you are using [Nightwatch 2.3.6]
(https://github.com/nightwatchjs/nightwatch/releases/tag/v2.3.6) 
or greater this is now pre-included as a default plugin in the 
Nightwatch installation so you can skip this install section**

Since the package.json has a dependency on Nightwatch v3.8.0, nightwatch-axe-verbose plugin should already be included by default. But the project readme was missing an additional step that might help resolve this.

Can you please run npm install before running npx nightwatch --verbose test/testSelectors.js? And let me know if the issue still persists.

[daveverwer]

Thanks @msuzoagu. I did try a npm install yesterday before running it as I assumed it would be needed. I did it again today and it installed nothing new but worked well this time… Computers! 😂

I now see the same error you see 😬

[msuzoagu]

@daveverwer computers ... everyone thinks they are "logical" but only we know how temperamental they can be :laughing:

[msuzoagu]

Hi @finestructure! Following up on our discussion today, I have finished up a version that accepts a url as command line arg; see branch commandline-args.

Let me know if that works for your use case so that I move towards incorporating CI/CD.

[finestructure]

I'm going to go ahead and close this issue as the investigation is complete. And not only that, thanks to @msuzoagu's work we also have a replacement for our existing doc test in place which should soon go live: https://gitlab.com/finestructure/spi-doctest

Thanks again for your help with this, @msuzoagu !