Add tests via TestDriverAI

[chottuthejimmy]

Description

This PR contains the test cases to test the basic functionality of the stirling.io webpage. The tests are written using testdriverai a next gen automation AI agent for end-to-end testing of web and desktop application.

Overview

This PR adds TestDriver tests that I generated using the testdriverai npm package. Instead of using selectors, TestDriver uses prompts and AI vision to complete tasks. This means the tests will still work even when code and design changes.

Another huge benefit is we can test PDF input and output because tests operate at the OS level. You can see the schema of supported commands here.

Test Cases

• Opens Google Chrome and interacts with various UI elements like buttons and icons.
• Verifies opening and viewing of a PDF file.
• Navigates back to the homepage after certain actions.
• Changes the language from English to Spanish and back, verifying text translations.
• Toggles the website theme using sun and moon icons.
• Searches for specific terms and verifies correct navigation to related pages.
• Hovers over buttons like "Merge" and checks for visual feedback (background changes).

Setup

In order to complete this PR you’ll need to add a API key as a GitHub action secret. Follow the directions here for more info. Once you sign up, let us know your email and we’ll enroll you in the open source program for free Pro account.

Here's how to add more tests :

• Install the testdriverai package via npm and start by creating tests by just prompting the cli using natural language prompts ( you can check the prompts field in the test file for motivation )
• Once you are done writing the tests, just push it to the repo, here's where the magic happens.
• Whenever there's a new PR, the VMs (maintained by us) will clone the feature branch and performs the tests and returns the result along with the screen recordings and its logs (thanks to dashcam.io)
• Also feel free to check out our docs on how to improve performance and techniques like parallel testing

What’s next

We can set this up to build feature branches using our prerun scripts.

Checklist

• [x] I have read the Contribution Guidelines
• [x] I have performed a self-review of my own code
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] My changes generate no new warnings
• [ ] I have read the section Add New Translation Tags (for new translation tags only)

[ianjennings]

@Frooodle We need your approval to run the workflow.

[Frooodle]

I believe I need the key: ${{secrets.TESTDRIVER_API_KEY}}

As well? Additionally we should set it up to run on PR code not on web instance :p

[ianjennings]

Here is a preview of the test execution:

Watch testdriverai/Stirling-PDF/refs/heads/main on Dashcam

[chottuthejimmy]

Hey @Frooodle could you have another look at this.

It's a go from our side!

Cheers!

[Frooodle]

Sorry been very busy should be able to properly check it end of week

[Frooodle]

Added key but job still failed, does account need upgraded?

[ianjennings]

Added key but job still failed, does account need upgraded?

The API is reporting the key a null. Are you sure PRs have perms to access secrets? Or the variable name is correct?

[chottuthejimmy]

Hey @Frooodle could please make sure you have got the API set up properly. I would be happy to hop on a call and help you out, so let me know.

[Frooodle]

Secret is there along side other secrets used within this repo,

[chottuthejimmy]

Hey @Frooodle, so I tried to recreate the issue by forking and making a PR. I think its pretty much the API issue and nothing more. So could you try out a couple of things for us please.

Go to dashcam.io/team and click on the cycle button next to API Key, copy it and click on update. And replace the secret with this new key.

Try to re-run the failed job over here

If that doesn't work, make sure the right permissions are set here for PRs to access the secrets, or you can just merge this PR if you do not wish to change the permission.

[Frooodle]

All seems okay my side from what I can see, Can we add a - name: Debug - Print masked secret run: | echo "Secret is: ${{ secrets.TESTDRIVER_API_KEY }}"

to the job just to see if it can grab it or if its being lost in the call to the github action

[chottuthejimmy]

yes, lets try that. I made a commit, could try to run the workflow here

[Frooodle]

Hmm printing nope I'll check more my side

[chottuthejimmy]

Hmm printing nope I'll check more my side

It printed yes for me when I tried on my fork

[chottuthejimmy]

Is it because of this code ??

[chottuthejimmy]

@Frooodle could you take a look at this sometime today please!

[Frooodle]

Merging into repo branch to test it

[Frooodle]

and that instantly worked.. looks like it had to be integrated first to have access to secret, it works fine now

[Frooodle]

going to monitor PR and see ETA runtimes etc, will need to have it run a docker container and test the actual code not stirlingpdf.io deployment before merging into main also due to runtime might make more sense for this to be pre-release instead of precommit

[Frooodle]

https://app.dashcam.io/replay/6706e85e67f936f4742aef4d?share=3NOziidOgB0CDw1kNjVaA

Did show some package.json error

[chottuthejimmy]

and that instantly worked.. looks like it had to be integrated first to have access to secret, it works fine now

Told you hear JK.

[chottuthejimmy]

Did show some package.json error

We are unable to access the recordings (as they belong to your org / acc) please accept my request access so that we can have a look into it