Add tests via TestDriverAI (#1957)

[Frooodle]

• initial Commit

• update prerun

• tweak the prompt

• update the test

• finetune prompt

• change the prompt

• minor change to retry test

• add debug

---

Description

Please provide a summary of the changes, including relevant motivation and context.

Closes #(issue_number)

Checklist

• [ ] I have read the Contribution Guidelines
• [ ] I have performed a self-review of my own code
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] My changes generate no new warnings
• [ ] I have read the section Add New Translation Tags (for new translation tags only)

[github-actions[bot]]

🚀 Translation Verification Summary

📋 Checking with the file messages_en_GB.properties from the main-branch - Checking the pr-branch

🗂️ Checking File: messages_ca_CA.properties...

• Test 1 Status: ❌ Failed
  • Issue: Missing lines! Comments, empty lines, or translation strings are missing. Details: 1242 (reference) vs 1241 (current).
• Test 2 Status: ❌ Failed
  • Issue: There are keys in messages_en_GB.properties navbar.search that are not present in messages_ca_CA.properties!

---

❌ Overall Check Status: Failed

[chottuthejimmy]

Hey @Frooodle hope you are doing well mate. long time 😅 , sorry.....

So I tried to a few stuff and was very close to becoming the guy in the meme who burns all the money on aws. One of the things that looked promising was running docker on another linux instance and accessing it in the windows VM via the private IP , but the issue is 1) we do not want to act as a build server , 2) its not scalable for us, given the dynamic customers we work with.

So I am here to ask for you help and advice, what do you think would be a nice way for you to build and host the docker image (max we would need it live for like 15 - 20 min) who's public url we can just reference in our windows VM

I tried a few stuff like ngrok, but the issue is its not persistent, the url goes down once the job terminates, we need that url for our testing job :(

[Frooodle]

Hey I have been testing this on my side I recently grabbed a VPS and used it for our pr testing via using /deploy-pr as a comment in any PR

Was using this also as a test to then integrate with your scripts so we can throw it on a cheap VPS and deploy it before hand

[Frooodle]

https://github.com/Stirling-Tools/Stirling-PDF/pull/2433#issuecomment-2548073262 comment where deployment is in use

https://github.com/Stirling-Tools/Stirling-PDF/blob/main/.github/workflows/PR-Demo-Comment.yml code

for example

[Frooodle]

I think its unreasonable to suggest your side to do any building or deploying on this scale when each type can be so different so I agree it should be on our side it gets deployed etc

[chottuthejimmy]

You are awesome dude. You saved my Christmas haha

[chottuthejimmy]

Nice. Now what do you want from me ? How can I help you ?

[Frooodle]

I think its all good, its on me to pair the deployment script I have done to run before the testdriverAI side and then should be good

[Frooodle]

🫡

[chottuthejimmy]

Hey @Frooodle how's it going mate. Anything you need help with ??

[Frooodle]

Knew I forgot something.. give me 30 minutes.. :) I'll speed it out

[Frooodle]

Cool testing now, https://github.com/Stirling-Tools/Stirling-PDF/actions/runs/12753015589/job/35543898659 will have this as a pre-release test instead of per commit

Can you link a doc (or update ours) on how we update this test as we go forward so others can adapt, will need to add a small writeup to our https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md

[Frooodle]

@chottuthejimmy We recently upgraded our security practices to use hard coded versions for our github actions, I see your latested release was 0.0.3 9 months ago and you advise people use branch 'main' instead, could you make a new release? We are following best practices from https://www.bestpractices.dev/en/projects/9847 and https://scorecard.dev/viewer/?uri=github.com/Stirling-Tools/Stirling-PDF

[Frooodle]

https://github.com/marketplace/actions/testdriver-ai

[Frooodle]

Cool i think everything else other than release tag and doc is good! Just finished the rest of security changes and running final test once that first one has finished

[ianjennings]

@Frooodle tagged 1.0.0 and released!

Cool i think everything else other than release tag and doc is good! Just finished the rest of security changes and running final test once that first one has finished

[ianjennings]

You can update to use [email protected]

[Frooodle]

Thanks for all the work, will merge this once latest run has finished!

[Frooodle]

@ianjennings @chottuthejimmy sorry it took so long appreciate the help along the way!

[chottuthejimmy]

Heyy @Frooodle

Can you link a doc (or update ours) on how we update this test as we go forward so others can adapt, will need to add a small writeup to our https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md

I bet folks are going to contribute a ton to this haha. It's actually really that simple. Anyways, if you meant like some guidelines about how to write the prompt / create tests locally, there are a few docs that I can link to and write a bit based on my experience of generating tests so far. You got any tips for me ? First time writing a contribution docs, and that too for something like testdriver, it def feels like we are writing a new framework or something 😅

[Frooodle]

Honestly All docs are iterative Whatever you add can be improved on certainly by the open source community.

As long as you can give examples when it's something complex it should be fine