apify-docs icon indicating copy to clipboard operation
apify-docs copied to clipboard

chore: test Academy exercises

Open honzajavorek opened this issue 3 weeks ago • 18 comments

This is a proof of concept how we could test exercises in the academy:

  • Exercises target real world websites, so they can easily break without us knowing
  • We could test them e.g. weekly, so it's not too noisy
  • The solution should be as simple as possible, and capable running both JavaScript and Python
  • While working on the PoC I actually discovered one exercise which is broken 95% of time due to aggressive anti-scraping protections, so I changed it
  • The test can be executed with bats -r --print-output-on-failure .
  • Bats is a simple testing framework based on Bash, which allows to run arbitrary programs and evaluate their output - https://github.com/bats-core/bats-core Not sure yet how to get it inside the CI, but on macOS it's just brew install bats

Todo:

  • [x] Implement Python exercise
  • [x] Implement JS exercise
  • [x] Implement GitHub Action
  • [x] Discuss with the team whether we want this
  • [x] Document the solution
  • [x] Port the rest of the exercises
  • [x] Change the YAML to do just crons

[!NOTE] Adds a monthly GitHub Action and Bats-based suite to run Academy JS/Python exercise solutions, embeds solutions into lessons, and updates docs/ignore files.

  • CI:
    • Add monthly and manual workflow .github/workflows/test-academy.yml to run Academy exercises via Node (npm) and Python (uv).
  • Testing:
    • Introduce Bats-based tests for Academy exercises: sources/academy/**/exercises/test.bats (JS & Python) executing solutions and asserting outputs.
    • New npm script test:academy and dev dependency bats.
  • Docs/Academy content:
    • Embed executable exercise solutions using CodeBlock + !!raw-loader across JS/Python lessons (04–12), replacing inline snippets.
    • Update several exercises (e.g., switch example from AliExpress to LEGO) and add multiple new solution files (Cheerio/BeautifulSoup/Crawlee).
  • Repo:
    • Update .gitignore to exclude exercise artifacts (storage, node_modules, package*.json, dataset.json).
    • Document testing process in CONTRIBUTING.md (broken links check + Academy exercises CI).

Written by Cursor Bugbot for commit 118a55065eb6993fcad36815d31aa72ef73bd767. Configure here.

honzajavorek avatar Nov 18 '25 13:11 honzajavorek

Preview for this PR was built for commit 05099c1 and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account avatar Nov 21 '25 16:11 apify-service-account

From my perspective, we should design them in a way that they run automatically and do not require manual start-up. Otherwise, we'll forget about it eventually.

tomnosek avatar Nov 24 '25 10:11 tomnosek

Sure, I'd make a GitHub Action, which runs like once a week or once a month (depends on our ability to fix the exercises, doesn't make sense to run them too often).

honzajavorek avatar Nov 24 '25 12:11 honzajavorek

(Creating such GitHub Action is a matter of a few lines and I'll add it to this PR)

honzajavorek avatar Nov 24 '25 12:11 honzajavorek

Preview for this PR was built for commit cf913593 and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account avatar Nov 24 '25 13:11 apify-service-account

Preview for this PR was built for commit cb4959b2 and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account avatar Nov 24 '25 13:11 apify-service-account

Preview for this PR was built for commit 5e78a4db and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account avatar Nov 24 '25 13:11 apify-service-account

Preview for this PR was built for commit 463457b9 and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account avatar Nov 24 '25 13:11 apify-service-account

Preview for this PR was built for commit 0214184a and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account avatar Nov 24 '25 13:11 apify-service-account

Preview for this PR was built for commit b930d1f and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account avatar Nov 24 '25 17:11 apify-service-account

Preview for this PR was built for commit 4fccbf45 and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account avatar Nov 25 '25 09:11 apify-service-account

Preview for this PR was built for commit 96ae391 and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account avatar Nov 25 '25 12:11 apify-service-account

Preview for this PR was built for commit 44d7b1a and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account avatar Nov 25 '25 12:11 apify-service-account

Preview for this PR was built for commit ddf09129 and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account avatar Nov 25 '25 13:11 apify-service-account

Preview for this PR was built for commit 6c5f05bb and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account avatar Nov 25 '25 14:11 apify-service-account

Preview for this PR was built for commit 118a550 and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account avatar Nov 25 '25 14:11 apify-service-account

I think this should be ready now. The Bats tests currently do fail, because there are failing exercises, so that's expected. But the testing infrastructure is solid and this PR is about the infrastructure. I've set the frequency of the tests to monthly and let's see. I didn't want to snowball this PR, so I recorded the failures and other issues separately and I'll work on them in subsequent PRs:

  • https://github.com/apify/apify-docs/issues/2112
  • https://github.com/apify/apify-docs/issues/2113

honzajavorek avatar Nov 25 '25 14:11 honzajavorek

Preview for this PR was built for commit 8c875d1 and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account avatar Nov 25 '25 15:11 apify-service-account