[WIP] Cap and update versions in CI

[bouwkast]

What does this PR do?

Adds outdated.js and latests.json to be used to cap versions pulled in during testing via mocha.js.

To generate or update latests.json: node scripts/outdated.js fix

To just validate latests.json: node scripts/outdated.js check

Idea is to use mocha.js to try and enforce these capped versions.

TODO:

• Review/Remove the register/install_plugin_modules from Bryan
  • We don't want to cap the instrumented version at this time which I think these do
• Create script to automatically call node scripts/outdated.js fix and make a PR
• Test this as I don't know how :)

Motivation

We'd like to cap the versions pulled in during testing to keep CI more stable. We'd like to create a GitHub Action that runs consistently to keep these updated and get better visibility of unsupported versions.

Plugin Checklist

• [ ] Unit tests.
• [ ] TypeScript definitions.
• [ ] TypeScript tests.
• [ ] API documentation.
• [ ] CircleCI jobs/workflows.
• [ ] Plugin is exported.

Additional Notes

[github-actions[bot]]

Overall package size

Self size: 8.83 MB Deduped: 95.03 MB No deduping: 95.54 MB

Dependency sizes

| name | version | self size | total size | |------|---------|-----------|------------| | @datadog/libdatadog | 0.4.0 | 29.44 MB | 29.44 MB | | @datadog/native-appsec | 8.4.0 | 19.25 MB | 19.26 MB | | @datadog/native-iast-taint-tracking | 3.3.0 | 13.77 MB | 13.78 MB | | @datadog/pprof | 5.5.1 | 9.79 MB | 10.17 MB | | protobufjs | 7.2.5 | 2.77 MB | 5.16 MB | | @datadog/native-iast-rewriter | 2.8.0 | 2.6 MB | 2.74 MB | | @opentelemetry/core | 1.14.0 | 872.87 kB | 1.47 MB | | @datadog/native-metrics | 3.1.0 | 1.06 MB | 1.46 MB | | @opentelemetry/api | 1.8.0 | 1.21 MB | 1.21 MB | | import-in-the-middle | 1.11.2 | 112.74 kB | 835.4 kB | | source-map | 0.7.4 | 226 kB | 226 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | lru-cache | 7.18.3 | 133.92 kB | 133.92 kB | | pprof-format | 2.1.0 | 111.69 kB | 111.69 kB | | @datadog/sketches-js | 2.1.0 | 109.9 kB | 109.9 kB | | lodash.sortby | 4.7.0 | 75.76 kB | 75.76 kB | | ignore | 5.3.2 | 53.63 kB | 53.63 kB | | shell-quote | 1.8.1 | 44.96 kB | 44.96 kB | | istanbul-lib-coverage | 3.2.0 | 29.34 kB | 29.34 kB | | rfdc | 1.3.1 | 25.21 kB | 25.21 kB | | @isaacs/ttlcache | 1.4.1 | 25.2 kB | 25.2 kB | | tlhunter-sorted-set | 0.1.0 | 24.94 kB | 24.94 kB | | limiter | 1.1.5 | 23.17 kB | 23.17 kB | | dc-polyfill | 0.1.4 | 23.1 kB | 23.1 kB | | retry | 0.13.1 | 18.85 kB | 18.85 kB | | semifies | 1.0.0 | 15.84 kB | 15.84 kB | | jest-docblock | 29.7.0 | 8.99 kB | 12.76 kB | | crypto-randomuuid | 1.0.0 | 11.18 kB | 11.18 kB | | ttl-set | 1.0.0 | 4.61 kB | 9.69 kB | | path-to-regexp | 0.1.12 | 6.6 kB | 6.6 kB | | koalas | 1.0.2 | 6.47 kB | 6.47 kB | | module-details-from-path | 1.0.3 | 4.47 kB | 4.47 kB |

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

[pr-commenter[bot]]

Benchmarks

Benchmark execution time: 2025-02-27 21:11:30

Comparing candidate commit 611eafd8d890b02300ed4d9e52866cce00ee1456 in PR branch steven/cap-and-update with baseline commit 2c9ab130df5db7459ce85bceae3fe892d7cc7fcc in branch master.

Found 72 performance improvements and 8 performance regressions! Performance is the same for 808 metrics, 15 unstable metrics.

scenario:appsec-iast-no-vulnerability-control-18

• 🟩 cpu_user_time [-93.927ms; -81.720ms] or [-16.097%; -14.005%]
• 🟩 execution_time [-100.115ms; -76.096ms] or [-14.729%; -11.195%]
• 🟩 instructions [-182.4M instructions; -160.1M instructions] or [-13.427%; -11.785%]
• 🟩 max_rss_usage [-5.555MB; -4.530MB] or [-7.024%; -5.728%]

scenario:appsec-iast-no-vulnerability-control-20

• 🟩 cpu_user_time [-105.584ms; -95.714ms] or [-18.544%; -16.811%]
• 🟩 execution_time [-108.868ms; -96.591ms] or [-16.810%; -14.914%]
• 🟩 instructions [-164.4M instructions; -145.8M instructions] or [-13.608%; -12.067%]
• 🟩 max_rss_usage [-4.382MB; -3.916MB] or [-5.716%; -5.109%]

scenario:appsec-iast-no-vulnerability-control-22

• 🟩 cpu_user_time [-94.538ms; -83.902ms] or [-18.496%; -16.415%]
• 🟩 execution_time [-98.833ms; -83.412ms] or [-16.563%; -13.979%]
• 🟩 instructions [-162.7M instructions; -140.7M instructions] or [-13.106%; -11.332%]
• 🟩 max_rss_usage [-6.865MB; -6.163MB] or [-7.940%; -7.128%]

scenario:appsec-iast-no-vulnerability-iast-enabled-always-active-18

• 🟩 cpu_user_time [-146.310ms; -124.212ms] or [-11.039%; -9.372%]
• 🟩 execution_time [-154.233ms; -122.829ms] or [-10.539%; -8.393%]
• 🟩 instructions [-222.5M instructions; -204.6M instructions] or [-8.388%; -7.716%]

scenario:appsec-iast-no-vulnerability-iast-enabled-always-active-20

• 🟩 cpu_user_time [-171.243ms; -150.683ms] or [-12.411%; -10.921%]
• 🟩 execution_time [-176.022ms; -159.001ms] or [-11.631%; -10.506%]
• 🟩 instructions [-187.7M instructions; -164.7M instructions] or [-7.204%; -6.321%]

scenario:appsec-iast-no-vulnerability-iast-enabled-always-active-22

• 🟩 cpu_user_time [-117.317ms; -99.646ms] or [-11.303%; -9.600%]
• 🟩 execution_time [-122.877ms; -102.547ms] or [-10.495%; -8.758%]
• 🟩 instructions [-186.4M instructions; -159.0M instructions] or [-7.285%; -6.215%]

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-18

• 🟩 cpu_user_time [-119.010ms; -105.990ms] or [-9.316%; -8.297%]
• 🟩 execution_time [-127.729ms; -105.043ms] or [-9.045%; -7.438%]
• 🟩 instructions [-217.7M instructions; -185.1M instructions] or [-8.775%; -7.462%]

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-20

• 🟩 cpu_user_time [-154.073ms; -139.944ms] or [-11.532%; -10.474%]
• 🟩 execution_time [-165.110ms; -149.838ms] or [-11.197%; -10.161%]
• 🟩 instructions [-196.4M instructions; -173.8M instructions] or [-8.050%; -7.121%]

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-22

• 🟩 cpu_user_time [-113.604ms; -95.885ms] or [-11.283%; -9.524%]
• 🟩 execution_time [-113.440ms; -97.365ms] or [-9.980%; -8.566%]
• 🟩 instructions [-180.4M instructions; -147.8M instructions] or [-7.558%; -6.193%]

scenario:appsec-iast-with-vulnerability-control-18

• 🟩 cpu_user_time [-108.011ms; -84.805ms] or [-13.188%; -10.355%]
• 🟩 execution_time [-137.754ms; -112.848ms] or [-10.379%; -8.502%]
• 🟩 instructions [-205.5M instructions; -186.1M instructions] or [-8.570%; -7.763%]

scenario:appsec-iast-with-vulnerability-control-20

• 🟩 cpu_user_time [-111.652ms; -95.136ms] or [-14.087%; -12.003%]
• 🟩 execution_time [-131.650ms; -115.507ms] or [-12.269%; -10.764%]
• 🟩 instructions [-173.2M instructions; -153.6M instructions] or [-8.986%; -7.969%]

scenario:appsec-iast-with-vulnerability-control-22

• 🟩 cpu_user_time [-94.998ms; -75.989ms] or [-13.603%; -10.881%]
• 🟩 execution_time [-109.188ms; -91.269ms] or [-11.302%; -9.447%]
• 🟩 instructions [-156.9M instructions; -131.8M instructions] or [-8.085%; -6.790%]

scenario:appsec-iast-with-vulnerability-iast-enabled-always-active-18

• 🟩 cpu_user_time [-373.648ms; -332.388ms] or [-19.665%; -17.494%]
• 🟩 execution_time [-492.655ms; -400.410ms] or [-17.606%; -14.309%]
• 🟩 instructions [-525.7M instructions; -486.4M instructions] or [-11.733%; -10.856%]

scenario:appsec-iast-with-vulnerability-iast-enabled-always-active-20

• 🟩 cpu_user_time [-324.603ms; -290.027ms] or [-17.243%; -15.406%]
• 🟩 execution_time [-389.631ms; -341.329ms] or [-15.905%; -13.934%]
• 🟩 instructions [-381.0M instructions; -349.8M instructions] or [-9.632%; -8.845%]
• 🟩 max_rss_usage [-11.939MB; -9.492MB] or [-9.332%; -7.420%]

scenario:appsec-iast-with-vulnerability-iast-enabled-always-active-22

• 🟩 cpu_user_time [-200.460ms; -171.078ms] or [-13.557%; -11.570%]
• 🟩 execution_time [-241.658ms; -195.355ms] or [-12.094%; -9.777%]
• 🟩 instructions [-376.8M instructions; -339.9M instructions] or [-9.670%; -8.723%]
• 🟩 max_rss_usage [-11.815MB; -8.328MB] or [-9.259%; -6.526%]

scenario:appsec-iast-with-vulnerability-iast-enabled-default-config-18

• 🟩 cpu_user_time [-322.228ms; -288.756ms] or [-18.063%; -16.187%]
• 🟩 execution_time [-437.636ms; -359.124ms] or [-16.850%; -13.827%]
• 🟩 instructions [-390.7M instructions; -338.9M instructions] or [-9.753%; -8.459%]

scenario:appsec-iast-with-vulnerability-iast-enabled-default-config-20

• 🟩 cpu_user_time [-273.327ms; -244.887ms] or [-15.410%; -13.806%]
• 🟩 execution_time [-328.660ms; -284.316ms] or [-14.517%; -12.558%]
• 🟩 instructions [-299.6M instructions; -251.4M instructions] or [-8.484%; -7.121%]
• 🟩 max_rss_usage [-11.526MB; -8.197MB] or [-9.037%; -6.427%]

scenario:appsec-iast-with-vulnerability-iast-enabled-default-config-22

• 🟩 cpu_user_time [-140.482ms; -111.107ms] or [-10.530%; -8.328%]
• 🟩 execution_time [-169.013ms; -126.415ms] or [-9.542%; -7.137%]
• 🟩 instructions [-275.6M instructions; -234.3M instructions] or [-7.957%; -6.766%]

scenario:plugin-graphql-control-18

• 🟥 cpu_user_time [+16.974ms; +24.417ms] or [+5.647%; +8.123%]
• 🟥 execution_time [+20.007ms; +24.468ms] or [+5.735%; +7.014%]

scenario:plugin-graphql-with-async-hooks-18

• 🟥 cpu_user_time [+23.254ms; +35.781ms] or [+6.177%; +9.504%]
• 🟥 execution_time [+27.414ms; +39.607ms] or [+6.379%; +9.217%]

scenario:plugin-q-control-20

• 🟥 cpu_user_time [+60.460ms; +70.335ms] or [+12.103%; +14.080%]
• 🟥 execution_time [+60.722ms; +68.921ms] or [+10.593%; +12.024%]

scenario:plugin-q-control-22

• 🟥 cpu_user_time [+37.347ms; +46.578ms] or [+8.280%; +10.327%]
• 🟥 execution_time [+37.490ms; +44.868ms] or [+7.195%; +8.611%]

scenario:plugin-q-with-tracer-18

• 🟩 cpu_user_time [-147.882ms; -127.854ms] or [-16.513%; -14.277%]
• 🟩 execution_time [-155.065ms; -134.985ms] or [-15.648%; -13.622%]
• 🟩 instructions [-309.8M instructions; -269.9M instructions] or [-16.475%; -14.354%]
• 🟩 max_rss_usage [-9.804MB; -9.315MB] or [-6.534%; -6.209%]

scenario:plugin-q-with-tracer-20

• 🟩 cpu_user_time [-116.228ms; -97.629ms] or [-13.591%; -11.416%]
• 🟩 execution_time [-122.264ms; -103.811ms] or [-12.864%; -10.923%]
• 🟩 instructions [-298.8M instructions; -266.0M instructions] or [-17.476%; -15.555%]
• 🟩 max_rss_usage [-10.389MB; -9.955MB] or [-6.969%; -6.678%]

scenario:plugin-q-with-tracer-22

• 🟩 cpu_user_time [-178.094ms; -159.180ms] or [-20.317%; -18.160%]
• 🟩 execution_time [-189.201ms; -170.006ms] or [-19.424%; -17.454%]
• 🟩 instructions [-316.9M instructions; -267.0M instructions] or [-17.793%; -14.987%]
• 🟩 max_rss_usage [-17.051MB; -15.597MB] or [-10.857%; -9.931%]

[codecov[bot]]

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 73.01%. Comparing base (2c9ab13) to head (611eafd). :warning: Report is 641 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4906      +/-   ##
==========================================
- Coverage   80.46%   73.01%   -7.45%     
==========================================
  Files         491      428      -63     
  Lines       21850    19372    -2478     
==========================================
- Hits        17581    14144    -3437     
- Misses       4269     5228     +959     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[datadog-datadog-prod-us1[bot]]

Datadog Report

Branch report: steven/cap-and-update Commit report: aca1e66 Test service: dd-trace-js-integration-tests

:x: 51 Failed (0 Known Flaky), 464 Passed, 0 Skipped, 7m 9.78s Total Time

:x: Failed Tests (51)

This report shows up to 5 failed tests.

• cucumber@latest commonJS correctly calculates test code owners when working directory is not repository root - integration-tests/cucumber/cucumber.spec.js - Details

  Expand for error

  annot read properties of undefined (reading 'content')
  

• cucumber@latest commonJS correctly calculates test code owners when working directory is not repository root - integration-tests/cucumber/cucumber.spec.js - Details

  Expand for error

  annot read properties of undefined (reading 'content')
  

• cucumber@latest commonJS reporting via agentless intelligent test runner can report git metadata - integration-tests/cucumber/cucumber.spec.js - Details

  Expand for error

  imeout
  

• cucumber@latest commonJS reporting via agentless intelligent test runner can report git metadata - integration-tests/cucumber/cucumber.spec.js - Details

  Expand for error

  imeout
  

• cucumber@latest commonJS reporting via agentless intelligent test runner can skip suites received by the intelligent test runner API and still reports code coverage - integration-tests/cucumber/cucumber.spec.js - Details

  Expand for error

  imeout
  

[BridgeAR]

I very much like the overall idea here. Is that still worked upon?

[BridgeAR]

Seems like this is doing something similar to https://github.com/DataDog/dd-trace-js/pull/4819?

[rochdev]

Closing as this was superseded by #6169