dd-trace-rb icon indicating copy to clipboard operation
dd-trace-rb copied to clipboard

Add parametric system-tests workflow

Open vpellan opened this issue 1 year ago • 1 comments
trafficstars

What does this PR do?

This PR adds a GitHub workflow for parametrics system-tests

Motivation:

The SCA Env Var system-tests is run as a parametric test, and I wanted to test it on the CI.

Additional Notes:

The FORCE_TESTS env var does not work as the -F option on system-tests only work for the DEFAULT scenario right now. There are currently 4 tests that needs to be fixed:

  • [ ] tests/parametric/test_dynamic_configuration.py::TestDynamicConfigSamplingRules::test_trace_sampling_rules_with_tags
  • [ ] tests/parametric/test_otel_span_methods.py::Test_Otel_Span_Methods::test_otel_span_started_with_link_from_datadog_headers
  • [ ] tests/parametric/test_otel_span_methods.py::Test_Otel_Span_Methods::test_otel_span_started_with_link_from_w3c_headers
  • [ ] tests/parametric/test_otel_span_methods.py::Test_Otel_Span_Methods::test_otel_span_started_with_link_from_other_spans

vpellan avatar Jun 10 '24 13:06 vpellan

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 98.11%. Comparing base (eab4b6e) to head (eddd956). Report is 2 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #3698   +/-   ##
=======================================
  Coverage   98.11%   98.11%           
=======================================
  Files        1225     1225           
  Lines       72743    72743           
  Branches     3479     3479           
=======================================
  Hits        71369    71369           
  Misses       1374     1374           

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov-commenter avatar Jun 10 '24 13:06 codecov-commenter

Hey @vpellan is this PR still relevant?

ivoanjo avatar Dec 09 '24 09:12 ivoanjo

Hey @vpellan is this PR still relevant?

This is still relevant (it is useful to run parametric system-tests) but needs to be reworked. We still have it in AppSec backlog (although it is not specific to AppSec)

vpellan avatar Dec 10 '24 08:12 vpellan

Ack! Do you want to keep this PR around as reference, or shall we close it for now?

ivoanjo avatar Dec 10 '24 08:12 ivoanjo

Ack! Do you want to keep this PR around as reference, or shall we close it for now?

I think it's better to keep it as reference, we can close it to reduce the number of open PR but I may reopen it at some point

vpellan avatar Dec 10 '24 14:12 vpellan

Nah, no rush, let's keep it open then! :)

ivoanjo avatar Dec 10 '24 14:12 ivoanjo

Well actually this has been fixed by a61a09e (#4174) so we can close this PR

vpellan avatar Dec 11 '24 13:12 vpellan