Add "continue-on-error" CI machines to test automotive scanners

[polybassa]

Hi, I would appreciate having the automotive scanners tests be part of the CI pipeline. However, I understand that some flaky tests cause problems for the overall pipeline.

Would it be possible, to have some CI-machines for the automotive scanners only, which are allowed to fail?

[codecov[bot]]

Codecov Report

Merging #3735 (09e63ca) into master (1989211) will decrease coverage by 1.39%. The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3735      +/-   ##
==========================================
- Coverage   86.01%   84.61%   -1.40%     
==========================================
  Files         301      296       -5     
  Lines       68390    62084    -6306     
==========================================
- Hits        58828    52535    -6293     
+ Misses       9562     9549      -13     

Impacted Files	Coverage Δ
scapy/contrib/isotp/isotp_scanner.py	18.18% <0.00%> (-71.60%)	:arrow_down:
scapy/contrib/cansocket_python_can.py	59.37% <0.00%> (-23.13%)	:arrow_down:
...apy/contrib/automotive/scanner/staged_test_case.py	84.87% <0.00%> (-6.73%)	:arrow_down:
scapy/contrib/automotive/uds_ecu_states.py	85.71% <0.00%> (-5.72%)	:arrow_down:
scapy/contrib/automotive/gm/gmlan_ecu_states.py	91.30% <0.00%> (-4.35%)	:arrow_down:
scapy/contrib/automotive/scanner/executor.py	79.63% <0.00%> (-4.08%)	:arrow_down:
scapy/contrib/automotive/scanner/enumerator.py	86.46% <0.00%> (-3.87%)	:arrow_down:
scapy/contrib/automotive/obd/services.py	93.10% <0.00%> (-3.45%)	:arrow_down:
scapy/contrib/automotive/ecu.py	92.00% <0.00%> (-2.67%)	:arrow_down:
scapy/contrib/automotive/obd/obd.py	94.44% <0.00%> (-1.86%)	:arrow_down:
... and 14 more

[p-l-]

Hi @polybassa can you please rebase against current master? :pray:

[polybassa]

Done

[polybassa]

Ok.. just the codecov returned an error. All other CI-machines returned passed, even if a scanner-test failed

[polybassa]

@gpotter2 What are your thoughts about this PR?

[gpotter2]

This is a decent idea. (sorry for the delay) I'd prefer to be able to keep the matrix clean though (for github ci). Do you think we could just add something similar to appveyor's UT_FLAGS but for that too ?

[polybassa]

Thanks for your response.

I'm not quite sure if I understand your request correctly.

For GitHub you would prefer a test without matrix?

For appveyor, you ask if we can integrate the allow to fail in UT_FLAGS?

-- Viele Grüße

Nils Weiß

On October 5, 2022 10:15:53 PM GMT+02:00, gpotter2 @.***> wrote:

This is a decent idea. (sorry for the delay) I'd prefer to be able to keep the matrix clean though (for github ci). Do you think we could just add something similar to appveyor's UT_FLAGS but for that too ?

-- Reply to this email directly or view it on GitHub: https://github.com/secdev/scapy/pull/3735#issuecomment-1268923481 You are receiving this because you were mentioned.

Message ID: @.***>

[polybassa]

@gpotter2 Do you like it more, now?

[gpotter2]

It's much better ! I see you bumped into the weird case where undefined values don't really work as expected...

[polybassa]

Yes, apparently the online example on GitHub was not working as intended.

Viele Grüße

Nils Weiß

On October 6, 2022 8:37:18 PM GMT+02:00, gpotter2 @.***> wrote:

It's much better ! I see you bumped into the weird case where undefined values don't really work as expected...

-- Reply to this email directly or view it on GitHub: https://github.com/secdev/scapy/pull/3735#issuecomment-1270520814 You are receiving this because you were mentioned.

Message ID: @.***>

[gpotter2]

Thanks a lot ! That's a great improvement

[polybassa]

Thank you for your patience

Viele Grüße

Nils Weiß

On October 6, 2022 8:44:19 PM GMT+02:00, gpotter2 @.***> wrote:

Thanks a lot ! That's a great improvement

-- Reply to this email directly or view it on GitHub: https://github.com/secdev/scapy/pull/3735#issuecomment-1270533404 You are receiving this because you were mentioned.

Message ID: @.***>