Unexpected state of AET report when every outgoing HTTP request ends with a TCP failure

[jwadolowski]

Describe the bug

I just deployed AET Swarm stack on top of infrastructure, which does not have access to the internet whatsoever (every request attempt ends with TCP connection reset).

That was deployment from scratch, so MongoDB was empty and there was no patterns to compare against.

Generated report was a bit strange - URLs w/ view port checks were all green:

More complex ones (view ports + JS errors + markup + accessibility) were reported as red, however only in certain sections:

• Version used: AET 3.3.0

To Reproduce

1. Make sure your AET instance does not have access to the internet
2. Run suite of your choice

Expected behaviour

All red report + information that requested resources could not be fetched. Under certain circumstances (markup/accessibility checks disabled) it can lead to a false impression that everything is fine.

Screenshots

See above

Additional context

N/A

[malaskowski]

Hi @jwadolowski , I guess that's expected behaviour when screenshot comparison is always green for the very first run of AET (there is nothing to compare, AET just collected patterns). Screenshot collection and comparison are fairly simple - open the page in the browser, make screenshot compare to pattern (if there is no pattern - there's assumption that current screenshot will be the pattern). I don't see much sense in implementing special mechanics that will check if the page is not blank or loaded at all for screenshot comparisons. First of all, that scenario is marginal. Second of all - another checks are for that (e.g. properly used status-codes should return errors in that case) that can be combined with screen in one test.

[jwadolowski]

What I'm trying to say is that's kinda misleading - no content was fetched, but all of a sudden the report is green :) Exactly the same situation can happen if someone made a typo in the domain name.

I'd say that processing shouldn't even go to comparison stage if given request ends with a timeout/DNS failure/TLS error/TCP reset/whatever.

[malaskowski]

Ok, I think I understand your idea better now. I would still try to keep collectors (such as screen/status-code etc.) as simple as possible and avoid extending their logic with additional checks. But the check if AET were able to reach the desired website could be done when using open command (which should not be hard to implement). Source collector behaviour is different (and behaves as you suggest), because it does not use the webdriver to access the website (it connects directly to fetch the markup before browser's processing).