chromium-bidi
chromium-bidi copied to clipboard
GoogleChromeLabs
Should we author e2e tests in a forward-compatible manner?
For example, adding new attributes to local end structures is not a breaking changes in the spec but it will be a breaking change for e2e (and maybe for some wpt) tests that use assert === to check the entire structure shape. I believe we should treat all local end structs as extensible in our tests to avoid modifying all tests when there is a new attribute added.
cc @sadym-chromium
Historically, the strict assertions were done in order to be aware of any protocol changes. This was reasonable when the amount of e2e tests was relatively low and the protocol was changing quite quick.
The proposed transition allowing for flexibility in adding fields in the responses without causing test failures. However, this shift introduces the possibility of introducing new fields without explicit awareness. A significant concern is that developers often rely on tests rather than directly examining the raw protocol.
Having said that, in my experience I don't recall this was a problem, so I'm fine if we change protocol assertions from == to <=, if that reduces friction with an acceptable level of risks.
Note that there is a mechanism for <= assertion in place, but it is not very ergonomic.
To increase usability, we can consider introducing something like recursive_compare.
In "<=" I mean "is sub-dictionary" comparison. It is already implemented in 2 ways:
- In mapper:
AnyWithEntries, but it is not recursive. - In WPT:
recursive_compare.