code-dot-org
code-dot-org copied to clipboard
Upgrade eyes_selenium to 6.3.7
While upgrading Ruby to 3.3 (PR), we discovered two compatibility issues between our old version of eyes_selenium 3.18.4 and Ruby 3.3.
- eyes_selenium 3.18.4 has a dependency on the state_machine gem, which is not compatible with Ruby >= 3.2, and appears not to be receiving updates either. state_machine gem appears to have been forked into the state_machines gem, which is used by newer eyes_selenium versions and is fixed.
- eyes_selenium passes the faraday gem a hardcoded SSL cert for the applitools server in a way that breaks openssl on Ruby >= 3.1.
A previous (merged to test) attempt saw numerous eyes failures: applitools eyes failures during eyes_selenium upgrade
Possible Timing Issues
Solution eval dialog not popped up
- gamelab multiple choice contained level - Step 5/5: finished level with contained level
- maze multi contained level - Step 4/4: level run
- maze free response contained level - Step 4/4: level run
- applab contained level - step 5/5: finished level with contained level
Videos not rendered
- ai landing page - step 1/1: initial load
- dance tutorial landing - step 1/1: initial load
- minecraft tutorial landing - step 1/1: initial load
- app lab tutorial landing - step 1/1: initial load
Timing Etc
Microscopic differences in font rendering:
Example 1
Example 2
Possible example 3
This is a microscopic difference, there's a little pinch of white pixel in the non-basline.
Time travelling
"Saved In a few seconds" whereas baseline is "Saved a few seconds ago" 🤯
Teacher Feedback tab not present
Viewport Issues
Captured full page but baseline only shows viewport
Captured page wider than baseline
- V2 Progress Dashboard Assessments - step 2/3: feedback given icon is displayed
- V2 Progress Dashboard Assessments - step 3/3: keep working icon is displayed
Baseline includes empty capture beyond viewport
In this case, it almost seems like the baseline PNG is wrong:
- toggle on hidden maze level - step 2/3: view as student
- maze student has not starterd - step 1/1: student not started warning
- V2 progress dashboard - step 1/1: V2 progress dashboard, note this one is slightly different
- game lab student has not started - step 1/1: student not started warning
- learn landing page - step 1/1: initial load
- edit a function - step 1/2: add a new block to the function
Captured page cuts off viewport
Note that this may be capturing wider than the viewport. Both of these might be considered buggy, perhaps this is CSS stitching mode or something?