Fix randomly failing specs

[svbergerem]

RSpec

• [ ] ConversationsController#create mobile with non-mutual contact does not create a message spec, example
• [ ] ConversationsController#create mobile with non-mutual contact responds with an error message spec, example

Cucumber

• [x] features/desktop/aspect_navigation.feature, Scenario: Aspects selection is remembered through site navigation spec, ~example~, example Step: I should see "..." aspect selected, Error: expected to find visible css "li[data-aspect_id='1'] .selected" ...
• [ ] features/desktop/comments.feature, Scenario: comment on a status show page spec, example Step: I make a show page comment "...", Error: Unable to find button ...
• [ ] features/desktop/manages_aspects.feature: Scenario: sorting the aspects spec, example Step: And I have turned off jQuery effects, Error: ReferenceError: $ is not defined
• [ ] features/desktop/notifications.feature, Scenario: I mark a notification as read spec, comment
• [ ] features/desktop/oidc_implicit_flow.feature, Scenario: Application is authorized spec, example Step: I parse the bearer tokens and use it to access user info, Error: expected `nil.present?` ...
• [ ] features/desktop/post_preview.feature, Scenario: preview a post with location spec, example Step: Unable to find field "location_address"
• [ ] features/desktop/signs_up.feature, Scenario: User signs up with an already existing username and email and then tries to sign in (Issue #6136) spec, example Step: I fill in the new user form with an existing email and username, Error: Unable to find field ...
• [ ] features/mobile/getting_started.feature, Scenario: new user adds a profile photo and tags spec, example Step: I should see "..." within "...", Error: expected to find text ...
• [ ] features/mobile/reactions.feature, Scenario: like on a mobile post spec, example Step: I should see a "...", Error: expected to find css "a.like-action.active" ...

---

Broken cucumber steps:

• [ ] Step: I confirm the alert after ... (step) Error: Unable to find modal dialog
  • features/desktop/reshare.feature, Scenario: Resharing a post from a single post page spec, example
  • features/desktop/reshare.feature, Scenario: Resharing a post from a single post page that is reshared spec, example
  • features/desktop/single_post_view_moderation.feature, Scenario: block a contact spec, example
  • features/desktop/single_post_view_moderation.feature, Scenario: report a contact spec, example

---

Fixed:

RSpec

• [x] Notifications::Mentioned.notify calls filter_mentions on self spec, example

Cucumber

• [x] features/desktop/activity_stream.feature, Scenario: delete a comment spec, example (Fix: #7478)
• [x] features/desktop/blocks_user.feature, Scenario: Blocking a user from the profile page spec, example (Fix: #7379)
• [x] features/desktop/invitations.feature, Scenario: sends an invitation from the stream spec, ~example~, example Step: I should see "..." within "...", Error: Unable to find css ...
• [x] features/desktop/mentions_from_profile_page.feature, Scenario: mentioning while posting to all aspects spec. ~example~, example, example (Fix: #7491)
• [x] features/desktop/mentions_from_profile_page.feature, Scenario: mentioning while posting to just one aspect spec, ~example~, example (Fix: #7491)
• [x] features/mobile/reshare.feature, Scenario: Delete original reshared post spec, ~example~, example Step: I sign in as "..." on the mobile website, Error: expected to find css "#menu-badge" ...
• [x] Step: I submit the publisher (step) Error: expected "mention_popup publisher row" to include "closed"
  • features/desktop/signs_up.feature, Scenario: new user without any tags posts first status message spec, example
  • features/desktop/signs_up.feature, Scenario: new user with some tags posts first status message spec, example
  • features/desktop/single_post_view_moderation.feature, Scenario: hide a contact's post spec, example

[cmrd-senya]

I'll copy my comment from #7355 here because it is more relevant here:

Will it be useful for random cukes failure debugging if we make screenshot and save HTML when the exceptions are raised (expectations aren't met)? Beside making screenshots we'll also need to save this data after travis builds, which can be achieved by uploading generated files (screenshots, htmls) to S3. I think it can be useful because it's quite hard to reproduce this random failures locally. So getting some more information from faulty runs might be useful.

[Flaburgan]

About mentioning while posting to all aspects and mentioning while posting to just one aspect the problem mainly comes to the fact that we reload the page after the submission. Why do we do that? So a fix can be to remove this reload and directly display the success message. On the other hand, this popup always seemed weird to me. Why don't we redirect to the stream page with the publisher prefilled?

[SuperTux88]

"features/desktop/invitations.feature, Scenario: sends an invitation from the stream" failed again (see here), so I unchecked it again. (sorry @Flaburgan)

[SuperTux88]

I also found some new random failing test, and added them in the list above:

• features/desktop/activity_stream.feature, Scenario: delete a comment
• features/desktop/aspect_navigation.feature, Scenario: Aspects selection is remembered through site navigation
• features/desktop/reshare.feature, Scenario: Delete original reshared post
• features/desktop/single_post_view_moderation.feature, Scenario: block a contact
• features/desktop/single_post_view_moderation.feature, Scenario: report a contact

[Flaburgan]

It would be cool to have some metrics to know which ones fail the most, to be able to order them and pick first the most failing ones. Does travis allow something like that?

[SuperTux88]

No, travis doesn't have that, but you can simply look at the last few builds of develop/next-minor, and look what failed there. (and ignore that green develop-build, it was an accident ;) Or maybe your changes improved something? ;) )

[Flaburgan]

So looking at the pr build and the 3 last next-minor builds 1, 2 and 3 it looks like https://github.com/diaspora/diaspora/pull/7398 improved the situation, I check the mentioning while posting to all aspects and mentioning while posting to just one aspect checkboxes.

[SuperTux88]

So I unchecked the mentions_from_profile_page.feature again, because they are failing again (or still failing), but with a different error, so I added new examples.

I also added a new example for invitations.feature with the new error.

[SuperTux88]

I updated this issue by checking all failed travis runs from july. I linked to newer examples and also added a few new flaky tests. Most of the failed tests in july were activity_stream.feature and mentions_from_profile_page.feature, so thanks @Flaburgan and @fwoelm for fixing them. One other test that failed often was signs_up.feature (new user with or without any tags ...), which is not fixed yet.

I also restructured the issue description to group tests that fail because of the same reason.

[FinnWoelm]

I ran some more tests. I did a total of 10 travis CI tests, each with 4 builds (2x db, 2x ruby version) for a total of 40 complete cucumber tests.

Half were run with the current Capybara.max_wait_time of 30s. And the other half were run with a Capybara.max_wait_time of 300s. This was done through two separate PRs which you can check out in detail by clicking on their respective links.

I wanted to test whether the failures were due to Travis being slow and Capybara timing out. This does not seem to be the case. There was a roughly equal amount of failures in both.

Since it seems relevant to this thread, here are the overall failures from the 40 tests.

Summary

CI Tests

Successes: 1 (10%) Failures: 9 (90%)

Builds

Successes: 23 (58%) Failures: 17 (42%) > Ruby 2.3.1 & PSQL: 4 > Ruby 2.3.1 & MySQL: 5 > Ruby 2.1 & PSQL: 5 > Ruby 2.1 & MySQL: 3

Failed Features

Count: 21

features/desktop/signs_up.feature:56 (x4)

  expected "mention_popup publisher row" to include "closed" (RSpec::Expectations::ExpectationNotMetError)

  ./features/support/publishing_cuke_helpers.rb:31:in `submit_publisher'

  ./features/step_definitions/posts_steps.rb:85:in `/^I submit the publisher$/'

  features/desktop/signs_up.feature:66:in `And I submit the publisher'

features/desktop/comments.feature:45 (x3)

  Unable to find button "Comment" (Capybara::ElementNotFound)

  ./features/support/publishing_cuke_helpers.rb:116:in `make_comment'

  ./features/support/publishing_cuke_helpers.rb:110:in `block in comment_on_show_page'

  ./features/support/publishing_cuke_helpers.rb:109:in `comment_on_show_page'

  ./features/step_definitions/comment_steps.rb:15:in `/^I make a show page comment "([^"]*)"$/'

  features/desktop/comments.feature:50:in `When I make a show page comment "I think that’s a cat"'

• 3x features/mobile/getting_started.feature:19

   expected to find text "#men" in "" (RSpec::Expectations::ExpectationNotMetError)
  
  ./features/step_definitions/web_steps.rb:101:in `block (3 levels) in <top (required)>'
  
  ./features/step_definitions/web_steps.rb:13:in `block in with_scope'
  
  ./features/step_definitions/web_steps.rb:12:in `with_scope'
  
  ./features/step_definitions/web_steps.rb:100:in `block (2 levels) in <top (required)>'
  
  ./features/step_definitions/web_steps.rb:99:in `each'
  
  ./features/step_definitions/web_steps.rb:99:in `/^(?:|I )should see (\".+?\"[\s]*)(?:[\s]+within[\s]* "([^"]*)")?$/'
  
  features/mobile/getting_started.feature:26:in `Then I should see "#men" within "ul#as-selections-tags"'
  

• 2x features/desktop/aspect_navigation.feature:18 @ line 24

  expected to find css "li[data-aspect_id='1'] .selected" but there were no matches (RSpec::Expectations::ExpectationNotMetError)
  
  ./features/step_definitions/aspects_steps.rb:111:in `block (2 levels) in <top (required)>'
  
  ./features/step_definitions/aspects_steps.rb:110:in `/^I should see "([^"]*)" aspect selected$/'
  
  features/desktop/aspect_navigation.feature:24:in `Then I should see "Besties" aspect selected'
  

• 2x features/desktop/aspect_navigation.feature:18 @ line 25

   expected #has_no_css?("li[data-aspect_id='2'] .selected") to return true, got false (RSpec::Expectations::ExpectationNotMetError)
  
  ./features/step_definitions/aspects_steps.rb:118:in `block (2 levels) in <top (required)>'
  
  ./features/step_definitions/aspects_steps.rb:117:in `/^I should see "([^"]*)" aspect unselected$/'
  
  features/desktop/aspect_navigation.feature:25:in `And I should see "Unicorns" aspect unselected'
  

• 2x features/desktop/invitations.feature:53

  Unable to find css "#invitationsModalLabel" (Capybara::ElementNotFound)
  
  ./features/step_definitions/web_steps.rb:12:in `with_scope'
  
  ./features/step_definitions/web_steps.rb:100:in `block (2 levels) in <top (required)>'
  
  ./features/step_definitions/web_steps.rb:99:in `each'
  
  ./features/step_definitions/web_steps.rb:99:in `/^(?:|I )should see (\".+?\"[\s]*)(?:[\s]+within[\s]* "([^"]*)")?$/'      
  

• 1x features/desktop/notifications.feature:113 @ line 122

  expected to find css ".read .unread-toggle" but there were no matches (RSpec::Expectations::ExpectationNotMetError)
  
  ./features/step_definitions/custom_web_steps.rb:179:in `block (2 levels) in <top (required)>'
  
  ./features/step_definitions/web_steps.rb:16:in `with_scope'
  
  ./features/step_definitions/custom_web_steps.rb:178:in `/^(?:|I )should see a "([^\"]*)"(?: within "([^\"]*)")?$/'
  
  features/desktop/notifications.feature:122:in `Then I should see a ".read .unread-toggle"'
  

• 1x features/desktop/notifications.feature:113 @ line 122

  One or more errors were raised in the Javascript code on the page. If you don't care about these errors, you can ignore them by setting js_errors: false in your Poltergeist configuration (see documentation for details).
  
  TypeError: null is not an object (evaluating 'that.$element.off')
  
  TypeError: null is not an object (evaluating 'that.$element.off')
  
      at http://127.0.0.1:9887/assets/main.js:54993
  
      at http://127.0.0.1:9887/assets/main.js:54839 in complete
  
      at http://127.0.0.1:9887/assets/jquery3.js:4951 in fn
  
      at http://127.0.0.1:9887/assets/main.js:53330 in handle
  
      at http://127.0.0.1:9887/assets/jquery3.js:5202 in dispatch
  
      at http://127.0.0.1:9887/assets/jquery3.js:5010 in handle
  
      at http://127.0.0.1:9887/assets/jquery3.js:8172 in trigger
  
      at http://127.0.0.1:9887/assets/jquery3.js:8240
  
      at http://127.0.0.1:9887/assets/jquery3.js:369 in each
  
      at http://127.0.0.1:9887/assets/jquery3.js:158 in each
  
      at http://127.0.0.1:9887/assets/jquery3.js:8239 in trigger
  
      at http://127.0.0.1:9887/assets/main.js:53316 in callback (Capybara::Poltergeist::JavascriptError)
  
  ./features/step_definitions/custom_web_steps.rb:179:in `block (2 levels) in <top (required)>'
  
  ./features/step_definitions/web_steps.rb:16:in `with_scope'
  
  ./features/step_definitions/custom_web_steps.rb:178:in `/^(?:|I )should see a "([^\"]*)"(?: within "([^\"]*)")?$/'
  
  features/desktop/notifications.feature:122:in `Then I should see a ".read .unread-toggle"'
  

• 1x features/desktop/signs_up.feature:47

  expected "mention_popup publisher row" to include "closed" (RSpec::Expectations::ExpectationNotMetError)
  
  ./features/support/publishing_cuke_helpers.rb:31:in `submit_publisher'
  
  ./features/step_definitions/posts_steps.rb:85:in `/^I submit the publisher$/'
  
  features/desktop/signs_up.feature:53:in `And I submit the publisher'
  

• 1x features/desktop/signs_up.feature:11

  Unable to find modal dialog (Capybara::ModalNotFound)
  
  ./features/step_definitions/custom_web_steps.rb:140:in `/^I confirm the alert after (.*)$/'
  
  features/desktop/signs_up.feature:14:in `And I confirm the alert after I follow "awesome_button"'
  

• 1x features/desktop/reshare.feature:25

  Unable to find modal dialog (Capybara::ModalNotFound)
  
  ./features/step_definitions/custom_web_steps.rb:140:in `/^I confirm the alert after (.*)$/'
  
  features/desktop/reshare.feature:30:in `And I confirm the alert after I click on selector "a.reshare"'
  

[FinnWoelm]

@cmrd-senya Taking screenshots on every failure would be super useful!

In the meantime, a hacky workaround to get a picture 'on demand' is to use the following line in whatever step you want to get a screenshot:

# :full => true takes a fullscreen picture
# If you leave out :full, it will give you just the current viewport
puts page.driver.render_base64(:png, :full => true)

This will print the base64 encoded image to the Travis log which you can then copy and paste into a base64 image decoder, such as http://base64online.org/decode/

This functionality is provided by poltergeist: https://github.com/teampoltergeist/poltergeist#taking-screenshots-with-some-extensions

[SuperTux88]

@fwoelm thanks for your work here. I added features/desktop/notifications.feature:113 to the list, because it was new in your tests.

And now lets see what random fails occur again after merging #7497, but at least invitations.feature:53, aspect_navigation.feature:18 and the Unable to find modal dialog are still there.

[SuperTux88]

I think Scenario: sends an invitation from the stream is the next most annoying randomly failing test (at least I see that very often).

[weex]

Here's some bash to use github's CLI to get the most common recent failures in the CI workflow:

gh run list -w CI -R diaspora/diaspora | grep completed | grep failure | cut -f7 | while read -r line ; do
    gh run view $line --log-failed | grep Failing -A 1 | cut -f3 | cut -d' ' -f2-
done