Flaky tests

原文：https://docs.gitlab.com/ee/development/testing_guide/flaky_tests.html

• What’s a flaky test?

• Quarantined tests

  • Quarantine tests on the CI

• Automatic retries and flaky tests detection

• Problems we had in the past at GitLab
  • Time-sensitive flaky tests
  • Array order expectation
  • Feature tests
    • Capybara viewport size related issues
    • Capybara JS driver related issues
    • PhantomJS / WebKit related issues
    • Capybara expectation times out

• Resources

Flaky tests

What’s a flaky test?

这个测试有时会失败，但是如果您重试了多次，它最终会通过.

Quarantined tests

当master的测试频繁失败时，应创建〜" master：broken"问题 . 如果无法及时修复测试，则会影响所有开发人员的工作效率，因此应通过分配:quarantine元数据将其放入隔离:quarantine .

这意味着除非使用--tag quarantine运行，否则它将被跳过：

bin/rspec --tag quarantine 

在对隔离进行测试之前，应确保存在〜" master：broken"问题，这样它才不会永远留在隔离区.

Once a test is in quarantine, there are 3 choices:

• 测试应该固定（即摆脱其脆弱性）吗？
• 是否应将测试移至较低级别的测试？
• 是否应该完全删除该测试（例如，因为已经有一个较低级别的测试，或者它正在复制另一个相同级别的测试，或者它测试的太多等等）？

Quarantine tests on the CI

隔离测试在允许失败的专用作业中在 CI 上运行：

• rspec-pg-quarantine （CE 和 EE）
• rspec-pg-quarantine-ee （仅适用于 EE）

Automatic retries and flaky tests detection

在 CI 上，我们使用RSpec :: Retry自动重试失败的示例几次（有关准确的重试次数，请参阅spec/spec_helper.rb ）.

我们还使用自制的RspecFlaky::Listener侦听RspecFlaky::Listener ，该RspecFlaky::Listener将RspecFlaky::Listener示例记录在master上的 JSON 报告文件中（ retrieve-tests-metadata和update-tests-metadata作业）.

这最初是在以下网址中实现的： https : //gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13021 .

如果要在本地启用重试，则可以使用RETRIES环境变量. 例如RETRIES=1 bin/rspec ...将重试失败的示例一次.

Problems we had in the past at GitLab

• rspec-retry is biting us when some API specs fail: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9825

• Sporadic RSpec failures due to PG::UniqueViolation: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9846

  • Follow-up: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10688
  • Capybara.reset_session! should be called before requests are blocked: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12224

• FFaker 生成测试尚未准备就绪的时髦数据（测试应该是可预测的，所以很糟糕！）：

  • 使spec/mailers/notify_spec.rb更加健壮 ： https : spec/mailers/notify_spec.rb
  • spec/requests/api/commits_spec.rb暂时失败 ： https : spec/requests/api/commits_spec.rb
  • 用序列替换 FFaker 工厂数据 ： https : //gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10184
  • spec / finders / issues_finder_spec.rb 中的暂时失败 ： https : //gitlab.com/gitlab-org/gitlab-foss/-//merge_requests/10404

Time-sensitive flaky tests

• https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10046
• https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10306

Array order expectation

• https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10148

Feature tests

• Be sure to create all the data the test need before starting exercise: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12059
• Bis: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12604
• Bis: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12664
• Assert against the underlying database state instead of against a page’s content: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10934

• 在 JS 测试中，当元素在 Capybara 发送点击的确切时间移动时，移动元素会导致 Capybara 误点击

  • 由于窗口大小和滚动位置，下拉菜单向上或向下渲染
  • 延迟加载的图像可能导致水豚误点击

• Triggering JS events before the event handlers are set up

• Wait for the image to be lazy-loaded when asserting on a Markdown image’s src attribute

Capybara viewport size related issues

• Transient failure of spec/features/issues/filtered_search/filter_issues_spec.rb: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10411

Capybara JS driver related issues

• Don’t wait for AJAX when no AJAX request is fired: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10454
• Bis: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12626

PhantomJS / WebKit related issues

• 记忆在屋顶！ （TL; DR：加载图像但阻止图像请求！）： https : //gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12003

Capybara expectation times out

• Test imports a project (via Sidekiq) that is growing over time, leading to timeouts when the import takes longer than 60 seconds

Resources

• Flaky Tests: Are You Sure You Want to Rerun Them?
• How to Deal With and Eliminate Flaky Tests
• Tips on Treating Flakiness in your Rails Test Suite
• ‘Flaky’ tests: a short story
• Using Insights to Discover Flaky, Slow, and Failed Tests

Return to Testing documentation