DEV: Remove flaky tests report when there are too many failures (#25031)

Why this change?

Currently we only rerun failing tests to check if they are flaky tests
when there are 10 or less failing tests. When there are more than 10
failing tests in the first run, we assume that the odds of those tests
being flaky are low and do not rerun the tests. However, there was a bug
where we do not clean up the potential flaky tests being logged when
there are too many test failures. This resulted in those test failures
being treated as flaky tests.

What does this change do?

Clean up the flaky tests report when we do not rerun the tests.
This commit is contained in:
Alan Guo Xiang Tan 2023-12-26 09:23:17 +08:00 committed by GitHub
parent d03f6727b1
commit cfec408bc1
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 5 additions and 0 deletions

View File

@ -11,6 +11,10 @@ module TurboTests
.map { |failed_example| failed_example["location_rerun_argument"] }
end
def self.remove_flaky_tests
File.delete(PATH) if File.exist?(PATH)
end
# This method should only be called by a formatter registered with `TurboTests::Runner` and logs the failed examples
# to `PATH`. See `FailedExample#to_h` for the details of each example that is logged.
#

View File

@ -103,6 +103,7 @@ module TurboTests
return rerun_failed_examples(@reporter.failed_examples)
else
STDOUT.puts "Retry and log flaky tests was enabled but ignored because there are more than #{retry_failed_examples_threshold} failures."
Flaky::Manager.remove_flaky_tests
end
end