Commit Graph

80 Commits

Author SHA1 Message Date
Alan Guo Xiang Tan 433cb7092d
DEV: Use documentation format for core specs on CI (#22531)
The documentation format makes it easier to link a failing test to the
process it was launched in.
2023-07-11 10:12:51 +08:00
Alan Guo Xiang Tan 7b40ec859f
DEV: Reduce parallel test processors for system tests to 4 (#22456)
Why this change?

This is abit of a trial and error but we're starting to see selenium
session not created errors on CI. One of the reason for this is that the
system has run out of resources to create a new tab.

This commit reduces the number of parallel test processors in an attempt
to increase the amount of resources available to each test process and
hopefully lead to more stable CI system tests.
2023-07-06 10:34:44 +08:00
Alan Guo Xiang Tan b764c53cc0
Revert "DEV: Reduce number of parallel test processors to 3 for system tests (#22423)" (#22453)
This reverts commit 865f7a9852.

The flakiness that we have been seeing and fixing on CI were not related
to system resource problems. Therefore, we can bump this up back to 5.
2023-07-06 08:47:43 +08:00
Alan Guo Xiang Tan 858cc6aff2
DEV: Log server errors encountered when running system tests on CI (#22427)
Why is this change required?

We've been seeing flaky tests due to server errors on CI but are unable
to debug it because we do not log any of the errors. This change gives
us a fighting chance the next time we encounter a server error during
system test runs.

See
https://github.com/discourse/discourse/actions/runs/5459248864/jobs/9935049920?pr=22424
for an example of server errors encountered during system tests.
2023-07-05 09:46:48 +08:00
Alan Guo Xiang Tan 865f7a9852
DEV: Reduce number of parallel test processors to 3 for system tests (#22423)
This is an experiment to see if not saturating resources on a runner
with 8 vCPU leads to more stability on our CI.
2023-07-05 08:08:46 +08:00
David Taylor 251d6f0627
DEV: Introduce Zeitwerk reloading check in CI (#22151)
This should help us to catch the most catastrophic cases where plugins fail to auto-reload in development mode.
2023-06-16 14:33:14 +01:00
Alan Guo Xiang Tan 5897709a90
DEV: Use runtime info to split test files for parallel testing (#22060)
Using the runtime information, we will be able to more efficiently group
the test files across the test processes hence leading to better
utilization of resources.
2023-06-12 09:07:17 +08:00
Daniel Waterworth 67afd85aae
Revert "DEV: Use runtime info to split test files for parallel testing (#21896)" (#22016)
This reverts commit 14ed971db6.

This prevented the core backend tests from running in GitHub CI
2023-06-08 15:13:26 -05:00
Alan Guo Xiang Tan 14ed971db6
DEV: Use runtime info to split test files for parallel testing (#21896)
Using the runtime information, we will be able to more efficiently group
the test files across the test processes hence leading to better
utilization of resources.
2023-06-05 08:01:41 +08:00
Alan Guo Xiang Tan ba251dec6b
DEV: Increase number of parallel process for system test to 5 on CI (#21878)
4dd053a69c addressed most of the
instability we were seeing with system tests on CI and locally. Let's
try pushing the number of parallel processes up to squeeze as much time
savings as possible from the runner.
2023-06-01 10:41:11 +08:00
Alan Guo Xiang Tan 2c1c7d1e4f DEV: Use `--profile` when running system tests in CI 2023-05-30 13:46:14 +09:00
Alan Guo Xiang Tan 123a77a2bc
DEV: Correct Capybara default max wait time setting in CI (#21801)
We were not setting capybara's default max wait time correctly in CI due
to a spelling error.

This regressd in fc17045876
2023-05-29 10:41:24 +08:00
Alan Guo Xiang Tan 038318bffc
DEV: Bump CAPYBARA_DEFAULT_MAX_WAIT_TIME to 10 seconds on CI (#21711)
We're running on pretty crappy hardware on Github's CI and this has an
impact on the stability of our system tests on CI. Therefore, we are
bumping `CABPYARA_DEFAULT_MAX_WAIT_TIME` to 10 seconds to account for
the less than ideal hardware we're running the system tests on.

This change trades off speed for stability but speed is already bad on
CI so stability is more important for our case.
2023-05-25 09:25:08 +08:00
Jarek Radosz fc17045876
DEV: Clean up workflow files (#21526) 2023-05-12 14:00:04 +02:00
Jarek Radosz 19ac90536f
DEV: Restore the documentation format in system tests (#21471) 2023-05-12 11:13:52 +02:00
David Taylor d6f94e0916
DEV: Revert buildjet (#21475)
This reverts commits 17603794b5 and 41bdf8cfcd.
2023-05-11 22:25:30 +02:00
Natalie Tay 17603794b5
DEV: Reduce plugin system test parallel processors (#21466) 2023-05-10 15:43:41 +08:00
Natalie Tay 41bdf8cfcd
DEV: Use BuildJet for some test jobs (#21404) 2023-05-10 10:39:27 +08:00
Alan Guo Xiang Tan 1f6d57ab93
DEV: Run turbo rspecs with verbose output on CI (#21379)
In CI, we the output to be as verbose as possible so that when it fails we have the necessary information to debug the failures.
2023-05-04 10:34:02 +08:00
David Taylor c6f5b4297d
DEV: Use `--frozen-lockfile` in GitHub CI (#21338)
This will help us to catch missing lockfile changes before they are merged into `main`
2023-05-02 16:25:22 +01:00
Joffrey JAFFEUX e495a2fc3f
DEV: Enable parallel system specs in GitHub actions CI (#21251)
Also skips/improves few flakey specs
2023-04-26 13:02:19 +02:00
Jarek Radosz 00630e4c74
DEV: Remove `RUBY_GLOBAL_METHOD_CACHE_SIZE` (#21249)
It doesn't do anything since ruby 3.0.0.preview1. It was removed in https://github.com/ruby/ruby/pull/2888
2023-04-26 10:39:39 +02:00
Joffrey JAFFEUX d4c6457065
DEV: increase plugin system tests timeout (#21247) 2023-04-26 10:08:10 +02:00
Alan Guo Xiang Tan 3cb9fd739a
DEV: Run system tests with documentation format on github actions (#21069)
Allows us to see the tests which have timed out
2023-04-12 14:47:05 +08:00
Daniel Waterworth 52999d1422
DEV: Show which system tests are being executed in CI (#20795)
... to see if it's always the same test causing timeouts
2023-03-23 15:12:14 -05:00
David Taylor 150a6601c0
DEV: Check Zeitwerk eager loading in GitHub CI (#20699)
In production, `eager_load=true`. This sometimes leads to boot errors which are not present in dev/test environments. Running `zeitwerk:check` in CI will help us to pick up on any errors early.

This commit also introduces a `DISCOURSE_ZEITWERK_EAGER_LOAD` environment variable to make it easier to toggle the behaviour when developing locally.
2023-03-16 14:22:16 +00:00
Daniel Waterworth 5324216740 DEV: Use rspec instead of turbo_rspec with one core 2023-03-15 16:46:48 -05:00
Rafael dos Santos Silva c3ae555ac2
DEV: Show correct version of Ruby we are using on CI (#20415) 2023-02-22 15:39:49 -03:00
David Taylor a6b680f4fe
DEV: Fix GitHub CI permissions issues (#20069)
The `git` version in our discourse_test docker image was recently updated to include a permissions check before running any git commands. For this to pass, the owner of the discourse directory needs to match the user running any git commands.

Under GitHub actions, by default the working directory is created with uid=1000 as the owner. We run all our tests as `root`, so this mismatch causes git to raise the permissions error. We can't switch to run the entire workflow as the `discourse (uid=1000)` user because our discourse_test image is not configured to allow `discourse` access to postgres/redis directories. For now, this commit updates the working directory's owner to match the user running the workflow.
2023-01-30 15:39:43 +00:00
David Taylor b96869d5fb
DEV: Disable parallel system specs in GitHub actions (#20023)
We have some flakiness which needs to be resolved. Followup to e717529d80
2023-01-26 14:37:47 +00:00
David Taylor e717529d80
DEV: Enable parallel system specs in GitHub actions CI (#19584) 2023-01-26 13:26:02 +00:00
Rafael dos Santos Silva 60ebbfd7e7
DEV: Stop testing with Ruby 3.2 for now (#19909) 2023-01-18 12:04:49 -03:00
Rafael dos Santos Silva 076b3a6514
DEV: Key bundler CI cache on Ruby version (#19868) 2023-01-13 11:39:49 -03:00
Rafael dos Santos Silva 8e7e6e14c7
DEV: Add Ruby 3.2 to test matrix (#19862)
* DEV: Add Ruby 3.2 to test matrix

* DEV: Update test name
2023-01-13 09:22:33 -03:00
Alan Guo Xiang Tan e58277adf3
DEV: Increase Capybara.default_max_wait_time on github actions (#19750)
Our working theory is that system tests on Github run on much less
powerful hardware as compared to running the tests on our work machines.
Hopefully, increasing the wait time now will help reduce some flakes
that we're seeing on Github.
2023-01-05 08:50:35 +08:00
David Taylor 335893ae91
DEV: Correct private-fork -> private-mirror (#19560)
Followup to 7eb9482ba9
2022-12-21 16:52:35 +00:00
David Taylor 7eb9482ba9
DEV: Skip 'push' workflow events for discourse-private-mirror (#19556)
We don't want 'push' workflows to run on this private fork (which is used for developing security-fixes before public disclosure)
2022-12-21 16:44:38 +00:00
Martin Brennan 8b3c6cd396
DEV: Fix github workflow system spec screenshot location (#19435)
These screenshots are located at paths like:

/__w/discourse/discourse/tmp/capybara/failures_r_spec_example_groups_quoting_chat_message_transcripts_copying_quote_transcripts_with_the_clipboard_quotes_multiple_chat_messages_into_a_topic_134.png

not /tmp/screenshots. This should fix the issue. Also makes plugin system specs
use documentation format and profile.
2022-12-13 15:36:30 +10:00
Alan Guo Xiang Tan 5103268312
DEV: Run system tests with documentation and profiling on actions (#19271)
When a test takes too long, we want to know which test and at what step
2022-12-01 05:54:17 +08:00
David Taylor f8939bd294
DEV: Bump `@actions/checkout` to v3 in frontend tests (#18989)
v2 uses Node 12, which is deprecated
2022-11-11 13:31:28 +00:00
David Taylor 70a990da03
DEV: Update GitHub actions set-output uses (#18988)
https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-11-11 13:12:08 +00:00
Joffrey JAFFEUX 11f3618b80
DEV: initial system tests for chat and plugins (#18881)
This is a very basic to ensure it's working and open future possible work
2022-11-04 15:06:24 +01:00
Martin Brennan 57caf08e13
DEV: Minimal first pass of rails system test setup (#16311)
This commit introduces rails system tests run with chromedriver, selenium,
and headless chrome to our testing toolbox.

We use the `webdrivers` gem and `selenium-webdriver` which is what
the latest Rails uses so the tests run locally and in CI out of the box.

You can use `SELENIUM_VERBOSE_DRIVER_LOGS=1` to show extra
verbose logs of what selenium is doing to communicate with the system
tests.

By default JS logs are verbose so errors from JS are shown when
running system tests, you can disable this with
`SELENIUM_DISABLE_VERBOSE_JS_LOGS=1`

You can use `SELENIUM_HEADLESS=0` to run the system
tests inside a chrome browser instead of headless, which can be useful to debug things
and see what the spec sees. See note above about `bin/ember-cli` to avoid
surprises.

I have modified `bin/turbo_rspec` to exclude `spec/system` by default,
support for parallel system specs is a little shaky right now and we don't
want them slowing down the turbo by default either.

### PageObjects and System Tests

To make querying and inspecting parts of the page easier
and more reusable inbetween system tests, we are using the
concept of [PageObjects](https://www.selenium.dev/documentation/test_practices/encouraged/page_object_models/) in
our system tests. A "Page" here is generally corresponds to
an overarching ember route, e.g. "Topic" for `/t/324345/some-topic`,
and this contains logic for querying components within the topic
such as "Posts".

I have also split "Modals" into their own entity. Further down the
line we may want to explore creating independent "Component"
contexts.

Capybara DSL should be included in each PageObject class,
reference for this can be found at https://rubydoc.info/github/teamcapybara/capybara/master#the-dsl

For system tests, since they are so slow, we want to focus on
the "happy path" and not do every different possible context
and branch check using them. They are meant to be overarching
tests that check a number of things are correct using the full stack
from JS and ember to rails to ruby and then the database.

### CI Setup

Whenever a system spec fails, a screenshot
is taken and a build artifact is produced _after the entire CI run is complete_,
which can be downloaded from the Actions UI in the repo.

Most importantly, a step to build the Ember app using Ember CLI
is needed, otherwise the JS assets cannot be found by capybara:

```
- name: Build Ember CLI
  run: bin/ember-cli --build
```

A new `--build` argument has been added to `bin/ember-cli` for this
case, which is not needed locally if you already have the discourse
rails server running via `bin/ember-cli -u` since the whole server is built and
set up by default.

Co-authored-by: David Taylor <david@taylorhq.com>
2022-09-28 11:48:16 +10:00
David Taylor 885e133cac
DEV: Cache `turbo_rspec_runtime.log` in CI (#18318) 2022-09-21 22:13:25 +01:00
David Taylor a11aea4fe2
DEV: Update github workflow for 8-core workers (#18271) 2022-09-21 18:13:13 +01:00
David Taylor b0a9d8b761
DEV: Improve labelling of Firefox Evergreen/ESR CI runs (#18313)
Both versions are used with `--headless`, so labelling one "Firefox" and the other "Firefox Headless" doesn't really make sense. Evergreen / ESR are better descriptions.
2022-09-21 15:34:26 +01:00
David Taylor 42d226f727
DEV: Ensure GitHub workflows cancel cleanly (#18314)
We added `always()` on some steps so that they run even if previous steps fail. That helps give us a picture of all failures in one run, rather than having to re-run the workflow after fixing the first failure.

However, when we explicitly cancel a job, we should skip running these steps. `!cancelled()` is a better substitute for `always()` in this case.
2022-09-21 14:32:21 +01:00
David Taylor e06b9d4a52
DEV: Remove support for legacy plugin JS compilation pipeline (#18293)
This became the default in b1755137
2022-09-21 12:38:02 +01:00
David Taylor ef39193a06
DEV: Add `rake plugins:turbo_spec` task (#18289)
This leans on our existing `turbo_rspec` implementation to run plugin specs in parallel on all available cores
2022-09-20 15:42:54 +01:00
David Taylor 33a2624f09
DEV: Introduce flag for compiling Plugin JS with Ember CLI (#17965)
When `EMBER_CLI_PLUGIN_ASSETS=1`, plugin application JS will be compiled via Ember CLI. In this mode, the existing `register_asset` API will cause any registered JS files to be made available in `/plugins/{plugin-name}_extra.js`. These 'extra' files will be loaded immediately after the plugin app JS file, so this should not affect functionality.

Plugin compilation in Ember CLI is implemented as an addon, similar to the existing 'admin' addon. We bypass the normal Ember CLI compilation process (which would add the JS to the main app bundle), and reroute the addon Broccoli tree into a separate JS file per-plugin. Previously, Sprockets would add compiled templates directly to `Ember.TEMPLATES`. Under Ember CLI, they are compiled into es6 modules. Some new logic in `discourse-boot.js` takes care of remapping the new module names into the old-style `Ember.TEMPLATES`.

This change has been designed to be a like-for-like replacement of the old plugin compilation system, so we do not expect any breakage. Even so, the environment variable flag will allow us to test this in a range of environments before enabling it by default.

A manual silence implementation is added for the build-time `ember-glimmer.link-to.positional-arguments` deprecation while we work on a better story for plugins.
2022-08-22 09:56:39 +01:00