Using the runtime information, we will be able to more efficiently group
the test files across the test processes hence leading to better
utilization of resources.
Why is this change required?
By default, `RSpec` comes with a `--profile=[COUNT]` option as well but
enabling that option means that the entire test suite needs to be
executed. This does not work so well for `turbo_rspec` which splits our
test files into various "buckets" for the tests to be executed in
multiple processes. Therefore, this commit adds a similar
`--profile=[COUNT]` option to `turbo_rspec` but will only profile the
tests being executed. Examples:
`LOAD_PLUGINS=1 bin/turbo_rspec --profile plugins/*/spec/system`
or
`LOAD_PLUGINS=1 bin/turbo_rspec --profile=20 plugins/*/spec/system`
This commit 57caf08e13 broke
`bin/turbo_rspec` timing recording via `TurboTests::Runner`,
because we changed to using all `spec/*` folders except
`spec/system` as default for the runner, rather than
the old `['spec']` array, which is what `TurboTests::Runner`
was relying on to determine whether to record test run
time with `ParallelTests::RSpec::RuntimeLogger`.
Instead, we can just pass a new `use_runtime_info` boolean to the
runner class and use it when running against the default set of
spec files using `bin/turbo_rspec` and the turbo rspec rake task.
This commit introduces rails system tests run with chromedriver, selenium,
and headless chrome to our testing toolbox.
We use the `webdrivers` gem and `selenium-webdriver` which is what
the latest Rails uses so the tests run locally and in CI out of the box.
You can use `SELENIUM_VERBOSE_DRIVER_LOGS=1` to show extra
verbose logs of what selenium is doing to communicate with the system
tests.
By default JS logs are verbose so errors from JS are shown when
running system tests, you can disable this with
`SELENIUM_DISABLE_VERBOSE_JS_LOGS=1`
You can use `SELENIUM_HEADLESS=0` to run the system
tests inside a chrome browser instead of headless, which can be useful to debug things
and see what the spec sees. See note above about `bin/ember-cli` to avoid
surprises.
I have modified `bin/turbo_rspec` to exclude `spec/system` by default,
support for parallel system specs is a little shaky right now and we don't
want them slowing down the turbo by default either.
### PageObjects and System Tests
To make querying and inspecting parts of the page easier
and more reusable inbetween system tests, we are using the
concept of [PageObjects](https://www.selenium.dev/documentation/test_practices/encouraged/page_object_models/) in
our system tests. A "Page" here is generally corresponds to
an overarching ember route, e.g. "Topic" for `/t/324345/some-topic`,
and this contains logic for querying components within the topic
such as "Posts".
I have also split "Modals" into their own entity. Further down the
line we may want to explore creating independent "Component"
contexts.
Capybara DSL should be included in each PageObject class,
reference for this can be found at https://rubydoc.info/github/teamcapybara/capybara/master#the-dsl
For system tests, since they are so slow, we want to focus on
the "happy path" and not do every different possible context
and branch check using them. They are meant to be overarching
tests that check a number of things are correct using the full stack
from JS and ember to rails to ruby and then the database.
### CI Setup
Whenever a system spec fails, a screenshot
is taken and a build artifact is produced _after the entire CI run is complete_,
which can be downloaded from the Actions UI in the repo.
Most importantly, a step to build the Ember app using Ember CLI
is needed, otherwise the JS assets cannot be found by capybara:
```
- name: Build Ember CLI
run: bin/ember-cli --build
```
A new `--build` argument has been added to `bin/ember-cli` for this
case, which is not needed locally if you already have the discourse
rails server running via `bin/ember-cli -u` since the whole server is built and
set up by default.
Co-authored-by: David Taylor <david@taylorhq.com>
Also:
* Remove an unused method (#fill_email)
* Replace a method that was used just once (#generate_username) with `SecureRandom.alphanumeric`
* Remove an obsolete dev puma `tmp/restart` file logic
* File.exists? is deprecated and removed in Ruby 3.2 in favor of
File.exist?
* Dir.exists? is deprecated and removed in Ruby 3.2 in favor of
Dir.exist?
On new macs, `localhost` resolves to IPV6 of `::1` and unfortunately
unicorn doesn't bind to IPv6 by default.
This seems to be the path of least resistance. By using 127.0.0.1 we
force IPv4 which works great.
Trying to use a local test hostname other than localhost
(e.g. discourse.test )for discourse development was difficult due
the fact that localhost was hardcoded in a few places. This patch
uses existing environment variables to allow a developer to use a
different domain when developing.
Signed-off-by: Ryan Lerch <rlerch@redhat.com>
The `-depth` flag is incorrect on Linux, it does not take an argument
and causes an error and results in no plugins ever being found.
Copied from `man find`:
```
The global options occur after the list of start points, and so are not the same kind of option as -L, for example.
-d A synonym for -depth, for compatibility with FreeBSD, NetBSD, MacOS X and OpenBSD.
-depth Process each directory's contents before the directory itself. The -delete action also implies -depth.
...
-maxdepth levels
Descend at most levels (a non-negative integer) levels of directories below the starting-points. Using -maxdepth 0 means
only apply the tests and actions to the starting-points themselves.
```
Previously we would need to launch unicorn separately this achieves
the same goal by making 2 modifications:
1. If -u is supplied ember-cli binary will launch and monitor ember cli and unicorn
2. We suppress 200 requests to keep console clean (we may consider moving to development rails logs)
Also cleans out output a bit by supplying silent flags to yarn.
Running a development environment using Docker's qemu architecture emulation is currently not possible because `inotify` is not supported: https://github.com/docker/for-mac/issues/5321
- Add `d/ember-cli`, and publish port 4200
- Remove `d/sidekiq`. Sidekiq is now started with the rails server
- Move all `docker exec` logic into a single place, so we have one place to set environment variable pass-throughs
- Use `exec` for all bash scripts, so that return statuses are passed back correctly
- Avoid using `bin/bash -c` unnecessarily, because it makes escaping arguments difficult
Sometimes plugins directories will end up with other symlinks (e.g. inside node_modules folders). This logic does not work with deeply nested symlinks, and they are unlikely to be necessary for the plugin to work. Therefore we should only look for symlinks in the top-level of the `plugins` directory
Some people have noticed that if we change the packages in package.json
that they have to manually run `yarn install` or Discourse won't work.
This adds `yarn install` to the `bin/ember-cli` helper we run. It seems
quite fast if there is nothing to install so it shouldn't hurt to do
this every time we start the server.
This fixes the following error I've been seeing lately in RubyMine:
> Error:Your `bin/bundle` was not generated by Bundler, so this binstub cannot run.
> Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.
The auto restart logic was sending a USR2 to the parent process without checking what the parent process actually was. In some situations, it might not be the `bin/unicorn` supervisor.
This commit switches to use a global variable for the supervisor PID. This will be much less prone to unexpected behavior.
(inspired by martin-brennan's recent adventure with `tags` exclusion)
**tl;dr: some of it is obsolete, some is dev-env specific and should live in *your* ~/.gitignore.**
---
To enable global gitignore (`~/.gitignore`) run:
```bash
git config --global core.excludesFile '~/.gitignore'
```
Then create that file and add your VIM/VSCode/Sublime/Emacs/Eclipse/RubyMine/JetBrains/macOS/Arch/Windows 95-specific stuff.
---
Reasons for removal:
* `bin` - never really ignored, all the files are checked in
* `.DS_Store`, `._.DS_Store` - OS specific
* `.sass-cache/*`, `/.bundle`, `/bundle/*`, `/cache`, `/logfile`, `!/plugins/discourse-nginx-performance-report`, `MiniProfiler/Ruby/rack-mini-profiler-2.0.1a.gem`, `config/fog_credentials.yml`, `/public/stylesheet-cache/*`, `script/download_db`, `script/refresh_db`, `config/version.rb` - no longer used?
* `/log/*.log` - covered by /log
* `bootsnap-load-path-cache`, `bootsnap-compile-cache` - bootsnap now keeps its ~~trash~~ temporary files in `/tmp`
* `/.project`, `/.buildpath`, `/.byebug_history`, `/.idea`, `discourse.sublime-workspace`, `*~`, `*.swp`, `*.swo`, `*.swm`, `config/multisite1.yml`, `.rvmrc`, `.ruby-version`, `.ruby-gemset`, `.rbenv`, `bundler_stubs/*`, `*.db`, `*.iml`, `*.swn`, `/package-lock.json`, `.vscode`, `.envrc`, `tags` - dev-env specific, see the note at the top of the commit
Using `bundle exec` will slow down server startup by at least 0.5s. `bin/unicorn` has built-in handling of bundler dependencies, so it is better to launch `bin/rails s` or `bin/unicorn` directly.
In Discourse, `rails s` ultimately launches the `bin/unicorn` script. However, the overhead of `rails` launching `bin/rails`, and then in turn `bin/unicorn` can be non-trivial. Therefore it is much better to run `bin/unicorn` directly.
This commit prints a warning message to STDERR when `rails s` is used. Functionality is unchanged.
We really want to encourage all developers to use Ember CLI for local
development and testing. This will display an error page if they are not
with instructions on how to start the local server.
To disable it, you can set `NO_EMBER_CLI=1` as an ENV variable
This little helper script allows for easy ember cli development.
To see the options run `bin/ember-cli -h`
It allows you to proxy try.discourse.org with the `bin/ember-cli --try`
switch, which effectively allows for some development without a rails installed.
It passes on arguments to ember-cli so you can customize port and so on.
It makes the assumption that on local people are using `bin/unicorn` for
development. (it includes some extra discourse specific helpers)
We were having issues in development mode where the JS code had errors due to a bad cache. When starting a server in development mode in bin/unicorn we now get the git sha of the discourse HEAD and get a git sha of all plugins, and store them in a file. If the sha has changed then we delete tmp/cache to refresh the assets cache.
Alan Guo Xiang Tan
Jarek Radosz
Daniel Waterworth
dsims
Martin Brennan
Loïc Guitaut
David Taylor
Gerhard Schlager
Peter Zhu
Robin Ward
Ryan Lerch
Arpit Jalan
Jacob Mischka
Penar Musaraj
Sam
Jeff Wong
Alessandro Bertacco
Kane York
Martin Brennan