Commit Graph

114 Commits

Author SHA1 Message Date
Alan Guo Xiang Tan 5897709a90
DEV: Use runtime info to split test files for parallel testing (#22060)
Using the runtime information, we will be able to more efficiently group
the test files across the test processes hence leading to better
utilization of resources.
2023-06-12 09:07:17 +08:00
Daniel Waterworth 67afd85aae
Revert "DEV: Use runtime info to split test files for parallel testing (#21896)" (#22016)
This reverts commit 14ed971db6.

This prevented the core backend tests from running in GitHub CI
2023-06-08 15:13:26 -05:00
Alan Guo Xiang Tan 14ed971db6
DEV: Use runtime info to split test files for parallel testing (#21896)
Using the runtime information, we will be able to more efficiently group
the test files across the test processes hence leading to better
utilization of resources.
2023-06-05 08:01:41 +08:00
Alan Guo Xiang Tan b00edf3ea0 DEV: Add `--profile=[COUNT]` option for `turbo_rspec`
Why is this change required?

By default, `RSpec` comes with a `--profile=[COUNT]` option as well but
enabling that option means that the entire test suite needs to be
executed. This does not work so well for `turbo_rspec` which splits our
test files into various "buckets" for the tests to be executed in
multiple processes. Therefore, this commit adds a similar
`--profile=[COUNT]` option to `turbo_rspec` but will only profile the
tests being executed. Examples:

`LOAD_PLUGINS=1 bin/turbo_rspec --profile plugins/*/spec/system`

or

`LOAD_PLUGINS=1 bin/turbo_rspec --profile=20 plugins/*/spec/system`
2023-05-30 13:46:14 +09:00
Jarek Radosz bf8939f7ad
DEV: add `--seed` to turbo_rspec, tweak CI output (#21598) 2023-05-17 11:22:31 +02:00
Daniel Waterworth 852475dca4
DEV: Add yarn-app shim to run yarn using the app modules (#20983) 2023-04-05 13:12:58 -05:00
dsims 6415bab455
DEV: Run yarn install before db migrations in `d/boot_dev` (#20866)
Yarn dependencies need to be present before booting Rails
2023-04-05 18:49:50 +01:00
Martin Brennan 4ab1f76499
DEV: Fix bin/turbo_rspec runtime recording (#20407)
This commit 57caf08e13 broke
`bin/turbo_rspec` timing recording via `TurboTests::Runner`,
because we changed to using all `spec/*` folders except
`spec/system` as default for the runner, rather than
the old `['spec']` array, which is what `TurboTests::Runner`
was relying on to determine whether to record test run
time with `ParallelTests::RSpec::RuntimeLogger`.

Instead, we can just pass a new `use_runtime_info` boolean to the
runner class and use it when running against the default set of
spec files using `bin/turbo_rspec` and the turbo rspec rake task.
2023-02-23 07:47:11 +10:00
Loïc Guitaut f7c57fbc19 DEV: Enable `unless` cops
We discussed the use of `unless` internally and decided to enforce
available rules from rubocop to restrict its most problematic uses.
2023-02-21 10:30:48 +01:00
Martin Brennan 57caf08e13
DEV: Minimal first pass of rails system test setup (#16311)
This commit introduces rails system tests run with chromedriver, selenium,
and headless chrome to our testing toolbox.

We use the `webdrivers` gem and `selenium-webdriver` which is what
the latest Rails uses so the tests run locally and in CI out of the box.

You can use `SELENIUM_VERBOSE_DRIVER_LOGS=1` to show extra
verbose logs of what selenium is doing to communicate with the system
tests.

By default JS logs are verbose so errors from JS are shown when
running system tests, you can disable this with
`SELENIUM_DISABLE_VERBOSE_JS_LOGS=1`

You can use `SELENIUM_HEADLESS=0` to run the system
tests inside a chrome browser instead of headless, which can be useful to debug things
and see what the spec sees. See note above about `bin/ember-cli` to avoid
surprises.

I have modified `bin/turbo_rspec` to exclude `spec/system` by default,
support for parallel system specs is a little shaky right now and we don't
want them slowing down the turbo by default either.

### PageObjects and System Tests

To make querying and inspecting parts of the page easier
and more reusable inbetween system tests, we are using the
concept of [PageObjects](https://www.selenium.dev/documentation/test_practices/encouraged/page_object_models/) in
our system tests. A "Page" here is generally corresponds to
an overarching ember route, e.g. "Topic" for `/t/324345/some-topic`,
and this contains logic for querying components within the topic
such as "Posts".

I have also split "Modals" into their own entity. Further down the
line we may want to explore creating independent "Component"
contexts.

Capybara DSL should be included in each PageObject class,
reference for this can be found at https://rubydoc.info/github/teamcapybara/capybara/master#the-dsl

For system tests, since they are so slow, we want to focus on
the "happy path" and not do every different possible context
and branch check using them. They are meant to be overarching
tests that check a number of things are correct using the full stack
from JS and ember to rails to ruby and then the database.

### CI Setup

Whenever a system spec fails, a screenshot
is taken and a build artifact is produced _after the entire CI run is complete_,
which can be downloaded from the Actions UI in the repo.

Most importantly, a step to build the Ember app using Ember CLI
is needed, otherwise the JS assets cannot be found by capybara:

```
- name: Build Ember CLI
  run: bin/ember-cli --build
```

A new `--build` argument has been added to `bin/ember-cli` for this
case, which is not needed locally if you already have the discourse
rails server running via `bin/ember-cli -u` since the whole server is built and
set up by default.

Co-authored-by: David Taylor <david@taylorhq.com>
2022-09-28 11:48:16 +10:00
David Taylor ef39193a06
DEV: Add `rake plugins:turbo_spec` task (#18289)
This leans on our existing `turbo_rspec` implementation to run plugin specs in parallel on all available cores
2022-09-20 15:42:54 +01:00
David Taylor d262775c3e
DEV: make `bin/ember-cli -u` terminate unicorn when ember-cli fails (#18172) 2022-09-02 22:51:51 +01:00
Jarek Radosz 16f22e3c36
DEV: Add `--forward-host` option to `bin/ember-cli` (#17244)
This allows to e.g. test multisite setup in a local dev environment. Also fixes some minor proxy issues.
2022-06-28 21:20:14 +02:00
Jarek Radosz 39a025c7af
DEV: Allow newer versions of node (#17261)
It should now properly work with 18.x, so we should start moving into direction of it being the default.
2022-06-28 20:52:31 +02:00
Gerhard Schlager 8442a07c13
DEV: Compatibility with TruffleRuby (#16641) 2022-05-05 09:50:02 +08:00
Jarek Radosz 5a50f18c0c
DEV: Avoid `$` globals (#15453)
Also:
* Remove an unused method (#fill_email)
* Replace a method that was used just once (#generate_username) with `SecureRandom.alphanumeric`
* Remove an obsolete dev puma `tmp/restart` file logic
2022-01-08 23:39:46 +01:00
Peter Zhu c5fd8c42db
DEV: Fix methods removed in Ruby 3.2 (#15459)
* File.exists? is deprecated and removed in Ruby 3.2 in favor of
File.exist?
* Dir.exists? is deprecated and removed in Ruby 3.2 in favor of
Dir.exist?
2022-01-05 18:45:08 +01:00
Jarek Radosz de3680eb5c
DEV: Re-allow node 17, with a warning (#15083) 2021-11-24 21:16:33 +01:00
Jarek Radosz c75224e3d9
DEV: Update supported node versions (#15073)
13 and 15 are no longer supported by node, and issues with discourse dependencies prevent us from using 17. (for now)
2021-11-24 18:18:35 +01:00
Robin Ward 5998e69b9c FIX: Use `127.0.0.1` instead of `localhost` for ember CLI
On new macs, `localhost` resolves to IPV6 of `::1` and unfortunately
unicorn doesn't bind to IPv6 by default.

This seems to be the path of least resistance. By using 127.0.0.1 we
force IPv4 which works great.
2021-11-09 16:02:51 -05:00
Ryan Lerch 1fffe941bf
remove some hardcoded 'localhost's from dev environment (#14801)
Trying to use a local test hostname other than localhost
(e.g. discourse.test )for discourse development was difficult due
the fact that localhost was hardcoded in a few places. This patch
uses existing environment variables to allow a developer to use a
different domain when developing.

Signed-off-by: Ryan Lerch <rlerch@redhat.com>
2021-11-03 11:26:44 +08:00
Arpit Jalan 016524e244
DEV: use mailhog in our docker dev environment (#14549) 2021-10-08 11:10:46 +05:30
David Taylor 89994cff40 DEV: Allow Ember CLI for `rake qunit:test` and `rake plugin:qunit`
To use Ember CLI, set QUNIT_EMBER_CLI=1
2021-09-21 18:10:04 +01:00
Jacob Mischka f9247dabcc
Replace `-depth` -> `-maxdepth` in boot_dev (#14046)
The `-depth` flag is incorrect on Linux, it does not take an argument
and causes an error and results in no plugins ever being found.

Copied from `man find`:

```
The global options occur after the list of start points, and so are not the same kind of option as -L, for example.

       -d     A synonym for -depth, for compatibility with FreeBSD, NetBSD, MacOS X and OpenBSD.

       -depth Process each directory's contents before the directory itself.  The -delete action also implies -depth.

       ...

       -maxdepth levels
              Descend at most levels (a non-negative integer) levels of directories below the starting-points.  Using -maxdepth 0  means
              only apply the tests and actions to the starting-points themselves.
```
2021-08-16 13:28:54 +08:00
David Taylor 2955d64703
DEV: Allow annotations to work in symlinked plugins, add binstub (#13635) 2021-07-05 15:43:10 +01:00
Penar Musaraj fbe004cc63
DEV: Do not add proxy argument when running ember-cli test (#13498) 2021-06-23 10:03:52 -04:00
David Taylor 3b6d6c7024
DEV: Set DISCOURSE_PORT when spawning unicorn via `ember-cli -u` (#13346)
This means that Discourse will use the ember-cli proxy's port number in various places like auth redirects and emails
2021-06-09 15:32:28 +01:00
Penar Musaraj 513bfc3a6c
DEV: bin/ember-cli standalone by default (#13344) 2021-06-09 09:48:43 -04:00
Sam 0241748876
DEV: ember-cli -u can be used to run a standalone dev discourse (#13336)
Previously we would need to launch unicorn separately this achieves
the same goal by making 2 modifications:

1. If -u is supplied ember-cli binary will launch and monitor ember cli and unicorn
2. We suppress 200 requests to keep console clean (we may consider moving to development rails logs)


Also cleans out output a bit by supplying silent flags to yarn.
2021-06-09 12:44:33 +10:00
David Taylor 78f9d47ab1 DEV: Add non-x86_64 warning to `d/boot_dev`
Running a development environment using Docker's qemu architecture emulation is currently not possible because `inotify` is not supported: https://github.com/docker/for-mac/issues/5321
2021-05-21 16:51:10 +01:00
David Taylor 73050d9049 DEV: Update docker development binstubs
- Add `d/ember-cli`, and publish port 4200
- Remove `d/sidekiq`. Sidekiq is now started with the rails server
- Move all `docker exec` logic into a single place, so we have one place to set environment variable pass-throughs
- Use `exec` for all bash scripts, so that return statuses are passed back correctly
- Avoid using `bin/bash -c` unnecessarily, because it makes escaping arguments difficult
2021-05-21 16:51:10 +01:00
David Taylor ab23412113 DEV: Only mount top-level plugin symlinks
Sometimes plugins directories will end up with other symlinks (e.g. inside node_modules folders). This logic does not work with deeply nested symlinks, and they are unlikely to be necessary for the plugin to work. Therefore we should only look for symlinks in the top-level of the `plugins` directory
2021-05-21 16:51:10 +01:00
Robin Ward f165f98cf7
DEV: Run `yarn install` when running `bin/ember-cli` (#13102)
Some people have noticed that if we change the packages in package.json
that they have to manually run `yarn install` or Discourse won't work.

This adds `yarn install` to the `bin/ember-cli` helper we run. It seems
quite fast if there is nothing to install so it shouldn't hurt to do
this every time we start the server.
2021-05-20 14:30:22 -04:00
Gerhard Schlager 4c26dd09e4
DEV: Update `bin/bundle` (#13067)
This fixes the following error I've been seeing lately in RubyMine:

> Error:Your `bin/bundle` was not generated by Bundler, so this binstub cannot run.
> Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.
2021-05-14 19:54:10 +02:00
David Taylor b6b27bc383
DEV: Improve auto-restart parent process detection logic (#13068)
The auto restart logic was sending a USR2 to the parent process without checking what the parent process actually was. In some situations, it might not be the `bin/unicorn` supervisor.

This commit switches to use a global variable for the supervisor PID. This will be much less prone to unexpected behavior.
2021-05-14 18:17:31 +01:00
Jeff Wong 7a0e5c8cb9
FIX: properly filter ember-cli args (#13011)
Was previously rejecting all args.
2021-05-10 10:39:54 -07:00
Penar Musaraj 49de5dadf7
DEV: Ember CLI postinstall hook and shortcut (#12987) 2021-05-10 09:34:24 -04:00
Jarek Radosz 53c06442ef
DEV: Clean up .gitignore (#12981)
(inspired by martin-brennan's recent adventure with `tags` exclusion)

**tl;dr: some of it is obsolete, some is dev-env specific and should live in *your* ~/.gitignore.**

---

To enable global gitignore (`~/.gitignore`) run:
```bash
git config --global core.excludesFile '~/.gitignore'
```

Then create that file and add your VIM/VSCode/Sublime/Emacs/Eclipse/RubyMine/JetBrains/macOS/Arch/Windows 95-specific stuff.

---

Reasons for removal:

* `bin` - never really ignored, all the files are checked in
* `.DS_Store`, `._.DS_Store` - OS specific
* `.sass-cache/*`, `/.bundle`, `/bundle/*`, `/cache`, `/logfile`, `!/plugins/discourse-nginx-performance-report`, `MiniProfiler/Ruby/rack-mini-profiler-2.0.1a.gem`, `config/fog_credentials.yml`, `/public/stylesheet-cache/*`, `script/download_db`, `script/refresh_db`, `config/version.rb` - no longer used?
* `/log/*.log` - covered by /log
* `bootsnap-load-path-cache`, `bootsnap-compile-cache` - bootsnap now keeps its ~~trash~~ temporary files in `/tmp`
* `/.project`, `/.buildpath`, `/.byebug_history`, `/.idea`, `discourse.sublime-workspace`, `*~`, `*.swp`, `*.swo`, `*.swm`, `config/multisite1.yml`, `.rvmrc`, `.ruby-version`, `.ruby-gemset`, `.rbenv`, `bundler_stubs/*`, `*.db`, `*.iml`, `*.swn`, `/package-lock.json`, `.vscode`, `.envrc`, `tags` - dev-env specific, see the note at the top of the commit
2021-05-10 13:43:13 +02:00
Jeff Wong bc3c3d56e0
DEV: do not cd into yarn dir, use --cwd argument to yarn instead (#12986)
* DEV: do not cd into yarn dir, use --cwd argument to yarn instead

Co-authored-by: Robin Ward <robin.ward@gmail.com>
2021-05-07 09:58:57 -07:00
David Taylor ac1d344368
DEV: Update server startup warning to be more specific (#12949)
Using `bundle exec` will slow down server startup by at least 0.5s. `bin/unicorn` has built-in handling of bundler dependencies, so it is better to launch `bin/rails s` or `bin/unicorn` directly.
2021-05-05 13:33:11 +01:00
David Taylor 0188d53f3a
DEV: Add performance warning message when running `rails s` (#12912)
In Discourse, `rails s` ultimately launches the `bin/unicorn` script. However, the overhead of `rails` launching `bin/rails`, and then in turn `bin/unicorn` can be non-trivial. Therefore it is much better to run `bin/unicorn` directly.

This commit prints a warning message to STDERR when `rails s` is used. Functionality is unchanged.
2021-04-30 11:50:06 +01:00
Robin Ward 51f872f13a
DEV: Require Ember CLI to be used in development mode (#12738)
We really want to encourage all developers to use Ember CLI for local
development and testing. This will display an error page if they are not
with instructions on how to start the local server.

To disable it, you can set `NO_EMBER_CLI=1` as an ENV variable
2021-04-29 14:13:36 -04:00
Sam 307cb1d114
DEV: add helper script to run ember cli (#12005)
This little helper script allows for easy ember cli development.

To see the options run `bin/ember-cli -h`

It allows you to proxy try.discourse.org with the `bin/ember-cli --try`
switch, which effectively allows for some development without a rails installed.

It passes on arguments to ember-cli so you can customize port and so on.

It makes the assumption that on local people are using `bin/unicorn` for
development. (it includes some extra discourse specific helpers)
2021-02-09 16:33:14 +11:00
Arpit Jalan bbf6cc4f18
DEV: require fileutils gem on boot (#11542) 2020-12-21 16:31:06 +11:00
Robin Ward 2eb9c0f3dd FIX: `notify_file_change` was outputting a command to vim
The check for `socat` needs to be silenced too, or vim will output the
path to `socat` every time you save a file.
2020-12-10 10:03:57 -05:00
Alessandro Bertacco dc2f5064e2
DEV: support for relative symlinks for plugins in docker dev (#10955)
These changes gives the possibility to use relative symlinks for plugins. Previously only absolute symlinks was supported.
2020-10-21 09:39:53 +11:00
David Taylor 46d1c91e1a
DEV: Make notify_file_change fallback to `nc` when `socat` missing (#10903) 2020-10-13 17:03:24 +01:00
Kane York 4635be10c8 DEV: Add docker cleanup script to d/ folder 2020-03-01 12:09:07 -08:00
Sam Saffron d8412f409a DEV: resolve symlinks in docker dev
symlinks where not fully resolved leading to docker not booting when you
had symlinks in the plugins directory pointing at relative paths.
2020-01-13 10:33:34 +11:00
Martin Brennan aa2b3e71b4 Minor refactor of bin/unicorn cache clean
* move git ls-files for plugins folder into one line
2019-11-20 08:47:51 +10:00