Commit Graph

924 Commits

Author SHA1 Message Date
Sam Saffron a5b582f686
DEV: demux stdout when running multisite migrate
This avoids mixing up output in such a way that we can not tell which
site ran which migrations

Avoids threads all fighting for output
2020-04-09 12:50:14 +10:00
Sam Saffron 708dd97dfd
PERF: speed up migrations on multisite
Previously we were migrating multisites serially, this is extremely slow
especially when 200 dbs are involved.

The new implementation defaults to running 20 migrations concurrently, leading
to a 20x speedup.

We also amended it so errors are printed out last, something that makes
debugging failures easier.

This is code specific to Discourse cause we integrate SeedFu with our
migrations and can not include this in the multisite gem.
2020-04-09 11:58:20 +10:00
Sam Saffron f062ebf274
DEV: reduce noise in logs regarding migration
If a migration performs no changes it should not output stuff.

Previously we would output information about seeds which was very noisy.

On multisite this was particularly bad
2020-04-09 10:05:22 +10:00
Sam Saffron 8a50ab45aa
DEV: use REPORT_REQUESTS=1 to find all requests
We can use this to profile our pretender and ensure nothing is superfluous
and nothing is missing
2020-04-02 16:01:49 +11:00
Martin Brennan 6f978bc95c
FIX: First pass to improve efficiency of secure uploads rake task (#9284)
Get rid of harmful each loop over uploads to update. Instead we put all the unique access control posts for the uploads into a map for fast access (vs using the slow .find through array) and look up the post when it is needed when looping through the uploads in batches.

On a Discourse instance with ~93k uploads, a simplified version of the old method takes > 1 minute, and a simplified version of the new method takes ~18s and uses a lot less memory.
2020-03-26 15:59:57 +10:00
Martin Brennan efd5fb665b
DEV: Fix flaky time sensitive uploads.rake specs (#9283)
Also fix issues in spec where certain uploads were not considered secure
2020-03-26 13:31:39 +10:00
Martin Brennan 097851c135
FIX: Change secure media to encompass attachments as well (#9271)
If the “secure media” site setting is enabled then ALL files uploaded to Discourse (images, video, audio, pdf, txt, zip etc. etc.) will follow the secure media rules. The “prevent anons from downloading files” setting will no longer have any bearing on upload security. Basically, the feature will more appropriately be called “secure uploads” instead of “secure media”.

This is being done because there are communities out there that would like all attachments and media to be secure based on category rules but still allow anonymous users to download attachments in public places, which is not possible in the current arrangement.
2020-03-26 07:16:02 +10:00
Joffrey JAFFEUX 86eb03af85
DEV: allows to run up/down migrations of a plugin (#9241)
This is mostly useful while developing a plugin, to avoid manual actions of deleting tables and schema_migrations rows.

Usage:
bundle exec rake plugin:migrate:down[discourse-calendar]
2020-03-19 19:30:08 +01:00
Martin Brennan 16799da580
FIX: Improve bookmark modal on mobile and bookmark sync rake task (#9221)
* Improve the bookmark mobile on modal so it doesn't go all the way to the edge and the custom datetime input is easier to use
* Improve the rake task for syncing so it does not error for topics that no longer exist and batches 2000 inserts at a time, clearing the array each time
2020-03-17 15:15:22 +10:00
Sam Saffron a14313e9d0
FIX: RANDOM_PASSWORD not working rake admin:create
We regressed behavior of this env var which enable admin account
creation with a random password
2020-03-15 17:24:11 +11:00
Robin Ward c9d1577087 Let's not log the username/password
This could easily be seen by someone who shouldn't.
2020-03-11 12:48:10 -04:00
Gerhard Schlager 8fa8bab9ff FIX: Don't optimize icons during db:migrate when restoring backup
Uploads are extracted after the DB migration, so this could lead to a failure during the restore. Site icons get optimized after extracting uploads.
2020-03-04 16:59:49 +01:00
Martin Brennan 0388653a4d
DEV: Upload and secure media retroactive rake task improvements (#9027)
* Add uploads:sync_s3_acls rake task to ensure the ACLs in S3 are the correct (public-read or private) setting based on upload security

* Improved uploads:disable_secure_media to be more efficient and provide better messages to the user.

* Rename uploads:ensure_correct_acl task to uploads:secure_upload_analyse_and_update as it does more than check the ACL

* Many improvements to uploads:secure_upload_analyse_and_update

* Make sure that upload.access_control_post is unscoped so deleted posts are still fetched, because they still affect the security of the upload.

* Add escape hatch for capture_stdout in the form of RAILS_ENABLE_TEST_STDOUT. If provided the capture_stdout code will be ignored, so you can see the output if you need.
2020-03-03 10:03:58 +11:00
Martin Brennan cfd56e9159 Include access control post when loading uploads in rake task
* to avoid N+1 query
2020-02-18 10:35:15 +10:00
Martin Brennan 9dcc454a07
FIX: Improvements and fixes for update_upload_acl rake task (#8980)
The rake task was broken, because the addition of the
UploadSecurity check returned true/false instead of the
upload ID to determine which uploads to set secure.
Also it was rebaking the posts in the wrong place and
pretty inefficiently at that. Also it was rebaking before
the upload was being changed to secure in the DB.
This also updates the task to set the access_control_post_id
for all uploads. the first post the upload is linked to is used
for the access control. if the upload doesn't get changed to
secure this doesn't affect anything.
Added a spec for the rake task to cover common cases.
2020-02-17 14:21:43 +10:00
Martin Brennan e1e74abd4f
FEATURE: Improving bookmarks part 2 -- Topic Bookmarking (#8954)
### UI Changes

If `SiteSetting.enable_bookmarks_with_reminders` is enabled:

* Clicking "Bookmark" on a topic will create a new Bookmark record instead of a post + user action
* Clicking "Clear Bookmarks" on a topic will delete all the new Bookmark records on a topic
* The topic bookmark buttons control the post bookmark flags correctly and vice-versa
Disabled selecting the "reminder type" for bookmarks in the UI because the backend functionality is not done yet (of sending users notifications etc.)

### Other Changes

* Added delete bookmark route (but no UI yet)
* Added a rake task to sync the old PostAction bookmarks to the new Bookmark table, which can be run as many times as we want for a site (it will not create duplicates).
2020-02-13 16:26:02 +10:00
Joffrey JAFFEUX 87e92da085
DEV: makes popper.js part of javascript rake task (#8847) 2020-02-04 15:34:46 +01:00
Gerhard Schlager 4e8be6f18b FIX: uploads:s3_migration_status rake task was broken 2020-01-28 22:10:25 +01:00
Roman Rizzi 8eb2147f1f
FIX: Ensure sourcemap's source is correct. Uses the full assets path this time. (#8774) 2020-01-23 14:44:00 -03:00
romanrizzi 1de5b4ad64 Revert "FIX: Ensure sourcemap's source is correct"
This reverts commit 4553a0f6e0.
2020-01-22 16:31:34 -03:00
Roman Rizzi 4553a0f6e0
FIX: Ensure sourcemap's source is correct 2020-01-22 12:57:28 -03:00
Gerhard Schlager ab07b945c2
Merge pull request #8736 from gschlager/rename_reply_id_column
REFACTOR: Rename `post_replies.reply_id` column to `post_replies.reply_post_id`
2020-01-17 17:24:49 +01:00
Martin Brennan 7c32411881
FEATURE: Secure media allowing duplicated uploads with category-level privacy and post-based access rules (#8664)
### General Changes and Duplication

* We now consider a post `with_secure_media?` if it is in a read-restricted category.
* When uploading we now set an upload's secure status straight away.
* When uploading if `SiteSetting.secure_media` is enabled, we do not check to see if the upload already exists using the `sha1` digest of the upload. The `sha1` column of the upload is filled with a `SecureRandom.hex(20)` value which is the same length as `Upload::SHA1_LENGTH`. The `original_sha1` column is filled with the _real_ sha1 digest of the file. 
* Whether an upload `should_be_secure?` is now determined by whether the `access_control_post` is `with_secure_media?` (if there is no access control post then we leave the secure status as is).
* When serializing the upload, we now cook the URL if the upload is secure. This is so it shows up correctly in the composer preview, because we set secure status on upload.

### Viewing Secure Media

* The secure-media-upload URL will take the post that the upload is attached to into account via `Guardian.can_see?` for access permissions
* If there is no `access_control_post` then we just deliver the media. This should be a rare occurrance and shouldn't cause issues as the `access_control_post` is set when `link_post_uploads` is called via `CookedPostProcessor`

### Removed

We no longer do any of these because we do not reuse uploads by sha1 if secure media is enabled.

* We no longer have a way to prevent cross-posting of a secure upload from a private context to a public context.
* We no longer have to set `secure: false` for uploads when uploading for a theme component.
2020-01-16 13:50:27 +10:00
Gerhard Schlager e474cda321 REFACTOR: Restoring of backups and migration of uploads to S3 2020-01-14 11:41:35 +01:00
David Taylor c8d438cc63
DEV: Allow CSP to be enabled during QUnit tests (#8668)
The QUnit rake task starts a server in test mode. We need a tweak to allow dynamic CSP hostnames in test mode. This tweak is already present in development mode.

To allow CSP to work, the browser host/port must match what the server sees. Therefore we need to disable the enforce_hostname middleware in test mode. To keep rspec and production as similar as possible, we skip enforce_hostname using an environment variable.

Also move the qunit rake task to use unicorn, for consistency with development and production.
2020-01-07 12:22:58 +00:00
Martin Brennan abca91cc4d
FEATURE: Add rake task to disable secure media (#8669)
* Add a rake task to disable secure media. This sets all uploads to `secure: false`, changes the upload ACL to public, and rebakes all the posts using the uploads to make sure they point to the correct URLs. This is in a transaction for each upload with the upload being updated the last step, so if the task fails it can be resumed.
* Also allow viewing media via the secure url if secure media is disabled, redirecting to the normal CDN url, because otherwise media links will be broken while we go and rebake all the posts + update ACLs
2020-01-07 12:27:24 +10:00
Blake Erickson b7b85f9ade
FEATURE: Turn csp on by default (#8665)
* turn csp on by default

* remove csp migration for new sites now that is is on by default

* Ensure CSP is off before starting qunit
2020-01-06 13:42:21 -07:00
Blake Erickson fc6b093dce FIX: Ensure CSP is off for qunit
If CSP is turned on qunit won't run at all, not even via the CLI, this
is causing the js tests to fail.

Follow up to: 3193b0f6e6
2020-01-03 18:43:19 -07:00
Sam Saffron 412e1ebbe2 DEV: correct parallel specs rake tasks
This used to work due to side effects.

`rake parallel:migrate` used to work very inconsistently and would only migrate
some of the databases.

This introduces the recommended change to db.yml so the correct database is
found based off TEST_ENV_NUMBER if for some reason we did not set it using
RAILS_DB

Also avoids a bunch of schema dumping which is not needed when migrating
parallel specs



DB number 1 is very odd cause for whatever reason parallel spec is not
setting it.
2019-12-31 14:07:55 +11:00
Robin Ward 9fc1a754fe DEV: Error message if multisite DB can't be created 2019-12-19 11:52:27 -05:00
Robin Ward 480e2ec523 FIX: If we run `db:migrate` on its own, it should load the environment 2019-12-16 14:46:33 -05:00
Hector Bustillos 42758379b9 FEATURE: Adds a message when the passwords doesn't match in rake admin:create (#8543) 2019-12-13 11:40:33 +11:00
David Taylor 4c9ca24ccf
FEATURE: Hash API keys in the database (#8438)
API keys are now only visible when first created. After that, only the first four characters are stored in the database for identification, along with an sha256 hash of the full key. This makes key usage easier to audit, and ensures attackers would not have access to the live site in the event of a database leak.

This makes the merge lower risk, because we have some time to revert if needed. Once the change is confirmed to be working, we will add a second commit to drop the `key` column.
2019-12-12 11:45:00 +00:00
David Taylor 67605b55d5 DEV: Prevent plugin loading when running core db migrations in tests
This is required because bin/rake automatically loads plugins when migrating. In our continuous integration, we don't want plugins to break the core build. They should only be loaded for the plugin build.
2019-12-10 16:11:49 +00:00
Neil Lalonde 891b71bdf5 Fix release notes rake task
It breaks when a commit message is "FIX:"
2019-12-05 11:53:10 -05:00
romanrizzi b13f427450 DEV: Drop uglifyJS V2 support 2019-12-04 15:58:38 -03:00
David Taylor 46d8fd3831 FIX: Allow for nil upload record when migrating to S3 2019-12-04 15:13:39 +00:00
Joffrey JAFFEUX 0d3d2c43a0
DEV: s/\$redis/Discourse\.redis (#8431)
This commit also adds a rubocop rule to prevent global variables.
2019-12-03 10:05:53 +01:00
Penar Musaraj 0a6b29a942 DEV: Allow skipping --fail-fast rspec option for plugin specs 2019-11-22 13:27:47 -05:00
Jeff Wong c6d8dbd4a9 Revert "FEATURE: Normalize the service worker route (#8359)"
This reverts commit 9799a651b6.
2019-11-20 14:10:17 -08:00
Jeff Wong 9799a651b6
FEATURE: Normalize the service worker route (#8359)
* FEATURE: Normalize the service worker route

Update cache headers so they are not immutable outside of the rails app

Add the ability to purge the service worker cache from localhost

Rails -> nginx will pass immutable flags so the file is cached until reloaded.
In most cases, nginx will have its cache flushed on rebuild (new image)

For those needing dynamic re-caching (such as upgrading via the UI),
a rake task for flushing the service worker script is provided
through `assets:flush_sw`
2019-11-20 11:33:41 -08:00
Penar Musaraj 102909edb3 FEATURE: Add support for secure media (#7888)
This PR introduces a new secure media setting. When enabled, it prevent unathorized access to media uploads (files of type image, video and audio). When the `login_required` setting is enabled, then all media uploads will be protected from unauthorized (anonymous) access. When `login_required`is disabled, only media in private messages will be protected from unauthorized access. 

A few notes: 

- the `prevent_anons_from_downloading_files` setting no longer applies to audio and video uploads
- the `secure_media` setting can only be enabled if S3 uploads are already enabled and configured
- upload records have a new column, `secure`, which is a boolean `true/false` of the upload's secure status
- when creating a public post with an upload that has already been uploaded and is marked as secure, the post creator will raise an error
- when enabling or disabling the setting on a site with existing uploads, the rake task `uploads:ensure_correct_acl` should be used to update all uploads' secure status and their ACL on S3
2019-11-18 11:25:42 +10:00
Robin Ward 397852a9a6 FIX: Support for `rake db:rollback` with plugins
For this to work we need to overwrite `db:rollback` in our Rakefile like
we do for migrate, so that it removes the load_config dependency. This
allows our custom migration paths to work.
2019-11-15 16:38:46 -05:00
romanrizzi 0ba0aa00e7 DEV: Use a bigger suite for warming up the tmp folder 2019-11-13 18:34:10 -03:00
Simon Cossar 544a1379b6
Reference posts table for deleted_at and user_id (#8332) 2019-11-11 11:03:57 -05:00
Jay Pfaffman 63bd07492e FIX: Update api_key rake task for recent changes
New usage: 

```
rake api_key:get_or_create_master["Onboarding Key"]
```
2019-11-08 23:34:15 +00:00
Sam Saffron d8f7f363cd FEATURE: stop updating last_posted_at on users for messages and whispers
This ensures we only update last_posted_at which is user facing for non messages
and non whispers.

We still update this date for secure categories, we do not revert it for
deleted posts.
2019-10-31 09:01:26 +11:00
Krzysztof Kotlarek c32bd8ae48 FEATURE: Remove attachments and truncate raw field for incoming emails (#8253)
Adds the settings: 

raw_email_max_length, raw_rejected_email_max_length, delete_rejected_email_after_days. 

These settings control retention of the "raw" emails logs.

raw_email_max_length ensures that if we get incoming email that is huge we will truncate it removing uploads from the raw log.

raw_rejected_email_max_length introduces an even more aggressive truncation for rejected incoming mail. 

delete_rejected_email_after_days controls how many days we will keep rejected emails for (default 90)
2019-10-30 16:54:35 +11:00
romanrizzi 197ad8cd6e DEV: Run a faster test when warming up 2019-10-28 17:19:55 -03:00
romanrizzi d76d0e75ec DEV: Move warmup inside docker rake task 2019-10-25 16:31:05 -03:00
Gerhard Schlager 452d541b74 DEV: Add env variable to allow easier debugging of smoke tests
Set `DEBUG_NODE=1` when running `rake smoke:test` and use your favorite tool to debug the smoke tests. See https://nodejs.org/en/docs/guides/debugging-getting-started/ for more information.

The debugger will break at the beginning of the smoke tests when the env variable is set.
2019-10-24 17:48:15 +02:00
Krzysztof Kotlarek f34a0141c7 FIX: Correct path to ImportExport module (#8227)
During the move from Classic autoloader to Zeitwerk import_export module was moved to correct file name convention.
427d54b2b0 (diff-d896ec33b95afb7fae9f8bfe73d0580b)

Problem is that export/import is still using old path to require that module

Meta: https://meta.discourse.org/t/topic-and-category-export-import/38930/40
2019-10-23 17:27:14 +11:00
Daniel Waterworth 55a1394342 DEV: pluck_first
Doing .pluck(:column).first is a very common pattern in Discourse and in
most cases, a limit cause isn't being added. Instead of adding a limit
clause to all these callsites, this commit adds two new methods to
ActiveRecord::Relation:

pluck_first, equivalent to limit(1).pluck(*columns).first

and pluck_first! which, like other finder methods, raises an exception
when no record is found
2019-10-21 12:08:20 +01:00
Roman Rizzi 10478cb395
DEV: Bump uglifyjs (#7834)
* Rewrite uglifyjs command to work with 3.x

* Use ES5 syntax in plain JS files

* Use the older command if uglifyJS V2.x is installed
2019-10-09 10:02:49 -03:00
Sam Saffron 6471fbd453 FIX: maxminddb:get task no longer working
Following our zeitwerk upgrade we needed to be more explicit about the
require
2019-10-04 09:47:33 +10:00
Krzysztof Kotlarek 427d54b2b0 DEV: Upgrading Discourse to Zeitwerk (#8098)
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains. 

We no longer need to use Rails "require_dependency" anywhere and instead can just use standard 
Ruby patterns to require files.

This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
Joffrey JAFFEUX cb8fa46970
DEV: prevents populate task to crash (#8111)
Generated emails/names/sentences were crashing with the following error:

```
Can not transliterate strings with ASCII-8BIT encoding
```
2019-09-22 05:18:48 -07:00
Sam Saffron 1ca257be79 DEV: db:migrate no longer works after db:schema:load
In Rails 6 due to internal changes, the following sequence no longer works:

```
RAILS_ENV=test bin/rake db:migrate
RAILS_ENV=test bin/rake db:schema:dump
dropdb discourse_test
createdb discourse_test
RAILS_ENV=test bin/rake db:schema:load
RAILS_ENV=test bin/rake db:migrate
```

What appears to be happening is that our tracking of plugin migrations is
being missed on schema:dump or load.

A more comprehensive fix restoring schema:dump / load support will be
investigated.
2019-09-18 13:17:49 +10:00
Sam Saffron 025d4ee91f FIX: Rails 6 multisite migrations and plugin migrations
Prior to this change plugin migrations were not working and multisite
migrations not working.

Rails internals changed so we need to account for it.

Specifically semantics of `db:migrate` in rails changed so it is sort of
a "multisite:migrate".
2019-09-13 09:17:54 +10:00
Roman Rizzi 568232052e
DEV: Remove FlagQuery class and old code (#8064) 2019-09-12 13:21:33 -03:00
Gerhard Schlager 56c6e8e961 FIX: POP3 doesn't work with TLS 1.3 2019-09-11 18:43:19 +02:00
Michael Brown 503a11cc88 FIX: inline_uploads and subfolder (#8076)
* FIX: inline_uploads and subfolder

* if subfolder, also look for images with a path containing
  cdn_url + relative_url_root

* FIX: migrate_to_s3 task and subfolder
2019-09-11 11:50:48 +10:00
Gerhard Schlager 874985db71 DEV: Ignore duplicate prefixes when generating release notes 2019-09-04 15:45:48 +02:00
Arpit Jalan 75f37ac16a Rename `lazyYT` plugin directory name to `lazy-yt` 2019-08-21 14:35:14 +05:30
Sam Saffron 7632fe0b58 PERF: rake posts:rebake_uncooked_posts runs inline
Running this inline makes more sense otherwise there is extreme risk in
saturating sidekiq queue.

This also reworks ordering and selection so we double check if a post needs
rebaking prior to rebaking, this unlocks the ability to run this rake task
from multiple consoles.
2019-08-13 10:28:42 +10:00
Rafael dos Santos Silva 606c0ed14d
FIX: S3 uploads were missing a cache-control header (#7902)
Admins still need to run the rake task to fix the files who where uploaded previously.
2019-08-06 14:55:17 -03:00
Sam Saffron 4c6a0313f2 FEATURE: improve stats provided by rake db:stats
New stats include index size and sort by index + table size

Eg:

```
table_name                  | row_estimate | table_size | index_size | total_size
--------------------------------------------------------------------------
topic_views                 | 18260776     | 796 MB     | 2015 MB    | 2811 MB
post_timings                | 15813837     | 668 MB     | 1153 MB    | 1821 MB
posts                       | 386890       | 1532 MB    | 215 MB     | 1747 MB
post_search_data            | 317575       | 580 MB     | 162 MB     | 742 MB
incoming_emails             | 34412        | 734 MB     | 7160 kB    | 741 MB
incoming_links              | 5169302      | 306 MB     | 377 MB     | 683 MB
email_logs                  | 5            | 32 kB      | 418 MB     | 418 MB
topic_users                 | 2410166      | 217 MB     | 155 MB     | 372 MB
user_actions                | 1361253      | 101 MB     | 266 MB     | 366 MB
...

```
2019-08-02 17:25:42 +10:00
Vinoth Kannan e44d56e4d2 DEV: raise error only when 'STOP_ON_ERROR' env variable is available. 2019-08-01 23:54:06 +05:30
Daniel Waterworth 20bc4a38a5
FIX: DistributedMutex (#7953) 2019-08-01 09:12:05 +01:00
Guo Xiang Tan a1a61fcd2d DEV: Use `File#file?` in `posts:missing_uploads` rake task. 2019-08-01 10:59:34 +08:00
Rishabh dcb47d902b
REFACTOR: Rename SiteSetting.disable_edit_notifications to disable_system_edit_notifications (#7958)
* REFACTOR: Rename SiteSetting.disable_edit_notifications to disable_system_edit_notifications

- The older name could cause some confusion because the setting does not disable all edit notifications, only system ones.

* FIX: Add frozen_string_literal: true in the migration

* DEV: Deprecate 'disable_edit_notifications'
2019-07-31 20:20:41 +05:30
Penar Musaraj a1fb718340 DEV: Tweak plugin rake task console message
Show "Allowing write to all repos!" message only once.
2019-07-30 09:53:56 -04:00
Guo Xiang Tan 8a64b0c8e8 Revert "DEV: Remove unused kwarg and properly check for local missing uploads."
This reverts commit 97769f3d02.

The code is confusing but this change is quite risky. Defer for now
until we can look at it properly.
2019-07-29 14:35:34 +08:00
Guo Xiang Tan 97769f3d02 DEV: Remove unused kwarg and properly check for local missing uploads. 2019-07-29 14:21:06 +08:00
Guo Xiang Tan e9dc48ccae DEV: Fix multisite support for `posts:missing_uploads`. 2019-07-29 10:56:11 +08:00
Blake Erickson d26aa6e71e REFACTOR: Cleanup rake tasks based on feedback
Follow up to: [FEATURE: Create a rake task for destroying categories][1]

- `Discourse.system_user` is my friend
- Remove puts statements from rake tasks that don't return anything
- `for_each` is also my friend
- Use `human_users` to also exclude discobot
- Sort/format categories:list

[1]: 092eeb5ca3
2019-07-18 19:15:01 -06:00
Blake Erickson 092eeb5ca3 FEATURE: Create a rake task for destroying categories
Created a rake task for destroying multiple categories along with any
subcategories and topics the belong to those categories.

Also created a rake task for listing all of your categories.

Refactored existing destroy rake tasks to use new logging method, that
allows for puts output in the console but prevents it from showing in
the specs.
2019-07-17 12:44:14 -06:00
Rafael dos Santos Silva 1221d34284
FEATURE: Make Discourse work offline with WorkboxJS (#7870) 2019-07-15 13:05:55 -03:00
Gerhard Schlager ce8e099639 FEATURE: Use configured quotation marks in fancy topic title 2019-07-12 21:10:10 +02:00
Joffrey JAFFEUX 550e811652
DEV: allows lodash to be updated with `rake javascript:update` (#7881) 2019-07-11 16:57:03 +02:00
Penar Musaraj befcf67c90 DEV: run db:create and db:migrate on turbo specs
This ensures multisite specs run correctly when running ./bin/turbo_rspec
2019-07-08 14:27:51 -04:00
Sam Saffron b9a2a4d904 DEV: Avoid emoji for symbols
Rake task needs to work across a wide variety of terminal emulators.
Requiring an emoji font is superflous.
2019-07-05 09:41:15 +10:00
Ralph Rooding 1318e0b288 FEATURE: Rake themes installer (#7848)
* Delete remote_theme when deleting the theme

* Install themes and theme components through rake

* Removed unnecessary test
2019-07-04 14:33:05 -04:00
Joffrey JAFFEUX e9bb13c630
DEV: updates jquery-color to prevent it to pull old jquery version (#7857) 2019-07-04 15:03:21 +02:00
Daniel Waterworth d6aa92e98e DEV: Add a verbose option to ./bin/turbo_rspec 2019-06-27 15:49:21 +01:00
Daniel Waterworth b3ec3fac93 FIX: turbo_rspec doesn't accept these options 2019-06-26 17:05:27 +01:00
Daniel Waterworth 68d54d818a DEV: Add USE_TURBO option to docker rake task (#7811) 2019-06-26 11:54:39 -04:00
Guo Xiang Tan d21594f4f7 Revert changes added by mistake in 2b19e2acc8. 2019-06-25 15:25:12 +08:00
Guo Xiang Tan 2b19e2acc8 Fix typo in a0aeabbb94. 2019-06-25 15:18:57 +08:00
Guo Xiang Tan a0aeabbb94 DEV: Safer replacement for old scheme uploads for `posts:missing_uploads`. 2019-06-25 11:34:26 +08:00
Guo Xiang Tan f3c13d80c9 DEV: Don't copy file when trying to recover missing uploads. 2019-06-24 16:32:59 +08:00
Guo Xiang Tan 2cc5561504 DEV: Correct description in `posts:missing_uploads` rake task. 2019-06-24 14:57:43 +08:00
Sam Saffron 5bc92296be DEV: lint a bunch of files we missed 2019-06-21 11:33:41 +10:00
Daniel Waterworth e18ce56f4b DEV: Add a new way to run specs in parallel with better output (#7778)
* DEV: Add a new way to run specs in parallel with better output

This commit:

 1. adds a new executable, `bin/interleaved_rspec` which works much like
    `rspec`, but runs the tests in parallel.

 2. adds a rake task, `rake interleaved:spec` which runs the whole test
    suite.

 3. makes autospec use this new wrapper by default. You can disable this
    by running `PARALLEL_SPEC=0 rake autospec`.

It works much like the `parallel_tests` gem (and relies on it), but
makes each subprocess use a machine-readable formatter and parses this
output in order to provide a better overall summary.

(It's called interleaved, because parallel was taken and naming is
hard).

* Make popen3 invocation safer

* Use FileUtils instead of shelling out

* DRY up reporter

* Moved summary logic into Reporter

* s/interleaved/turbo/g

* Move Reporter into its own file

* Moved run into its own class

* Moved Runner into its own file

* Move JsonRowsFormatter under TurboTests

* Join on threads at the end

* Acted on feedback from eviltrout
2019-06-21 10:59:01 +10:00
Penar Musaraj 4bbb43cb2b DEV: pull plugin repos before linting them 2019-06-14 13:50:15 -04:00
Sam Saffron ecb2fd8222 DEV: improve db:create to catch all cases
- No RAILS_ENV - create multisite / dev / test
- `test` RAILS_ENV - create multisite / test
2019-06-14 15:47:05 +10:00
Sam Saffron a01488ae67 DEV: improve on rake db:create
Followup on 3af00a65 which broke build
2019-06-14 15:06:07 +10:00
Sam Saffron 3af00a65e6 FIX: site settings loading default values when no db
This fixes a condition where an intermittent db connection could cause
invalid site settings to be stored

It also removes a catch all we had.

Somewhere around Rails 5 `db:create` started wanting full environment
this is a problem for Discourse since it needs to boot up data from the
db.

This removes the catch all and surgically adds a db / redis bypass to
db:create task.
2019-06-14 14:21:07 +10:00
Guo Xiang Tan eaa7527933 DEV: Switch `posts:inline_uploads` scope to be more generic. 2019-06-14 11:53:34 +08:00
Penar Musaraj 66b15b9d87 DEV: support installing plugin gems in test images
- adds plugin:install_all_gems rake task

- adds UPDATE_ALL_PLUGINS env variable in docker:test
2019-06-13 09:58:57 -04:00