Commit Graph

924 Commits

Author SHA1 Message Date
Gerhard Schlager 452d541b74 DEV: Add env variable to allow easier debugging of smoke tests
Set `DEBUG_NODE=1` when running `rake smoke:test` and use your favorite tool to debug the smoke tests. See https://nodejs.org/en/docs/guides/debugging-getting-started/ for more information.

The debugger will break at the beginning of the smoke tests when the env variable is set.
2019-10-24 17:48:15 +02:00
Krzysztof Kotlarek f34a0141c7 FIX: Correct path to ImportExport module (#8227)
During the move from Classic autoloader to Zeitwerk import_export module was moved to correct file name convention.
427d54b2b0 (diff-d896ec33b95afb7fae9f8bfe73d0580b)

Problem is that export/import is still using old path to require that module

Meta: https://meta.discourse.org/t/topic-and-category-export-import/38930/40
2019-10-23 17:27:14 +11:00
Daniel Waterworth 55a1394342 DEV: pluck_first
Doing .pluck(:column).first is a very common pattern in Discourse and in
most cases, a limit cause isn't being added. Instead of adding a limit
clause to all these callsites, this commit adds two new methods to
ActiveRecord::Relation:

pluck_first, equivalent to limit(1).pluck(*columns).first

and pluck_first! which, like other finder methods, raises an exception
when no record is found
2019-10-21 12:08:20 +01:00
Roman Rizzi 10478cb395
DEV: Bump uglifyjs (#7834)
* Rewrite uglifyjs command to work with 3.x

* Use ES5 syntax in plain JS files

* Use the older command if uglifyJS V2.x is installed
2019-10-09 10:02:49 -03:00
Sam Saffron 6471fbd453 FIX: maxminddb:get task no longer working
Following our zeitwerk upgrade we needed to be more explicit about the
require
2019-10-04 09:47:33 +10:00
Krzysztof Kotlarek 427d54b2b0 DEV: Upgrading Discourse to Zeitwerk (#8098)
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains. 

We no longer need to use Rails "require_dependency" anywhere and instead can just use standard 
Ruby patterns to require files.

This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
Joffrey JAFFEUX cb8fa46970
DEV: prevents populate task to crash (#8111)
Generated emails/names/sentences were crashing with the following error:

```
Can not transliterate strings with ASCII-8BIT encoding
```
2019-09-22 05:18:48 -07:00
Sam Saffron 1ca257be79 DEV: db:migrate no longer works after db:schema:load
In Rails 6 due to internal changes, the following sequence no longer works:

```
RAILS_ENV=test bin/rake db:migrate
RAILS_ENV=test bin/rake db:schema:dump
dropdb discourse_test
createdb discourse_test
RAILS_ENV=test bin/rake db:schema:load
RAILS_ENV=test bin/rake db:migrate
```

What appears to be happening is that our tracking of plugin migrations is
being missed on schema:dump or load.

A more comprehensive fix restoring schema:dump / load support will be
investigated.
2019-09-18 13:17:49 +10:00
Sam Saffron 025d4ee91f FIX: Rails 6 multisite migrations and plugin migrations
Prior to this change plugin migrations were not working and multisite
migrations not working.

Rails internals changed so we need to account for it.

Specifically semantics of `db:migrate` in rails changed so it is sort of
a "multisite:migrate".
2019-09-13 09:17:54 +10:00
Roman Rizzi 568232052e
DEV: Remove FlagQuery class and old code (#8064) 2019-09-12 13:21:33 -03:00
Gerhard Schlager 56c6e8e961 FIX: POP3 doesn't work with TLS 1.3 2019-09-11 18:43:19 +02:00
Michael Brown 503a11cc88 FIX: inline_uploads and subfolder (#8076)
* FIX: inline_uploads and subfolder

* if subfolder, also look for images with a path containing
  cdn_url + relative_url_root

* FIX: migrate_to_s3 task and subfolder
2019-09-11 11:50:48 +10:00
Gerhard Schlager 874985db71 DEV: Ignore duplicate prefixes when generating release notes 2019-09-04 15:45:48 +02:00
Arpit Jalan 75f37ac16a Rename `lazyYT` plugin directory name to `lazy-yt` 2019-08-21 14:35:14 +05:30
Sam Saffron 7632fe0b58 PERF: rake posts:rebake_uncooked_posts runs inline
Running this inline makes more sense otherwise there is extreme risk in
saturating sidekiq queue.

This also reworks ordering and selection so we double check if a post needs
rebaking prior to rebaking, this unlocks the ability to run this rake task
from multiple consoles.
2019-08-13 10:28:42 +10:00
Rafael dos Santos Silva 606c0ed14d
FIX: S3 uploads were missing a cache-control header (#7902)
Admins still need to run the rake task to fix the files who where uploaded previously.
2019-08-06 14:55:17 -03:00
Sam Saffron 4c6a0313f2 FEATURE: improve stats provided by rake db:stats
New stats include index size and sort by index + table size

Eg:

```
table_name                  | row_estimate | table_size | index_size | total_size
--------------------------------------------------------------------------
topic_views                 | 18260776     | 796 MB     | 2015 MB    | 2811 MB
post_timings                | 15813837     | 668 MB     | 1153 MB    | 1821 MB
posts                       | 386890       | 1532 MB    | 215 MB     | 1747 MB
post_search_data            | 317575       | 580 MB     | 162 MB     | 742 MB
incoming_emails             | 34412        | 734 MB     | 7160 kB    | 741 MB
incoming_links              | 5169302      | 306 MB     | 377 MB     | 683 MB
email_logs                  | 5            | 32 kB      | 418 MB     | 418 MB
topic_users                 | 2410166      | 217 MB     | 155 MB     | 372 MB
user_actions                | 1361253      | 101 MB     | 266 MB     | 366 MB
...

```
2019-08-02 17:25:42 +10:00
Vinoth Kannan e44d56e4d2 DEV: raise error only when 'STOP_ON_ERROR' env variable is available. 2019-08-01 23:54:06 +05:30
Daniel Waterworth 20bc4a38a5
FIX: DistributedMutex (#7953) 2019-08-01 09:12:05 +01:00
Guo Xiang Tan a1a61fcd2d DEV: Use `File#file?` in `posts:missing_uploads` rake task. 2019-08-01 10:59:34 +08:00
Rishabh dcb47d902b
REFACTOR: Rename SiteSetting.disable_edit_notifications to disable_system_edit_notifications (#7958)
* REFACTOR: Rename SiteSetting.disable_edit_notifications to disable_system_edit_notifications

- The older name could cause some confusion because the setting does not disable all edit notifications, only system ones.

* FIX: Add frozen_string_literal: true in the migration

* DEV: Deprecate 'disable_edit_notifications'
2019-07-31 20:20:41 +05:30
Penar Musaraj a1fb718340 DEV: Tweak plugin rake task console message
Show "Allowing write to all repos!" message only once.
2019-07-30 09:53:56 -04:00
Guo Xiang Tan 8a64b0c8e8 Revert "DEV: Remove unused kwarg and properly check for local missing uploads."
This reverts commit 97769f3d02.

The code is confusing but this change is quite risky. Defer for now
until we can look at it properly.
2019-07-29 14:35:34 +08:00
Guo Xiang Tan 97769f3d02 DEV: Remove unused kwarg and properly check for local missing uploads. 2019-07-29 14:21:06 +08:00
Guo Xiang Tan e9dc48ccae DEV: Fix multisite support for `posts:missing_uploads`. 2019-07-29 10:56:11 +08:00
Blake Erickson d26aa6e71e REFACTOR: Cleanup rake tasks based on feedback
Follow up to: [FEATURE: Create a rake task for destroying categories][1]

- `Discourse.system_user` is my friend
- Remove puts statements from rake tasks that don't return anything
- `for_each` is also my friend
- Use `human_users` to also exclude discobot
- Sort/format categories:list

[1]: 092eeb5ca3
2019-07-18 19:15:01 -06:00
Blake Erickson 092eeb5ca3 FEATURE: Create a rake task for destroying categories
Created a rake task for destroying multiple categories along with any
subcategories and topics the belong to those categories.

Also created a rake task for listing all of your categories.

Refactored existing destroy rake tasks to use new logging method, that
allows for puts output in the console but prevents it from showing in
the specs.
2019-07-17 12:44:14 -06:00
Rafael dos Santos Silva 1221d34284
FEATURE: Make Discourse work offline with WorkboxJS (#7870) 2019-07-15 13:05:55 -03:00
Gerhard Schlager ce8e099639 FEATURE: Use configured quotation marks in fancy topic title 2019-07-12 21:10:10 +02:00
Joffrey JAFFEUX 550e811652
DEV: allows lodash to be updated with `rake javascript:update` (#7881) 2019-07-11 16:57:03 +02:00
Penar Musaraj befcf67c90 DEV: run db:create and db:migrate on turbo specs
This ensures multisite specs run correctly when running ./bin/turbo_rspec
2019-07-08 14:27:51 -04:00
Sam Saffron b9a2a4d904 DEV: Avoid emoji for symbols
Rake task needs to work across a wide variety of terminal emulators.
Requiring an emoji font is superflous.
2019-07-05 09:41:15 +10:00
Ralph Rooding 1318e0b288 FEATURE: Rake themes installer (#7848)
* Delete remote_theme when deleting the theme

* Install themes and theme components through rake

* Removed unnecessary test
2019-07-04 14:33:05 -04:00
Joffrey JAFFEUX e9bb13c630
DEV: updates jquery-color to prevent it to pull old jquery version (#7857) 2019-07-04 15:03:21 +02:00
Daniel Waterworth d6aa92e98e DEV: Add a verbose option to ./bin/turbo_rspec 2019-06-27 15:49:21 +01:00
Daniel Waterworth b3ec3fac93 FIX: turbo_rspec doesn't accept these options 2019-06-26 17:05:27 +01:00
Daniel Waterworth 68d54d818a DEV: Add USE_TURBO option to docker rake task (#7811) 2019-06-26 11:54:39 -04:00
Guo Xiang Tan d21594f4f7 Revert changes added by mistake in 2b19e2acc8. 2019-06-25 15:25:12 +08:00
Guo Xiang Tan 2b19e2acc8 Fix typo in a0aeabbb94. 2019-06-25 15:18:57 +08:00
Guo Xiang Tan a0aeabbb94 DEV: Safer replacement for old scheme uploads for `posts:missing_uploads`. 2019-06-25 11:34:26 +08:00
Guo Xiang Tan f3c13d80c9 DEV: Don't copy file when trying to recover missing uploads. 2019-06-24 16:32:59 +08:00
Guo Xiang Tan 2cc5561504 DEV: Correct description in `posts:missing_uploads` rake task. 2019-06-24 14:57:43 +08:00
Sam Saffron 5bc92296be DEV: lint a bunch of files we missed 2019-06-21 11:33:41 +10:00
Daniel Waterworth e18ce56f4b DEV: Add a new way to run specs in parallel with better output (#7778)
* DEV: Add a new way to run specs in parallel with better output

This commit:

 1. adds a new executable, `bin/interleaved_rspec` which works much like
    `rspec`, but runs the tests in parallel.

 2. adds a rake task, `rake interleaved:spec` which runs the whole test
    suite.

 3. makes autospec use this new wrapper by default. You can disable this
    by running `PARALLEL_SPEC=0 rake autospec`.

It works much like the `parallel_tests` gem (and relies on it), but
makes each subprocess use a machine-readable formatter and parses this
output in order to provide a better overall summary.

(It's called interleaved, because parallel was taken and naming is
hard).

* Make popen3 invocation safer

* Use FileUtils instead of shelling out

* DRY up reporter

* Moved summary logic into Reporter

* s/interleaved/turbo/g

* Move Reporter into its own file

* Moved run into its own class

* Moved Runner into its own file

* Move JsonRowsFormatter under TurboTests

* Join on threads at the end

* Acted on feedback from eviltrout
2019-06-21 10:59:01 +10:00
Penar Musaraj 4bbb43cb2b DEV: pull plugin repos before linting them 2019-06-14 13:50:15 -04:00
Sam Saffron ecb2fd8222 DEV: improve db:create to catch all cases
- No RAILS_ENV - create multisite / dev / test
- `test` RAILS_ENV - create multisite / test
2019-06-14 15:47:05 +10:00
Sam Saffron a01488ae67 DEV: improve on rake db:create
Followup on 3af00a65 which broke build
2019-06-14 15:06:07 +10:00
Sam Saffron 3af00a65e6 FIX: site settings loading default values when no db
This fixes a condition where an intermittent db connection could cause
invalid site settings to be stored

It also removes a catch all we had.

Somewhere around Rails 5 `db:create` started wanting full environment
this is a problem for Discourse since it needs to boot up data from the
db.

This removes the catch all and surgically adds a db / redis bypass to
db:create task.
2019-06-14 14:21:07 +10:00
Guo Xiang Tan eaa7527933 DEV: Switch `posts:inline_uploads` scope to be more generic. 2019-06-14 11:53:34 +08:00
Penar Musaraj 66b15b9d87 DEV: support installing plugin gems in test images
- adds plugin:install_all_gems rake task

- adds UPDATE_ALL_PLUGINS env variable in docker:test
2019-06-13 09:58:57 -04:00
Guo Xiang Tan 9bce3d5e65 DEV: Multisite support for `rake posts:inline_uploads`. 2019-06-13 16:13:45 +08:00
Sam Saffron 215216df48 FIX: stop re-downloading backup maxmind
We omitted updating the mmdb_time which changed causing a constant
re-download of the maxmind db
2019-06-13 17:42:26 +10:00
Penar Musaraj e47ca6633b DEV: Fix plugin:update_all task
Reenable subtask so it loops through all plugins.

Only run for plugins that are git repositories.
2019-06-12 17:21:56 -04:00
Guo Xiang Tan d49c193289 FIX: `posts:inline_uploads` rake task needs to bypass bump. 2019-06-12 09:13:31 +08:00
Guo Xiang Tan 864222372f Remove incorrectly committed byebug.
Follow up to 49b921147a.
2019-06-11 16:33:49 +08:00
Guo Xiang Tan 49b921147a FIX: Undefined method in `posts:inline_uploads`. 2019-06-11 16:28:41 +08:00
Guo Xiang Tan 1cf3bf016c Make rubocop happy. 2019-06-10 09:49:02 +08:00
Guo Xiang Tan 29259b46ae DEV: Verbose mode for `posts:inline_uploads` rake task. 2019-06-10 08:59:11 +08:00
Joffrey JAFFEUX 8cd13b22a1
DEV: allows smoke test to run on macOS (#7735) 2019-06-07 18:20:48 +02:00
Bianca Nenciu f63b8bb79d FIX: Periodically ensure consistency of categories. (#7663) 2019-06-06 11:30:52 +02:00
Penar Musaraj f00275ded3 FEATURE: Support private attachments when using S3 storage (#7677)
* Support private uploads in S3
* Use localStore for local avatars
* Add job to update private upload ACL on S3
* Test multisite paths
* update ACL for private uploads in migrate_to_s3 task
2019-06-06 13:27:24 +10:00
Gerhard Schlager f7a2648694 FEATURE: Migrate uploads to S3 during restore 2019-06-04 15:47:36 +02:00
Gerhard Schlager fb121815a5 DEV: Allow rebaking oncooked posts of single site 2019-06-04 15:47:36 +02:00
Guo Xiang Tan 1991af2abb DEV: Switch `InlineUploads` to a regexp based implementation. 2019-06-04 15:54:25 +08:00
Guo Xiang Tan d93e5fb00d DEV: Class that converts MD with old attachment links to new MD. 2019-06-04 15:54:25 +08:00
Arpit Jalan af20d616de FEATURE: new rake task to invalidate broken images in post 2019-06-04 12:40:51 +05:30
David Taylor 81dcadf788
DEV: Stop compiling themes during DB migration. Recompile on cdn change. (#7676)
This is an improved implementation for bc8b7b13
2019-06-03 16:38:02 +01:00
Rafael dos Santos Silva 725588f835 FIX: migrate_to_s3 wasn't IAM profile aware 2019-06-01 12:09:46 -03:00
Arpit Jalan 388433ec06 DEV: remove unused method (rake task deleted in 6a274fd) 2019-05-31 11:16:10 +05:30
Arpit Jalan 5c526e5abb FEATURE: new rake task to watch all replied topics 2019-05-31 11:08:00 +05:30
Guo Xiang Tan 6a274fd2cd DEV: Remove redundant rake task.
Onceoff job created in 028121b95b
2019-05-31 10:05:13 +08:00
Joffrey JAFFEUX 630e9814bc
datetime is not available at this point (#7630) 2019-05-29 14:06:32 +02:00
Joffrey JAFFEUX 6439004161
DEV: do not use STDERR to print tests timestamps (#7629) 2019-05-29 13:28:02 +02:00
Joffrey JAFFEUX 5d47d1833b
DEV: prints timestamp at every docker:test commands (#7628) 2019-05-29 12:20:33 +02:00
Vinoth Kannan e12ae453e9 FIX: verify the exitence of s3_object properly without db name 2019-05-29 15:10:36 +05:30
Vinoth Kannan 9a9a06e34b DEV: add option to skip etag verification on 'migrate_to_s3' rake task 2019-05-29 14:16:36 +05:30
Vinoth Kannan b3779dc377 DEV: rename 'uploads:missing' rake task into 'uploads:missing_files'.
To improve the readability.
2019-05-28 23:30:43 +05:30
Ralph Rooding 04abf18194 Fixed typo in S3 rake task 2019-05-28 16:57:55 +08:00
Sam Saffron 9a232e1a0a FEATURE: use GIVE_UP=1 to inform rake posts:missing_uploads you are done
We need this give up for cases where uploads can not be recovered

This also improves the recovery routines
2019-05-28 14:44:59 +10:00
Sam Saffron c780d36ddc DEV: Correct posts:recover_uploads_from_index
Was creating a blank index file by mistake
2019-05-28 12:39:39 +10:00
Sam Saffron 3de8c58134 DEV: clean up backup maxmind path
This follows up on 6580025a

reuses a list and removes a map that was erronuously there.
2019-05-28 11:47:58 +10:00
Sam Saffron 7429700389 FIX: ensure we can download maxmind without redis or db config
This also corrects FileHelper.download so it supports "follow_redirect"
correctly (it used to always follow 1 redirect) and adds a `validate_url`
param that will bypass all uri validation if set to false (default is true)
2019-05-28 10:28:57 +10:00
Sam Saffron e4e2acf148 FIX: rake maxminddb:get broke cause it required environment 2019-05-28 08:59:28 +10:00
Sam Saffron 6580025af9 FEATURE: add backup directory for mmdb files
This new `DISCOURSE_MAXMIND_BACKUP_PATH` can be used a secondary location
for maxmind db. That way a build machine, for example can cache it on the
host and reuse between builds.

Also per 5bfeef77 added proper error raising for download fails from
dedicated rake task

This also moves "refresh_maxmind_db_during_precompile_days" to a global
setting, it did not make sense in a site setting
2019-05-27 16:51:24 +10:00
Régis Hanol 2f5938c3bf DEV: use FileHelper to download MaxMindDB 2019-05-24 18:11:39 +02:00
Régis Hanol 8c9ffee3af TEMP: log more information when failing to download db 2019-05-24 17:33:52 +02:00
Régis Hanol 9ed2c54e27 TEMP: log information when failing to download db 2019-05-24 17:26:04 +02:00
Régis Hanol 0e24cb0f78
DEV: don't skip discourse-calendar plugin specs 2019-05-24 11:35:22 +02:00
Sam Saffron 9ce06d53d1 DEV: improve upload recovery rake task
- Better error handling
- Automatically generates index file if missing
- Improved output
2019-05-24 13:36:03 +10:00
Guo Xiang Tan 791241e0b2 DEV: Fix rake task to delete old export topics. 2019-05-24 11:20:53 +08:00
Sam Saffron 76f7e90e23 FEATURE: rake posts:recover_uploads_from_index improvements
We now recover more smartly based on failed db remaps as well.
2019-05-24 12:38:38 +10:00
Sam Saffron 1059aafc55 FEATURE: rake task for recovery of uploads from an index file
`rake posts:recover_uploads_from_index`

Searches through all missing uploads in the cluster, if it finds one it
tries to find it in the "upload index file" and creates a new upload for
it.
2019-05-23 17:08:22 +10:00
Sam Saffron 7ce58df7bf lint the file
somehow I did not notice this on save
2019-05-23 15:28:41 +10:00
Sam Saffron a5ce9cb470 FEATURE: fix_relative_upload_links now multisite safety
This also finds `<img src="/uploads/xyz` HTML images in raw and corrects
them. Also handles some cross multisite recovery and provides better output
2019-05-23 15:09:16 +10:00
Sam Saffron 96e58125a7 FEATURE: rename task to posts:destroy_old_data_exports
Previously we were only catching one type of data export, the new job will
catch every csv export we have

Job is pretty safe as it filters on system user id / pm with a particular
slug
2019-05-23 15:02:56 +10:00
Sam Saffron 954293655f FEATURE: rake posts:destroy_old_user_data_exports
Historically we would keep the user data export posts around but delete
the uploads.

This leaves a lot of broken uploads in the system.

This rake task allows us to clean up old mess.
2019-05-23 11:11:37 +10:00
Sam Saffron e8799f0ba4 DEV: improve uploads:recover job so it stores a map of old to new sha
Previous attempt created broken images
2019-05-22 15:51:09 +10:00
Sam Saffron ebcb571de7 FIX: allow upload recovery to recover uploads with sha mismatch
Filename on disk may mismatch sha of file in some old 1X setups. This will
attempt to recover file even if sha1 mismatches. We had an old bug that
caused this.

This also adds `uploads:fix_relative_upload_links` which attempts to replace
urls of the format `/upload/default/...` with `upload://`
2019-05-22 15:24:36 +10:00
Sam Saffron f772ecc597 DEV: Correct missing output detailing failure
uploads:s3_migration_status was failing but not returning proper output
2019-05-22 12:58:54 +10:00
Guo Xiang Tan 5429c9b5e9 DEV: Fix incorrectly hardcoded value in rake task. 2019-05-22 09:01:25 +08:00
Sam Saffron a9af77d650 PERF: ensure priority for rake rebake tasks is ultra low
Flooding the default queue with lots of jobs is never a good thing. This
reduces the risk of sidekiq going into a bad state post rebake
2019-05-22 10:31:49 +10:00
Sam Saffron d6daa60d0a FEATURE: multisite support for posts:rebake_uncooked_posts 2019-05-22 10:04:54 +10:00
Sam Saffron a2428bd862 FEATURE: warn about sidekiq overload prior to migrating
Also makes pre-flight check ensure there is no giant backlog of posts that
need to be cooked
2019-05-22 10:04:33 +10:00
Sam Saffron 5fdc7b7ca2 Correct 59012fc0
Logic was flipped here by mistake, oops
2019-05-22 09:48:03 +10:00
Sam Saffron 73f178a634 FEATURE: posts:rebake_uncooked_posts to look at mismatching baked_version
also amends flagging onebox updates to set baked_version to nil
2019-05-22 09:43:31 +10:00
Sam Saffron 4f296608da FEATURE: add uploads:s3_migration_status for looking at current status
Also a few minor cleanups and better progress reporting
2019-05-22 09:00:32 +10:00
Sam Saffron 59012fc0f7 PERF: mark posts for rebake vs forcing a rebake inline when migrating to s3
Rebaking posts can be expensive instead of blocking here simply mark posts
for rebake.

We can then work through them faster in other jobs, plus this should not
hold of a datacenter migration.
2019-05-22 08:39:25 +10:00
Vinoth Kannan 338b927f95 DEV: include local uploads while finding missing uploads
in rake task 'posts:missing_uploads'
2019-05-21 22:07:08 +05:30
Vinoth Kannan 7b82850f66 FIX: migrate_to_s3 task should remap attachment links too. 2019-05-21 21:58:11 +05:30
Sam Saffron 8360415453 FEATURE: big warning for uploads missing which can be very very slow on S3 2019-05-21 16:11:56 +10:00
Sam Saffron cb86d8279a FEATURE: add toggle for uploads:missing so it can skip external
Validating s3 uploads in uploads:missing can be very expensive, allow to
bypass.
2019-05-21 16:11:56 +10:00
Sam Saffron f4d4f7871e FEATURE: make posts:missing_uploads multisite friendly
Previously this rake job would only run on a single site which is a bit
misleading

This also adds `VERBOSE=1 rake posts:missing_uploads` that will provide a
full report of missing uploads
2019-05-21 12:45:51 +10:00
David Taylor 0fbff66d97 DEV: Correct rake task syntax error 2019-05-20 17:43:30 +01:00
David Taylor 31902159af DEV: Allow skipping failed migrations when running S3 migration
Use the SKIP_FAILED environment variable to skip failed sites. Use with caution - make sure you go back and re-run the failed migrations afterwards.
2019-05-20 17:25:56 +01:00
David Taylor a15cca9a0f DEV: Improve error message for posts:missing_uploads during S3 migration 2019-05-20 16:09:22 +01:00
Vinoth Kannan 2bfc0cf145 FIX: skip old scheme upload URLs while validating s3 uploads remap 2019-05-20 19:13:41 +05:30
Vinoth Kannan 9f43852609 FIX: remove the unwanted code. 2019-05-20 19:10:21 +05:30
David Taylor 41bc90dd3e DEV: Add progress indicator for post rebake during s3 migration
Now that we run sidekiq jobs synchronously, this is important
2019-05-20 14:19:58 +01:00
David Taylor 77a06209cb DEV: Skip S3 migration if all uploads are already migrated
This makes the task resumable in a multisite context
2019-05-20 14:17:37 +01:00
Vinoth Kannan bc0c4b7253 FIX: should not migrate the system uploads to s3 2019-05-20 14:27:34 +05:30
Vinoth Kannan be3fb85a04 DEV: add post migration checks and raise error if failed. 2019-05-20 14:18:28 +05:30
Sam Saffron 50e6ee55a9 DEV: add "WAIT_FOR_URL" env to smoke test
This allows you to wait up to N seconds for the smoke test url to come up
in some cases you want to kick off the smoke test prior to having the smoke
test env ready to accept connections
2019-05-20 17:23:09 +10:00
Sam Saffron 08371db0cc FIX: ensure we don't queue any jobs during s3 migration
Previously we could flood sidekiq with jobs which is not ideal.

This ensures we are 100% done when the job is done.
2019-05-20 16:28:50 +10:00
Joffrey JAFFEUX 9ce9d72e71
DEV: makes hidepassed default when running qunit (#7558)
Mostly useful when not running headless, but I endup doing it a lot when debugging, one less thing to check.
2019-05-16 14:37:01 +02:00
Joffrey JAFFEUX 4a9756ff3f
DEV: sets rack server to test env when using rake qunit:test (#7554) 2019-05-16 10:44:29 +02:00
David Taylor 5605dba85c DEV: Automatically annotate plugin models alongside core models 2019-05-13 16:37:47 +01:00
David Taylor 750c125707 DEV: Include foreign keys in model annotations 2019-05-13 15:53:42 +01:00
Guo Xiang Tan c00dab89e4 Fix the build take 2. 2019-05-13 11:22:48 +08:00
Guo Xiang Tan 8c4caac2e4 Fix frozen string error in `plugin:install_all_official`. 2019-05-13 10:19:02 +08:00
Guo Xiang Tan 8165ceb320 Make rubocop happy. 2019-05-13 09:55:44 +08:00
Sam Saffron 30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
David Taylor 20daa76829 DEV: Change priority to ultra_low for post-s3-migration rebake 2019-05-10 18:37:45 +08:00
Guo Xiang Tan 626808e100 Fix incorrect condition in `groups:grant_badge` rake task.
`#find` raises an error if the id given to it is invalid. As a result,
the conditional to check whether a `group` or `badge` is `present?` will
not be executed if any of the ids are invalid.

Follow up to
6ba914033c.
2019-05-09 14:53:13 +08:00
Vinoth Kannan 87cd4701b8 FEATURE: option to skip posts with ignored missing uploads 2019-05-09 05:11:15 +05:30
Penar Musaraj a92c925378 Use faster Brotli compression for non-default locale files 2019-05-08 14:55:24 -04:00
Penar Musaraj d9ab722d05 Revert "Revert "PERF: Skip compressing locales for faster rebuilds (#7501)" (#7502)"
This reverts commit 6ea811c923.
2019-05-08 14:55:06 -04:00
Joffrey JAFFEUX 6ea811c923
Revert "PERF: Skip compressing locales for faster rebuilds (#7501)" (#7502) 2019-05-08 07:54:21 +02:00
Penar Musaraj 22d6203eb7 DEV: Use moment-timezone with 10-year range data
We do not need to use the full timezone dataset (it is a 950kb file!), only previous/next 5 years should suffice.
2019-05-07 23:42:49 -04:00
Sam Saffron 5c0bb29461 DEV: re-enable compression for node asset compilation
#b9d82818 makes enormous improvements to our bootstrap time, however going
to still keep compress for now despite the cost and watch it for a few weeks
2019-05-08 10:24:57 +10:00
Penar Musaraj b9d8281818 PERF: Skip compressing locales for faster rebuilds (#7501)
* Do not brotli all locales in precompile

* Try without gzip

* uglify without compressing, always gzip

* skip uglify for unused locales

* FIX: Uglifier needs harmony for ES6 compatibility

* Use node uglifier if available

* Minor refactor
2019-05-08 10:22:19 +10:00
Guo Xiang Tan 2adbec1b3c PERF: Speed up migrate_to_s3 rake task by deleting optimized image record.
No point moving all optimized image files to tombstone when the store is
changing. Also, `destroy_all` can easily blow memory since we are no
loading in batches.
2019-05-07 16:10:32 +08:00
Sam Saffron 9be70a22cd DEV: introduce new API to look up dynamic site setting
This removes all uses of both `send` and `public_send` from consumers of
SiteSetting and instead introduces a `get` helper for dynamic lookup

This leads to much cleaner and safer code long term as we are always explicit
to test that a site setting is really there before sending an arbitrary
string to the class

It also removes a couple of risky stubs from the auth provider test
2019-05-07 11:00:30 +10:00
Arpit Jalan 6ba914033c FEATURE: rake task to bulk grant badge to members of a group 2019-05-06 12:41:50 +05:30
Vinoth Kannan e8fafbc123 List and restore missing post uploads from S3 inventory. 2019-05-04 01:16:20 +05:30
Penar Musaraj a5b804fec4 DEV: Add --profile and --fail-fast when running plugin specs, tweak parallel test balancing 2019-05-01 22:47:18 -04:00
David Taylor 0e303c7f5d
FEATURE: Automatically generate optimized site metadata icons (#7372)
This change automatically resizes icons for various purposes. Admins can now upload `logo` and `logo_small`, and everything else will be auto-generated. Specific icons can still be uploaded separately if required.

## Core

- Adds an SiteIconManager module which manages automatic resizing and fallback

- Icons are looked up in the OptimizedImage table at runtime, and then cached in Redis. If the resized version is missing for some reason, then most icons will fall back to the original files. Some icons (e.g. PWA Manifest) will return `nil` (because an incorrectly sized icon is worse than a missing icon). 

- `SiteSetting.site_large_icon_url` will return the optimized version, including any fallback. `SiteSetting.large_icon` continues to return the upload object. This means that (almost) no changes are required in core/plugins to support this new system.

- Icons are resized whenever a relevant site setting is changed, and during post-deploy migrations

## Wizard

- Allows `requiresRefresh` wizard steps to reload data via AJAX instead of a full page reload

- Add placeholders to the **icons** step of the wizard, which automatically update from the "Square Logo"

- Various copy updates to support the changes

- Remove the "upload-time" resizing for `large_icon`. This is no longer required.

## Site Settings UX

- Move logo/icon settings under a new "Branding" tab

- Various copy changes to support the changes

- Adds placeholder support to the `image-uploader` component

- Automatically reloads site settings after saving. This allows setting placeholders to change based on changes to other settings

- Upload site settings will be assigned a placeholder if SiteIconManager `responds_to?` an icon of the same name

## Dashboard Warnings

- Remove PWA icon and PWA title warnings. Both are now handled automatically.

## Bonus

- Updated the sketch logos to use @awesomerobot's new high-res designs
2019-05-01 14:44:45 +01:00
David Taylor bc8b7b13ef DEV: Recompile theme fields after db:migrate 2019-04-30 11:52:30 +01:00
Vinoth Kannan c448e014fe Recover missing old scheme uploads from tombstone directory too. 2019-04-29 11:33:57 +05:30
Vinoth Kannan 8698a10920 Don't raise error if a record already exists. 2019-04-25 21:48:39 +05:30
Arpit Jalan 5a70367a05 fix the build 2019-04-25 20:19:52 +05:30