Commit Graph

924 Commits

Author SHA1 Message Date
Sam Saffron a9af77d650 PERF: ensure priority for rake rebake tasks is ultra low
Flooding the default queue with lots of jobs is never a good thing. This
reduces the risk of sidekiq going into a bad state post rebake
2019-05-22 10:31:49 +10:00
Sam Saffron d6daa60d0a FEATURE: multisite support for posts:rebake_uncooked_posts 2019-05-22 10:04:54 +10:00
Sam Saffron a2428bd862 FEATURE: warn about sidekiq overload prior to migrating
Also makes pre-flight check ensure there is no giant backlog of posts that
need to be cooked
2019-05-22 10:04:33 +10:00
Sam Saffron 5fdc7b7ca2 Correct 59012fc0
Logic was flipped here by mistake, oops
2019-05-22 09:48:03 +10:00
Sam Saffron 73f178a634 FEATURE: posts:rebake_uncooked_posts to look at mismatching baked_version
also amends flagging onebox updates to set baked_version to nil
2019-05-22 09:43:31 +10:00
Sam Saffron 4f296608da FEATURE: add uploads:s3_migration_status for looking at current status
Also a few minor cleanups and better progress reporting
2019-05-22 09:00:32 +10:00
Sam Saffron 59012fc0f7 PERF: mark posts for rebake vs forcing a rebake inline when migrating to s3
Rebaking posts can be expensive instead of blocking here simply mark posts
for rebake.

We can then work through them faster in other jobs, plus this should not
hold of a datacenter migration.
2019-05-22 08:39:25 +10:00
Vinoth Kannan 338b927f95 DEV: include local uploads while finding missing uploads
in rake task 'posts:missing_uploads'
2019-05-21 22:07:08 +05:30
Vinoth Kannan 7b82850f66 FIX: migrate_to_s3 task should remap attachment links too. 2019-05-21 21:58:11 +05:30
Sam Saffron 8360415453 FEATURE: big warning for uploads missing which can be very very slow on S3 2019-05-21 16:11:56 +10:00
Sam Saffron cb86d8279a FEATURE: add toggle for uploads:missing so it can skip external
Validating s3 uploads in uploads:missing can be very expensive, allow to
bypass.
2019-05-21 16:11:56 +10:00
Sam Saffron f4d4f7871e FEATURE: make posts:missing_uploads multisite friendly
Previously this rake job would only run on a single site which is a bit
misleading

This also adds `VERBOSE=1 rake posts:missing_uploads` that will provide a
full report of missing uploads
2019-05-21 12:45:51 +10:00
David Taylor 0fbff66d97 DEV: Correct rake task syntax error 2019-05-20 17:43:30 +01:00
David Taylor 31902159af DEV: Allow skipping failed migrations when running S3 migration
Use the SKIP_FAILED environment variable to skip failed sites. Use with caution - make sure you go back and re-run the failed migrations afterwards.
2019-05-20 17:25:56 +01:00
David Taylor a15cca9a0f DEV: Improve error message for posts:missing_uploads during S3 migration 2019-05-20 16:09:22 +01:00
Vinoth Kannan 2bfc0cf145 FIX: skip old scheme upload URLs while validating s3 uploads remap 2019-05-20 19:13:41 +05:30
Vinoth Kannan 9f43852609 FIX: remove the unwanted code. 2019-05-20 19:10:21 +05:30
David Taylor 41bc90dd3e DEV: Add progress indicator for post rebake during s3 migration
Now that we run sidekiq jobs synchronously, this is important
2019-05-20 14:19:58 +01:00
David Taylor 77a06209cb DEV: Skip S3 migration if all uploads are already migrated
This makes the task resumable in a multisite context
2019-05-20 14:17:37 +01:00
Vinoth Kannan bc0c4b7253 FIX: should not migrate the system uploads to s3 2019-05-20 14:27:34 +05:30
Vinoth Kannan be3fb85a04 DEV: add post migration checks and raise error if failed. 2019-05-20 14:18:28 +05:30
Sam Saffron 50e6ee55a9 DEV: add "WAIT_FOR_URL" env to smoke test
This allows you to wait up to N seconds for the smoke test url to come up
in some cases you want to kick off the smoke test prior to having the smoke
test env ready to accept connections
2019-05-20 17:23:09 +10:00
Sam Saffron 08371db0cc FIX: ensure we don't queue any jobs during s3 migration
Previously we could flood sidekiq with jobs which is not ideal.

This ensures we are 100% done when the job is done.
2019-05-20 16:28:50 +10:00
Joffrey JAFFEUX 9ce9d72e71
DEV: makes hidepassed default when running qunit (#7558)
Mostly useful when not running headless, but I endup doing it a lot when debugging, one less thing to check.
2019-05-16 14:37:01 +02:00
Joffrey JAFFEUX 4a9756ff3f
DEV: sets rack server to test env when using rake qunit:test (#7554) 2019-05-16 10:44:29 +02:00
David Taylor 5605dba85c DEV: Automatically annotate plugin models alongside core models 2019-05-13 16:37:47 +01:00
David Taylor 750c125707 DEV: Include foreign keys in model annotations 2019-05-13 15:53:42 +01:00
Guo Xiang Tan c00dab89e4 Fix the build take 2. 2019-05-13 11:22:48 +08:00
Guo Xiang Tan 8c4caac2e4 Fix frozen string error in `plugin:install_all_official`. 2019-05-13 10:19:02 +08:00
Guo Xiang Tan 8165ceb320 Make rubocop happy. 2019-05-13 09:55:44 +08:00
Sam Saffron 30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
David Taylor 20daa76829 DEV: Change priority to ultra_low for post-s3-migration rebake 2019-05-10 18:37:45 +08:00
Guo Xiang Tan 626808e100 Fix incorrect condition in `groups:grant_badge` rake task.
`#find` raises an error if the id given to it is invalid. As a result,
the conditional to check whether a `group` or `badge` is `present?` will
not be executed if any of the ids are invalid.

Follow up to
6ba914033c.
2019-05-09 14:53:13 +08:00
Vinoth Kannan 87cd4701b8 FEATURE: option to skip posts with ignored missing uploads 2019-05-09 05:11:15 +05:30
Penar Musaraj a92c925378 Use faster Brotli compression for non-default locale files 2019-05-08 14:55:24 -04:00
Penar Musaraj d9ab722d05 Revert "Revert "PERF: Skip compressing locales for faster rebuilds (#7501)" (#7502)"
This reverts commit 6ea811c923.
2019-05-08 14:55:06 -04:00
Joffrey JAFFEUX 6ea811c923
Revert "PERF: Skip compressing locales for faster rebuilds (#7501)" (#7502) 2019-05-08 07:54:21 +02:00
Penar Musaraj 22d6203eb7 DEV: Use moment-timezone with 10-year range data
We do not need to use the full timezone dataset (it is a 950kb file!), only previous/next 5 years should suffice.
2019-05-07 23:42:49 -04:00
Sam Saffron 5c0bb29461 DEV: re-enable compression for node asset compilation
#b9d82818 makes enormous improvements to our bootstrap time, however going
to still keep compress for now despite the cost and watch it for a few weeks
2019-05-08 10:24:57 +10:00
Penar Musaraj b9d8281818 PERF: Skip compressing locales for faster rebuilds (#7501)
* Do not brotli all locales in precompile

* Try without gzip

* uglify without compressing, always gzip

* skip uglify for unused locales

* FIX: Uglifier needs harmony for ES6 compatibility

* Use node uglifier if available

* Minor refactor
2019-05-08 10:22:19 +10:00
Guo Xiang Tan 2adbec1b3c PERF: Speed up migrate_to_s3 rake task by deleting optimized image record.
No point moving all optimized image files to tombstone when the store is
changing. Also, `destroy_all` can easily blow memory since we are no
loading in batches.
2019-05-07 16:10:32 +08:00
Sam Saffron 9be70a22cd DEV: introduce new API to look up dynamic site setting
This removes all uses of both `send` and `public_send` from consumers of
SiteSetting and instead introduces a `get` helper for dynamic lookup

This leads to much cleaner and safer code long term as we are always explicit
to test that a site setting is really there before sending an arbitrary
string to the class

It also removes a couple of risky stubs from the auth provider test
2019-05-07 11:00:30 +10:00
Arpit Jalan 6ba914033c FEATURE: rake task to bulk grant badge to members of a group 2019-05-06 12:41:50 +05:30
Vinoth Kannan e8fafbc123 List and restore missing post uploads from S3 inventory. 2019-05-04 01:16:20 +05:30
Penar Musaraj a5b804fec4 DEV: Add --profile and --fail-fast when running plugin specs, tweak parallel test balancing 2019-05-01 22:47:18 -04:00
David Taylor 0e303c7f5d
FEATURE: Automatically generate optimized site metadata icons (#7372)
This change automatically resizes icons for various purposes. Admins can now upload `logo` and `logo_small`, and everything else will be auto-generated. Specific icons can still be uploaded separately if required.

## Core

- Adds an SiteIconManager module which manages automatic resizing and fallback

- Icons are looked up in the OptimizedImage table at runtime, and then cached in Redis. If the resized version is missing for some reason, then most icons will fall back to the original files. Some icons (e.g. PWA Manifest) will return `nil` (because an incorrectly sized icon is worse than a missing icon). 

- `SiteSetting.site_large_icon_url` will return the optimized version, including any fallback. `SiteSetting.large_icon` continues to return the upload object. This means that (almost) no changes are required in core/plugins to support this new system.

- Icons are resized whenever a relevant site setting is changed, and during post-deploy migrations

## Wizard

- Allows `requiresRefresh` wizard steps to reload data via AJAX instead of a full page reload

- Add placeholders to the **icons** step of the wizard, which automatically update from the "Square Logo"

- Various copy updates to support the changes

- Remove the "upload-time" resizing for `large_icon`. This is no longer required.

## Site Settings UX

- Move logo/icon settings under a new "Branding" tab

- Various copy changes to support the changes

- Adds placeholder support to the `image-uploader` component

- Automatically reloads site settings after saving. This allows setting placeholders to change based on changes to other settings

- Upload site settings will be assigned a placeholder if SiteIconManager `responds_to?` an icon of the same name

## Dashboard Warnings

- Remove PWA icon and PWA title warnings. Both are now handled automatically.

## Bonus

- Updated the sketch logos to use @awesomerobot's new high-res designs
2019-05-01 14:44:45 +01:00
David Taylor bc8b7b13ef DEV: Recompile theme fields after db:migrate 2019-04-30 11:52:30 +01:00
Vinoth Kannan c448e014fe Recover missing old scheme uploads from tombstone directory too. 2019-04-29 11:33:57 +05:30
Vinoth Kannan 8698a10920 Don't raise error if a record already exists. 2019-04-25 21:48:39 +05:30
Arpit Jalan 5a70367a05 fix the build 2019-04-25 20:19:52 +05:30
Vinoth Kannan a65ce81e25 Don't check same upload url twice in the rake task 2019-04-25 20:06:10 +05:30
Arpit Jalan 0d331d33c3 DEV: do not use find_each for `avatars:clean` task
https://review.discourse.org/t/fix-show-available-details-when-rescuing-error/2865
2019-04-25 18:46:34 +05:30
Vinoth Kannan 1724c27713 FIX: reload the 'post' model to retrive raw field value. 2019-04-25 02:09:27 +05:30
Vinoth Kannan aed683390c FIX: Don't treat 'upload_patterns' as constant to make the rake task compatible with multisite 2019-04-25 02:06:20 +05:30
Vinoth Kannan b0bb1d95a5 trying to recover old scheme uploads on 'posts:missing_uploads' rake task 2019-04-25 01:16:18 +05:30
Vinoth Kannan 3dae12dec1 reduce unlinked count from missing post upload count 2019-04-19 21:53:00 +05:30
Vinoth Kannan 4fcefc3615 add unlinked missing post upload count in stat 2019-04-19 20:39:21 +05:30
Vinoth Kannan ba6369edc5 Add more missing upload stats in rake task 2019-04-19 20:14:40 +05:30
Vinoth Kannan 48cedf3ebe fix: retrieve posts in order by id.
so we can find starting and ending id of the batch.
2019-04-19 18:30:59 +05:30
Vinoth Kannan ce8d30f667 PERF: use 'delete_all' method instead of 'destroy_all' 2019-04-19 18:00:01 +05:30
Vinoth Kannan 5de483a1b9 PERF: Very post upload's existence by preloaded upload sha1s array 2019-04-19 17:27:16 +05:30
Arpit Jalan 989325ea7c FIX: show available details when rescuing error 2019-04-15 14:09:52 +05:30
Arpit Jalan 07b961ab13 DEV: rescue avatar rake tasks with error backtrace 2019-04-15 14:06:29 +05:30
Arpit Jalan a423e9b14a FIX: rescue `avatars:refresh` rake task 2019-04-14 14:26:15 +05:30
David Taylor 24ef4f7b2b Remove support for disable_jump_reply user setting (#7359) 2019-04-12 09:03:06 +10:00
Sam Saffron 84f12ac3f7 DEV: No need to carry two brotli implementations around
old version of brotli binary is long gone, require brotli 1.0.0 and up
2019-04-11 16:53:29 +10:00
Arpit Jalan a20f58554b IMPORT: create category definitions in `import:ensure_consistency` task 2019-04-11 12:06:37 +05:30
Sam Saffron baa7a9836c FEATURE: remove "COMPRESS_BROTLI" optional behavior
The compress brotli functionality is no longer optional, this has worked
well for years. The name of the ENV var is also confusing cause it does
not have a `DISCOURSE_` prefix which caused issues with the web upgrader

Brotli support is now unconditionally on
2019-04-11 12:36:18 +10:00
Vinoth Kannan 8d5c900142 DEV: add unique missing uploads index in post custom fields
https://review.discourse.org/t/feature-send-missing-post-uploads-stat-to-prometheus/2609/6?u=vinothkannans
2019-04-10 18:09:35 +05:30
Bianca Nenciu c523a12187 FIX: Fix assets:precompile rake task. 2019-04-10 14:03:26 +03:00
Bianca Nenciu 4555d0c598 FEATURE: Refresh MaxmindDb during assets:precompile. (#7340) 2019-04-10 11:37:29 +02:00
Bianca Nenciu b5008586c5 DEV: Remove HTML parser from Tautologistics. (#7344) 2019-04-10 11:21:22 +02:00
Vinoth Kannan d0fe42e2ef FIX: should look through posts for image markdown
Downloaded onebox images only included in the cooked HTML content. So we have to check 'post.cooked' instead of 'raw'. bfdd0fe64c
2019-04-10 13:52:35 +05:30
Vinoth Kannan 4878ee95e5 DEV: optimize sql query to narrow down the filtering of post with uploads 2019-04-10 12:58:10 +05:30
Vinoth Kannan bfdd0fe64c FIX: empty the missing list on each post loop 2019-04-10 11:20:26 +05:30
Vinoth Kannan 914ada1c74 DEV: convert scheduled job EnsurePostUploadsExistence into a rake task 2019-04-09 02:07:35 +05:30
Robin Ward b58867b6e9 FEATURE: New 'Reviewable' model to make reviewable items generic
Includes support for flags, reviewable users and queued posts, with REST API
backwards compatibility.

Co-Authored-By: romanrizzi <romanalejandro@gmail.com>
Co-Authored-By: jjaffeux <j.jaffeux@gmail.com>
2019-03-28 12:45:10 -04:00
Guo Xiang Tan 71e431de9e DEV: Fix hardcoded value introduced in cc2bac86e9. 2019-03-26 07:45:21 +08:00
Gerhard Schlager cd4fd447ca Make Rubocop happy 2019-03-25 17:04:45 +01:00
Guo Xiang Tan cc2bac86e9 FIX: Dry run broken for `uploads:migrate_to_s3`. 2019-03-25 22:38:24 +08:00
Guo Xiang Tan 19c3c25db1 FIX: Handle BBCode in `migrate_to_s3` task as well. 2019-03-22 16:47:06 +08:00
Guo Xiang Tan 2d34be24be FIX: Rebake lightbox and use short upload urls in `migrate_to_s3`. 2019-03-22 13:09:59 +08:00
Joffrey JAFFEUX c18a6069c2
DEV: upgrades bootstrap-modal (#7235) 2019-03-21 14:18:12 +01:00
Guo Xiang Tan 4e594f2b2b FIX: Destroy optimized images in `upload:migrate_to_s3` rake task.
`OptimizedImage` are currently not renegerated when the image changes
store.
2019-03-21 16:50:15 +08:00
Guo Xiang Tan 8e5e5d7d35 DEV: Make setting up of multisite DB in test env clearer. 2019-03-21 09:58:07 +08:00
Sam Saffron 600313373b DEV: skip creating multisite db when running JS tests only 2019-03-21 12:19:12 +11:00
Penar Musaraj ed74774001 DEV: rebalance parallel tests (again) 2019-03-19 10:21:50 -04:00
Penar Musaraj 60b6c67c1c DEV: rebalance parallel tests (take 2) 2019-03-19 09:59:03 -04:00
Penar Musaraj c8e1d3c2e7 DEV: rebalance parallel tests 2019-03-19 09:23:10 -04:00
Gerhard Schlager 8a99d59422 FIX: seed data classes couldn't always be found 2019-03-18 23:13:47 +01:00
Gerhard Schlager 3fd04df781
FEATURE: Locale support for seeded categories and topics (#7110) 2019-03-18 21:09:13 +01:00
Joffrey JAFFEUX d91b47064e
FIX: windows has no country flags (#7199) 2019-03-18 19:28:49 +01:00
Gerhard Schlager 81108ee783 DEV: Increase timeout for JS tests and add env variable 2019-03-18 17:39:53 +01:00
Régis Hanol 9dbca41152 FIX: don't check system uploads in migrate_to_s3 rake task 2019-03-18 11:01:18 +01:00
Joffrey JAFFEUX e1f5620b04
do not skip until we re-merge ember 3.7 (#7183) 2019-03-15 18:31:24 +01:00
Penar Musaraj b43f1687ee FIX: Update user options in rake task 2019-03-15 11:41:16 -04:00
Joffrey JAFFEUX aa4b84925a FEATURE: updates emoji to latest 12.0 version (#7095)
* improved emoji support

- always optimize images as part of the task
- use the unicode standard ordering/naming for sections

* UX: more height for when there are recently used
2019-03-15 16:12:07 +01:00
Régis Hanol a6a52d1523 FIX: images:compress rake task
Optimized a few unoptimized PNGs.
2019-03-15 15:23:57 +01:00
Kelvin 45a97ad82f Fix SMTP connection test
This test did not support 'no auth' use case and other auth methods except 'login'. I fixed it by simply making the call to start() in the right way. 
As shown in the source code of Net::SMTP (https://github.com/ruby/ruby/blob/ruby_2_5/lib/net/smtp.rb#L452), the start() function does accept the 'user' and 'secret' arguments. Also, in do_start() function (https://github.com/ruby/ruby/blob/ruby_2_5/lib/net/smtp.rb#L542), it automatically checks the auth method and args, skips the authentication if 'user' is not provided, and selects the right auth method from 'plain', 'login' or 'cram_md5'. This is exactly all of what we should do in a connection test and the odd 'auth_login' call in the previous code makes problems. 
BTW, I am using 'localhost' as the third argument, which is the same as the default value in start(). This parameter is the domain address sent along with the 'ehlo' command in SMTP protocol. I have seen some documents, e.g. https://github.com/tpn/msmtp/blob/master/doc/msmtp.1#L455, saying that 'localhost' is fine. It works for me.
2019-03-15 09:50:51 -04:00
Joffrey JAFFEUX 9a5f08ee61
DEV: skip discourse-chat-integration tests (#7164)
Suspected of causing timeout issues in our test suite with Ember 3.7
2019-03-13 16:50:56 +01:00
Penar Musaraj 49cdb072d7 DEV: Use `--profile` and `--fail-fast` in CI only 2019-03-11 22:04:47 -04:00
Guo Xiang Tan d82876896e FIX: `uploads:migrate_to_s3` broken for GlobalSetting using file provider. 2019-03-11 14:21:35 +08:00
David Taylor bc52437153 FIX: Add additional rescue for assets:precompile
This error can be caused if assets are precompiled before the database has been migrated
2019-03-07 14:03:05 +00:00
Régis Hanol ad87b0d662
Make "uploads:recover_from_tombstone" call the "uploads:recover" rake task 2019-03-07 14:15:30 +01:00
Penar Musaraj edffd0097f
DEV: Support running Ruby tests in parallel (#7087) 2019-02-28 16:36:22 -05:00
Penar Musaraj 9bf11a7c02
FEATURE: Add localized timezone names (#7004)
Uses github.com/discourse/moment-timezone-names-translations to translate timezone names.
Plugins can also provide their own timezone name translations.
2019-02-25 14:40:02 -05:00
Guo Xiang Tan 433f07fcb3 Fix confusing ENV in `migrate_to_s3` rake task. 2019-02-22 14:44:46 +08:00
Rishabh e69634ec3a FIX: Use s3_endpoint in migrate_to_s3 when not using S3
- overrides :region and uses :endpoint when SiteSetting.s3_endpoint is provided
- Now, we can use the new rake task with DigitalOcean Spaces
- I've tested that it's compatible with/without bucket folder path
- I've tested that it's compatible with S3 and it doesn't break S3 for non-default regions
- follow-up on 97e17fe0
2019-02-21 15:42:48 +05:30
Vinoth Kannan 0472bd4adc FIX: Remove 'backfill_etags' keyword argument from 'uploads:missing' rake task
And etags backfilling code is optimized
2019-02-15 00:34:35 +05:30
Penar Musaraj e55c19d8ce DEV: update ACE Editor to 1.4.2
Remove duplicate "src-min" folder and fix rake task

Remove unused "snippets" folder
2019-02-14 11:05:28 -05:00
Penar Musaraj 090e9c8432
FIX: make composer full screen shortcut work when inputs have focus (#6907)
- Uses a Mousetrap plugin for global shortcuts
- Implemented for search `ctrl+alt+f` and composer fullscreen `shift+f11` shortcuts
2019-02-14 00:19:27 -05:00
Vinoth Kannan 7b5931013a Update rake task to backfill etags from s3 inventory 2019-02-14 05:18:06 +05:30
Penar Musaraj 5cef5b34ef DEV: Yarn-manage moment and moment-timezone libraries 2019-02-12 13:57:52 -05:00
Joffrey JAFFEUX 831d7d2f87
run tests on discourse-code-review (#6988) 2019-02-08 11:39:45 +01:00
Penar Musaraj 995c87a91e Don't run specs on code-review again 2019-02-07 16:57:05 -05:00
Penar Musaraj d639cadb7a
DEV: Plugin scss errors should break precompile (#6974) 2019-02-07 09:27:42 -05:00
David Taylor 68173cd234 Revert "DEV: Allow rake plugin:spec to traverse symlinks"
This reverts commit dfd63b185f.

https://meta.discourse.org/t/108110/11
2019-02-01 12:40:29 +00:00
David Taylor dfd63b185f DEV: Allow rake plugin:spec to traverse symlinks 2019-02-01 10:40:41 +00:00
Arpit Jalan 82b7795f36 Revert "don't run specs on code-review for now"
This reverts commit cb493b66e0.
2019-01-29 11:49:42 +05:30
Joshua Rosenfeld cb493b66e0
don't run specs on code-review for now
code-review is currently causing failing builds. Remove from testing until it is fixed
2019-01-24 11:48:19 -05:00
Guo Xiang Tan 0cf2df3028 Fix remap in `migrate_to_s3` rake task.
The current way of doing the remap only allows to run the rake task
once. Running the rake task more than once will end up badly.
2019-01-23 15:50:44 +08:00
Guo Xiang Tan 07850994d3 Add ENV to skip multisite prefix when migrating to s3. 2019-01-23 15:19:50 +08:00
Guo Xiang Tan 979d03aa68 Remove s3 bucket check in `migrate_to_s3` task.
Bucket creation is expected to be handled by the user. If the bucket
does not exist, the script will fail anyway.
2019-01-23 15:04:51 +08:00
Guo Xiang Tan 99cd3ff6ee FIX: `migrate_to_s3` task not setting the right content_disposition. 2019-01-23 15:04:47 +08:00
Rishabh 97e17fe084 FIX: Use ENV values instead of 'S3Helper.s3_options' in migrate_to_s3
This commit makes the rake task operational for all regions for s3. If we declare s3_endpoint as https://s3.amazonaws.com while
creating an instance of Aws::S3::Client, head_bucket fails for all s3 regions apart from us-east-1. The commit manually defines all
parameters for Aws::S3::Client apart from s3_endpoint to bypass this problem make this task usable for AWS S3.

Removing s3_endpoint from the payload means that custom endpoints like Minio/DO Spaces for will not work in the meantime and we'll
have to add support for a custom `s3_endpoint` in the future.

This commit follows up on 60790eb0.
2019-01-20 20:55:27 +05:30
Penar Musaraj 020e0d623a DEV: run "yarn install" before lint tests 2019-01-17 14:17:44 -05:00
Michael Brown 664c282400 FIX: improve rake emails:test task
* FIX: rake emails:test to bypass the "STARTTLS required" message

* FIX: too much j

* hint meta.discourse.org to people with unknown errors from the mail test
2019-01-17 13:41:10 +11:00
Rishabh 60790eb006 FIX: Use GlobalSetting values instead of ENV variables in migrate_to_s3
TIL how GlobalSetting works in sync with environment variables
Also fixes a small bug where bucket value was being used when it could have been nil
2019-01-16 14:40:38 +05:30
Rishabh ff8f9dc1c9 FIX: prefix should precede folder path (follow-up on 10fbb07e) 2019-01-15 15:58:19 +05:30
Rishabh 10fbb07e1a FIX: include folder name in prefix for listing files on S3 (follow-up on 3ec38f5a)
Fix the destination url in remap since it's already a part of s3_base_url
2019-01-15 15:23:55 +05:30
Penar Musaraj b28d68efbf DEV: Use yarn to manage Highlight.js dependency
Moves Highlight.js files to vendor/assets/javascripts

Adds Highlight.js in yarn package management

Removes old rake task and reliance on NPM to build Highlight.js

Highlight.js is now integrated in the "javascript:update" rake task
2019-01-14 12:46:35 -05:00
Florian Heidenreich 39983f9d95 FIX: list staff users within the last month 2019-01-14 10:29:00 -05:00
Régis Hanol 3ec38f5a3b Revert "FIX: migrate_to_s3 rake task with folder path"
This reverts commit 97fd12e8af.
2019-01-08 19:44:31 +01:00
Régis Hanol 97fd12e8af
FIX: migrate_to_s3 rake task with folder path 2019-01-08 18:56:18 +01:00
Rishabh f181e9cc08
FIX: Add compatibility for bucket folder paths in migrate_to_s3 task (#6855)
* FIX: Add compatibility for bucket folder paths in migrate_to_s3 task
* Refactor bucket_name split logic into S3Helper
2019-01-08 20:04:48 +05:30
Rishabh efc481d9c0 DEV: Use puts instead of printing newline (follow up on c5b7bda1) 2019-01-05 01:20:00 +05:30
Rishabh c5b7bda198 DEV: Show migrate_to_s3 output on a new line 2019-01-04 18:09:54 +05:30
Vinoth Kannan 902f535111 FIX: upload method in S3Helper will expect a file object param 2019-01-04 15:30:45 +05:30
Sam e2dca641c6 handle exceptions in s3:correct_acl task
We need to handle arbitrary exceptions in this task, especially since the
task is not easily resumable.

Simply output problem uploads as you hit them for now.
2019-01-04 08:32:09 +11:00
Sam 05a3e3670f FEATURE: add rake task that resets ACL on every object in S3
Some previous migrations to S3 may have bad ACLs set on objects. This
introduces a new rake task (`rake s3:correct_acl`) that will reset ACL on
every S3 object.

Vast majority of users will never have to run it, but if you have ACL issues
this is the atomic solution.
2019-01-04 08:13:43 +11:00
Gerhard Schlager a474bf966c DEV: Skip prettier in single plugin test if patterns aren't found
This reverts d06ca90c94
It didn't work because yarn doesn't return the original exit code of the failed command.
2019-01-02 14:36:09 +01:00
Gerhard Schlager d06ca90c94 DEV: Ignore prettier error when there are no ES6 files 2019-01-02 11:30:09 +01:00
Guo Xiang Tan 9e50813252 FIX: Pass all necessary options in `migrate_to_s3` rake task. 2019-01-02 09:11:23 +08:00
Régis Hanol 5381096bfd PERF: new 'migrate_to_s3' rake task 2018-12-26 17:34:49 +01:00
Sam Saffron abbbcb2622 DEV: allow creating users with random passwords via rake
Use: `RANDOM_PASSWORD=1 bin/rake admin:create`

Handy in conjunction with dev mode /session/username/become.
2018-12-18 11:43:16 +08:00
David Taylor 77fe57f923 FEATURE: Rake task to list users which have been staff in the past month 2018-12-14 16:29:12 +00:00
Sam 0ca61242b8 FEATURE: polyfill intersection observer for IE11 / iOS Safari
This feature is used for defer loading of images and in future for post cloaking

This gives us a polyfill so we can safely use the feature in problem browsers

The polyfill supports "polling" but it does not appear we need it yet.

If we discover anything odd here, consider setting poll interval per:

https://github.com/w3c/IntersectionObserver/tree/master/polyfill

```
var io = new IntersectionObserver(callback);
io.POLL_INTERVAL = 100; // Time in milliseconds.
```

Keeping the mutation observer cause we often mutate the DOM
2018-12-12 15:36:08 +11:00
Rishabh 314c084e5d DEV: nuke code dependent comments from scheduler.rake 2018-12-10 16:16:19 +05:30
Rishabh 4767a5a26e
DEV: Update incorrect Job frequency 2018-12-10 13:26:19 +05:30
Régis Hanol 3c9c95ac83 Update Rubocop to 0.60 2018-12-04 10:48:16 +01:00
David Taylor 404acef6e3 DEV: Move `run-qunit.js` out of the vendor directory 2018-12-03 16:16:37 +00:00
Penar Musaraj 03deda2147
Upgrade to FontAwesome 5 (take two) (#6673)
* Add missing icons to set

* Revert FA5 revert

 This reverts commit 42572ff

* use new SVG syntax in locales

* Noscript page changes (remove login button, center "powered by" footer text)

* Cast wider net for SVG icons in settings

- include any _icon setting for SVG registry (offers better support for plugin settings)

- let themes store multiple pipe-delimited icons in a setting

- also replaces broken onebox image icon with SVG reference in cooked post processor

* interpolate icons in locales

* Fix composer whisper icon alignment

* Add support for stacked icons

* SECURITY: enforce hostname to match discourse hostname

This ensures that the hostname rails uses for various helpers always matches
the Discourse hostname

* load SVG sprite with pre-initializers

* FIX: enable caching on SVG sprites

* PERF: use JSONP for SVG sprites so they are served from CDN

This avoids needing to deal with CORS for loading of the SVG

Note, added the svg- prefix to the filename so we can quickly tell in
dev tools what the file is

* Add missing SVG sprite JSONP script to CSP

* Upgrade to FA 5.5.0

* Add support for all FA4.7 icons

- adds complete frontend and backend for renamed FA4.7 icons

- improves performance of SvgSprite.bundle and SvgSprite.all_icons

* Fix group avatar flair preview

- adds an endpoint at /svg-sprites/search/:keyword

- adds frontend ajax call that pulls icon in avatar flair preview even when it is not in subset

* Remove FA 4.7 font files
2018-11-26 16:49:57 -05:00
Vinoth Kannan fd272eee44 FEATURE: Make uploads:missing task compatible with s3 uploads 2018-11-27 00:54:51 +05:30
Kyle E. Mitchell 15e793fd3b FEATURE: Terms of Service v1.0.0
Co-authored-by: Gerhard Schlager <mail@gerhard-schlager.at>
2018-11-21 00:45:16 +01:00
Vinoth Kannan bc41057949 minor copy edit 2018-11-20 12:07:56 +05:30
Vinoth Kannan 1a9a2bd5c1 DEV: Report the missing uploads count 2018-11-19 12:06:46 +05:30
Guo Xiang Tan a895626d10 DB based S3 settings fixes for `s3:upload_assets`. 2018-11-15 15:48:40 +08:00
Guo Xiang Tan 68b640f6a9 Allow `s3:upload_assets` to use site settings from DB take 2. 2018-11-15 15:10:39 +08:00
Guo Xiang Tan c56d682f45 Allow `s3:upload_assets` to use site settings from DB. 2018-11-15 15:06:19 +08:00
Rishabh 4a12cfaecb Remove trailing whitespace for Rubocop 2018-11-13 17:19:26 +05:30
Brian Helba ea94323766 FIX: 'migrate_from_s3' rake task should respect max sizes (#6598)
Rather than hardcode a maximum size of 20MB for uploads migrated from S3, the task should use site settings for this value.
2018-11-13 12:27:38 +01:00
Gerhard Schlager 24e5be3f0c FIX: Relative links in translations should work with subfolder 2018-11-08 23:31:05 +00:00
Guo Xiang Tan 14ff47f6f1 Fix typo. 2018-11-08 16:42:12 +08:00
Guo Xiang Tan 7290145641 PERF: Speed up `migrate_to_s3` rake task.
* Prioritizes non-image uploads
* Does one remap per upload instead of 3 remaps previously
* Every 100 uploads migrated, do 2 remaps which fixes broken
  URLs
* Exclude email_logs table from remap
2018-11-08 16:39:56 +08:00
Sam 42572ff138 Revert font awesome 5 changes
We are still pushing ahead on this 100% just need a bit longer to prepare
all plugins
2018-11-08 16:12:18 +11:00
Guo Xiang Tan 0232a3b5e5 PERF: Exclude tables when remapping in `migrate_to_s3` rake task. 2018-11-08 12:37:36 +08:00
Penar Musaraj 005e1ecb9b
FEATURE: Update Font Awesome to v5.4.1 and SVGs (#6557)
* First take on subsetting svg icons

* FontAwesome 5 svg subset WIP

* Include icons from plugins/badges into svg sprite subset

* add svg icon support to themes

* Add spec for SvgSprite

* Misc. SVG icon fixes

* Use FA5 svgs in local-dates plugin

* CSS adjustments, fix SVG icons in group flair

* Use SVG icons in poll plugin

* Add SVG icons to /wizard
2018-11-07 13:05:43 -05:00
Bianca Nenciu e1e392f15b DEV: Use DiscourseIpInfo for all IP queries. (#6482)
* DEV: Use DiscourseIpInfo for all IP queries.

* UX: Use latitude and longitude for more precision.
2018-10-30 22:08:57 +00:00
David Taylor bafe3cd99a Revert "DEV: Lint official plugins in CI (#6519)"
This reverts commit 093cab2db0.
2018-10-22 22:30:33 +01:00
David Taylor 093cab2db0
DEV: Lint official plugins in CI (#6519) 2018-10-22 21:28:38 +01:00
Gerhard Schlager 341836eb42 Fix the rake task and importer instead 2018-10-17 16:48:09 +02:00
Matt Palmer b1d7582abe Run specs on discourse-prometheus-alert-receiver
Sam wants to watch the world burn.
2018-10-17 15:32:57 +11:00
Vinoth Kannan 4c2331260e run specs on discourse-calendar plugin 2018-10-16 01:26:24 +05:30
Arpit Jalan d76658ff8c FEATURE: new rake task to anonymize all users 2018-10-15 16:19:25 +05:30
Guo Xiang Tan 84d4c81a26 FEATURE: Support backup uploads/downloads directly to/from S3.
This reverts commit 3c59106bac.
2018-10-15 09:43:31 +08:00
Joshua Rosenfeld fd58ca1903
remove manually mapping of discourse-logster-rate-limit-checker, repo renamed to match 2018-10-12 15:56:23 -04:00
Joshua Rosenfeld 43a7b08a48
don't run specs on prometheus-alert-receiver for now either 2018-10-12 12:20:49 -04:00
Joshua Rosenfeld ca71815bb2
don't run specs on calendar for now 2018-10-12 12:12:21 -04:00
Joshua Rosenfeld 9d4916012a
manually map discourse-perspective too 2018-10-12 10:32:07 -04:00
Joshua Rosenfeld 2cb176f1ed
manually map discourse-logster-rate-limit-checker
also remove obsolete plugins
2018-10-12 10:27:24 -04:00
Chris Butler ddbed82bca Exit with an error if there are errors with settings import (#6475)
If the site_settings import has any errors or settings that are not found, this
will cause the task to exit with a non-zero exit code.

This is useful when using this task as part of automated configuration deployment,
where you may not want to continue with the process if a setting fails to
import.
2018-10-11 15:57:57 +08:00
Guo Xiang Tan 3c59106bac Revert "FEATURE: Support backup uploads/downloads directly to/from S3."
This reverts commit c29a4dddc1.

We're doing a beta bump soon so un-revert this after that is done.
2018-10-11 11:08:23 +08:00
Gerhard Schlager c29a4dddc1 FEATURE: Support backup uploads/downloads directly to/from S3. 2018-10-11 10:38:43 +08:00
Guo Xiang Tan 918633aa12 FIX: `upload:migrate_to_s3` rake task not remapping properly. 2018-10-10 15:09:21 +08:00
Sam 9b01e2b855 FIX: maxmind download task does not need environment 2018-10-10 11:16:28 +11:00
Bianca Nenciu 1d26a473e7 FEATURE: Show "Recently used devices" in user preferences (#6335)
* FEATURE: Added MaxMindDb to resolve IP information.

* FEATURE: Added browser detection based on user agent.

* FEATURE: Added recently used devices in user preferences.

* DEV: Added acceptance test for recently used devices.

* UX: Do not show 'Show more' button if there aren't more tokens.

* DEV: Fix unit tests.

* DEV: Make changes after code review.

* Add more detailed unit tests.

* Improve logging messages.

* Minor coding style fixes.

* DEV: Use DropdownSelectBoxComponent and run Prettier.

* DEV: Fix unit tests.
2018-10-09 22:21:41 +08:00
Guo Xiang Tan 77c293f82d Prefer `Process.clock_gettime` over `Time.now`. 2018-10-05 10:25:52 +08:00
Penar Musaraj dc8a16b43b
Use Yarn to manage dependencies, cleanup unused bootstrap libs (#6436) 2018-10-03 10:15:43 -04:00
Guo Xiang Tan 13dde44c72 FIX: Limit concurrent `assets:precompile` to number of available cores. 2018-10-03 16:46:21 +08:00
David Taylor b8c0e77908 DEV: apply plugin migrations when testing in Travis and Docker
In `RAILS_ENV=test`, plugins are not loaded by default. Therefore we need to explicitly specify `LOAD_PLUGINS=1` when we want to apply plugin migrations.
2018-09-13 12:39:54 +01:00
David Taylor 9461c99210 DEV: Add travis_fold to termination stage 2018-09-12 18:59:48 +01:00
David Taylor 26bd67a865 DEV: Add travis_fold statements to docker_test 2018-09-12 17:52:58 +01:00
Guo Xiang Tan 24c55bd613 Add dry run option to `UploadRecovery`. 2018-09-12 21:53:01 +08:00
Guo Xiang Tan c053f8ccf6 New rake task `uploads:recover`. 2018-09-12 01:52:30 -07:00
Guo Xiang Tan 6d01e0aa04 DEV: Print the error class in `uploads:list_posts_with_broken_images`. 2018-09-12 01:06:51 -07:00
Blake Erickson 1d41f3c3fb
Merge pull request #6380 from discourse/rake-destroy-sub-category
FIX: Allow `rake destroy:topics` to delete topics in sub-categories
2018-09-10 10:26:04 -06:00
Guo Xiang Tan 94ff428571 Pass the right value to rake task. 2018-09-10 20:07:28 +08:00
David Taylor 84fc7abb73 FIX: Allow `rake destroy:topics` to delete topics in sub-categories 2018-09-10 12:52:14 +01:00
Guo Xiang Tan 4a966c639d DEV: Update `uploads:list_posts_with_broken_images` to recover from tombstone. 2018-09-10 17:01:41 +08:00
Guo Xiang Tan 68572b8afc Print error messages on why upload fails to save. 2018-09-10 16:02:13 +08:00
Guo Xiang Tan 0aca80e92a Fixes to `uploads:list_posts_with_broken_images`. 2018-09-10 15:16:29 +08:00
Guo Xiang Tan 8496537590 Add `RECOVER_FROM_S3` to `uploads:list_posts_with_broken_images` rake task. 2018-09-10 15:14:30 +08:00
Gerhard Schlager f13c34aaed Adds a check for invalid message formats to rake i18n:check 2018-09-06 17:27:17 +02:00
Gerhard Schlager 3c09026fe4 Minor improvements to rake i18n:check 2018-09-06 16:54:30 +02:00
Guo Xiang Tan 72834f19ff DEV: Add rake tasks to list posts with broken images. 2018-09-05 16:54:15 +08:00
Gerhard Schlager f4956f79a5 Make Rubocop happy 2018-09-05 01:04:43 +02:00
Gerhard Schlager 6658a6601c Remove unused rake task 2018-09-05 00:47:39 +02:00
Gerhard Schlager 0d8c72d8c4 DEV: Add rake task to check locale files for errors 2018-09-05 00:47:39 +02:00
Guo Xiang Tan 8dc1463ab3 Enable `Lint/ShadowingOuterLocalVariable` for Rubocop. 2018-09-04 10:16:42 +08:00
Guo Xiang Tan 492b6dc306 DEV: `yarn exec` doesn't do what I think it does. 2018-08-30 15:55:36 +08:00
Guo Xiang Tan 720726ea68 Use `yarn exec` when running eslint and prettier. 2018-08-29 11:03:42 +08:00
Gerhard Schlager 6588a45880 Make Rubocop happy 2018-08-23 15:09:20 +02:00
Gerhard Schlager 409ee66839 Add optional "ignore_case" parameter to posts:remap rake task 2018-08-23 14:49:17 +02:00
Guo Xiang Tan 0f07494b12
Merge pull request #6259 from pfaffman/patch-2
FIX: disable_2fa fix method selection
2018-08-21 17:29:42 +08:00
Gerhard Schlager 802ad0dc61 Add rake task for updating static pages 2018-08-16 19:51:20 +02:00
Joffrey JAFFEUX 3fbf7b164d
FEATURE: emoji 11.0 "recently added" update (#6275)
Adds support for emojis in the http://www.unicode.org/emoji/charts-11.0/emoji-released.html list
2018-08-16 11:59:45 +02:00
Arpit Jalan 8e3b685aa4 make rubocop happy 2018-08-16 14:03:46 +05:30
Arpit Jalan df55e5bac4 optimize categories:create_definition task 2018-08-16 14:00:07 +05:30
Arpit Jalan 78e14abb32 new rake task to bulk tag all topics in a category 2018-08-16 13:56:57 +05:30
Arpit Jalan 777055f960 new rake task to create category definition 2018-08-16 13:50:52 +05:30
Guo Xiang Tan 33f4aa2835 Fix `migrate_to_s3` rake task, `File#close!` is not a valid method. 2018-08-16 13:26:58 +08:00
Jay Pfaffman 71a1d75d7e
FIX: disable_2fa fix method selection
The previous code resulted in

    NameError: undefined local variable or method `totp' for main:Object

I now understand what @tgxworld meant about we should only disable totp when I submitted this before.

This is the kind of Ruby stuff that I still don't understand well,(perhaps this isn't the most Ruby way to do this?) but this does what I think is supposed to happen. And it worked just now.
2018-08-10 14:45:40 -07:00
Sam 1fc2597626 better error handling for upload extension fixer 2018-08-10 11:28:22 +10:00
Guo Xiang Tan 0879610ffd Add missing require in `uploads:fix_incorrect_extensions`. 2018-08-08 15:39:37 +08:00
Guo Xiang Tan 17047806b9 Add a rake task to fix uploads with wrong extension. 2018-08-08 13:15:17 +08:00
Régis Hanol 14bbd5d167 FIX: use the right URL when downloading the file from S3 2018-08-02 10:04:41 +02:00
Régis Hanol 0f1137dafa FIX: 'migrate_from_s3' rake task wasn't handling short urls 2018-08-01 22:58:46 +02:00
Neil Lalonde b829452c75
Merge pull request #6209 from discourse/mini_scheduler
REFACTOR: extract scheduler to the mini_scheduler gem
2018-08-01 10:28:24 -04:00
Neil Lalonde 4ad7ce70ce REFACTOR: extract scheduler to the mini_scheduler gem 2018-07-31 17:12:55 -04:00
Sam f0c203a5cf clean up previous commit 2018-07-31 14:50:02 +10:00
Jay Pfaffman b55d9e63a0 Rake mail test debugging (#6171)
* FEATURE: rake emails:test add debugging

* Oops! Remove safety code

* more language tweaks
2018-07-31 14:45:59 +10:00
Gerhard Schlager e42038eae2 Rake task called method with wrong arguments 2018-07-24 00:10:09 +02:00
Jay Pfaffman 0ed2834c2d FEATURE: Add users:disable_2factor rake task
https://meta.discourse.org/t/admin-locked-out-of-2fa/92156/2?u=pfaffman
2018-07-16 09:56:55 +08:00
Arpit Jalan 536cef86f4 PERF: do not carry post ids in memory when rebaking all posts 2018-07-11 14:34:33 +05:30
Guo Xiang Tan 59a2767de8 Ensure that we restore the site setting in posts:rebake rake task. 2018-07-06 16:22:54 +08:00
Neil Lalonde 92000bc8a0 FEATURE: add a rake task to recalculate user stats post_count and topic_count 2018-07-05 10:38:46 -04:00
Sam d1b21aa73b add timings to asset precompile 2018-07-04 09:42:38 +10:00
Gerhard Schlager 23acddc9fa Add descriptions for rake tasks 2018-06-27 20:24:46 +02:00
Joffrey JAFFEUX 2f7960bd2a
DEV: updates prettier and displays linters/prettifiers version in CI 2018-06-20 18:34:49 +02:00
Joffrey JAFFEUX 8126b603e4
fix prettier 2018-06-20 18:26:43 +02:00
Joffrey JAFFEUX c5c1b45d19
higher loglevel for prettier 2018-06-20 16:51:48 +02:00
Sam cb824a6b33 DEV: remove all calls to SqlBuilder use DB.build instead
This is part of the migration to mini_sql, SqlBuilder.new is being
deprecated and replaced with DB.build
2018-06-20 17:53:49 +10:00
Sam 5f64fd0a21 DEV: remove exec_sql and replace with mini_sql
Introduce new patterns for direct sql that are safe and fast.

MiniSql is not prone to memory bloat that can happen with direct PG usage.
It also has an extremely fast materializer and very a convenient API

- DB.exec(sql, *params) => runs sql returns row count
- DB.query(sql, *params) => runs sql returns usable objects (not a hash)
- DB.query_hash(sql, *params) => runs sql returns an array of hashes
- DB.query_single(sql, *params) => runs sql and returns a flat one dimensional array
- DB.build(sql) => returns a sql builder

See more at: https://github.com/discourse/mini_sql
2018-06-19 16:13:36 +10:00
Joffrey JAFFEUX 913d97e780
fix prettier 2018-06-18 20:10:48 +02:00
Joffrey JAFFEUX 6f2c5ea1f2
increases loglevel to debug CI only failure 2018-06-18 19:25:54 +02:00
Joffrey JAFFEUX 03a7d532cf
DEV: introduces prettier for es6 files 2018-06-15 17:03:24 +02:00
Guo Xiang Tan b91a3c12a5 Revert "Smoke test should wait up to 5 minutes before it fails"
This reverts commit 34d4712133.
2018-06-11 08:47:57 +08:00
Joffrey JAFFEUX d96c775833
enforces prettier 2018-06-08 17:38:29 +02:00
Gerhard Schlager 4ba910ac54 Update username only after successful user anonymization 2018-06-08 15:50:07 +02:00
Joffrey JAFFEUX 22c186a792
disables prettier enforcing for now 2018-06-08 15:44:00 +02:00