Commit Graph

1242 Commits

Author SHA1 Message Date
Penar Musaraj eae8b0465c
DEV: Upgrade ACE Editor to 1.4.12, remove extra ACE files (#10431) 2020-08-13 21:53:27 -04:00
Sam Saffron 5be26c7d24
DEV: add error handling in case download fails 2020-08-13 13:48:23 +10:00
Jeff Wong 787ad7d84d FIX: allow plugin pinning to fetch missing commits
Add update for fetching git commits if they do not exist, eg with
clone --depth 1 - only can fetch via git fetch --depth 1 {remote} {ref}
the ref needs to be a full, non-ambiguous reference.
2020-08-12 20:03:37 -07:00
Sam Saffron 5011435ec7
DEV: do not correct sha when correctly uploads 2020-08-13 11:52:57 +10:00
Sam Saffron f48fa30ecd
DEV: fix_missing_s3 attempts to re-download if missing
unverified uploads get re-downloaded and corrected if they exist in the task
2020-08-13 11:22:14 +10:00
David Taylor 451b9b245f
DEV: Rename new upload rake tasks
These tasks are s3-specific, so update the names to make that very clear
2020-08-12 23:26:13 +01:00
Sam Saffron ec173a72d9
DEV: add more counts to analyze missing 2020-08-12 17:28:58 +10:00
Sam Saffron 97fe68e980
DEV: look for avatars when analyzing missing uploads 2020-08-12 14:04:21 +10:00
Sam Saffron 990cd8c2e4
FEATURE: introduce tasks for dealing with legacy broken uploads
`rake uploads:analyze_missing` can be used get rich information regarding
uploads missing from s3 (where verified is false)

`rake uploads:fix_missing` is a work in progress task for automatically
correcting certain historic issues. At the moment it simply rebakes all
posts with missing uploads, but it will improve over time
2020-08-12 13:33:04 +10:00
Daniel Waterworth abebc4e05e DEV: Use verbose mode when running tests in CI 2020-08-07 17:04:55 +01:00
Jarek Radosz cd4f251891
FEATURE: Poll breakdown 2.0 (#10345)
The poll breakdown modal replaces the grouped pie charts feature.

Includes:

* MODAL: Untangle `onSelectPanel`
Previously modal-tab component would call on click the onSelectPanel callback with itself (modal-tab) as `this` which severely limited its usefulness. Now showModal binds the callback to its controller.

"The PR includes a fix/change to d-modal (b7f6ec6) that hasn't been extracted to a separate PR because it's not currently possible to test a change like this in abstract, i.e. with dynamically created controllers/components in tests. The percentage/count toggle test for the poll breakdown feature is essentially a test for that d-modal modification."
2020-08-06 17:57:06 +02:00
Guo Xiang Tan 850d5696f2
FIX: Disable concurrent migration for `multisite:migrate`.
Rails migration is not thread safe.
2020-08-05 15:12:56 +08:00
Guo Xiang Tan 1d281353b6
FIX: Don't run seeds if multisite migration fails. 2020-08-05 15:12:56 +08:00
Sam Saffron e8328968bb
DEV: allow qunit:test to run concurrently
Previously we were fighting on pid file which makes it hard to run multiple
2020-08-05 16:57:12 +10:00
Régis Hanol ef92b72ff6 DEV: ensure 'posts:reorder_posts' rake task work on single topic
when the topic_id argument is provided.
2020-08-04 14:57:49 +02:00
Régis Hanol 712ab33ff8 DEV: fix 'posts:reorder' rake task
Running the reorder rake task was triggering the following error:

PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "post_timings_unique"

I re-worked the queries and refactored to use the same couple of queries for all similar tables/columns.
2020-08-04 11:07:07 +02:00
Krzysztof Kotlarek e0d9232259
FIX: use allowlist and blocklist terminology (#10209)
This is a PR of the renaming whitelist to allowlist and blacklist to the blocklist.
2020-07-27 10:23:54 +10:00
Robin Ward 5e5973cabe FIX: Our test build of highlight.js was broken
Some definitions rely on others, in particular the c/cpp/c-like ones,
and we were appending the bundle of all files in the folder.

Instead for testing I've limited us to just three definitions. This has
the benefit of being a lot smaller to download/parse in test mode too.
2020-07-20 15:34:24 -04:00
Robin Ward 9889b7277f FIX: Silence route-recognizer source map errors in development mode 2020-07-15 15:42:04 -04:00
Robin Ward e8bf304f05 FIX: Add popper sourcemap 2020-07-15 15:29:12 -04:00
Jeff Wong 339549d14a
Support plugin and Theme compatibility version manifests (#9995)
Adds a new rake task `plugin:checkout_compatible_all` and
`plugin:checkout_compatible[plugin-name]` that check out compatible plugin
versions.

Supports a .discourse-compatibility file in the root of plugins and themes that
list out a plugin's compatibility with certain discourse versions:

eg: .discourse-compatibility
```
2.5.0.beta6: some-git-hash
2.4.4.beta4: some-git-tag
2.2.0: git-reference
```

This ensures older Discourse installs are able to find and install older
versions of plugins without intervention, through the manifest only.

It iterates through the versions in descending order. If the current Discourse
version matches an item in the manifest, it checks out the listed plugin target.
If the Discourse version is greater than an item in the manifest, it checks out
the next highest version listed in the manifest.

If no versions match, it makes no change.
2020-07-06 14:48:00 -07:00
Blake Erickson 69803599a9 DEV: Refactor seed data filter
Added a small helper class to for seed data because we need to add the
same filter to multisite:migrate as we have in db:migrate. Having this
filter in both places means we can get rid of the SKIP_SEED flag.
2020-06-26 14:36:50 -06:00
Guo Xiang Tan c16ad39f8e
DEV: Run seeds irregardless of post deploy migration flag.
Follow up to 01937b2d
2020-06-26 11:04:34 +08:00
Guo Xiang Tan 01937b2de2
Revert "FIX: Seed needs to run before optimizing site icons."
This reverts commit 715ddf3861.
2020-06-26 11:03:47 +08:00
Guo Xiang Tan 715ddf3861
FIX: Seed needs to run before optimizing site icons. 2020-06-26 08:58:53 +08:00
Blake Erickson 01b6349a67
DEV: Add skip seed flag (#10116)
* add a flag to skip seed

* only seed when running post deployment migrations
2020-06-25 10:14:58 -06:00
Guo Xiang Tan 0384b6d910
FIX: `multisite:migrate` failing to properly seed data. 2020-06-23 09:10:02 +08:00
Dan Ungureanu 2e1efbde52
FIX: Pass local scope variable to inner function 2020-06-22 19:23:59 +03:00
David Taylor e29afa200a
FIX: Cleanup migrations with timestamps in the future
A future-dated migration was accidently introduced by me in 45c399f0. This was removed in b9762afc, but other migrations had already been generated based on its incorrect date. This commit removes the offending data in the schema_migrations table, and corrects the version in the published_pages migration.

This commit also adds a check to db:migrate which raises an error when invalid migration timestamps are used.
2020-06-17 15:58:22 +01:00
Guo Xiang Tan 45eb97c202
FIX: Thread safety issues with `multisite:migrate` and `SeedFu`. 2020-06-17 16:15:43 +08:00
Jarek Radosz 669c940ec3 Revert "DEV: Remove the remaining ENV["TRAVIS"] usage (#10041)"
This reverts commit 78aff841e3.

See https://review.discourse.org/t/dev-remove-the-remaining-env-travis-usage-10041/12737/4?u=cvx
2020-06-16 19:42:00 +02:00
Jarek Radosz 78aff841e3
DEV: Remove the remaining ENV["TRAVIS"] usage (#10041) 2020-06-16 17:41:15 +02:00
Blake Erickson a2713578dd DEV: Allow plugins to exclude seed data
This allows plugins to specify if they would like to filter out any seed
data files from running during migrations.
2020-06-15 15:30:25 -06:00
Kane York 5f3fd23fa8
DEV: Add stub flush_sw task to prevent rebuild errors (#10021) 2020-06-10 13:07:37 -07:00
Joffrey JAFFEUX d4caf69ed7
DEV: makes SKIP_INSTALL_PLUGINS called last and global (#9990) 2020-06-05 17:59:23 +02:00
Sam Saffron 1c48853ede
DEV: allow skipping of a single plugin when installing
SKIP_INSTALL_PLUGINS can be used to tell plugin:install_all_official
to skip a plugin.

Comma seperated list
2020-06-05 18:26:25 +10:00
Joshua Rosenfeld 76af25f753
FEATURE: Support append when bulk tagging via rake (#9978)
New argument option for the `tags:bulk_tag_category` rake task.
Backwards compatible - defaults to false (no append) if no argument given.
2020-06-04 09:33:48 -04:00
Arpit Jalan 07774894a6 DEV: notify when the rake ask is completed 2020-06-04 12:00:06 +05:30
Arpit Jalan a93d24501c FIX: base import script was not updating first_post_created_at column
FEATURE: new rake task to update first_post_created_at column

The not-equal operator (`<>`) in PostgreSQL does not compare values
with NULL. We should instead use `IS DISTINCT FROM` when comparing
values with NULL.
2020-06-04 11:26:40 +05:30
Michael K Johnson 81e6bc7a0f
FEATURE: Add uploads:batch_migrate_from_s3 task to limit total posts migrated at once (#9933)
Allow limiting the number of migrations to do at once, both to do migrations that
have impact limited to multiple off-peak usage hours to reduce user impact from
a migration, and to allow tests that do only a very small number for test
purposes. ("Give me a ping, Vasili. One ping only, please.")
2020-06-04 09:48:11 +10:00
Gerhard Schlager f683c5d0e0 DEV: Check English locale for errors in CI
Moves the most important checks into a linter. It gets executed by Lefthook as well as the docker rake task and Github actions. Doing those checks in rspec takes too long and it produces errors when the discourse:test Docker image contains old, invalid locale files.
2020-06-03 21:54:58 +02:00
Sam Saffron 57a3d4e0d2
FEATURE: whitelist theme repo mode (experimental)
In some restricted setups all JS payloads need tight control.

This setting bans admins from making changes to JS on the site and
requires all themes be whitelisted to be used.

There are edge cases we still need to work through in this mode
hence this is still not supported in production and experimental.

Use an example like this to enable:

`DISCOURSE_WHITELISTED_THEME_REPOS="https://repo.com/repo.git,https://repo.com/repo2.git"`

By default this feature is not enabled and no changes are made.

One exception is that default theme id was missing a security check
this was added for correctness.
2020-06-03 13:19:57 +10:00
Aman Gupta Karmani 6358e79f0f
FIX: catch NoMethodError during precompile_css too (#9940)
Ran into this on heroku:

    NoMethodError: undefined method `enabled?' for #<Theme:0x0000558f069e7718>
    vendor/bundle/ruby/2.6.0/gems/activemodel-6.0.1/lib/active_model/attribute_methods.rb:431:in `method_missing'
    app/models/theme.rb:155:in `block (2 levels) in transform_ids'
    vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/delegation.rb:85:in `each'
    vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/delegation.rb:85:in `each'
    vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/query_methods.rb:260:in `select'
    vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/query_methods.rb:260:in `select'
    app/models/theme.rb:155:in `block in transform_ids'
    app/models/theme.rb:105:in `get_set_cache'
    app/models/theme.rb:140:in `transform_ids'
    app/models/theme.rb:321:in `list_baked_fields'
    app/models/theme.rb:317:in `resolve_baked_field'
    lib/stylesheet/manager.rb:289:in `theme_digest'
2020-06-02 15:18:03 +10:00
Joffrey JAFFEUX 7635c18a14
DEV: ensures highlightjs is correctly tested (#9923) 2020-05-29 13:05:44 +02:00
Michael Brown d9a02d1336
Revert "Revert "Merge branch 'master' of https://github.com/discourse/discourse""
This reverts commit 20780a1eee.

* SECURITY: re-adds accidentally reverted commit:
  03d26cd6: ensure embed_url contains valid http(s) uri
* when the merge commit e62a85cf was reverted, git chose the 2660c2e2 parent to land on
  instead of the 03d26cd6 parent (which contains security fixes)
2020-05-23 00:56:13 -04:00
Jeff Atwood 20780a1eee Revert "Merge branch 'master' of https://github.com/discourse/discourse"
This reverts commit e62a85cf6f, reversing
changes made to 2660c2e21d.
2020-05-22 20:25:56 -07:00
Martin Brennan 779dc30d2e FIX: Join bookmarks migration on users to avoid missing user records
* the post_actions table has no FK to users, so if a user has been
  deleted we may end up with dangling post_action records, which then
  interferes with the bookmarks migration because bookmarks DO have
  an FK to users
2020-05-19 16:31:05 +10:00
Kane York 869f9b20a2
PERF: Dematerialize topic_reply_count (#9769)
* PERF: Dematerialize topic_reply_count

It's only ever used for trust level promotions that run daily, or compared to 0. We don't need to track it on every post creation.

* UX: Add symbol in TL3 report if topic reply count is capped

* DEV: Drop user_stats.topic_reply_count column
2020-05-14 15:42:00 -07:00
Robin Ward 42b300b9be DEV: Skip I18n in plugins until they can be updated 2020-05-14 10:40:53 -04:00
Rafael dos Santos Silva 1c0c425f38
DEV: Detect correct data dir using ENV 2020-05-12 13:10:57 -03:00
Robin Ward 8311374a5a DEV: Replace `version.js.erb` with pre generated file 2020-05-11 15:43:09 -04:00
Robin Ward d2b16a7618
DEV: Generate `emoji/data` on demand (#9744)
These emoji rarely change and can be re-generated when we make changes
to them.
2020-05-11 15:29:46 -04:00
Robin Ward 34e5f807ec DEV: User UPPER_CASE for constants 2020-05-08 15:50:55 -04:00
Robin Ward 7f373e8b93 DEV: Don't use `js.erb` for constants
Adds a new rake task to auto generate a constants.js file with the
constants present. This makes migrating to Ember CLI easier, but also
slightly speeds up asset compilation by having to do less work.

If the constants change you need to run:
`rake javascripts:update_constants`
2020-05-08 14:14:01 -04:00
David Taylor 72ad701df0
DEV: Stub #flush in StdOutDemux for multisite:migrate
https://meta.discourse.org/t/multisite-migrate-error/150579/2
2020-05-06 11:58:35 +01:00
Krzysztof Kotlarek 9bff0882c3
FEATURE: Nokogumbo (#9577)
* FEATURE: Nokogumbo

Use Nokogumbo HTML parser.
2020-05-05 13:46:57 +10:00
Robin Ward 08fbf199ad FIX: S3 rake task can ignore yarn.lock 2020-04-29 13:14:04 -04:00
Robin Ward 9ec908950d DEV: Better error handling for s3 task 2020-04-29 12:54:39 -04:00
Penar Musaraj ec2943c5bc
DEV: Update jquery.fileupload and dependencies (#9466) 2020-04-28 10:39:29 -04:00
Sam Saffron 6b62d75c50
FIX: remove unsuppored rake task
rebuilding user_actions is not something that should be done.

Plugins such as solved and assigned extend it, there are tons of
little rules that were not captured in `user_actions:rebuild`
2020-04-28 16:27:42 +10:00
Rafael dos Santos Silva b810fc282d FEATURE: Digital Ocean Spaces compatibility for Object Storage
DO does not implement tagging support for S3 objects. Removing our default
empty tag fixes compatibility.

The expire_missing_assets rake task can't be used with that service still,
but this patch allows normal operation.
2020-04-22 13:57:14 -03:00
Martin Brennan 628ba9d1e2
FEATURE: Promote bookmarks with reminders to core functionality (#9369)
The main thrust of this PR is to take all the conditional checks based on the `enable_bookmarks_with_reminders` away and only keep the code from the `true` path, making bookmarks with reminders the core bookmarks feature. There is also a migration to create `Bookmark` records out of `PostAction` bookmarks for a site.

### Summary

* Remove logic based on whether enable_bookmarks_with_reminders is true. This site setting is now obsolete, the old bookmark functionality is being removed. Retain the setting and set the value to `true` in a migration.
* Use the code from the rake task to create a database migration that creates bookmarks from post actions.
* Change the bookmark report to read from the new table.
* Get rid of old endpoints for bookmarks
* Link to the new bookmarks list from the user summary page
2020-04-22 13:44:19 +10:00
tshenry e41f1e6ec2
FEATURE: Rake task to export groups (#9450) 2020-04-17 14:59:54 -07:00
Martin Brennan 51672b9121
FIX: Minor bookmark with reminder issue cleanup (#9436)
* Count user summary bookmarks from new Bookmark table if bookmarks with reminders enabled
* Update topic user bookmarked column when new topic bookmark changed
* Make in:bookmarks search work with new bookmarks
* Fix batch inserts for bookmark rake task (and thus migration). We were only inserting one bookmark at a time, completely defeating the purpose of batching!
2020-04-16 11:32:21 +10:00
Mark VanLandingham f07c4a781c
DEV: Add options to theme install rake task - more options (#9394) 2020-04-14 09:17:00 -05:00
Sam Saffron e2284cf739
Revert "We have had errors reported due to migrations breaking and are reverting"
This reverts commit 8b46f14744.

It corrects the reason for the revert:

We rely on SafeMigrate existing cause we call it from migrations,
Zeitwerk will autoload it.

Instead of previous pattern we explicitly bypass all the hacks in
production mode.

We need to disable SafeMigrate cause it is not thread safe.

A thread safe implementation is possible but not worth the effort,
we catch the issues in dev and test.
2020-04-14 11:31:07 +10:00
Robin Ward 8b46f14744 We have had errors reported due to migrations breaking and are reverting
these series of commits.

See:
https://meta.discourse.org/t/new-installation-fails-on-migration/147425

This reverts commit 80e832662b.
This reverts commit de5f2d3308.
This reverts commit 6b192d29fa.
This reverts commit a5b582f686.
This reverts commit 708dd97dfd.
2020-04-09 14:18:56 -04:00
Sam Saffron 80e832662b
DEV: fake stdout must respond to close
If it does not respond to close Logger will refuse to use it
2020-04-09 14:36:01 +10:00
Sam Saffron a5b582f686
DEV: demux stdout when running multisite migrate
This avoids mixing up output in such a way that we can not tell which
site ran which migrations

Avoids threads all fighting for output
2020-04-09 12:50:14 +10:00
Sam Saffron 708dd97dfd
PERF: speed up migrations on multisite
Previously we were migrating multisites serially, this is extremely slow
especially when 200 dbs are involved.

The new implementation defaults to running 20 migrations concurrently, leading
to a 20x speedup.

We also amended it so errors are printed out last, something that makes
debugging failures easier.

This is code specific to Discourse cause we integrate SeedFu with our
migrations and can not include this in the multisite gem.
2020-04-09 11:58:20 +10:00
Sam Saffron f062ebf274
DEV: reduce noise in logs regarding migration
If a migration performs no changes it should not output stuff.

Previously we would output information about seeds which was very noisy.

On multisite this was particularly bad
2020-04-09 10:05:22 +10:00
Sam Saffron 8a50ab45aa
DEV: use REPORT_REQUESTS=1 to find all requests
We can use this to profile our pretender and ensure nothing is superfluous
and nothing is missing
2020-04-02 16:01:49 +11:00
Martin Brennan 6f978bc95c
FIX: First pass to improve efficiency of secure uploads rake task (#9284)
Get rid of harmful each loop over uploads to update. Instead we put all the unique access control posts for the uploads into a map for fast access (vs using the slow .find through array) and look up the post when it is needed when looping through the uploads in batches.

On a Discourse instance with ~93k uploads, a simplified version of the old method takes > 1 minute, and a simplified version of the new method takes ~18s and uses a lot less memory.
2020-03-26 15:59:57 +10:00
Martin Brennan efd5fb665b
DEV: Fix flaky time sensitive uploads.rake specs (#9283)
Also fix issues in spec where certain uploads were not considered secure
2020-03-26 13:31:39 +10:00
Martin Brennan 097851c135
FIX: Change secure media to encompass attachments as well (#9271)
If the “secure media” site setting is enabled then ALL files uploaded to Discourse (images, video, audio, pdf, txt, zip etc. etc.) will follow the secure media rules. The “prevent anons from downloading files” setting will no longer have any bearing on upload security. Basically, the feature will more appropriately be called “secure uploads” instead of “secure media”.

This is being done because there are communities out there that would like all attachments and media to be secure based on category rules but still allow anonymous users to download attachments in public places, which is not possible in the current arrangement.
2020-03-26 07:16:02 +10:00
Joffrey JAFFEUX 86eb03af85
DEV: allows to run up/down migrations of a plugin (#9241)
This is mostly useful while developing a plugin, to avoid manual actions of deleting tables and schema_migrations rows.

Usage:
bundle exec rake plugin:migrate:down[discourse-calendar]
2020-03-19 19:30:08 +01:00
Martin Brennan 16799da580
FIX: Improve bookmark modal on mobile and bookmark sync rake task (#9221)
* Improve the bookmark mobile on modal so it doesn't go all the way to the edge and the custom datetime input is easier to use
* Improve the rake task for syncing so it does not error for topics that no longer exist and batches 2000 inserts at a time, clearing the array each time
2020-03-17 15:15:22 +10:00
Sam Saffron a14313e9d0
FIX: RANDOM_PASSWORD not working rake admin:create
We regressed behavior of this env var which enable admin account
creation with a random password
2020-03-15 17:24:11 +11:00
Robin Ward c9d1577087 Let's not log the username/password
This could easily be seen by someone who shouldn't.
2020-03-11 12:48:10 -04:00
Gerhard Schlager 8fa8bab9ff FIX: Don't optimize icons during db:migrate when restoring backup
Uploads are extracted after the DB migration, so this could lead to a failure during the restore. Site icons get optimized after extracting uploads.
2020-03-04 16:59:49 +01:00
Martin Brennan 0388653a4d
DEV: Upload and secure media retroactive rake task improvements (#9027)
* Add uploads:sync_s3_acls rake task to ensure the ACLs in S3 are the correct (public-read or private) setting based on upload security

* Improved uploads:disable_secure_media to be more efficient and provide better messages to the user.

* Rename uploads:ensure_correct_acl task to uploads:secure_upload_analyse_and_update as it does more than check the ACL

* Many improvements to uploads:secure_upload_analyse_and_update

* Make sure that upload.access_control_post is unscoped so deleted posts are still fetched, because they still affect the security of the upload.

* Add escape hatch for capture_stdout in the form of RAILS_ENABLE_TEST_STDOUT. If provided the capture_stdout code will be ignored, so you can see the output if you need.
2020-03-03 10:03:58 +11:00
Martin Brennan cfd56e9159 Include access control post when loading uploads in rake task
* to avoid N+1 query
2020-02-18 10:35:15 +10:00
Martin Brennan 9dcc454a07
FIX: Improvements and fixes for update_upload_acl rake task (#8980)
The rake task was broken, because the addition of the
UploadSecurity check returned true/false instead of the
upload ID to determine which uploads to set secure.
Also it was rebaking the posts in the wrong place and
pretty inefficiently at that. Also it was rebaking before
the upload was being changed to secure in the DB.
This also updates the task to set the access_control_post_id
for all uploads. the first post the upload is linked to is used
for the access control. if the upload doesn't get changed to
secure this doesn't affect anything.
Added a spec for the rake task to cover common cases.
2020-02-17 14:21:43 +10:00
Martin Brennan e1e74abd4f
FEATURE: Improving bookmarks part 2 -- Topic Bookmarking (#8954)
### UI Changes

If `SiteSetting.enable_bookmarks_with_reminders` is enabled:

* Clicking "Bookmark" on a topic will create a new Bookmark record instead of a post + user action
* Clicking "Clear Bookmarks" on a topic will delete all the new Bookmark records on a topic
* The topic bookmark buttons control the post bookmark flags correctly and vice-versa
Disabled selecting the "reminder type" for bookmarks in the UI because the backend functionality is not done yet (of sending users notifications etc.)

### Other Changes

* Added delete bookmark route (but no UI yet)
* Added a rake task to sync the old PostAction bookmarks to the new Bookmark table, which can be run as many times as we want for a site (it will not create duplicates).
2020-02-13 16:26:02 +10:00
Joffrey JAFFEUX 87e92da085
DEV: makes popper.js part of javascript rake task (#8847) 2020-02-04 15:34:46 +01:00
Gerhard Schlager 4e8be6f18b FIX: uploads:s3_migration_status rake task was broken 2020-01-28 22:10:25 +01:00
Roman Rizzi 8eb2147f1f
FIX: Ensure sourcemap's source is correct. Uses the full assets path this time. (#8774) 2020-01-23 14:44:00 -03:00
romanrizzi 1de5b4ad64 Revert "FIX: Ensure sourcemap's source is correct"
This reverts commit 4553a0f6e0.
2020-01-22 16:31:34 -03:00
Roman Rizzi 4553a0f6e0
FIX: Ensure sourcemap's source is correct 2020-01-22 12:57:28 -03:00
Gerhard Schlager ab07b945c2
Merge pull request #8736 from gschlager/rename_reply_id_column
REFACTOR: Rename `post_replies.reply_id` column to `post_replies.reply_post_id`
2020-01-17 17:24:49 +01:00
Martin Brennan 7c32411881
FEATURE: Secure media allowing duplicated uploads with category-level privacy and post-based access rules (#8664)
### General Changes and Duplication

* We now consider a post `with_secure_media?` if it is in a read-restricted category.
* When uploading we now set an upload's secure status straight away.
* When uploading if `SiteSetting.secure_media` is enabled, we do not check to see if the upload already exists using the `sha1` digest of the upload. The `sha1` column of the upload is filled with a `SecureRandom.hex(20)` value which is the same length as `Upload::SHA1_LENGTH`. The `original_sha1` column is filled with the _real_ sha1 digest of the file. 
* Whether an upload `should_be_secure?` is now determined by whether the `access_control_post` is `with_secure_media?` (if there is no access control post then we leave the secure status as is).
* When serializing the upload, we now cook the URL if the upload is secure. This is so it shows up correctly in the composer preview, because we set secure status on upload.

### Viewing Secure Media

* The secure-media-upload URL will take the post that the upload is attached to into account via `Guardian.can_see?` for access permissions
* If there is no `access_control_post` then we just deliver the media. This should be a rare occurrance and shouldn't cause issues as the `access_control_post` is set when `link_post_uploads` is called via `CookedPostProcessor`

### Removed

We no longer do any of these because we do not reuse uploads by sha1 if secure media is enabled.

* We no longer have a way to prevent cross-posting of a secure upload from a private context to a public context.
* We no longer have to set `secure: false` for uploads when uploading for a theme component.
2020-01-16 13:50:27 +10:00
Gerhard Schlager e474cda321 REFACTOR: Restoring of backups and migration of uploads to S3 2020-01-14 11:41:35 +01:00
David Taylor c8d438cc63
DEV: Allow CSP to be enabled during QUnit tests (#8668)
The QUnit rake task starts a server in test mode. We need a tweak to allow dynamic CSP hostnames in test mode. This tweak is already present in development mode.

To allow CSP to work, the browser host/port must match what the server sees. Therefore we need to disable the enforce_hostname middleware in test mode. To keep rspec and production as similar as possible, we skip enforce_hostname using an environment variable.

Also move the qunit rake task to use unicorn, for consistency with development and production.
2020-01-07 12:22:58 +00:00
Martin Brennan abca91cc4d
FEATURE: Add rake task to disable secure media (#8669)
* Add a rake task to disable secure media. This sets all uploads to `secure: false`, changes the upload ACL to public, and rebakes all the posts using the uploads to make sure they point to the correct URLs. This is in a transaction for each upload with the upload being updated the last step, so if the task fails it can be resumed.
* Also allow viewing media via the secure url if secure media is disabled, redirecting to the normal CDN url, because otherwise media links will be broken while we go and rebake all the posts + update ACLs
2020-01-07 12:27:24 +10:00
Blake Erickson b7b85f9ade
FEATURE: Turn csp on by default (#8665)
* turn csp on by default

* remove csp migration for new sites now that is is on by default

* Ensure CSP is off before starting qunit
2020-01-06 13:42:21 -07:00
Blake Erickson fc6b093dce FIX: Ensure CSP is off for qunit
If CSP is turned on qunit won't run at all, not even via the CLI, this
is causing the js tests to fail.

Follow up to: 3193b0f6e6
2020-01-03 18:43:19 -07:00
Sam Saffron 412e1ebbe2 DEV: correct parallel specs rake tasks
This used to work due to side effects.

`rake parallel:migrate` used to work very inconsistently and would only migrate
some of the databases.

This introduces the recommended change to db.yml so the correct database is
found based off TEST_ENV_NUMBER if for some reason we did not set it using
RAILS_DB

Also avoids a bunch of schema dumping which is not needed when migrating
parallel specs



DB number 1 is very odd cause for whatever reason parallel spec is not
setting it.
2019-12-31 14:07:55 +11:00
Robin Ward 9fc1a754fe DEV: Error message if multisite DB can't be created 2019-12-19 11:52:27 -05:00
Robin Ward 480e2ec523 FIX: If we run `db:migrate` on its own, it should load the environment 2019-12-16 14:46:33 -05:00
Hector Bustillos 42758379b9 FEATURE: Adds a message when the passwords doesn't match in rake admin:create (#8543) 2019-12-13 11:40:33 +11:00
David Taylor 4c9ca24ccf
FEATURE: Hash API keys in the database (#8438)
API keys are now only visible when first created. After that, only the first four characters are stored in the database for identification, along with an sha256 hash of the full key. This makes key usage easier to audit, and ensures attackers would not have access to the live site in the event of a database leak.

This makes the merge lower risk, because we have some time to revert if needed. Once the change is confirmed to be working, we will add a second commit to drop the `key` column.
2019-12-12 11:45:00 +00:00
David Taylor 67605b55d5 DEV: Prevent plugin loading when running core db migrations in tests
This is required because bin/rake automatically loads plugins when migrating. In our continuous integration, we don't want plugins to break the core build. They should only be loaded for the plugin build.
2019-12-10 16:11:49 +00:00
Neil Lalonde 891b71bdf5 Fix release notes rake task
It breaks when a commit message is "FIX:"
2019-12-05 11:53:10 -05:00
romanrizzi b13f427450 DEV: Drop uglifyJS V2 support 2019-12-04 15:58:38 -03:00
David Taylor 46d8fd3831 FIX: Allow for nil upload record when migrating to S3 2019-12-04 15:13:39 +00:00
Joffrey JAFFEUX 0d3d2c43a0
DEV: s/\$redis/Discourse\.redis (#8431)
This commit also adds a rubocop rule to prevent global variables.
2019-12-03 10:05:53 +01:00
Penar Musaraj 0a6b29a942 DEV: Allow skipping --fail-fast rspec option for plugin specs 2019-11-22 13:27:47 -05:00
Jeff Wong c6d8dbd4a9 Revert "FEATURE: Normalize the service worker route (#8359)"
This reverts commit 9799a651b6.
2019-11-20 14:10:17 -08:00
Jeff Wong 9799a651b6
FEATURE: Normalize the service worker route (#8359)
* FEATURE: Normalize the service worker route

Update cache headers so they are not immutable outside of the rails app

Add the ability to purge the service worker cache from localhost

Rails -> nginx will pass immutable flags so the file is cached until reloaded.
In most cases, nginx will have its cache flushed on rebuild (new image)

For those needing dynamic re-caching (such as upgrading via the UI),
a rake task for flushing the service worker script is provided
through `assets:flush_sw`
2019-11-20 11:33:41 -08:00
Penar Musaraj 102909edb3 FEATURE: Add support for secure media (#7888)
This PR introduces a new secure media setting. When enabled, it prevent unathorized access to media uploads (files of type image, video and audio). When the `login_required` setting is enabled, then all media uploads will be protected from unauthorized (anonymous) access. When `login_required`is disabled, only media in private messages will be protected from unauthorized access. 

A few notes: 

- the `prevent_anons_from_downloading_files` setting no longer applies to audio and video uploads
- the `secure_media` setting can only be enabled if S3 uploads are already enabled and configured
- upload records have a new column, `secure`, which is a boolean `true/false` of the upload's secure status
- when creating a public post with an upload that has already been uploaded and is marked as secure, the post creator will raise an error
- when enabling or disabling the setting on a site with existing uploads, the rake task `uploads:ensure_correct_acl` should be used to update all uploads' secure status and their ACL on S3
2019-11-18 11:25:42 +10:00
Robin Ward 397852a9a6 FIX: Support for `rake db:rollback` with plugins
For this to work we need to overwrite `db:rollback` in our Rakefile like
we do for migrate, so that it removes the load_config dependency. This
allows our custom migration paths to work.
2019-11-15 16:38:46 -05:00
romanrizzi 0ba0aa00e7 DEV: Use a bigger suite for warming up the tmp folder 2019-11-13 18:34:10 -03:00
Simon Cossar 544a1379b6
Reference posts table for deleted_at and user_id (#8332) 2019-11-11 11:03:57 -05:00
Jay Pfaffman 63bd07492e FIX: Update api_key rake task for recent changes
New usage: 

```
rake api_key:get_or_create_master["Onboarding Key"]
```
2019-11-08 23:34:15 +00:00
Sam Saffron d8f7f363cd FEATURE: stop updating last_posted_at on users for messages and whispers
This ensures we only update last_posted_at which is user facing for non messages
and non whispers.

We still update this date for secure categories, we do not revert it for
deleted posts.
2019-10-31 09:01:26 +11:00
Krzysztof Kotlarek c32bd8ae48 FEATURE: Remove attachments and truncate raw field for incoming emails (#8253)
Adds the settings: 

raw_email_max_length, raw_rejected_email_max_length, delete_rejected_email_after_days. 

These settings control retention of the "raw" emails logs.

raw_email_max_length ensures that if we get incoming email that is huge we will truncate it removing uploads from the raw log.

raw_rejected_email_max_length introduces an even more aggressive truncation for rejected incoming mail. 

delete_rejected_email_after_days controls how many days we will keep rejected emails for (default 90)
2019-10-30 16:54:35 +11:00
romanrizzi 197ad8cd6e DEV: Run a faster test when warming up 2019-10-28 17:19:55 -03:00
romanrizzi d76d0e75ec DEV: Move warmup inside docker rake task 2019-10-25 16:31:05 -03:00
Gerhard Schlager 452d541b74 DEV: Add env variable to allow easier debugging of smoke tests
Set `DEBUG_NODE=1` when running `rake smoke:test` and use your favorite tool to debug the smoke tests. See https://nodejs.org/en/docs/guides/debugging-getting-started/ for more information.

The debugger will break at the beginning of the smoke tests when the env variable is set.
2019-10-24 17:48:15 +02:00
Krzysztof Kotlarek f34a0141c7 FIX: Correct path to ImportExport module (#8227)
During the move from Classic autoloader to Zeitwerk import_export module was moved to correct file name convention.
427d54b2b0 (diff-d896ec33b95afb7fae9f8bfe73d0580b)

Problem is that export/import is still using old path to require that module

Meta: https://meta.discourse.org/t/topic-and-category-export-import/38930/40
2019-10-23 17:27:14 +11:00
Daniel Waterworth 55a1394342 DEV: pluck_first
Doing .pluck(:column).first is a very common pattern in Discourse and in
most cases, a limit cause isn't being added. Instead of adding a limit
clause to all these callsites, this commit adds two new methods to
ActiveRecord::Relation:

pluck_first, equivalent to limit(1).pluck(*columns).first

and pluck_first! which, like other finder methods, raises an exception
when no record is found
2019-10-21 12:08:20 +01:00
Roman Rizzi 10478cb395
DEV: Bump uglifyjs (#7834)
* Rewrite uglifyjs command to work with 3.x

* Use ES5 syntax in plain JS files

* Use the older command if uglifyJS V2.x is installed
2019-10-09 10:02:49 -03:00
Sam Saffron 6471fbd453 FIX: maxminddb:get task no longer working
Following our zeitwerk upgrade we needed to be more explicit about the
require
2019-10-04 09:47:33 +10:00
Krzysztof Kotlarek 427d54b2b0 DEV: Upgrading Discourse to Zeitwerk (#8098)
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains. 

We no longer need to use Rails "require_dependency" anywhere and instead can just use standard 
Ruby patterns to require files.

This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
Joffrey JAFFEUX cb8fa46970
DEV: prevents populate task to crash (#8111)
Generated emails/names/sentences were crashing with the following error:

```
Can not transliterate strings with ASCII-8BIT encoding
```
2019-09-22 05:18:48 -07:00
Sam Saffron 1ca257be79 DEV: db:migrate no longer works after db:schema:load
In Rails 6 due to internal changes, the following sequence no longer works:

```
RAILS_ENV=test bin/rake db:migrate
RAILS_ENV=test bin/rake db:schema:dump
dropdb discourse_test
createdb discourse_test
RAILS_ENV=test bin/rake db:schema:load
RAILS_ENV=test bin/rake db:migrate
```

What appears to be happening is that our tracking of plugin migrations is
being missed on schema:dump or load.

A more comprehensive fix restoring schema:dump / load support will be
investigated.
2019-09-18 13:17:49 +10:00
Sam Saffron 025d4ee91f FIX: Rails 6 multisite migrations and plugin migrations
Prior to this change plugin migrations were not working and multisite
migrations not working.

Rails internals changed so we need to account for it.

Specifically semantics of `db:migrate` in rails changed so it is sort of
a "multisite:migrate".
2019-09-13 09:17:54 +10:00
Roman Rizzi 568232052e
DEV: Remove FlagQuery class and old code (#8064) 2019-09-12 13:21:33 -03:00
Gerhard Schlager 56c6e8e961 FIX: POP3 doesn't work with TLS 1.3 2019-09-11 18:43:19 +02:00
Michael Brown 503a11cc88 FIX: inline_uploads and subfolder (#8076)
* FIX: inline_uploads and subfolder

* if subfolder, also look for images with a path containing
  cdn_url + relative_url_root

* FIX: migrate_to_s3 task and subfolder
2019-09-11 11:50:48 +10:00
Gerhard Schlager 874985db71 DEV: Ignore duplicate prefixes when generating release notes 2019-09-04 15:45:48 +02:00
Arpit Jalan 75f37ac16a Rename `lazyYT` plugin directory name to `lazy-yt` 2019-08-21 14:35:14 +05:30
Sam Saffron 7632fe0b58 PERF: rake posts:rebake_uncooked_posts runs inline
Running this inline makes more sense otherwise there is extreme risk in
saturating sidekiq queue.

This also reworks ordering and selection so we double check if a post needs
rebaking prior to rebaking, this unlocks the ability to run this rake task
from multiple consoles.
2019-08-13 10:28:42 +10:00
Rafael dos Santos Silva 606c0ed14d
FIX: S3 uploads were missing a cache-control header (#7902)
Admins still need to run the rake task to fix the files who where uploaded previously.
2019-08-06 14:55:17 -03:00
Sam Saffron 4c6a0313f2 FEATURE: improve stats provided by rake db:stats
New stats include index size and sort by index + table size

Eg:

```
table_name                  | row_estimate | table_size | index_size | total_size
--------------------------------------------------------------------------
topic_views                 | 18260776     | 796 MB     | 2015 MB    | 2811 MB
post_timings                | 15813837     | 668 MB     | 1153 MB    | 1821 MB
posts                       | 386890       | 1532 MB    | 215 MB     | 1747 MB
post_search_data            | 317575       | 580 MB     | 162 MB     | 742 MB
incoming_emails             | 34412        | 734 MB     | 7160 kB    | 741 MB
incoming_links              | 5169302      | 306 MB     | 377 MB     | 683 MB
email_logs                  | 5            | 32 kB      | 418 MB     | 418 MB
topic_users                 | 2410166      | 217 MB     | 155 MB     | 372 MB
user_actions                | 1361253      | 101 MB     | 266 MB     | 366 MB
...

```
2019-08-02 17:25:42 +10:00
Vinoth Kannan e44d56e4d2 DEV: raise error only when 'STOP_ON_ERROR' env variable is available. 2019-08-01 23:54:06 +05:30
Daniel Waterworth 20bc4a38a5
FIX: DistributedMutex (#7953) 2019-08-01 09:12:05 +01:00
Guo Xiang Tan a1a61fcd2d DEV: Use `File#file?` in `posts:missing_uploads` rake task. 2019-08-01 10:59:34 +08:00
Rishabh dcb47d902b
REFACTOR: Rename SiteSetting.disable_edit_notifications to disable_system_edit_notifications (#7958)
* REFACTOR: Rename SiteSetting.disable_edit_notifications to disable_system_edit_notifications

- The older name could cause some confusion because the setting does not disable all edit notifications, only system ones.

* FIX: Add frozen_string_literal: true in the migration

* DEV: Deprecate 'disable_edit_notifications'
2019-07-31 20:20:41 +05:30
Penar Musaraj a1fb718340 DEV: Tweak plugin rake task console message
Show "Allowing write to all repos!" message only once.
2019-07-30 09:53:56 -04:00
Guo Xiang Tan 8a64b0c8e8 Revert "DEV: Remove unused kwarg and properly check for local missing uploads."
This reverts commit 97769f3d02.

The code is confusing but this change is quite risky. Defer for now
until we can look at it properly.
2019-07-29 14:35:34 +08:00
Guo Xiang Tan 97769f3d02 DEV: Remove unused kwarg and properly check for local missing uploads. 2019-07-29 14:21:06 +08:00
Guo Xiang Tan e9dc48ccae DEV: Fix multisite support for `posts:missing_uploads`. 2019-07-29 10:56:11 +08:00
Blake Erickson d26aa6e71e REFACTOR: Cleanup rake tasks based on feedback
Follow up to: [FEATURE: Create a rake task for destroying categories][1]

- `Discourse.system_user` is my friend
- Remove puts statements from rake tasks that don't return anything
- `for_each` is also my friend
- Use `human_users` to also exclude discobot
- Sort/format categories:list

[1]: 092eeb5ca3
2019-07-18 19:15:01 -06:00
Blake Erickson 092eeb5ca3 FEATURE: Create a rake task for destroying categories
Created a rake task for destroying multiple categories along with any
subcategories and topics the belong to those categories.

Also created a rake task for listing all of your categories.

Refactored existing destroy rake tasks to use new logging method, that
allows for puts output in the console but prevents it from showing in
the specs.
2019-07-17 12:44:14 -06:00
Rafael dos Santos Silva 1221d34284
FEATURE: Make Discourse work offline with WorkboxJS (#7870) 2019-07-15 13:05:55 -03:00
Gerhard Schlager ce8e099639 FEATURE: Use configured quotation marks in fancy topic title 2019-07-12 21:10:10 +02:00
Joffrey JAFFEUX 550e811652
DEV: allows lodash to be updated with `rake javascript:update` (#7881) 2019-07-11 16:57:03 +02:00
Penar Musaraj befcf67c90 DEV: run db:create and db:migrate on turbo specs
This ensures multisite specs run correctly when running ./bin/turbo_rspec
2019-07-08 14:27:51 -04:00
Sam Saffron b9a2a4d904 DEV: Avoid emoji for symbols
Rake task needs to work across a wide variety of terminal emulators.
Requiring an emoji font is superflous.
2019-07-05 09:41:15 +10:00
Ralph Rooding 1318e0b288 FEATURE: Rake themes installer (#7848)
* Delete remote_theme when deleting the theme

* Install themes and theme components through rake

* Removed unnecessary test
2019-07-04 14:33:05 -04:00
Joffrey JAFFEUX e9bb13c630
DEV: updates jquery-color to prevent it to pull old jquery version (#7857) 2019-07-04 15:03:21 +02:00
Daniel Waterworth d6aa92e98e DEV: Add a verbose option to ./bin/turbo_rspec 2019-06-27 15:49:21 +01:00
Daniel Waterworth b3ec3fac93 FIX: turbo_rspec doesn't accept these options 2019-06-26 17:05:27 +01:00
Daniel Waterworth 68d54d818a DEV: Add USE_TURBO option to docker rake task (#7811) 2019-06-26 11:54:39 -04:00
Guo Xiang Tan d21594f4f7 Revert changes added by mistake in 2b19e2acc8. 2019-06-25 15:25:12 +08:00
Guo Xiang Tan 2b19e2acc8 Fix typo in a0aeabbb94. 2019-06-25 15:18:57 +08:00
Guo Xiang Tan a0aeabbb94 DEV: Safer replacement for old scheme uploads for `posts:missing_uploads`. 2019-06-25 11:34:26 +08:00
Guo Xiang Tan f3c13d80c9 DEV: Don't copy file when trying to recover missing uploads. 2019-06-24 16:32:59 +08:00
Guo Xiang Tan 2cc5561504 DEV: Correct description in `posts:missing_uploads` rake task. 2019-06-24 14:57:43 +08:00
Sam Saffron 5bc92296be DEV: lint a bunch of files we missed 2019-06-21 11:33:41 +10:00
Daniel Waterworth e18ce56f4b DEV: Add a new way to run specs in parallel with better output (#7778)
* DEV: Add a new way to run specs in parallel with better output

This commit:

 1. adds a new executable, `bin/interleaved_rspec` which works much like
    `rspec`, but runs the tests in parallel.

 2. adds a rake task, `rake interleaved:spec` which runs the whole test
    suite.

 3. makes autospec use this new wrapper by default. You can disable this
    by running `PARALLEL_SPEC=0 rake autospec`.

It works much like the `parallel_tests` gem (and relies on it), but
makes each subprocess use a machine-readable formatter and parses this
output in order to provide a better overall summary.

(It's called interleaved, because parallel was taken and naming is
hard).

* Make popen3 invocation safer

* Use FileUtils instead of shelling out

* DRY up reporter

* Moved summary logic into Reporter

* s/interleaved/turbo/g

* Move Reporter into its own file

* Moved run into its own class

* Moved Runner into its own file

* Move JsonRowsFormatter under TurboTests

* Join on threads at the end

* Acted on feedback from eviltrout
2019-06-21 10:59:01 +10:00
Penar Musaraj 4bbb43cb2b DEV: pull plugin repos before linting them 2019-06-14 13:50:15 -04:00
Sam Saffron ecb2fd8222 DEV: improve db:create to catch all cases
- No RAILS_ENV - create multisite / dev / test
- `test` RAILS_ENV - create multisite / test
2019-06-14 15:47:05 +10:00
Sam Saffron a01488ae67 DEV: improve on rake db:create
Followup on 3af00a65 which broke build
2019-06-14 15:06:07 +10:00
Sam Saffron 3af00a65e6 FIX: site settings loading default values when no db
This fixes a condition where an intermittent db connection could cause
invalid site settings to be stored

It also removes a catch all we had.

Somewhere around Rails 5 `db:create` started wanting full environment
this is a problem for Discourse since it needs to boot up data from the
db.

This removes the catch all and surgically adds a db / redis bypass to
db:create task.
2019-06-14 14:21:07 +10:00
Guo Xiang Tan eaa7527933 DEV: Switch `posts:inline_uploads` scope to be more generic. 2019-06-14 11:53:34 +08:00
Penar Musaraj 66b15b9d87 DEV: support installing plugin gems in test images
- adds plugin:install_all_gems rake task

- adds UPDATE_ALL_PLUGINS env variable in docker:test
2019-06-13 09:58:57 -04:00
Guo Xiang Tan 9bce3d5e65 DEV: Multisite support for `rake posts:inline_uploads`. 2019-06-13 16:13:45 +08:00
Sam Saffron 215216df48 FIX: stop re-downloading backup maxmind
We omitted updating the mmdb_time which changed causing a constant
re-download of the maxmind db
2019-06-13 17:42:26 +10:00
Penar Musaraj e47ca6633b DEV: Fix plugin:update_all task
Reenable subtask so it loops through all plugins.

Only run for plugins that are git repositories.
2019-06-12 17:21:56 -04:00
Guo Xiang Tan d49c193289 FIX: `posts:inline_uploads` rake task needs to bypass bump. 2019-06-12 09:13:31 +08:00
Guo Xiang Tan 864222372f Remove incorrectly committed byebug.
Follow up to 49b921147a.
2019-06-11 16:33:49 +08:00
Guo Xiang Tan 49b921147a FIX: Undefined method in `posts:inline_uploads`. 2019-06-11 16:28:41 +08:00
Guo Xiang Tan 1cf3bf016c Make rubocop happy. 2019-06-10 09:49:02 +08:00
Guo Xiang Tan 29259b46ae DEV: Verbose mode for `posts:inline_uploads` rake task. 2019-06-10 08:59:11 +08:00
Joffrey JAFFEUX 8cd13b22a1
DEV: allows smoke test to run on macOS (#7735) 2019-06-07 18:20:48 +02:00
Bianca Nenciu f63b8bb79d FIX: Periodically ensure consistency of categories. (#7663) 2019-06-06 11:30:52 +02:00
Penar Musaraj f00275ded3 FEATURE: Support private attachments when using S3 storage (#7677)
* Support private uploads in S3
* Use localStore for local avatars
* Add job to update private upload ACL on S3
* Test multisite paths
* update ACL for private uploads in migrate_to_s3 task
2019-06-06 13:27:24 +10:00
Gerhard Schlager f7a2648694 FEATURE: Migrate uploads to S3 during restore 2019-06-04 15:47:36 +02:00
Gerhard Schlager fb121815a5 DEV: Allow rebaking oncooked posts of single site 2019-06-04 15:47:36 +02:00
Guo Xiang Tan 1991af2abb DEV: Switch `InlineUploads` to a regexp based implementation. 2019-06-04 15:54:25 +08:00
Guo Xiang Tan d93e5fb00d DEV: Class that converts MD with old attachment links to new MD. 2019-06-04 15:54:25 +08:00
Arpit Jalan af20d616de FEATURE: new rake task to invalidate broken images in post 2019-06-04 12:40:51 +05:30
David Taylor 81dcadf788
DEV: Stop compiling themes during DB migration. Recompile on cdn change. (#7676)
This is an improved implementation for bc8b7b13
2019-06-03 16:38:02 +01:00
Rafael dos Santos Silva 725588f835 FIX: migrate_to_s3 wasn't IAM profile aware 2019-06-01 12:09:46 -03:00
Arpit Jalan 388433ec06 DEV: remove unused method (rake task deleted in 6a274fd) 2019-05-31 11:16:10 +05:30
Arpit Jalan 5c526e5abb FEATURE: new rake task to watch all replied topics 2019-05-31 11:08:00 +05:30
Guo Xiang Tan 6a274fd2cd DEV: Remove redundant rake task.
Onceoff job created in 028121b95b
2019-05-31 10:05:13 +08:00
Joffrey JAFFEUX 630e9814bc
datetime is not available at this point (#7630) 2019-05-29 14:06:32 +02:00
Joffrey JAFFEUX 6439004161
DEV: do not use STDERR to print tests timestamps (#7629) 2019-05-29 13:28:02 +02:00
Joffrey JAFFEUX 5d47d1833b
DEV: prints timestamp at every docker:test commands (#7628) 2019-05-29 12:20:33 +02:00
Vinoth Kannan e12ae453e9 FIX: verify the exitence of s3_object properly without db name 2019-05-29 15:10:36 +05:30
Vinoth Kannan 9a9a06e34b DEV: add option to skip etag verification on 'migrate_to_s3' rake task 2019-05-29 14:16:36 +05:30
Vinoth Kannan b3779dc377 DEV: rename 'uploads:missing' rake task into 'uploads:missing_files'.
To improve the readability.
2019-05-28 23:30:43 +05:30
Ralph Rooding 04abf18194 Fixed typo in S3 rake task 2019-05-28 16:57:55 +08:00
Sam Saffron 9a232e1a0a FEATURE: use GIVE_UP=1 to inform rake posts:missing_uploads you are done
We need this give up for cases where uploads can not be recovered

This also improves the recovery routines
2019-05-28 14:44:59 +10:00
Sam Saffron c780d36ddc DEV: Correct posts:recover_uploads_from_index
Was creating a blank index file by mistake
2019-05-28 12:39:39 +10:00
Sam Saffron 3de8c58134 DEV: clean up backup maxmind path
This follows up on 6580025a

reuses a list and removes a map that was erronuously there.
2019-05-28 11:47:58 +10:00
Sam Saffron 7429700389 FIX: ensure we can download maxmind without redis or db config
This also corrects FileHelper.download so it supports "follow_redirect"
correctly (it used to always follow 1 redirect) and adds a `validate_url`
param that will bypass all uri validation if set to false (default is true)
2019-05-28 10:28:57 +10:00
Sam Saffron e4e2acf148 FIX: rake maxminddb:get broke cause it required environment 2019-05-28 08:59:28 +10:00
Sam Saffron 6580025af9 FEATURE: add backup directory for mmdb files
This new `DISCOURSE_MAXMIND_BACKUP_PATH` can be used a secondary location
for maxmind db. That way a build machine, for example can cache it on the
host and reuse between builds.

Also per 5bfeef77 added proper error raising for download fails from
dedicated rake task

This also moves "refresh_maxmind_db_during_precompile_days" to a global
setting, it did not make sense in a site setting
2019-05-27 16:51:24 +10:00
Régis Hanol 2f5938c3bf DEV: use FileHelper to download MaxMindDB 2019-05-24 18:11:39 +02:00
Régis Hanol 8c9ffee3af TEMP: log more information when failing to download db 2019-05-24 17:33:52 +02:00
Régis Hanol 9ed2c54e27 TEMP: log information when failing to download db 2019-05-24 17:26:04 +02:00
Régis Hanol 0e24cb0f78
DEV: don't skip discourse-calendar plugin specs 2019-05-24 11:35:22 +02:00
Sam Saffron 9ce06d53d1 DEV: improve upload recovery rake task
- Better error handling
- Automatically generates index file if missing
- Improved output
2019-05-24 13:36:03 +10:00
Guo Xiang Tan 791241e0b2 DEV: Fix rake task to delete old export topics. 2019-05-24 11:20:53 +08:00
Sam Saffron 76f7e90e23 FEATURE: rake posts:recover_uploads_from_index improvements
We now recover more smartly based on failed db remaps as well.
2019-05-24 12:38:38 +10:00
Sam Saffron 1059aafc55 FEATURE: rake task for recovery of uploads from an index file
`rake posts:recover_uploads_from_index`

Searches through all missing uploads in the cluster, if it finds one it
tries to find it in the "upload index file" and creates a new upload for
it.
2019-05-23 17:08:22 +10:00
Sam Saffron 7ce58df7bf lint the file
somehow I did not notice this on save
2019-05-23 15:28:41 +10:00
Sam Saffron a5ce9cb470 FEATURE: fix_relative_upload_links now multisite safety
This also finds `<img src="/uploads/xyz` HTML images in raw and corrects
them. Also handles some cross multisite recovery and provides better output
2019-05-23 15:09:16 +10:00
Sam Saffron 96e58125a7 FEATURE: rename task to posts:destroy_old_data_exports
Previously we were only catching one type of data export, the new job will
catch every csv export we have

Job is pretty safe as it filters on system user id / pm with a particular
slug
2019-05-23 15:02:56 +10:00
Sam Saffron 954293655f FEATURE: rake posts:destroy_old_user_data_exports
Historically we would keep the user data export posts around but delete
the uploads.

This leaves a lot of broken uploads in the system.

This rake task allows us to clean up old mess.
2019-05-23 11:11:37 +10:00
Sam Saffron e8799f0ba4 DEV: improve uploads:recover job so it stores a map of old to new sha
Previous attempt created broken images
2019-05-22 15:51:09 +10:00
Sam Saffron ebcb571de7 FIX: allow upload recovery to recover uploads with sha mismatch
Filename on disk may mismatch sha of file in some old 1X setups. This will
attempt to recover file even if sha1 mismatches. We had an old bug that
caused this.

This also adds `uploads:fix_relative_upload_links` which attempts to replace
urls of the format `/upload/default/...` with `upload://`
2019-05-22 15:24:36 +10:00
Sam Saffron f772ecc597 DEV: Correct missing output detailing failure
uploads:s3_migration_status was failing but not returning proper output
2019-05-22 12:58:54 +10:00
Guo Xiang Tan 5429c9b5e9 DEV: Fix incorrectly hardcoded value in rake task. 2019-05-22 09:01:25 +08:00
Sam Saffron a9af77d650 PERF: ensure priority for rake rebake tasks is ultra low
Flooding the default queue with lots of jobs is never a good thing. This
reduces the risk of sidekiq going into a bad state post rebake
2019-05-22 10:31:49 +10:00
Sam Saffron d6daa60d0a FEATURE: multisite support for posts:rebake_uncooked_posts 2019-05-22 10:04:54 +10:00
Sam Saffron a2428bd862 FEATURE: warn about sidekiq overload prior to migrating
Also makes pre-flight check ensure there is no giant backlog of posts that
need to be cooked
2019-05-22 10:04:33 +10:00
Sam Saffron 5fdc7b7ca2 Correct 59012fc0
Logic was flipped here by mistake, oops
2019-05-22 09:48:03 +10:00
Sam Saffron 73f178a634 FEATURE: posts:rebake_uncooked_posts to look at mismatching baked_version
also amends flagging onebox updates to set baked_version to nil
2019-05-22 09:43:31 +10:00
Sam Saffron 4f296608da FEATURE: add uploads:s3_migration_status for looking at current status
Also a few minor cleanups and better progress reporting
2019-05-22 09:00:32 +10:00
Sam Saffron 59012fc0f7 PERF: mark posts for rebake vs forcing a rebake inline when migrating to s3
Rebaking posts can be expensive instead of blocking here simply mark posts
for rebake.

We can then work through them faster in other jobs, plus this should not
hold of a datacenter migration.
2019-05-22 08:39:25 +10:00
Vinoth Kannan 338b927f95 DEV: include local uploads while finding missing uploads
in rake task 'posts:missing_uploads'
2019-05-21 22:07:08 +05:30
Vinoth Kannan 7b82850f66 FIX: migrate_to_s3 task should remap attachment links too. 2019-05-21 21:58:11 +05:30
Sam Saffron 8360415453 FEATURE: big warning for uploads missing which can be very very slow on S3 2019-05-21 16:11:56 +10:00
Sam Saffron cb86d8279a FEATURE: add toggle for uploads:missing so it can skip external
Validating s3 uploads in uploads:missing can be very expensive, allow to
bypass.
2019-05-21 16:11:56 +10:00
Sam Saffron f4d4f7871e FEATURE: make posts:missing_uploads multisite friendly
Previously this rake job would only run on a single site which is a bit
misleading

This also adds `VERBOSE=1 rake posts:missing_uploads` that will provide a
full report of missing uploads
2019-05-21 12:45:51 +10:00
David Taylor 0fbff66d97 DEV: Correct rake task syntax error 2019-05-20 17:43:30 +01:00
David Taylor 31902159af DEV: Allow skipping failed migrations when running S3 migration
Use the SKIP_FAILED environment variable to skip failed sites. Use with caution - make sure you go back and re-run the failed migrations afterwards.
2019-05-20 17:25:56 +01:00
David Taylor a15cca9a0f DEV: Improve error message for posts:missing_uploads during S3 migration 2019-05-20 16:09:22 +01:00
Vinoth Kannan 2bfc0cf145 FIX: skip old scheme upload URLs while validating s3 uploads remap 2019-05-20 19:13:41 +05:30
Vinoth Kannan 9f43852609 FIX: remove the unwanted code. 2019-05-20 19:10:21 +05:30
David Taylor 41bc90dd3e DEV: Add progress indicator for post rebake during s3 migration
Now that we run sidekiq jobs synchronously, this is important
2019-05-20 14:19:58 +01:00
David Taylor 77a06209cb DEV: Skip S3 migration if all uploads are already migrated
This makes the task resumable in a multisite context
2019-05-20 14:17:37 +01:00
Vinoth Kannan bc0c4b7253 FIX: should not migrate the system uploads to s3 2019-05-20 14:27:34 +05:30
Vinoth Kannan be3fb85a04 DEV: add post migration checks and raise error if failed. 2019-05-20 14:18:28 +05:30
Sam Saffron 50e6ee55a9 DEV: add "WAIT_FOR_URL" env to smoke test
This allows you to wait up to N seconds for the smoke test url to come up
in some cases you want to kick off the smoke test prior to having the smoke
test env ready to accept connections
2019-05-20 17:23:09 +10:00
Sam Saffron 08371db0cc FIX: ensure we don't queue any jobs during s3 migration
Previously we could flood sidekiq with jobs which is not ideal.

This ensures we are 100% done when the job is done.
2019-05-20 16:28:50 +10:00
Joffrey JAFFEUX 9ce9d72e71
DEV: makes hidepassed default when running qunit (#7558)
Mostly useful when not running headless, but I endup doing it a lot when debugging, one less thing to check.
2019-05-16 14:37:01 +02:00
Joffrey JAFFEUX 4a9756ff3f
DEV: sets rack server to test env when using rake qunit:test (#7554) 2019-05-16 10:44:29 +02:00
David Taylor 5605dba85c DEV: Automatically annotate plugin models alongside core models 2019-05-13 16:37:47 +01:00
David Taylor 750c125707 DEV: Include foreign keys in model annotations 2019-05-13 15:53:42 +01:00
Guo Xiang Tan c00dab89e4 Fix the build take 2. 2019-05-13 11:22:48 +08:00
Guo Xiang Tan 8c4caac2e4 Fix frozen string error in `plugin:install_all_official`. 2019-05-13 10:19:02 +08:00
Guo Xiang Tan 8165ceb320 Make rubocop happy. 2019-05-13 09:55:44 +08:00
Sam Saffron 30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
David Taylor 20daa76829 DEV: Change priority to ultra_low for post-s3-migration rebake 2019-05-10 18:37:45 +08:00
Guo Xiang Tan 626808e100 Fix incorrect condition in `groups:grant_badge` rake task.
`#find` raises an error if the id given to it is invalid. As a result,
the conditional to check whether a `group` or `badge` is `present?` will
not be executed if any of the ids are invalid.

Follow up to
6ba914033c.
2019-05-09 14:53:13 +08:00
Vinoth Kannan 87cd4701b8 FEATURE: option to skip posts with ignored missing uploads 2019-05-09 05:11:15 +05:30
Penar Musaraj a92c925378 Use faster Brotli compression for non-default locale files 2019-05-08 14:55:24 -04:00
Penar Musaraj d9ab722d05 Revert "Revert "PERF: Skip compressing locales for faster rebuilds (#7501)" (#7502)"
This reverts commit 6ea811c923.
2019-05-08 14:55:06 -04:00
Joffrey JAFFEUX 6ea811c923
Revert "PERF: Skip compressing locales for faster rebuilds (#7501)" (#7502) 2019-05-08 07:54:21 +02:00
Penar Musaraj 22d6203eb7 DEV: Use moment-timezone with 10-year range data
We do not need to use the full timezone dataset (it is a 950kb file!), only previous/next 5 years should suffice.
2019-05-07 23:42:49 -04:00
Sam Saffron 5c0bb29461 DEV: re-enable compression for node asset compilation
#b9d82818 makes enormous improvements to our bootstrap time, however going
to still keep compress for now despite the cost and watch it for a few weeks
2019-05-08 10:24:57 +10:00
Penar Musaraj b9d8281818 PERF: Skip compressing locales for faster rebuilds (#7501)
* Do not brotli all locales in precompile

* Try without gzip

* uglify without compressing, always gzip

* skip uglify for unused locales

* FIX: Uglifier needs harmony for ES6 compatibility

* Use node uglifier if available

* Minor refactor
2019-05-08 10:22:19 +10:00
Guo Xiang Tan 2adbec1b3c PERF: Speed up migrate_to_s3 rake task by deleting optimized image record.
No point moving all optimized image files to tombstone when the store is
changing. Also, `destroy_all` can easily blow memory since we are no
loading in batches.
2019-05-07 16:10:32 +08:00
Sam Saffron 9be70a22cd DEV: introduce new API to look up dynamic site setting
This removes all uses of both `send` and `public_send` from consumers of
SiteSetting and instead introduces a `get` helper for dynamic lookup

This leads to much cleaner and safer code long term as we are always explicit
to test that a site setting is really there before sending an arbitrary
string to the class

It also removes a couple of risky stubs from the auth provider test
2019-05-07 11:00:30 +10:00
Arpit Jalan 6ba914033c FEATURE: rake task to bulk grant badge to members of a group 2019-05-06 12:41:50 +05:30
Vinoth Kannan e8fafbc123 List and restore missing post uploads from S3 inventory. 2019-05-04 01:16:20 +05:30
Penar Musaraj a5b804fec4 DEV: Add --profile and --fail-fast when running plugin specs, tweak parallel test balancing 2019-05-01 22:47:18 -04:00
David Taylor 0e303c7f5d
FEATURE: Automatically generate optimized site metadata icons (#7372)
This change automatically resizes icons for various purposes. Admins can now upload `logo` and `logo_small`, and everything else will be auto-generated. Specific icons can still be uploaded separately if required.

## Core

- Adds an SiteIconManager module which manages automatic resizing and fallback

- Icons are looked up in the OptimizedImage table at runtime, and then cached in Redis. If the resized version is missing for some reason, then most icons will fall back to the original files. Some icons (e.g. PWA Manifest) will return `nil` (because an incorrectly sized icon is worse than a missing icon). 

- `SiteSetting.site_large_icon_url` will return the optimized version, including any fallback. `SiteSetting.large_icon` continues to return the upload object. This means that (almost) no changes are required in core/plugins to support this new system.

- Icons are resized whenever a relevant site setting is changed, and during post-deploy migrations

## Wizard

- Allows `requiresRefresh` wizard steps to reload data via AJAX instead of a full page reload

- Add placeholders to the **icons** step of the wizard, which automatically update from the "Square Logo"

- Various copy updates to support the changes

- Remove the "upload-time" resizing for `large_icon`. This is no longer required.

## Site Settings UX

- Move logo/icon settings under a new "Branding" tab

- Various copy changes to support the changes

- Adds placeholder support to the `image-uploader` component

- Automatically reloads site settings after saving. This allows setting placeholders to change based on changes to other settings

- Upload site settings will be assigned a placeholder if SiteIconManager `responds_to?` an icon of the same name

## Dashboard Warnings

- Remove PWA icon and PWA title warnings. Both are now handled automatically.

## Bonus

- Updated the sketch logos to use @awesomerobot's new high-res designs
2019-05-01 14:44:45 +01:00
David Taylor bc8b7b13ef DEV: Recompile theme fields after db:migrate 2019-04-30 11:52:30 +01:00
Vinoth Kannan c448e014fe Recover missing old scheme uploads from tombstone directory too. 2019-04-29 11:33:57 +05:30
Vinoth Kannan 8698a10920 Don't raise error if a record already exists. 2019-04-25 21:48:39 +05:30
Arpit Jalan 5a70367a05 fix the build 2019-04-25 20:19:52 +05:30
Vinoth Kannan a65ce81e25 Don't check same upload url twice in the rake task 2019-04-25 20:06:10 +05:30
Arpit Jalan 0d331d33c3 DEV: do not use find_each for `avatars:clean` task
https://review.discourse.org/t/fix-show-available-details-when-rescuing-error/2865
2019-04-25 18:46:34 +05:30
Vinoth Kannan 1724c27713 FIX: reload the 'post' model to retrive raw field value. 2019-04-25 02:09:27 +05:30
Vinoth Kannan aed683390c FIX: Don't treat 'upload_patterns' as constant to make the rake task compatible with multisite 2019-04-25 02:06:20 +05:30
Vinoth Kannan b0bb1d95a5 trying to recover old scheme uploads on 'posts:missing_uploads' rake task 2019-04-25 01:16:18 +05:30
Vinoth Kannan 3dae12dec1 reduce unlinked count from missing post upload count 2019-04-19 21:53:00 +05:30
Vinoth Kannan 4fcefc3615 add unlinked missing post upload count in stat 2019-04-19 20:39:21 +05:30
Vinoth Kannan ba6369edc5 Add more missing upload stats in rake task 2019-04-19 20:14:40 +05:30
Vinoth Kannan 48cedf3ebe fix: retrieve posts in order by id.
so we can find starting and ending id of the batch.
2019-04-19 18:30:59 +05:30
Vinoth Kannan ce8d30f667 PERF: use 'delete_all' method instead of 'destroy_all' 2019-04-19 18:00:01 +05:30
Vinoth Kannan 5de483a1b9 PERF: Very post upload's existence by preloaded upload sha1s array 2019-04-19 17:27:16 +05:30
Arpit Jalan 989325ea7c FIX: show available details when rescuing error 2019-04-15 14:09:52 +05:30
Arpit Jalan 07b961ab13 DEV: rescue avatar rake tasks with error backtrace 2019-04-15 14:06:29 +05:30
Arpit Jalan a423e9b14a FIX: rescue `avatars:refresh` rake task 2019-04-14 14:26:15 +05:30
David Taylor 24ef4f7b2b Remove support for disable_jump_reply user setting (#7359) 2019-04-12 09:03:06 +10:00
Sam Saffron 84f12ac3f7 DEV: No need to carry two brotli implementations around
old version of brotli binary is long gone, require brotli 1.0.0 and up
2019-04-11 16:53:29 +10:00
Arpit Jalan a20f58554b IMPORT: create category definitions in `import:ensure_consistency` task 2019-04-11 12:06:37 +05:30
Sam Saffron baa7a9836c FEATURE: remove "COMPRESS_BROTLI" optional behavior
The compress brotli functionality is no longer optional, this has worked
well for years. The name of the ENV var is also confusing cause it does
not have a `DISCOURSE_` prefix which caused issues with the web upgrader

Brotli support is now unconditionally on
2019-04-11 12:36:18 +10:00
Vinoth Kannan 8d5c900142 DEV: add unique missing uploads index in post custom fields
https://review.discourse.org/t/feature-send-missing-post-uploads-stat-to-prometheus/2609/6?u=vinothkannans
2019-04-10 18:09:35 +05:30
Bianca Nenciu c523a12187 FIX: Fix assets:precompile rake task. 2019-04-10 14:03:26 +03:00
Bianca Nenciu 4555d0c598 FEATURE: Refresh MaxmindDb during assets:precompile. (#7340) 2019-04-10 11:37:29 +02:00
Bianca Nenciu b5008586c5 DEV: Remove HTML parser from Tautologistics. (#7344) 2019-04-10 11:21:22 +02:00
Vinoth Kannan d0fe42e2ef FIX: should look through posts for image markdown
Downloaded onebox images only included in the cooked HTML content. So we have to check 'post.cooked' instead of 'raw'. bfdd0fe64c
2019-04-10 13:52:35 +05:30
Vinoth Kannan 4878ee95e5 DEV: optimize sql query to narrow down the filtering of post with uploads 2019-04-10 12:58:10 +05:30
Vinoth Kannan bfdd0fe64c FIX: empty the missing list on each post loop 2019-04-10 11:20:26 +05:30
Vinoth Kannan 914ada1c74 DEV: convert scheduled job EnsurePostUploadsExistence into a rake task 2019-04-09 02:07:35 +05:30
Robin Ward b58867b6e9 FEATURE: New 'Reviewable' model to make reviewable items generic
Includes support for flags, reviewable users and queued posts, with REST API
backwards compatibility.

Co-Authored-By: romanrizzi <romanalejandro@gmail.com>
Co-Authored-By: jjaffeux <j.jaffeux@gmail.com>
2019-03-28 12:45:10 -04:00
Guo Xiang Tan 71e431de9e DEV: Fix hardcoded value introduced in cc2bac86e9. 2019-03-26 07:45:21 +08:00
Gerhard Schlager cd4fd447ca Make Rubocop happy 2019-03-25 17:04:45 +01:00
Guo Xiang Tan cc2bac86e9 FIX: Dry run broken for `uploads:migrate_to_s3`. 2019-03-25 22:38:24 +08:00
Guo Xiang Tan 19c3c25db1 FIX: Handle BBCode in `migrate_to_s3` task as well. 2019-03-22 16:47:06 +08:00
Guo Xiang Tan 2d34be24be FIX: Rebake lightbox and use short upload urls in `migrate_to_s3`. 2019-03-22 13:09:59 +08:00
Joffrey JAFFEUX c18a6069c2
DEV: upgrades bootstrap-modal (#7235) 2019-03-21 14:18:12 +01:00
Guo Xiang Tan 4e594f2b2b FIX: Destroy optimized images in `upload:migrate_to_s3` rake task.
`OptimizedImage` are currently not renegerated when the image changes
store.
2019-03-21 16:50:15 +08:00
Guo Xiang Tan 8e5e5d7d35 DEV: Make setting up of multisite DB in test env clearer. 2019-03-21 09:58:07 +08:00
Sam Saffron 600313373b DEV: skip creating multisite db when running JS tests only 2019-03-21 12:19:12 +11:00
Penar Musaraj ed74774001 DEV: rebalance parallel tests (again) 2019-03-19 10:21:50 -04:00
Penar Musaraj 60b6c67c1c DEV: rebalance parallel tests (take 2) 2019-03-19 09:59:03 -04:00
Penar Musaraj c8e1d3c2e7 DEV: rebalance parallel tests 2019-03-19 09:23:10 -04:00
Gerhard Schlager 8a99d59422 FIX: seed data classes couldn't always be found 2019-03-18 23:13:47 +01:00
Gerhard Schlager 3fd04df781
FEATURE: Locale support for seeded categories and topics (#7110) 2019-03-18 21:09:13 +01:00
Joffrey JAFFEUX d91b47064e
FIX: windows has no country flags (#7199) 2019-03-18 19:28:49 +01:00
Gerhard Schlager 81108ee783 DEV: Increase timeout for JS tests and add env variable 2019-03-18 17:39:53 +01:00
Régis Hanol 9dbca41152 FIX: don't check system uploads in migrate_to_s3 rake task 2019-03-18 11:01:18 +01:00
Joffrey JAFFEUX e1f5620b04
do not skip until we re-merge ember 3.7 (#7183) 2019-03-15 18:31:24 +01:00
Penar Musaraj b43f1687ee FIX: Update user options in rake task 2019-03-15 11:41:16 -04:00
Joffrey JAFFEUX aa4b84925a FEATURE: updates emoji to latest 12.0 version (#7095)
* improved emoji support

- always optimize images as part of the task
- use the unicode standard ordering/naming for sections

* UX: more height for when there are recently used
2019-03-15 16:12:07 +01:00
Régis Hanol a6a52d1523 FIX: images:compress rake task
Optimized a few unoptimized PNGs.
2019-03-15 15:23:57 +01:00
Kelvin 45a97ad82f Fix SMTP connection test
This test did not support 'no auth' use case and other auth methods except 'login'. I fixed it by simply making the call to start() in the right way. 
As shown in the source code of Net::SMTP (https://github.com/ruby/ruby/blob/ruby_2_5/lib/net/smtp.rb#L452), the start() function does accept the 'user' and 'secret' arguments. Also, in do_start() function (https://github.com/ruby/ruby/blob/ruby_2_5/lib/net/smtp.rb#L542), it automatically checks the auth method and args, skips the authentication if 'user' is not provided, and selects the right auth method from 'plain', 'login' or 'cram_md5'. This is exactly all of what we should do in a connection test and the odd 'auth_login' call in the previous code makes problems. 
BTW, I am using 'localhost' as the third argument, which is the same as the default value in start(). This parameter is the domain address sent along with the 'ehlo' command in SMTP protocol. I have seen some documents, e.g. https://github.com/tpn/msmtp/blob/master/doc/msmtp.1#L455, saying that 'localhost' is fine. It works for me.
2019-03-15 09:50:51 -04:00
Joffrey JAFFEUX 9a5f08ee61
DEV: skip discourse-chat-integration tests (#7164)
Suspected of causing timeout issues in our test suite with Ember 3.7
2019-03-13 16:50:56 +01:00
Penar Musaraj 49cdb072d7 DEV: Use `--profile` and `--fail-fast` in CI only 2019-03-11 22:04:47 -04:00
Guo Xiang Tan d82876896e FIX: `uploads:migrate_to_s3` broken for GlobalSetting using file provider. 2019-03-11 14:21:35 +08:00
David Taylor bc52437153 FIX: Add additional rescue for assets:precompile
This error can be caused if assets are precompiled before the database has been migrated
2019-03-07 14:03:05 +00:00
Régis Hanol ad87b0d662
Make "uploads:recover_from_tombstone" call the "uploads:recover" rake task 2019-03-07 14:15:30 +01:00
Penar Musaraj edffd0097f
DEV: Support running Ruby tests in parallel (#7087) 2019-02-28 16:36:22 -05:00
Penar Musaraj 9bf11a7c02
FEATURE: Add localized timezone names (#7004)
Uses github.com/discourse/moment-timezone-names-translations to translate timezone names.
Plugins can also provide their own timezone name translations.
2019-02-25 14:40:02 -05:00
Guo Xiang Tan 433f07fcb3 Fix confusing ENV in `migrate_to_s3` rake task. 2019-02-22 14:44:46 +08:00
Rishabh e69634ec3a FIX: Use s3_endpoint in migrate_to_s3 when not using S3
- overrides :region and uses :endpoint when SiteSetting.s3_endpoint is provided
- Now, we can use the new rake task with DigitalOcean Spaces
- I've tested that it's compatible with/without bucket folder path
- I've tested that it's compatible with S3 and it doesn't break S3 for non-default regions
- follow-up on 97e17fe0
2019-02-21 15:42:48 +05:30
Vinoth Kannan 0472bd4adc FIX: Remove 'backfill_etags' keyword argument from 'uploads:missing' rake task
And etags backfilling code is optimized
2019-02-15 00:34:35 +05:30
Penar Musaraj e55c19d8ce DEV: update ACE Editor to 1.4.2
Remove duplicate "src-min" folder and fix rake task

Remove unused "snippets" folder
2019-02-14 11:05:28 -05:00
Penar Musaraj 090e9c8432
FIX: make composer full screen shortcut work when inputs have focus (#6907)
- Uses a Mousetrap plugin for global shortcuts
- Implemented for search `ctrl+alt+f` and composer fullscreen `shift+f11` shortcuts
2019-02-14 00:19:27 -05:00
Vinoth Kannan 7b5931013a Update rake task to backfill etags from s3 inventory 2019-02-14 05:18:06 +05:30
Penar Musaraj 5cef5b34ef DEV: Yarn-manage moment and moment-timezone libraries 2019-02-12 13:57:52 -05:00
Joffrey JAFFEUX 831d7d2f87
run tests on discourse-code-review (#6988) 2019-02-08 11:39:45 +01:00
Penar Musaraj 995c87a91e Don't run specs on code-review again 2019-02-07 16:57:05 -05:00
Penar Musaraj d639cadb7a
DEV: Plugin scss errors should break precompile (#6974) 2019-02-07 09:27:42 -05:00
David Taylor 68173cd234 Revert "DEV: Allow rake plugin:spec to traverse symlinks"
This reverts commit dfd63b185f.

https://meta.discourse.org/t/108110/11
2019-02-01 12:40:29 +00:00
David Taylor dfd63b185f DEV: Allow rake plugin:spec to traverse symlinks 2019-02-01 10:40:41 +00:00
Arpit Jalan 82b7795f36 Revert "don't run specs on code-review for now"
This reverts commit cb493b66e0.
2019-01-29 11:49:42 +05:30
Joshua Rosenfeld cb493b66e0
don't run specs on code-review for now
code-review is currently causing failing builds. Remove from testing until it is fixed
2019-01-24 11:48:19 -05:00
Guo Xiang Tan 0cf2df3028 Fix remap in `migrate_to_s3` rake task.
The current way of doing the remap only allows to run the rake task
once. Running the rake task more than once will end up badly.
2019-01-23 15:50:44 +08:00
Guo Xiang Tan 07850994d3 Add ENV to skip multisite prefix when migrating to s3. 2019-01-23 15:19:50 +08:00
Guo Xiang Tan 979d03aa68 Remove s3 bucket check in `migrate_to_s3` task.
Bucket creation is expected to be handled by the user. If the bucket
does not exist, the script will fail anyway.
2019-01-23 15:04:51 +08:00
Guo Xiang Tan 99cd3ff6ee FIX: `migrate_to_s3` task not setting the right content_disposition. 2019-01-23 15:04:47 +08:00
Rishabh 97e17fe084 FIX: Use ENV values instead of 'S3Helper.s3_options' in migrate_to_s3
This commit makes the rake task operational for all regions for s3. If we declare s3_endpoint as https://s3.amazonaws.com while
creating an instance of Aws::S3::Client, head_bucket fails for all s3 regions apart from us-east-1. The commit manually defines all
parameters for Aws::S3::Client apart from s3_endpoint to bypass this problem make this task usable for AWS S3.

Removing s3_endpoint from the payload means that custom endpoints like Minio/DO Spaces for will not work in the meantime and we'll
have to add support for a custom `s3_endpoint` in the future.

This commit follows up on 60790eb0.
2019-01-20 20:55:27 +05:30
Penar Musaraj 020e0d623a DEV: run "yarn install" before lint tests 2019-01-17 14:17:44 -05:00
Michael Brown 664c282400 FIX: improve rake emails:test task
* FIX: rake emails:test to bypass the "STARTTLS required" message

* FIX: too much j

* hint meta.discourse.org to people with unknown errors from the mail test
2019-01-17 13:41:10 +11:00
Rishabh 60790eb006 FIX: Use GlobalSetting values instead of ENV variables in migrate_to_s3
TIL how GlobalSetting works in sync with environment variables
Also fixes a small bug where bucket value was being used when it could have been nil
2019-01-16 14:40:38 +05:30
Rishabh ff8f9dc1c9 FIX: prefix should precede folder path (follow-up on 10fbb07e) 2019-01-15 15:58:19 +05:30
Rishabh 10fbb07e1a FIX: include folder name in prefix for listing files on S3 (follow-up on 3ec38f5a)
Fix the destination url in remap since it's already a part of s3_base_url
2019-01-15 15:23:55 +05:30
Penar Musaraj b28d68efbf DEV: Use yarn to manage Highlight.js dependency
Moves Highlight.js files to vendor/assets/javascripts

Adds Highlight.js in yarn package management

Removes old rake task and reliance on NPM to build Highlight.js

Highlight.js is now integrated in the "javascript:update" rake task
2019-01-14 12:46:35 -05:00
Florian Heidenreich 39983f9d95 FIX: list staff users within the last month 2019-01-14 10:29:00 -05:00