Commit Graph

7562 Commits

Author SHA1 Message Date
Daniel Waterworth 4a9ee25c56 DEV: Add logging for stack level too deep exception in HtmlToMarkdown 2020-07-09 12:25:00 +01:00
Martin Brennan e0713455ca
PERF: Load topic bookmarks for the user in user_post_bookmarks (#10197)
Instead of loading all of the user bookmarks using all the post IDs in a topic, load all the bookmarks for a user using the topic ID. This eliminates a costly WHERE ID IN query.
2020-07-09 15:46:52 +10:00
Martin Brennan 31e31ef449
SECURITY: Add content-disposition: attachment for SVG uploads
* strip out the href and xlink:href attributes from use element that
  are _not_ anchors in svgs which can be used for XSS
* adding the content-disposition: attachment ensures that
  uploaded SVGs cannot be opened and executed using the XSS exploit.
  svgs embedded using an img tag do not suffer from the same exploit
2020-07-09 13:31:48 +10:00
Bianca Nenciu bd842cd2b0
FEATURE: Parse images in email signatures (#10137)
* FEATURE: Parse images in email signatures

* DEV: Fix tests

* Code review
2020-07-08 15:50:30 +10:00
Guo Xiang Tan 2e1eafae06
FIX: Incorrect search blurb when advanced search filters are used. 2020-07-08 11:59:49 +08:00
Gerhard Schlager ab4c0a4970 FEATURE: Create SQL-only backup if there are no uploads
It doesn't make sense to compress the database dump twice if the backup doesn't contain any uploaded files.
2020-07-07 16:23:47 +02:00
Joffrey JAFFEUX 9b7000dbf1
FIX: ensures category order keeps consistent (#10165)
Before this change:
- first full page load would get category defaults defined un cateory settings
- a navigation to a topic and then back to categories list would reset defaut to the ones defined in discovery/topics
2020-07-07 09:56:38 +02:00
Jeff Wong 339549d14a
Support plugin and Theme compatibility version manifests (#9995)
Adds a new rake task `plugin:checkout_compatible_all` and
`plugin:checkout_compatible[plugin-name]` that check out compatible plugin
versions.

Supports a .discourse-compatibility file in the root of plugins and themes that
list out a plugin's compatibility with certain discourse versions:

eg: .discourse-compatibility
```
2.5.0.beta6: some-git-hash
2.4.4.beta4: some-git-tag
2.2.0: git-reference
```

This ensures older Discourse installs are able to find and install older
versions of plugins without intervention, through the manifest only.

It iterates through the versions in descending order. If the current Discourse
version matches an item in the manifest, it checks out the listed plugin target.
If the Discourse version is greater than an item in the manifest, it checks out
the next highest version listed in the manifest.

If no versions match, it makes no change.
2020-07-06 14:48:00 -07:00
Jarek Radosz cb048d284d FIX: Handle the case where upload goes missing during downsizing 2020-07-06 18:51:38 +02:00
David Taylor 7f2b5a446a
PERF: Remove post_upload recovery in daily EnsureS3UploadsExistence job (#10173)
This is a very expensive process, and it should only be required in exceptional circumstances. It is possible to run a similar recovery using `rake uploads:recover` (5284d41a8e/lib/upload_recovery.rb (L135-L184))
2020-07-06 16:26:40 +01:00
Jarek Radosz 64ce12a758
FIX: `OptimizedImage#filesize` (#10095)
`OptimizedImage#filesize` calls `Discourse.store.download` with an OptimizedImage as an argument. It would in turn attempt to call `#original_filename` and `#secure?` on that object. Both would fail as these methods do not exist on OptimizedImage, only on Upload. We didn't know about these issues because:
1. `#calculate_filesize` is not called often, because the filesize is saved on OptimizedImage creation, so it's used mostly for manual filesize recalculation
2. we were using `rescue nil` which swallows all errors
2020-07-06 17:01:29 +02:00
Daniel Waterworth 1bd8a075d8 FIX: Make Email::Styles operate on html documents instead of fragments
`Nokogiri::HTML.fragment` is a huge hack (a comment in the source code
admits this). The current behavior of `Email::Styles` is to try to
emulate `fragment` using nokogumbo, but it misses some edge cases. In
particular, meta tags in a email template don't make it through to the
final email.

Instead of treating the provided HTML as an indeterminate fragment, this
commit makes `Email::Styles` treat the HTML as a complete document. This
means that the generated HTML for an email will now always contain top
level structure (a doctype, html, head and body tags).

This new behavior is behind a hidden site setting for now and defaults
off.
2020-07-06 11:45:39 +01:00
David Taylor 977766e7a8
FEATURE: sso_overrides_(email|username|name) for all auth methods
These settings previously applied only to discourse-sso. Now they work for all external authentication methods.
2020-07-06 10:18:45 +01:00
David Taylor ec448a1516
DEV: Refactor Auth::Result for readability, recreate during signup flow 2020-07-06 10:18:41 +01:00
Vinoth Kannan 6d17765924 PERF: use post number to create canoncial path in mega topics.
We don't need page number accuracy in mega topics since it can be expensive.

06d426bd87
2020-07-06 10:31:19 +05:30
Sam Saffron 38a30a6e96
DEV: correct regression and correct tests
etag change in 31976ecf was incorrect, revert it

Also correct regression in test suite.
2020-07-06 10:56:19 +10:00
Sam Saffron 31976ecfeb
PERF: only update etag when it changes
Previously when synchronizing upload etags we would update every single one
regardless of change.
2020-07-06 10:40:04 +10:00
Vinoth Kannan 06d426bd87 FIX: skip hidden posts while generating canonical url.
Previously, while generating the topic page's canoncial url we used the current post number. It will create invalid canonical path if the topic has whsiper posts. Now we only taking the visible posts for current page index calculation.
2020-07-05 14:04:31 +05:30
romanrizzi 6b4cebed3e DEV: Instance#replace_flags block should be optional 2020-07-03 16:21:06 -03:00
Régis Hanol 48b4ed41f5 FIX: uploading an existing image as a site setting
The previous fix (f43c0a5d85) wasn't working for images that were already uploaded.
The "metadata" (eg. 'for_*' and 'secure' attributes) were not added to existing uploads.

Also used 'Upload.get_from_url' is the admin/site_setting controller to properly retrieve
an upload from its URL.

Fixed the Upload::URL_REGEX to use the \h (hexadecimal) for the SHA

Follow-up-to: f43c0a5d85
2020-07-03 19:16:54 +02:00
Martin Brennan 8ef782bdbd
FIX: Increase time of DOWNLOAD_URL_EXPIRES_AFTER_SECONDS to 5 minutes (#10160)
* Change S3Helper::DOWNLOAD_URL_EXPIRES_AFTER_SECONDS to 5 minutes, which controls presigned URL expiry and secure-media route cache time.
* This is done because of the composer preview refreshing while typing causes a lot of requests sent to our server because of the short URL expiry. If this ends up being not enough we can always increase the time or explore other avenues (e.g. GitHub has a 7 day validity for secure URLs)
2020-07-03 13:42:36 +10:00
Roman Rizzi 2df388ffd7
DEV: Plugins can extend ReviewableScore types. (#10156) 2020-07-02 11:47:43 -03:00
Guo Xiang Tan 6bab2acc9f
Fix typo.
Follow up to af52df2d
2020-07-02 14:23:10 +08:00
Guo Xiang Tan af52df2d96
DEV: Add hidden site setting for PG search ranking normalization. 2020-07-02 14:11:18 +08:00
Gerhard Schlager fc8e842773 FIX: Sometimes not all output of psql was logged during restores
There was a race condition which could prevent Discourse from logging the last couple of lines of output from psql.
2020-06-30 16:52:50 +02:00
Dan Ungureanu fe284ffd06
Revert "DEV: Remove useless code (#10130)"
Some oneboxes still generate empty P tags (video oneboxes).

This reverts commit c299d02287.
2020-06-29 13:56:28 +03:00
Régis Hanol 860deeb072 FIX: identify slug-less topic urls everywhere
In 91c89df6, I fixed the onebox to support local topics with a slug-less URL.
This commit fixes all the other spots (search, topic links and user badges) where we look up for a local topic.

Follow-up-to: 91c89df6
2020-06-29 12:31:20 +02:00
David Taylor 0edffcc47d
FIX: Correct version comparison logic when comparing stable to beta (#10135)
* FIX: Correct version comparison logic when comparing stable to beta

For example, version 1.3.0 should be considered higher than 1.3.0.beta3. So `Discourse.has_needed_version?('1.3.0', '1.3.0.beta3')` should return true

* Switch to use Gem::Version to compare versions
2020-06-29 17:52:33 +10:00
Dan Ungureanu c299d02287
DEV: Remove useless code (#10130)
protection is not needed and can easily be bypassed with empty divs anyway.
2020-06-29 17:49:30 +10:00
Blake Erickson 69803599a9 DEV: Refactor seed data filter
Added a small helper class to for seed data because we need to add the
same filter to multisite:migrate as we have in db:migrate. Having this
filter in both places means we can get rid of the SKIP_SEED flag.
2020-06-26 14:36:50 -06:00
Guo Xiang Tan c16ad39f8e
DEV: Run seeds irregardless of post deploy migration flag.
Follow up to 01937b2d
2020-06-26 11:04:34 +08:00
Guo Xiang Tan 01937b2de2
Revert "FIX: Seed needs to run before optimizing site icons."
This reverts commit 715ddf3861.
2020-06-26 11:03:47 +08:00
Guo Xiang Tan 715ddf3861
FIX: Seed needs to run before optimizing site icons. 2020-06-26 08:58:53 +08:00
Justin DiRose 768bb406ee
DEV: Add knowledge-explorer to official plugin list (#10127) 2020-06-25 14:52:14 -05:00
Blake Erickson 01b6349a67
DEV: Add skip seed flag (#10116)
* add a flag to skip seed

* only seed when running post deployment migrations
2020-06-25 10:14:58 -06:00
Sam Saffron 689568c216
FIX: invalid urls should not break store.has_been_uploaded?
Breaking this method has wide ramification including breaking
search indexing.
2020-06-25 15:00:15 +10:00
Sam Saffron 3cb41d5429
PERF: stop adding more topics to search when not needed
The logic of adding additional search results does not seem to be
needed anymore.

It appears to be a relic of an old implementation.

This saves an entire search query for every search made.
2020-06-25 12:31:12 +10:00
Neil Lalonde b9174c8e8a
Version bump to v2.6.0.beta1 2020-06-24 14:00:19 -04:00
Régis Hanol 7109d94ee7 FIX: properly invalidate inline oneboxes when rebaking
When rebaking a post we were invalidating _regular_ oneboxes but not inline oneboxes.

DEV: also renamed 'InlineOneboxer.purge' to 'InlineOneboxer.invalidate' to keep
the API consistent with 'Oneboxer.invalidate'
2020-06-24 11:54:54 +02:00
Guo Xiang Tan b28d97b64a
FIX: Bump onebox for twitch video and clips embedding fix. 2020-06-24 11:00:30 +08:00
Régis Hanol 91c89df68a FIX: onebox local topic when using slug-less URL
When linking to a topic in the same Discourse, we try to onebox the link to show the title
and other various information depending on whether it's a "standard" or "inline" onebox.

However, we were not properly detecting links to topics that had no slugs (eg. https://meta.discourse.org/t/1234).
2020-06-23 17:18:38 +02:00
Daniel Waterworth 368af327fa DEV: Reduce size of begin-rescue region
Follow-up-to: e3e7905d9e
2020-06-23 10:14:09 +01:00
Guo Xiang Tan 84c12d8f3d
DEV: Allow multisite apps to boot with readonly db. 2020-06-23 16:34:25 +08:00
Martin Brennan e92909aa77
FIX: Use ActionDispatch::Http::ContentDisposition for uploads content-disposition (#10108)
See https://meta.discourse.org/t/broken-pipe-error-when-uploading-to-a-s3-clone-a-pdf-with-a-name-containing-e-i-etc/155414

When setting content-disposition for attachment, use the ContentDisposition class to format it. This handles filenames with weird characters and localization (accented characters) correctly.
2020-06-23 17:10:56 +10:00
Guo Xiang Tan 1157d2a0ff
DEV: Print proper summary when errors have been reporter in turbo_rspec 2020-06-23 13:34:58 +08:00
Guo Xiang Tan e3e7905d9e
FIX: `TurboTests::Runner` not failing on errors. 2020-06-23 11:45:18 +08:00
Guo Xiang Tan 0384b6d910
FIX: `multisite:migrate` failing to properly seed data. 2020-06-23 09:10:02 +08:00
Robin Ward ac9577bcc7 FIX: Don't raise an exception if we can't update the user on demotion
This is causing issues when purging old users, if they are set up in the
exact condition where they will be demoted into another group, but also
do not have a primary email.
2020-06-22 15:43:59 -04:00
Dan Ungureanu 2e1efbde52
FIX: Pass local scope variable to inner function 2020-06-22 19:23:59 +03:00
Bianca Nenciu 68f767a557
FEATURE: Check if selectable avatars exist before enabling them (#10032) 2020-06-22 16:58:26 +03:00
Guo Xiang Tan 3370ef188e
FEATURE: Remove deprecated uploads url site settings.
The site settings have been replaced with direct image upload since
Discourse 2.3.
2020-06-22 14:32:29 +08:00
Aman Gupta Karmani 8a86705e51
FIX: handle heroku style HTTP_X_REQUEST_START (#10087) 2020-06-19 10:17:24 -04:00
Daniel Waterworth 9cf77372a2 FIX: Guardian#can_remove_allowed_users? shouldn't break for ownerless topics
A topic can outlive its original author. TopicGuardian should still work
in this situation.
2020-06-19 10:35:52 +01:00
Gerhard Schlager 2da6faf281 FEATURE: Add `before_email_send` event
Plugins can use it to customize the message (e.g. add header) before the email is sent.
2020-06-18 18:58:51 +02:00
Bernhard Suttner e31471585a
DEV: allow to have duplicate topic titles if categegory is different (#10034)
Co-authored-by: Robin Ward <robin.ward@gmail.com>

Co-authored-by: Robin Ward <robin.ward@gmail.com>
2020-06-18 11:19:47 -04:00
Bianca Nenciu db1bebddce
FIX: Hide the post history for TL4 (#10065) 2020-06-18 13:27:51 +03:00
Dan Ungureanu d21a08c284
DEV: Deprecate Category#url_with_id in favor of Category#url (#9972) 2020-06-18 11:32:14 +03:00
Robin Ward e8756e1a95 FIX: Muted/Ignore should prevent PMs regardless of case sensitivity 2020-06-17 14:26:14 -04:00
David Taylor e29afa200a
FIX: Cleanup migrations with timestamps in the future
A future-dated migration was accidently introduced by me in 45c399f0. This was removed in b9762afc, but other migrations had already been generated based on its incorrect date. This commit removes the offending data in the schema_migrations table, and corrects the version in the published_pages migration.

This commit also adds a check to db:migrate which raises an error when invalid migration timestamps are used.
2020-06-17 15:58:22 +01:00
Guo Xiang Tan 45eb97c202
FIX: Thread safety issues with `multisite:migrate` and `SeedFu`. 2020-06-17 16:15:43 +08:00
Guo Xiang Tan 828ceab64b
DEV: Make rubocop happy. 2020-06-17 15:47:05 +08:00
Martin Brennan e5da2d24e5
FIX: Add attachment content-disposition for all non-image files (#10058)
This will make it so the original filename is used when downloading all non-image files, bringing S3Store into line with the to_s3 migration and local storage. Video and audio files will still stream correctly in HTML players as well.

See https://meta.discourse.org/t/cannot-download-non-image-media-files-original-filenames-lost-when-uploaded-to-s3/152797 for a lot of extra context.
2020-06-17 11:16:37 +10:00
Krzysztof Kotlarek dcb816b548
FIX: add table name to topic view query (#10052)
When plugin is hooking into TopicView joining other tables, it may fail because `created_at` is potentially available on 2 tables. Therefore we should explicitly define which `created_at` we want.
2020-06-17 10:40:01 +10:00
Robin Ward 7f8c5cf70b FIX: Allow plugins to provide test directories with transpiled `.js` 2020-06-16 14:31:01 -04:00
Jarek Radosz 669c940ec3 Revert "DEV: Remove the remaining ENV["TRAVIS"] usage (#10041)"
This reverts commit 78aff841e3.

See https://review.discourse.org/t/dev-remove-the-remaining-env-travis-usage-10041/12737/4?u=cvx
2020-06-16 19:42:00 +02:00
Jarek Radosz 78aff841e3
DEV: Remove the remaining ENV["TRAVIS"] usage (#10041) 2020-06-16 17:41:15 +02:00
Gerhard Schlager 859d9b75a7 FIX: Restoring backup from PG12 could fail on PG10
The `EXECUTE FUNCTION` syntax for `CREATE TRIGGER` statements was introduced in PostgreSQL 11. We need to replace `EXECUTE FUNCTION` with `EXECUTE PROCEDURE` in order to be able to restore backups created with PG12 on PG10.
2020-06-16 16:04:14 +02:00
Gerhard Schlager 4cff4892e8 Locale checks were disabled by mistake 2020-06-16 16:04:14 +02:00
Dan Ungureanu 84dfaad137
FIX: Fill acting_user field instead of target_user in history 2020-06-16 13:30:58 +03:00
Blake Erickson a2713578dd DEV: Allow plugins to exclude seed data
This allows plugins to specify if they would like to filter out any seed
data files from running during migrations.
2020-06-15 15:30:25 -06:00
Guo Xiang Tan 58e52c0e4f
DEV: Use rails_failover gem for ActiveRecord and Redis failover handling 2020-06-15 15:47:07 +08:00
Guo Xiang Tan d8cd912769
DEV: Switch to db config to disable advisory locks. 2020-06-15 14:33:41 +08:00
Guo Xiang Tan 0ff86b00cb
DEV: Upgrade Redis to 4.2.1. 2020-06-15 10:05:22 +08:00
Gerhard Schlager 36a3675e0a Update translations 2020-06-14 23:39:33 +02:00
Joffrey JAFFEUX 4b793a1072
FIX: allows PM owner to remove any user if >= TL2 (#10036) 2020-06-12 12:54:28 +02:00
Andrew Schleifer b2c94cc8ea FIX: do not migrate backups in the new prefix 2020-06-12 02:56:07 +00:00
Guo Xiang Tan 78b5ab746c
DEV: No longer need to clear anon cache when toggling readonly mode. 2020-06-12 09:58:17 +08:00
Roman Rizzi a4bfa35420
DEV: Pass the is_image flag when triggering the before_upload_creation event (#10031) 2020-06-11 11:06:48 -03:00
Jarek Radosz 3d55f2e3b7
FIX: Improvements and fixes to the image downsizing script (#9950)
Fixed bugs, added specs, extracted the upload downsizing code to a class, added support for non-S3 setups, changed it so that images aren't downloaded twice.

This code has been tested on production and successfully resized ~180k uploads.

Includes:

* DEV: Extract upload downsizing logic
* DEV: Add support for non-S3 uploads
* DEV: Process only images uploaded by users
* FIX: Incorrect usage of `count` and `exist?` typo
* DEV: Spec S3 image downsizing
* DEV: Avoid downloading images twice
* DEV: Update filesizes earlier in the process
* DEV: Return false on invalid upload
* FIX: Download images that currently above the limit (If the image size limit is decreased, then there was no way to resize those images that now fall outside the allowed size range)
* Update script/downsize_uploads.rb (Co-authored-by: Régis Hanol <regis@hanol.fr>)
2020-06-11 14:47:59 +02:00
Dan Ungureanu b7e70850e4
FIX: Allow users to add emails which were deleted before 2020-06-11 14:54:11 +03:00
Guo Xiang Tan 2e8075bac3
FIX: Ignore Redis readonly errors in `RateLimiter#rollback!`.
This is similar to what we're doing in `RateLimiter#performed!`.
2020-06-11 15:13:11 +08:00
Guo Xiang Tan 90dc6c8ce5
DEV: Fix deprecation warning when using `Redis#close` with redis namespace. 2020-06-11 14:09:19 +08:00
Guo Xiang Tan 531eca985e
DEV: Log error backtrace in `Discourse.warn_exception` without logster. 2020-06-11 10:49:46 +08:00
Kane York 5f3fd23fa8
DEV: Add stub flush_sw task to prevent rebuild errors (#10021) 2020-06-10 13:07:37 -07:00
romanrizzi ac555610d9 DEV: Trigger an event before creating an upload 2020-06-10 16:30:53 -03:00
Neil Lalonde 81a265cecd
Version bump to v2.5.0.beta7 2020-06-10 13:38:18 -04:00
Dan Ungureanu 5bfe1ee4f1
FEATURE: Improve UX support for multiple email addresses (#9691) 2020-06-10 19:11:49 +03:00
Guo Xiang Tan a3dfd553a1
Revert "Bump redis to 4.2.0."
This reverts commit 98bc28cea2.
2020-06-10 14:52:05 +08:00
Guo Xiang Tan 98bc28cea2
Bump redis to 4.2.0. 2020-06-10 14:28:56 +08:00
Jeff Wong 70a88111dd
FIX: prevent re-flagging when we have reviewed flags before (#10010)
FIX: prevent re-flagging when we have reviewed flags before

Fixes an edge case where a review can be reflagged when:
User flags as inappropriate.
Moderator rejects the flag.
Another user re-flags the post as spam.

Before, anyone was able to re-flag as inappropriate despite it being flagged
previously. With this, users are unable to re-flag for the same reason
regardless of reviewable status.
2020-06-09 15:26:10 -07:00
Arpit Jalan 3094459cd9
FEATURE: multiple use invite links (#9813) 2020-06-09 20:49:32 +05:30
Guo Xiang Tan c35649ccf1
DEV: Remove Redis noop logging.
When Redis drops into readonly mode, it floods the log. The log isn't
actionable and we have other means of knowing that an app is in readonly
mode.
2020-06-09 16:45:34 +08:00
Guo Xiang Tan a1c13eb3c6
DEV: Redis failover should only clear redis recently readonly. 2020-06-09 16:36:31 +08:00
Guo Xiang Tan 4065cd198a
DEV: Print backtrace of error when plugin fails to initialize. 2020-06-09 10:25:43 +08:00
Joffrey JAFFEUX d4caf69ed7
DEV: makes SKIP_INSTALL_PLUGINS called last and global (#9990) 2020-06-05 17:59:23 +02:00
Sam Saffron 1c48853ede
DEV: allow skipping of a single plugin when installing
SKIP_INSTALL_PLUGINS can be used to tell plugin:install_all_official
to skip a plugin.

Comma seperated list
2020-06-05 18:26:25 +10:00
Guo Xiang Tan e82d4d8a75
DEV: Update rails_failover to avoid monkey patching Rails config. 2020-06-05 09:05:19 +08:00
Martin Brennan 36841210fb Move UploadSecurity public types into PUBLIC_TYPES constant
* this is so plugins can extend the public types if required
2020-06-05 10:56:57 +10:00
Penar Musaraj 2d880b42a3
UX: Add simple-list setting type (#9970) 2020-06-04 10:44:54 -04:00
Joshua Rosenfeld 76af25f753
FEATURE: Support append when bulk tagging via rake (#9978)
New argument option for the `tags:bulk_tag_category` rake task.
Backwards compatible - defaults to false (no append) if no argument given.
2020-06-04 09:33:48 -04:00
Guo Xiang Tan 248db310d3 DEV: Fix lint. 2020-06-04 17:25:00 +08:00
Guo Xiang Tan aaece34e8b DEV: Update rails_failover so that we can move middleware up the stack. 2020-06-04 17:14:13 +08:00
Guo Xiang Tan 8e1681d356 Bump rails_failover. 2020-06-04 15:22:35 +08:00
Arpit Jalan 07774894a6 DEV: notify when the rake ask is completed 2020-06-04 12:00:06 +05:30
Arpit Jalan a93d24501c FIX: base import script was not updating first_post_created_at column
FEATURE: new rake task to update first_post_created_at column

The not-equal operator (`<>`) in PostgreSQL does not compare values
with NULL. We should instead use `IS DISTINCT FROM` when comparing
values with NULL.
2020-06-04 11:26:40 +05:30
Guo Xiang Tan 2188ccccd5 DEV: Remove `initiating_user` keyword arg from `EmailUpdater`.
The guardian contains the acting user.
2020-06-04 13:21:56 +08:00
Guo Xiang Tan d3c972c30c FIX: Rescue from readonly errors when looking up auth tokens.
Since this is rare, we don't want to check for
`Discourse.pg_readonly_mode?` on every request since we have to reach
for Redis. Instead, just rescue the error here.
2020-06-04 09:12:49 +08:00
Michael K Johnson 81e6bc7a0f
FEATURE: Add uploads:batch_migrate_from_s3 task to limit total posts migrated at once (#9933)
Allow limiting the number of migrations to do at once, both to do migrations that
have impact limited to multiple off-peak usage hours to reduce user impact from
a migration, and to allow tests that do only a very small number for test
purposes. ("Give me a ping, Vasili. One ping only, please.")
2020-06-04 09:48:11 +10:00
Blake Erickson a89574ccb9 FIX: Inline error when converting html to markdown
Looks like some html elements like `aside` and `section` will throw an error
when checking if they are inline or not. The commit simply handles

```
Job exception: undefined method `inline?' for nil:NilClass
```

and adds a test for it.
2020-06-03 15:59:19 -06:00
Gerhard Schlager f683c5d0e0 DEV: Check English locale for errors in CI
Moves the most important checks into a linter. It gets executed by Lefthook as well as the docker rake task and Github actions. Doing those checks in rspec takes too long and it produces errors when the discourse:test Docker image contains old, invalid locale files.
2020-06-03 21:54:58 +02:00
Robin Ward ba00cc8ec4
DEV: Move `Discourse.getURL` and related functions to a module (#9966)
* DEV: Move `Discourse.getURL` and related functions to a module

* DEV: Remove `Discourse.getURL` and `Discourse.getURLWithCDN`

* FIX: `get-url` is required for server side code

* DEV: Deprecate `BaseUri` too.
2020-06-03 12:45:26 -04:00
Guo Xiang Tan 76ce814a62
DEV: Use `safe_each_connection` patch when preloading Rails.
This is to ensure we still boot even if there is a problem with any of
the databases.
2020-06-03 15:36:50 +08:00
Sam Saffron 57a3d4e0d2
FEATURE: whitelist theme repo mode (experimental)
In some restricted setups all JS payloads need tight control.

This setting bans admins from making changes to JS on the site and
requires all themes be whitelisted to be used.

There are edge cases we still need to work through in this mode
hence this is still not supported in production and experimental.

Use an example like this to enable:

`DISCOURSE_WHITELISTED_THEME_REPOS="https://repo.com/repo.git,https://repo.com/repo2.git"`

By default this feature is not enabled and no changes are made.

One exception is that default theme id was missing a security check
this was added for correctness.
2020-06-03 13:19:57 +10:00
Guo Xiang Tan 062db10c52
FIX: `EmailValidator` needs to validate format of email. 2020-06-03 10:34:37 +08:00
Robin Ward 4e3a84c687 FIX: If creating a flag for a watched word, include the reason 2020-06-02 11:49:02 -04:00
Guo Xiang Tan 439db7ca1e
DEV: Add `REDIS_RAILS_FAILOVER` env to test our new redis failover. 2020-06-02 17:24:14 +08:00
Aman Gupta Karmani 6358e79f0f
FIX: catch NoMethodError during precompile_css too (#9940)
Ran into this on heroku:

    NoMethodError: undefined method `enabled?' for #<Theme:0x0000558f069e7718>
    vendor/bundle/ruby/2.6.0/gems/activemodel-6.0.1/lib/active_model/attribute_methods.rb:431:in `method_missing'
    app/models/theme.rb:155:in `block (2 levels) in transform_ids'
    vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/delegation.rb:85:in `each'
    vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/delegation.rb:85:in `each'
    vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/query_methods.rb:260:in `select'
    vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/query_methods.rb:260:in `select'
    app/models/theme.rb:155:in `block in transform_ids'
    app/models/theme.rb:105:in `get_set_cache'
    app/models/theme.rb:140:in `transform_ids'
    app/models/theme.rb:321:in `list_baked_fields'
    app/models/theme.rb:317:in `resolve_baked_field'
    lib/stylesheet/manager.rb:289:in `theme_digest'
2020-06-02 15:18:03 +10:00
Guo Xiang Tan fe9c82994d
DEV: Actually disconnect from Redis connections after fork. 2020-06-02 11:40:16 +08:00
Neil Lalonde 09dc5eb5ea
Version bump to v2.5.0.beta6 2020-06-01 14:13:48 -04:00
Guo Xiang Tan 389bdcf5ab
DEV: Fix implementation for `DiscourseRedis#exists`. 2020-06-01 13:11:32 +08:00
Jarek Radosz 27ad562ff5 DEV: Rubocop fix 2020-06-01 06:07:07 +02:00
Jarek Radosz 7df688d108 FIX: Handle files removed between `glob` and `mtime` 2020-06-01 05:50:50 +02:00
Guo Xiang Tan df62407f35
DEV: Implement multiple keys support for `DiscourseRedis#exists`. 2020-06-01 11:20:26 +08:00
Guo Xiang Tan 3894555b2e
DEV: Prefer `Discourse.redis.reconnct`. 2020-06-01 10:55:53 +08:00
Robin Ward 2b2434b82d
Start Discourse in an initializer (#9930)
* DEV: To be pedantic, there is more than EMBER in there now

* DEV: Use less globals. Have `Discourse` start in an initializer

* DEV: Remove another global
2020-05-29 14:37:02 -04:00
David Taylor e159fb06df
FEATURE: Download remote images even for old posts (#9925)
When a post is rebaked, the admins expect it to work the same regardless of how old the post is.
2020-05-29 17:13:55 +01:00
David Taylor 28f46c171c
FIX: Pull hotlinked images even when edited by system users (#9890)
Previously the pull hotlinked images job was skipped after system edits. This ensured that we never had an infinite loop of system-edit/pull-hotlinked/system-edit/pull-hotlinked etc.

A side effect was that edits made by system for any other reason (e.g. API, removing full quotes) would prevent pulling hotlinked images. This commit removes the system edit check, and replaces it with another method to avoid an infinite job scheduling loop.
2020-05-29 13:07:47 +01:00
Joffrey JAFFEUX 7635c18a14
DEV: ensures highlightjs is correctly tested (#9923) 2020-05-29 13:05:44 +02:00
Guo Xiang Tan 3b311f6b15
DEV: Bump `rails_failover`. 2020-05-29 09:02:34 +08:00
Vinoth Kannan ce1491e830
UX: remove `in:unpinned` filter from advanced search page. (#9911) 2020-05-29 00:47:28 +05:30
Andrew Schleifer 74d28a43d1
new S3 backup layout (#9830)
* DEV: new S3 backup layout

Currently, with $S3_BACKUP_BUCKET of "bucket/backups", multisite backups
end up in "bucket/backups/backups/dbname/" and single-site will be in
"bucket/backups/".

Both _should_ be in "bucket/backups/dbname/"

- remove MULTISITE_PREFIX,
- always include dbname,
- method to move to the new prefix
- job to call the method

* SPEC: add tests for `VacateLegacyPrefixBackups` onceoff job.

Co-authored-by: Vinoth Kannan <vinothkannan@vinkas.com>
2020-05-29 00:28:23 +05:30
Guo Xiang Tan 6491db579b
DEV: Add `discourse-staff-alias` to official plugin list. 2020-05-28 16:43:15 +08:00
Blake Erickson 6548cd1a96 FIX: sending messages to groups with non-lowercase names
Fixes a regression in

e8fb9d4066

which caused a bug where you couldn't send a message to a group that
contained an Uppercase letter. Added a test case for this.

Bug report: https://meta.discourse.org/t/-/152999
2020-05-27 14:52:08 -06:00
Joffrey JAFFEUX 77801aa9be
FIX: allows to have custom emoji translation without static file (#9893) 2020-05-27 20:11:52 +02:00
Joffrey JAFFEUX 1d685c22af
REVERT: removes translate_emoji (#9889)
This API is actually used in some plugins.
2020-05-27 12:08:24 +02:00
Penar Musaraj b1c726be0d
Remove support for FontAwesome 4.7 icon names (#9871) 2020-05-26 14:53:32 -04:00
Gerhard Schlager 69ee94b526 FIX: XML files could be detected as SVG files 2020-05-26 18:18:20 +02:00
Jeff Wong 4dc6504234 DEV: Mark fontawesome-pro plugin as official 2020-05-26 08:50:55 -07:00
Neil Lalonde 788b8becde
Version bump to v2.5.0.beta5 2020-05-26 11:13:05 -04:00
Roman Rizzi b61a291cf3
FIX: returns false if the upload url is an invalid mailto link (#9877) 2020-05-26 10:32:48 -03:00
Guo Xiang Tan 878f06f1fe DEV: Remove custom connection reaper.
Rails 6 fixed the reaper to use one thread to reap all the connection pools.
2020-05-26 09:09:46 +08:00
Sam Saffron fc97f7e0e7
FIX: properly ban non human users from draft system
Previously we had a partial fix in place where non human users
were not allowed draft sequences, this left edges around where non
human users asked for drafts yet had none.

For example system could already have a few drafts in place.

This also removes and extensibility point we added that is not in use
2020-05-26 10:07:09 +10:00
Rafael dos Santos Silva b48299f81c
FEATURE: Add setting to disable automatic CORS rule install in S3 buckets (#9872) 2020-05-25 17:09:34 -03:00
Arpit Jalan e8fb9d4066 FIX: when creating new PM username/groupname should be case-insensitive
(take 2)

https://meta.discourse.org/t/case-sensitivity-in-links-to-groupname/147596
https://meta.discourse.org/t/remove-case-sensitive-in-adding-users-to-a-message/151275
2020-05-25 21:34:05 +05:30
Arpit Jalan 302b37c805 Revert "FIX: when creating new PM username/groupname should be case-insensitive"
This reverts commit 2be79d94f5.

This is affecting multiple code path. Investigating.
2020-05-25 20:10:14 +05:30
Arpit Jalan 30849c8b37 FIX: no need for downcasing second time 2020-05-25 19:20:15 +05:30
Arpit Jalan 2be79d94f5 FIX: when creating new PM username/groupname should be case-insensitive
https://meta.discourse.org/t/case-sensitivity-in-links-to-groupname/147596
https://meta.discourse.org/t/remove-case-sensitive-in-adding-users-to-a-message/151275
2020-05-25 19:04:59 +05:30
Vinoth Kannan 8e56197728
UX: use "icon-picker" & "image-uploader" fields to set group flair. (#9779) 2020-05-25 11:08:47 +05:30
Michael Brown d9a02d1336
Revert "Revert "Merge branch 'master' of https://github.com/discourse/discourse""
This reverts commit 20780a1eee.

* SECURITY: re-adds accidentally reverted commit:
  03d26cd6: ensure embed_url contains valid http(s) uri
* when the merge commit e62a85cf was reverted, git chose the 2660c2e2 parent to land on
  instead of the 03d26cd6 parent (which contains security fixes)
2020-05-23 00:56:13 -04:00
Jeff Atwood 20780a1eee Revert "Merge branch 'master' of https://github.com/discourse/discourse"
This reverts commit e62a85cf6f, reversing
changes made to 2660c2e21d.
2020-05-22 20:25:56 -07:00
Blake Erickson 03d26cd6f0 SECURITY: ensure embed_url contains valid http(s) uri 2020-05-22 14:54:56 -06:00
Martin Brennan f9d55b4941
FEATURE: Update the topic excerpt when the OP is rebaked (#9852)
* We now have a site setting "topic_excerpt_maxlength" that is used when the OP is created or revised to generate a topic excerpt.
* However, posts created before this setting was introduced cannot benefit from this change unless they are revised, and if the topic excerpt length setting is changed that situation is also not covererd.
* This PR makes a change to rebake! to update the topic excerpt IF the post is the OP.
2020-05-22 13:04:15 +10:00
Mark VanLandingham bd57ae83f1
DEV: Guardian for hiding about stats (#9841) 2020-05-20 10:26:20 -05:00
Arpit Jalan 4a533ec594
FIX: do not parameterize tag_id (#9839)
Parameterizing tag_id was breaking tags with non-ascii characters or emoji.

Bug report: https://meta.discourse.org/t/unable-to-see-pm-lists-for-non-ascii-tag/151723/4
2020-05-20 19:22:20 +05:30
Osama Sayegh 02f44def56
FIX: Don't blow up when trying to parse invalid or non-ASCII URLs (#9838)
* FIX: Don't blow up when trying to parseinvalid or non-ASCII URLs

Follow-up to 72f139191e
2020-05-20 12:46:27 +03:00
Sam Saffron b031e3220a
FIX: migrations should not fail when db is part migrated
When part migrated the translate overrides may have missing columns
in that case just silently ignore overrides
2020-05-20 15:28:01 +10:00
Martin Brennan 72f139191e
FIX: S3 store has_been_uploaded? was not taking into account s3 bucket path (#9810)
In some cases, between Discourse forums the hostname of a URL could match if they are hosting S3 files on the same bucket but the S3 bucket path might not. So e.g. https://testbucket.somesite.com/testpath/some/file/url.png vs https://testbucket.somesite.com/prodpath/some/file/url.png. So has_been_uploaded? was returning true for the second URL, even though it may have been uploaded on a different Discourse forum.

This is a very rare case but must be accounted for, because this impacts UrlHelper.is_local which mistakenly thinks the file has already been downloaded and thus allows the URL to be cooked, where we want to return the full URL to be downloaded using PullHotlinkedImages.
2020-05-20 10:40:38 +10:00
Gerhard Schlager 0a700d81fc FIX: Restoring backups could fail for database dumps > 8GiB
This is a temporary fix until we ship a new image with bsdtar.
2020-05-19 22:36:59 +02:00
Daniel Waterworth 38c05a4f07 FIX: Check backtrace and backtrace_locations before trying to print an error
... during plugin initialization.
2020-05-19 14:27:22 +01:00
Daniel Waterworth b4c5ff17a3 FIX: Show a useful message when starting a plugin fails without a backtrace 2020-05-19 14:03:54 +01:00
David Taylor 725e38f9d7
DEV: Allow plugins to request topic thumbnail sizes (#9828)
In plugin.rb, you can register new sizes like

```
register_topic_thumbnail_size [512, 512]
```

For more information about thumbnails see 03818e642a
2020-05-19 10:38:58 +01:00
Martin Brennan 779dc30d2e FIX: Join bookmarks migration on users to avoid missing user records
* the post_actions table has no FK to users, so if a user has been
  deleted we may end up with dangling post_action records, which then
  interferes with the bookmarks migration because bookmarks DO have
  an FK to users
2020-05-19 16:31:05 +10:00
Guo Xiang Tan 05beaa7f45
DEV: Remove override of `flushdb` in `DiscourseRedis`. 2020-05-19 10:26:23 +08:00
Gerhard Schlager 6d5e9db883 FIX: Restoring backup didn't clear cached translation overrides 2020-05-18 18:51:51 +02:00
Sam Saffron 862773ec83
FIX: do not remove stop words when using English locale
PG already handles English stop words, the list in cppjieba is
bigger than the list PG uses, which in turn causes confusion cause
words such as "volume" are stripped using cppijieba stop word list

We will follow up with another commit here to apply the Chinese
word stopwords, but for now to eliminate the confusion we are
skipping applying the stopword list when the dictionary in PG is
in English.
2020-05-18 10:54:56 +10:00
Robin Ward db8e872bda
DEV: Move select kit to an addon (#9797) 2020-05-15 16:07:35 -04:00
David Taylor 461b4e5cc6
DEV: Add framework for filtered plugin registers (#9763)
* DEV: Add framework for filtered plugin registers

Plugins often need to add values to a list, and we need to filter those lists at runtime to ignore values from disabled plugins. This commit provides a re-usable way to do that, which should make it easier to add new registers in future, and also reduce repeated code.

Follow-up commits will migrate existing registers to use this new system

* DEV: Migrate user and group custom field APIs to plugin registry

This gives us a consistent system for checking plugin enabled state, so we are repeating less logic. API changes are backwards compatible
2020-05-15 14:04:38 +01:00
Guo Xiang Tan 9ab5801a1b
FIX: Allow post migrations using `#change` to carry out unsafe migration 2020-05-15 14:23:27 +08:00
Sam Saffron 4601833e4e
PERF: ensure we run full GC on contexts
Prior to this change we would never clear memory from contexts and
rely on V8 reacting to pressure

This could lead to bloating of PrettyText and Transpiler contexts

This optimisations ensures that we will clear memory 2 seconds after
the last eval on the context
2020-05-15 14:01:54 +10:00
Sam Saffron 0cbaa8d813
FEATURE: extend duration allowed for download
Previously we would raise a warning in the logs if downloading
a file (from s3) takes longer than 60 seconds.

At scale this happens reasonably frequently.

1. Raised the duration to 3 minutes

2. Pulled the resizing mutex out of the downloading mutex
so we have less and clearer error logs
2020-05-15 12:45:47 +10:00
Guo Xiang Tan 245e8212ca
Revert "FIX: don't compute draft for (ro)bots 🤖 in topics list"
This reverts commit ec8f08d0da.
2020-05-15 10:40:35 +08:00
Blake Erickson 1a2b9435b0
DEV: Standardize table sorting verbiage (#9757)
* DEV: Standardize table sorting verbiage

This commit creates a common component that tables can use to make their
headers sortable. This commit also standardizes on using `desc` as the
default and passing in the `asc=true` flag to adjust the sorting
direction.

* Add deprecation warnings

Adds deprecation warnings if using previous params and maintains
backwards compatibility. Set the default sort value for group members to
be asc.

* switch group requests to use common table-header-toggle

* update fixture
2020-05-14 20:10:59 -06:00
Kane York 869f9b20a2
PERF: Dematerialize topic_reply_count (#9769)
* PERF: Dematerialize topic_reply_count

It's only ever used for trust level promotions that run daily, or compared to 0. We don't need to track it on every post creation.

* UX: Add symbol in TL3 report if topic reply count is capped

* DEV: Drop user_stats.topic_reply_count column
2020-05-14 15:42:00 -07:00
David Taylor 956d15d13f
UX: Do not use small onebox images as post/topic images 2020-05-14 18:01:43 +01:00
Mark VanLandingham bec6bace53
DEV: Allow plugins to hide user stats by new guardian method (#9772) 2020-05-14 11:57:35 -05:00
Régis Hanol ec8f08d0da FIX: don't compute draft for (ro)bots 🤖 in topics list 2020-05-14 17:17:42 +02:00
Robin Ward 42b300b9be DEV: Skip I18n in plugins until they can be updated 2020-05-14 10:40:53 -04:00
Mark VanLandingham a047004c9a
FIX: Specific email error for replies to digest emails (#9770) 2020-05-14 09:04:58 -05:00
Dan Ungureanu 3ed6a0e904
FIX: Detect Wayback Machine using user agent (#9777) 2020-05-14 21:10:07 +10:00
Guo Xiang Tan d226783405
Revert "DEV: Add `add_controller_callback` server side plugin api."
This reverts commit 8233610d92.

Too much messy around with blocks to support `before_action` and
`around_action`.
2020-05-14 15:26:50 +08:00
Robin Ward eab560fe2a
DEV: import I18n instead of global usage (#9768)
Co-authored-by: Mark VanLandingham <markvanlan@gmail.com>
Co-authored-by: Robin Ward <robin.ward@gmail.com>

Co-authored-by: Mark VanLandingham <markvanlan@gmail.com>
2020-05-13 16:23:41 -04:00
Joffrey JAFFEUX 9b85e75937
FIX: prevents SvgSprite.bundle to query ThemeField two times (#9762) 2020-05-13 17:11:44 +02:00
David Taylor ff331e845a
DEV: Prevent 'previous definition' warnings for PARAMETER_API_PATTERNS 2020-05-13 12:54:28 +01:00
David Taylor d4717f5d1e
DEV: Reduce repeated code in discourse_plugin_registry (#9705)
Use a helper method to simplify creating a new register. Previously this would require creating lots of different methods manually, and adding every register to the clear/reset functions
2020-05-13 12:25:34 +01:00
Blake Erickson 4078b22887 FIX: Handle missing provider return sso url
This commit prevents a 500 error from occurring if someone is trying to
setup their discourse instance as a sso provider and they don't pass in
a `return_sso_url` in their payload.
2020-05-12 18:16:50 -06:00
Rafael dos Santos Silva 1c0c425f38
DEV: Detect correct data dir using ENV 2020-05-12 13:10:57 -03:00
David Taylor 0344ad14c1
FIX: Add missing rss routes to parameter api whitelist 2020-05-12 16:08:35 +01:00
David Taylor 6230f5c554
FEATURE: Allow parameter authentication for UserApiKeys (#9742)
This refactors default_current_user_provider in a few ways:
- Introduce a generic `api_parameter_allowed?` method which checks for whitelisted routes/formats
- Only read the api_key parameter on allowed routes. It is now completely ignored on other routes (previously it would raise a 403)
- Start reading user_api_key parameter on allowed routes
- Refactor tests as end-end integration tests

A plugin API for PARAMETER_API_PATTERNS will be added soon
2020-05-12 13:35:36 +01:00
Guo Xiang Tan 8fb99f218d
FEATURE: Allow plugins to register a callback to ignore DraftSequence. 2020-05-12 14:25:20 +08:00
Guo Xiang Tan 3c423faafd
DEV: Avoid repeating draft key logic in multiple places. 2020-05-12 11:14:03 +08:00
Krzysztof Kotlarek 4c8bece104
FEATURE: default canonical URL (#9738)
For pages that do not specify canonical URL we will default to `https://SITENAME/PATH`. 

This ensures that if a URL is crawled on the CDN the search ranking will transfer to the main site.

Additionally we whitelist the `?page` param
2020-05-12 09:13:20 +10:00
Robin Ward 8311374a5a DEV: Replace `version.js.erb` with pre generated file 2020-05-11 15:43:09 -04:00
Robin Ward d2b16a7618
DEV: Generate `emoji/data` on demand (#9744)
These emoji rarely change and can be re-generated when we make changes
to them.
2020-05-11 15:29:46 -04:00
David Taylor 5fc51ed49c
DEV: Remove unused DiscoursePlugin class (#9715) 2020-05-11 15:46:54 +01:00
Guo Xiang Tan f293b28198
FEATURE: Pass instance of revisor to `post_edited` DiscourseEvent. 2020-05-11 16:45:40 +08:00
Guo Xiang Tan 8233610d92
DEV: Add `add_controller_callback` server side plugin api. 2020-05-11 16:45:08 +08:00
Bianca Nenciu 09caf614de DEV: Fix build
Rails.logger is not available when Discourse.deprecate is called from
enabled_site_setting_filter.

Follow up to db46018235.
2020-05-10 15:06:39 +03:00
Bianca Nenciu 8149bfbaf1
FEATURE: Filter settings by plugin (#9692) 2020-05-10 14:07:45 +03:00
Robin Ward 34e5f807ec DEV: User UPPER_CASE for constants 2020-05-08 15:50:55 -04:00
Robin Ward b2e4ca968b DEV: Remove `translate` API from plugin.
It appears to be completely unused at this point.
2020-05-08 14:14:01 -04:00
Robin Ward 7f373e8b93 DEV: Don't use `js.erb` for constants
Adds a new rake task to auto generate a constants.js file with the
constants present. This makes migrating to Ember CLI easier, but also
slightly speeds up asset compilation by having to do less work.

If the constants change you need to run:
`rake javascripts:update_constants`
2020-05-08 14:14:01 -04:00
Robin Ward 3cce1b4e78 FIX: Support transpiling js in plugins with a root admin folder 2020-05-08 11:10:54 -04:00
Krzysztof Kotlarek a3e1e9ef16
FIX: remove deprecated URI.escape (#9697)
During Nokogumbo changes I introduced back URI.escape which is deprecated.
2020-05-08 11:14:59 +10:00
Martin Brennan 22dffe6f69 FIX: Do not allow null options for bookmark manager 2020-05-08 15:24:59 +00:00
Robin Ward f9608c0af5 DEV: Remove INLINE_ONEBOX_* constants
There were two constants here, `INLINE_ONEBOX_LOADING_CSS_CLASS` and
`INLINE_ONEBOX_CSS_CLASS` that were both longer than the strings they
were DRYing up: `inline-onebox-loading` and `inline-onebox`

I normally appreciate constants, but in this case it meant that we had
a lot of JS imports resulting in many more lines of code (and CPU cycles
spent figuring them out.)

It also meant we had an `.erb` file and had to invoke Ruby to create the
JS file, which meant the app was harder to port to Ember CLI.

I removed the constants. It's less DRY but faster and simpler, and
arguably the loss of DRYness is not significant as you can still search
for the `inline-onebox-loading` and `inline-onebox` strings easily if
you are refactoring.
2020-05-07 16:14:38 -04:00
Vinoth Kannan 744bbf6904 FEATURE: exclude muted categories from the "top" topics list. 2020-05-08 00:34:53 +05:30
Joffrey JAFFEUX 23d585f255
REFACTOR: removes unreachable statement (#9680) 2020-05-07 16:37:02 +02:00
Martin Brennan 6fb0f36ce1
FEATURE: Optionally delete bookmark when reminder sent (#9637)
We now show an options gear icon next to the bookmark name.

When expanded we show the "delete bookmark when reminder sent" option. The value of this checkbox is saved in local storage for the user.

If this is ticked, when a reminder is sent for the bookmark the bookmark itself is deleted. This is so people can use the reminder functionality by itself.

Also remove the blue alert reminder section from the "Edit Bookmark" modal as it just added clutter, because the user can already see they had a reminder set:

Adds a default false boolean column `delete_when_reminder_sent` to bookmarks.
2020-05-07 13:37:39 +10:00
tshenry 0d394e6b76
FEATURE: Include category position when exporting categories (#9658) 2020-05-07 12:17:15 +10:00
Gerhard Schlager ec2f3169ff FIX: Using the `default_locale` in locale fallbacks caused problems
Locale files get precompiled after deployment and they contained translations from the `default_locale`. That's especially bad in multisites, because the initial `default_locale` is `en_US`. Sites where the `default_locale` isn't `en_US` could see missing translations. The same thing could happen when users are allowed to chose a different locale.

This change simplifies the logic by not using the `default_locale` in the locale chain. It always falls back to `en` in case of missing translations.
2020-05-06 22:59:07 +02:00
Robin Ward 01929e3505 DEV: Move `preload-store` to `discourse/lib/preload-store`
It's only used inside Discourse so it needn't be its own module
2020-05-06 15:28:06 -04:00
David Taylor 72ad701df0
DEV: Stub #flush in StdOutDemux for multisite:migrate
https://meta.discourse.org/t/multisite-migrate-error/150579/2
2020-05-06 11:58:35 +01:00
Martin Brennan fa572d3a7a
DEV: Remove all code referencing at_desktop bookmark reminders (#9650)
We have found no need for these reminder types, so we are removing the code for them.
2020-05-06 15:22:43 +10:00
Robin Ward 612284cef3
DEV: Remove `Discourse.RAW_TEMPLATES` (#9630)
We were sharing `Discourse` both as an application object and a
namespace which complicated things for Ember CLI. This patch
moves raw templates into `__DISCOURSE_RAW_TEMPLATES` and adds
a couple helper methods to create/remove them.
2020-05-05 12:15:03 -04:00
David Taylor 03818e642a
FEATURE: Include optimized thumbnails for topics (#9215)
This introduces new APIs for obtaining optimized thumbnails for topics. There are a few building blocks required for this:

- Introduces new `image_upload_id` columns on the `posts` and `topics` table. This replaces the old `image_url` column, which means that thumbnails are now restricted to uploads. Hotlinked thumbnails are no longer possible. In normal use (with pull_hotlinked_images enabled), this has no noticeable impact

- A migration attempts to match existing urls to upload records. If a match cannot be found then the posts will be queued for rebake

- Optimized thumbnails are generated during post_process_cooked. If thumbnails are missing when serializing a topic list, then a sidekiq job is queued

- Topic lists and topics now include a `thumbnails` key, which includes all the available images:
   ```
   "thumbnails": [
   {
     "max_width": null,
     "max_height": null,
     "url": "//example.com/original-image.png",
     "width": 1380,
     "height": 1840
   },
   {
     "max_width": 1024,
     "max_height": 1024,
     "url": "//example.com/optimized-image.png",
     "width": 768,
     "height": 1024
   }
   ]
  ```

- Themes can request additional thumbnail sizes by using a modifier in their `about.json` file:
   ```
    "modifiers": {
      "topic_thumbnail_sizes": [
        [200, 200],
        [800, 800]
      ],
      ...
  ```
  Remember that these are generated asynchronously, so your theme should include logic to fallback to other available thumbnails if your requested size has not yet been generated

- Two new raw plugin outlets are introduced, to improve the customisability of the topic list. `topic-list-before-columns` and `topic-list-before-link`
2020-05-05 09:07:50 +01:00
Krzysztof Kotlarek 9bff0882c3
FEATURE: Nokogumbo (#9577)
* FEATURE: Nokogumbo

Use Nokogumbo HTML parser.
2020-05-05 13:46:57 +10:00
Rafael dos Santos Silva d59d170452
FIX: PostgreSQL fallback was broken due to Rails masking exception (#9633)
The PR https://github.com/rails/rails/pull/36612 changes the raised exception
if the error message includes the target database name.

Since the error message contains the hostname, this could be triggered when
the hostname contains the database name.
2020-05-05 10:34:25 +10:00
Penar Musaraj 5706cab897
FIX: Improve digest email styling on Outlook 2016 (#9626) 2020-05-04 14:07:03 -04:00
Neil Lalonde c6b31464db
Version bump to v2.5.0.beta4 2020-05-04 11:44:42 -04:00
David Taylor 5901717531
DEV: Allow DB.after_commit to be used outside of a transaction
In this case, it will execute the given block immediately
2020-05-04 09:42:41 +01:00
Robin Ward e57fd283db DEV: Rename `deprecated` to the more appropriate app-boot 2020-05-01 15:19:19 -04:00
Martin Brennan bcc9ad6f57
FIX: Bookmark UI tweaks (#9604)
* When hovering over the bookmark icon for a post, show the name of the bookmark at the end of the tooltip _if_ it has been set.
* Order bookmarks by `updated_at DESC` in the user list and show that instead of created at.
2020-05-01 16:14:20 +10:00
Vinoth Kannan 5ff24b6891 FIX: do not raise error if 'class' attribute is not found. 2020-05-01 10:03:40 +05:30
Krzysztof Kotlarek 37e93914fc
FIX: the muted message should be sent after edit (#9593)
Recently, we added feature that we are sending `/muted` to users who muted specific topic just before `/latest` so the client knows to ignore those messages - https://github.com/discourse/discourse/pull/9482

Same `/muted` message should be included when the post is edited
2020-05-01 08:33:57 +10:00
Robin Ward d615de9139
DEV: Support for `import Handlebars from 'handlebars'`; (#9600)
* Remove Handlebars.SafeString usage

* DEV: Support for `import Handlebars from 'handlebars'`;

* FIX: Sprockets was broken when `node_modules` was present

By default the old version of sprockets looks for application.js
anywhere, including in a node_modules folder if this exists
(which it will when we move to Ember CLI.)
2020-04-30 16:41:02 -04:00
Rafael dos Santos Silva 08e4af6636 FEATURE: Add setting to controle the Expect header on S3 calls
Some providers don't implement the Expect: 100-continue support,
which results in a mismatch in the object signature.

With this settings, users can disable the header and use such providers.
2020-04-30 12:12:00 -03:00
Régis Hanol 501b19b6e0
FIX: server-side HtmlToMarkdown improvements (#9586)
TLDR; this commit vastly improves how whitespaces are handled when converting from HTML to Markdown.
It also adds support for converting HTML <tables> to markdown tables.

The previous 'remove_whitespaces!' method was traversing the whole HTML tree and used a heuristic to remove
leading and trailing whitespaces whenever it was appropriate (ie. mostly before and after HTML block elements)

It was a good idea, but it was very limited and leaded to bad conversion when the html had leading whitespaces on several lines for example.
One such example can be found [here](https://meta.discourse.org/t/86782).

For various reasons, most of the whitespaces in a HTML file is ignored when the page is being displayed in a browser.
The rules that the browsers follow are the [CSS' White Space Processing Rules](https://www.w3.org/TR/css-text-3/#white-space-rules).
They can be quite complicated when you take into account RTL languages and other various tidbits but they boils down to the following:

- Collapse whitespaces down to one space (0x20) inside an inline context (ie. nodes/tags that are being displaying on the same line)
- Remove any leading/trailing whitespaces inside an inline context

One quick & dirty way of getting this 90% solved would be to do 'HTML.gsub!(/[[:space:]]+/, " ")'.
We would also need to hoist <pre> elements in order to not mess with their whitespaces.
Unfortunately, this solution let some whitespaces creep around HTML tags which leads to more '.strip!' calls than I can bear.

I decided to "emulate" the browser's handling of whitespaces and came up with a solution in 4 parts

1. remove_not_allowed!

The HtmlToMarkdown library is recursively "visiting" all the nodes in the HTML in order to convert them to Markdown.
All the nodes that aren't handled by the library (eg. <script>, <style> or any non-textual HTML tags) are "swallowed".
In order to reduce the number of nodes visited, the method 'remove_not_allowed!' will automatically delete all the nodes
that have no "visitor" (eg. a 'visit_<tag>' method) defined.

2. remove_hidden!

Similar purpose as the previous method (eg. reducing number of nodes visited), there's no point trying to convert something that is hidden.
The 'remove_hidden!' method removes any nodes that was hidden using the "hidden" HTML attribute, some CSS or with a width or height equal to 0.

3. hoist_line_breaks!

The 'hoist_line_breaks!' method is there to handle <br> tags. I know those tiny <br> don't do much but they can be quite annoying.
The <br> tags are inline elements but they visually work like a block element (ie. they create a new line).
If you have the following HTML "<i>Foo<br>Bar</i>", it ends up visually similar to "<i>Foo</i><br><i>Bar</i>".
The latter being much more easy to process than the former, so that's what this method is doing.
The "hoist_line_breaks" will hoist <br> tags out of inline tags until their parent is a block element.

4. remove_whitespaces!

The "remove_whitespaces!" is where all the whitespace removal is happening. It's broken down into 4 methods as well

- remove_whitespaces!
- is_inline?
- collapse_spaces!
- remove_trailing_space!

The 'remove_whitespace!' method is recursively walking the HTML tree (skipping <pre> tags).
If a node has any children, they will be chunked into groups of inline elements vs block elements.
For each chunks of inline elements, it will call the "collapse_space!" and "remove_trailing_space!" methods.
For each chunks of block elements, it will call "remote_whitespace!" to keep walking the HTML tree recursively.

The "is_inline?" method determines whether a node is part of a inline context.
A node is inline iif it's a text node or it's an inline tag, but not <br>, and all its children are also inline.

The "collapse_spaces!" method will collapse any kind of (white) space into a single space (" ") character, even accros tags.
For example, if we have "  Foo \n<i> Bar </i>\t42", it will return "Foo <i>Bar </i>42".

Finally, the "remove_trailing_space!" method is there to remove any trailing space that might creep in at the end of the inline chunk.

This solution is not 100% bullet-proof.
It does not support RTL languages at all and has some caveats that I felt were not worth the work to get properly fixed.

FIX: better detection of hidden elements when converting HTML to Markdown
FIX: take into account the 'allowed_href_schemes' site setting when converting HTML <a> to Markdown
FIX: added support for 'mailto:' scheme when converting <a> from HTML to Markdown
FIX: added support for <img> dimensions when converting from HTML to Markdown
FIX: added support for <dl>, <dd> and <dt> when converting from HTML to Markdown
FIX: added support for multilines emphases, strongs and strikes when converting from HTML to Markdown
FIX: added support for <acronym> when converting from HTML to Markdown
DEV: remove unused 'sanitize' gem

Wow, did you just read all that?! Congratz, here's a cookie: 🍪.
2020-04-30 12:21:25 +02:00
Sam Saffron d0d5a138c3
DEV: stop freezing frozen strings
We have the `# frozen_string_literal: true` comment on all our
files. This means all string literals are frozen. There is no need
to call #freeze on any literals.

For files with `# frozen_string_literal: true`

```
puts %w{a b}[0].frozen?
=> true

puts "hi".frozen?
=> true

puts "a #{1} b".frozen?
=> true

puts ("a " + "b").frozen?
=> false

puts (-("a " + "b")).frozen?
=> true
```

For more details see: https://samsaffron.com/archive/2018/02/16/reducing-string-duplication-in-ruby
2020-04-30 16:48:53 +10:00
Dan Ungureanu c85018cdfd
Improve support for old browsers (#9515)
* FEATURE: Improve crawler view

* FIX: Make lazyYT crawler-friendly

* DEV: Rename discourse-internet-explorer to discourse-unsupported-browser

* DEV: Detect more unsupported browsers

Follow-up to 4eebbd2212.

* FIX: Hide browser update notice in print view
2020-04-29 21:40:21 +03:00
Robin Ward 08fbf199ad FIX: S3 rake task can ignore yarn.lock 2020-04-29 13:14:04 -04:00
Robin Ward 9ec908950d DEV: Better error handling for s3 task 2020-04-29 12:54:39 -04:00
Robin Ward cbb27241c4
DEV: Make `discourse-common` an Ember addon. (#9578)
This is to help with the migration to Ember CLI. In the current running
version of Discourse everything should be the same as before, just with
a few extra files that are not used. However, using Ember CLI this can
be installed as an Ember addon.

Co-Authored-By: Jarek Radosz <jradosz@gmail.com>
2020-04-29 12:18:21 -04:00
Robin Ward 1e603d7003 DEV: This file does not seem to be used anymore 2020-04-28 14:03:39 -04:00
Jarek Radosz c1c211365a
FIX: Improve clearing store cache (#9568)
1. Shorter
2. Simpler
3. Doesn't depend on external binaries
4. Doesn't fail on large amounts of files
5. Hopefully eliminates flaky spec errors
2020-04-28 17:24:04 +02:00
Penar Musaraj ec2943c5bc
DEV: Update jquery.fileupload and dependencies (#9466) 2020-04-28 10:39:29 -04:00
David Taylor ba616ffb50
DEV: Use a tmp directory for storing uploads in tests (#9554)
This avoids development-mode upload files from polluting the test environment
2020-04-28 14:03:04 +01:00
Sam Saffron 6b62d75c50
FIX: remove unsuppored rake task
rebuilding user_actions is not something that should be done.

Plugins such as solved and assigned extend it, there are tons of
little rules that were not captured in `user_actions:rebuild`
2020-04-28 16:27:42 +10:00
Martin Brennan 5108cf8ddf
FIX: Topic user bookmarked column logic was not correct (#9563)
Make sure the topic_user.bookmarked column is set correctly when user bookmarks/unbookmarks any post in a topic. For example, you bookmarked a post in the topic that was not the OP, the bookmark icon in the topic list would not be shown. Same if deleting a bookmark for the last bookmarked post in a topic, the bookmark icon in the topic list would not be removed.

Previously this was only setting it to true if bookmarking the OP/topic, which was not correct -- we want to show the icon on the topic list if any post is bookmarked.
Also set to false if unbookmarking the last bookmarked post in the topic.
Also in this PR is a migration to correct any out of sync topic_user.bookmarked columns, based on the new logic.
2020-04-28 16:19:25 +10:00
mentalstring 67f3fe14aa
FEATURE: support SSO website and location overrides
Add location and website + the ability to override using SSO using the `sso_overrides_location` and `sso_overrides_website` site settings.
2020-04-28 16:06:35 +10:00
Roman Rizzi 394babcae3
FIX: Only show the review page to users that can see it. Do not publish the reviewable count update message to everyone. (#9556) 2020-04-27 14:51:25 -03:00
Robin Ward 056327c0c9
DEV: Move `discourse.js` to `app/app.js` (#9545)
This is another thing to get our application in line with what Ember CLI
expects.
2020-04-27 13:28:10 -04:00
David Taylor f95609ae23
FIX: Add additional checks for automatic theme script CSP
- Skip absolute URLs
- Skip CDN URLs
- Skip invalid URLs
- Properly format protocol-less URLs
2020-04-27 15:56:29 +01:00
Benno 6e01acb3cb
FIX: Apply category priority for empty query (#9516) 2020-04-27 10:35:27 -04:00
Guo Xiang Tan 76fc48b318
DEV: Avoid an additional query in `SpamHandler`.
* Also run query against human users only
2020-04-27 13:03:57 +08:00
Sam Saffron 1f6eaf26a1
Revert "UX: replace closed topic icon with discourse-no-entry"
This reverts commit 040b8c00a4.

We decided to keep status quo for now
2020-04-25 13:12:56 +10:00
Kris 040b8c00a4 UX: replace closed topic icon with discourse-no-entry 2020-04-24 17:31:34 -04:00
David Taylor 8a112b7464
DEV: Automatically extend CSP when themes link to external scripts (#9531) 2020-04-24 09:47:01 +01:00
Martin Brennan cd1c7d7560
FIX: Copying image markdown for secure media loading full image (#9488)
* When copying the markdown for an image between posts, we were not adding the srcset and data-small-image attributes which are done by calling optimize_image! in cooked post processor
* Refactored the code which was confusing in its current state (the consider_for_reuse method was super confusing) and fixed the issue
2020-04-24 10:29:02 +10:00
Kris 9f52997be1 UX: Let's try X for the closed topic icon instead of a \ 2020-04-23 17:23:25 -04:00
Gerhard Schlager 67e96f6c54 DEV: discourse-backup-uploads-to-s3 plugin isn't supported anymore 2020-04-23 21:37:38 +02:00