Commit Graph

254 Commits

Author SHA1 Message Date
Martin Brennan fa6b87a1bf
SECURITY: Strip unrendered unicode bidirectional chars in code blocks ()
When rendering the markdown code blocks we replace the
offending characters in the output string with spans highlighting a textual
representation of the character, along with a title attribute with
information about why the character was highlighted.

The list of characters stripped by this fix, which are the bidirectional
characters considered relevant, are:

U+202A
U+202B
U+202C
U+202D
U+202E
U+2066
U+2067
U+2068
U+2069
2021-11-22 10:43:03 +10:00
Bianca Nenciu 1e992d9193
FIX: Do not check for duplicate links in Onebox ()
If a user posted a URL that appeared inside a Onebox, then the user
got a duplicate link notice. This was fixed by skipping those links in
Ruby.

If a user posted a URL that was Oneboxes and contained other links that
appeared in previous posts, then the user got a duplicate link notice.
This was fixed by skipping those links in JavaScript.
2021-06-18 18:55:24 +03:00
Bianca Nenciu d9484db718
FIX: Split link watched words from replace ()
It was not clear that replace watched words can be used to replace text
with URLs. This introduces a new watched word type that makes it easier
to understand.
2021-06-02 15:36:49 +10:00
Bianca Nenciu c1dfd76658
FIX: Make replace watched words work with wildcard ()
Watched words are always regular expressions, despite watched_words_
_regular_expressions being enabled or not. Internally, wildcard
characters are replaced with a regular expression that matches any non
whitespace character.
2021-05-18 12:09:47 +03:00
Osama Sayegh d56b2e85aa
FIX: Escape Font Awesome icons ()
This is not a security issue because regular users are not allowed to insert FA icons anywhere in the app. Admins can insert icons via custom badges, but they do have the ability to create themes with JS.
2021-03-17 16:11:40 +03:00
Rafael dos Santos Silva e06076268a
FIX: Missing quote in emoji cdn setting caused by 83f332b () 2021-03-03 17:39:00 -03:00
Rafael dos Santos Silva 83f332b5a5
FEATURE: Add a site setting to allow emojis to come from an external URL () 2021-03-02 16:04:16 -03:00
Bianca Nenciu 533800a87b
Add watched words of type "replace" ()
This commit includes other various improvements to watched words.

auto_silence_first_post_regex site setting was removed because it overlapped
with 'require approval' watched words.
2021-02-25 14:00:58 +02:00
Daniel Waterworth df8436cd7f
FIX: Don't add a slug to constructed quote urls ()
A topic with the slug 'topic' might exist and may end up being linked to
by mistake when malformed (i.e. cross-site) quotes are posted.
2021-02-11 12:21:13 -06:00
Bianca Nenciu 8ff9cdf390
FIX: Replace Vimeo iframes with a link in emails ()
This was implemented before, but it was not tested and broke at some
point (probably Nokogiri update).
2020-12-09 14:58:36 +02:00
Martin Brennan 879e4a9e29
FIX: Inline avatar style for onebox when embedding secure images ()
When embedding secure images that are inline-avatars for oneboxes we weren't applying the correct sizing/style.
2020-11-16 09:58:40 +10:00
Martin Brennan 27e94f2f98
FIX: Make secure image onebox check more robust ()
When embedding secure images which have been oneboxed, we checked to see if the image's parent's parent had the class onebox-body. This was not always effective as if the image does not get resized/optimized then it does not have the aspect-image div wrapping it. This would cause the image to embed in the email but be huge.

This PR changes the check to see if any of the image's ancestors have the class onebox-body, or if the image has the onebox-avatar class to account for variations in HTML structure.
2020-11-10 12:55:18 +10:00
Penar Musaraj c1f3bd6a1c
FIX: secure_media stripping on lightboxes, non-image links ()
- Fixes stripping of lightboxes with empty srcset attribute
- Does not fail when email has links with secure media URLs but no child image elements
2020-11-04 15:45:50 -05:00
Rafael dos Santos Silva b70b3e867c
FIX: Safely skip secure_media steps when it's not enabled ()
* FIX: Safely skip secure_media steps when it's not enabled

* DEV: Secure media tests should enable secure media
2020-11-03 13:53:15 -03:00
Martin Brennan 3655062c60
FIX: Ensure oneboxed secure images which are optimized and also lightboxed optimized images are embedded in email ()
We had an issue where onebox thumbnail was too large and thus was optimized, and we are using the image URLs in post to redact and re-embed, based on the sha1 in the URL. Optimized image URLs have extra stuff on the end like _99x99 so we were not parsing out the sha1 correctly. Another issue I found was for posts that have giant images, the original was being used to embed in the email and thus would basically never get included because it is huge.

For example the URL 787b17ea61_2_690x335.jpeg was not parsed correctly; we would end up with 787b17ea6140f4f022eb7f1509a692f2873cfe35_2_690x335.jpeg as the sha1 which would not find the image to re-embed that was already attached to the email.

This fix will use the first optimized image of the detected upload when we are redacting and then re-embedding to make sure we are not sending giant things in email. Also, I detect if it is a onebox thumbnail or the site icon and force appropriate sizes and styles.
2020-11-02 09:52:21 +10:00
Krzysztof Kotlarek dbec3792b7
FIX: pretty text allow list ()
Reword whitelist to allowlist in pretty-text.
This library is used by plugins so we need deprecation notice.
2020-10-28 13:22:06 +11:00
Krzysztof Kotlarek b2481adb40
FIX: persist secure image width and height if is given ()
`max-width: 50%; max-height: 400px;` is a good fallback, however, if width and height are given and are smaller than fallback -  we should persist that smaller size.
2020-10-22 13:25:09 +11:00
Daniel Waterworth 721ee36425
Replace `base_uri` with `base_path` ()
DEV: Replace instances of Discourse.base_uri with Discourse.base_path

This is clearer because the base_uri is actually just a path prefix. This continues the work started in 555f467.
2020-10-09 12:51:24 +01:00
Robin Ward e7abba9bbe FIX: Missing files from previous commit 2020-09-15 10:59:41 -04:00
Roman Rizzi efb9fd6ac0
FIX: Make sure rel attributes are correctly set. ()
We must guarantee that "rel=noopener" was set if "target=_blank" is present, which is not always the case for trusted users. Also, if the link contains the "nofollow" attribute, it has to have the "ugc" attribute as well.
2020-09-10 12:59:51 -03:00
Martin Brennan dede942007
FEATURE: Allow email image embed with secure media ()
This PR introduces a few important changes to secure media redaction in emails. First of all, two new site settings have been introduced:

* `secure_media_allow_embed_images_in_emails`: If enabled we will embed secure images in emails instead of redacting them.
* `secure_media_max_email_embed_image_size_kb`: The cap to the size of the secure image we will embed, defaulting to 1mb, so the email does not become too big. Max is 10mb. Works in tandem with `email_total_attachment_size_limit_kb`.

`Email::Sender` will now attach images to the email based on these settings. The sender will also call `inline_secure_images` in `Email::Styles` after secure media is redacted and attachments are added to replace redaction messages with attached images. I went with attachment and `cid` URLs because base64 image support is _still_ flaky in email clients.

All redaction of secure media is now handled in `Email::Styles` and calls out to `PrettyText.strip_secure_media` to do the actual stripping and replacing with placeholders. `app/mailers/group_smtp_mailer.rb` and `app/mailers/user_notifications.rb` no longer do any stripping because they are earlier in the pipeline than `Email::Styles`.

Finally the redaction notice has been restyled and includes a link to the media that the user can click, which will show it to them if they have the necessary permissions.

![image](https://user-images.githubusercontent.com/920448/92341012-b9a2c380-f0ff-11ea-860e-b376b4528357.png)
2020-09-10 09:50:16 +10:00
Robin Ward c4079780be REFACTOR: Remove `_.merge` 2020-09-03 10:03:33 -04:00
Krzysztof Kotlarek e0d9232259
FIX: use allowlist and blocklist terminology ()
This is a PR of the renaming whitelist to allowlist and blacklist to the blocklist.
2020-07-27 10:23:54 +10:00
Robin Ward ba00cc8ec4
DEV: Move `Discourse.getURL` and related functions to a module ()
* DEV: Move `Discourse.getURL` and related functions to a module

* DEV: Remove `Discourse.getURL` and `Discourse.getURLWithCDN`

* FIX: `get-url` is required for server side code

* DEV: Deprecate `BaseUri` too.
2020-06-03 12:45:26 -04:00
Joffrey JAFFEUX 77801aa9be
FIX: allows to have custom emoji translation without static file () 2020-05-27 20:11:52 +02:00
Sam Saffron 4601833e4e
PERF: ensure we run full GC on contexts
Prior to this change we would never clear memory from contexts and
rely on V8 reacting to pressure

This could lead to bloating of PrettyText and Transpiler contexts

This optimisations ensures that we will clear memory 2 seconds after
the last eval on the context
2020-05-15 14:01:54 +10:00
Robin Ward eab560fe2a
DEV: import I18n instead of global usage ()
Co-authored-by: Mark VanLandingham <markvanlan@gmail.com>
Co-authored-by: Robin Ward <robin.ward@gmail.com>

Co-authored-by: Mark VanLandingham <markvanlan@gmail.com>
2020-05-13 16:23:41 -04:00
Krzysztof Kotlarek a3e1e9ef16
FIX: remove deprecated URI.escape ()
During Nokogumbo changes I introduced back URI.escape which is deprecated.
2020-05-08 11:14:59 +10:00
Krzysztof Kotlarek 9bff0882c3
FEATURE: Nokogumbo ()
* FEATURE: Nokogumbo

Use Nokogumbo HTML parser.
2020-05-05 13:46:57 +10:00
Robin Ward d615de9139
DEV: Support for `import Handlebars from 'handlebars'`; ()
* Remove Handlebars.SafeString usage

* DEV: Support for `import Handlebars from 'handlebars'`;

* FIX: Sprockets was broken when `node_modules` was present

By default the old version of sprockets looks for application.js
anywhere, including in a node_modules folder if this exists
(which it will when we move to Ember CLI.)
2020-04-30 16:41:02 -04:00
Sam Saffron d0d5a138c3
DEV: stop freezing frozen strings
We have the `# frozen_string_literal: true` comment on all our
files. This means all string literals are frozen. There is no need
to call #freeze on any literals.

For files with `# frozen_string_literal: true`

```
puts %w{a b}[0].frozen?
=> true

puts "hi".frozen?
=> true

puts "a #{1} b".frozen?
=> true

puts ("a " + "b").frozen?
=> false

puts (-("a " + "b")).frozen?
=> true
```

For more details see: https://samsaffron.com/archive/2018/02/16/reducing-string-duplication-in-ruby
2020-04-30 16:48:53 +10:00
Roman Rizzi 7a2e8d3ead
DEV: Add the missing `app` subdirectory ()
* DEV: Add missing  directory to the Discourse ember app

* DEV: Resolve imports correctly
2020-04-23 10:07:54 -03:00
Robin Ward c150566506
Migrate pretty-text to `.js` extensions () 2020-03-20 09:55:42 -04:00
Sam Saffron 145c90419d
DEV: increase timeout for initial context eval
Due to JS refactors our initial payload transpile is taking a bit longer
instead of failing the test suite attempt to give this a bit more time
2020-03-20 15:48:25 +11:00
Robin Ward a3f0543f99
Support for transpiling `.js` files ()
* Remove some `.es6` from comments where it does not matter

* Use a post processor for transpilation

This will allow us to eventually use the directory structure to
transpile rather than the extension.

* FIX: Some errors and clean up in confirm-new-email

It would throw an error if the webauthn element wasn't present.
Also I changed things so that no-module is not explicitly
referenced.

* Remove `no-module`

Instead we allow a magic comment: `// discourse-skip-module` to prevent
the asset pipeline from creating a module.

* DEV: Enable babel transpilation based on directory

If it's in `app/assets/javascripts/dicourse` it will be transpiled
even without the `.es6` extension.

* REFACTOR: Remove Tilt/ES6ModuleTranspiler
2020-03-11 09:43:55 -04:00
Robin Ward 041168c9b6 FIX: Group mentions were not being cooked the same was as previewed
If a group mention could be notified on preview it was given an `<a>`
tag with the `.notify` class. When cooked it would display differently.
This patch makes the server side cooking match the client preview.
2020-02-18 15:45:02 -05:00
Penar Musaraj 708190802c FIX: Strip video oneboxes from excerpts
Follows up on 49843f327e and removes onebox markup for videos too in the excerpt, this was previously being counted as part of the excerpt length.
2020-02-17 13:52:23 -05:00
Jarek Radosz 6cfd16656f
FIX: Ignore group mentions inside quotes ()
Also includes:
* DEV: Reuse found elements
2020-02-10 18:31:42 +01:00
Penar Musaraj 49843f327e
FIX: Strip audio/video content from excerpt () 2020-02-06 15:08:13 -05:00
Martin Brennan 1150cd4621
FIX: Stop secure media URLs being censored too liberally in emails ()
For example /t/ URLs were being replaced if they contained secure-media-uploads so if you made a topic called "Secure Media Uploads Are Cool" the View Topic link in the user notifications would be stripped out.

Refactored code so this secure URL detection happens in one place.
2020-01-30 16:19:14 +10:00
Martin Brennan 45b37a8bd1
FIX: Resolve pull hotlinked image and broken link issues for secure media URLs ()
When pull_hotlinked_images tried to run on posts with secure media (which had already been downloaded from external sources) we were getting a 404 when trying to download the image because the secure endpoint doesn't allow anon downloads.

Also, we were getting into an infinite loop of pull_hotlinked_images because the job didn't consider the secure media URLs as "downloaded" already so it kept trying to download them over and over.

In this PR I have also refactored secure-media-upload URL checks and mutations into single source of truth in Upload, adding a SECURE_MEDIA_ROUTE constant to check URLs against too.
2020-01-24 11:59:30 +10:00
Gerhard Schlager 9ebb69e8eb FIX: Respect `enable_inline_emoji_translation` setting in titles 2019-12-03 20:39:11 +01:00
Penar Musaraj 102909edb3 FEATURE: Add support for secure media ()
This PR introduces a new secure media setting. When enabled, it prevent unathorized access to media uploads (files of type image, video and audio). When the `login_required` setting is enabled, then all media uploads will be protected from unauthorized (anonymous) access. When `login_required`is disabled, only media in private messages will be protected from unauthorized access. 

A few notes: 

- the `prevent_anons_from_downloading_files` setting no longer applies to audio and video uploads
- the `secure_media` setting can only be enabled if S3 uploads are already enabled and configured
- upload records have a new column, `secure`, which is a boolean `true/false` of the upload's secure status
- when creating a public post with an upload that has already been uploaded and is marked as secure, the post creator will raise an error
- when enabling or disabling the setting on a site with existing uploads, the rake task `uploads:ensure_correct_acl` should be used to update all uploads' secure status and their ACL on S3
2019-11-18 11:25:42 +10:00
Robin Ward 8d34f4bbd9 Revert "Revert Ember.run refactors"
This reverts commit fcb1ca52f9.
2019-10-30 09:48:24 -04:00
Robin Ward fcb1ca52f9 Revert Ember.run refactors
This reverts commit 5ca60fcb6b.
2019-10-29 17:10:47 -04:00
Robin Ward cfa5d123a3 Temporary fix: Don't use ember imports in pretty text 2019-10-29 16:45:05 -04:00
Krzysztof Kotlarek 427d54b2b0 DEV: Upgrading Discourse to Zeitwerk ()
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains. 

We no longer need to use Rails "require_dependency" anywhere and instead can just use standard 
Ruby patterns to require files.

This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
Sam Saffron e9cf0de588 FEATURE: do not replace &harr; with an emoji
&harr; technically can be an emoji, but usually will not be used in that way
2019-08-30 15:06:23 +10:00
David Taylor 39e0442de9 FIX: Various watched words improvements
- Client-side censoring fixed for non-chrome browsers. (Regular expression rewritten to avoid lookback)
- Regex generation is now done on the server, to reduce repeated logic, and make it easier to extend in plugins
- Censor tests are moved to ruby, to ensure everything works end-to-end
- If "watched words regular expressions" is enabled, warn the admin when the generated regex is invalid
2019-08-02 15:29:12 +01:00
Dan Ungureanu 9f5cfa192e
FEATURE: Allow Markdown in post notices. () 2019-07-09 14:42:02 +03:00