Commit Graph

243 Commits

Author SHA1 Message Date
Natalie Tay 4c46c7e334
DEV: Remove xlink hrefs (#15059) 2021-11-25 15:22:43 +11:00
Mark VanLandingham 4da23e811b
DEV: Create CookedProcessMixin to process generic cooked (#15029) 2021-11-22 13:32:12 -06:00
Arpit Jalan d1fc759ac4
FIX: remove 'crawl_images' site setting (#14646) 2021-10-19 17:12:29 +05:30
Martin Brennan dba6a5eabf
FEATURE: Humanize file size error messages (#14398)
The file size error messages for max_image_size_kb and
max_attachment_size_kb are shown to the user in the KB
format, regardless of how large the limit is. Since we
are going to support uploading much larger files soon,
this KB-based limit soon becomes unfriendly to the end
user.

For example, if the max attachment size is set to 512000
KB, this is what the user sees:

> Sorry, the file you are trying to upload is too big (maximum
size is 512000KB)

This makes the user do math. In almost all file explorers that
a regular user would be familiar width, the file size is shown
in a format based on the maximum increment (e.g. KB, MB, GB).

This commit changes the behaviour to output a humanized file size
instead of the raw KB. For the above example, it would now say:

> Sorry, the file you are trying to upload is too big (maximum
size is 512 MB)

This humanization also handles decimals, e.g. 1536KB = 1.5 MB
2021-09-22 07:59:45 +10:00
Penar Musaraj 726500bc59
FEATURE: Enable pausing images from Giphy and Tenor (#13185) 2021-05-27 15:00:38 -04:00
Penar Musaraj 29f3621f45
FIX: Disable lightboxing of animated images (#13099) 2021-05-20 15:19:44 -04:00
Penar Musaraj c11d75da87
FEATURE: Allow pausing animated images in posts (#12795)
Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2021-04-22 11:28:35 -04:00
Bianca Nenciu 0c8d658ba8
SECURITY: Prefer Loofah for processing cooked HTML 2021-02-24 17:17:49 +02:00
David Taylor 04c75d417b
UX: Skip github commit avatars for topic/post thumbnails (#12157)
GitHub oneboxes use `.onebox-avatar-inline`, not `.onebox-avatar`
2021-02-22 10:40:40 +00:00
David Taylor b770c30391
FEATURE: Allow onebox images to be used as topic thumbnails (#12050)
Still excludes GitHub avatars. Those were the original reason for adding
this broad exclusion. Context at https://meta.discourse.org/t/165713/4

If we find more oneboxes which are unsuitable for thumbnails, we can add
them to this selector.
2021-02-11 17:50:42 +00:00
David Taylor 830797a9c3
FEATURE: Allow post/topic thumbnails to be prioritized via markdown (#12044)
Previously we would always take the first image in a post to use as the
thumbnail. On media-heavy sites, users may want to manually select a
specific image as the topic thumbnail. This commit allows this to be
done via a `|thumbnail` attribute in markdown.

For example, in this case, bbb would be chosen as the thumbnail:

```
![alttext|100x100](upload://aaa)
![alttext|100x100|thumbnail](upload://bbb)
```
2021-02-11 15:44:41 +00:00
Bianca Nenciu e98c7b15d6
FIX: Do not optimize animated images in cooked posts (#11214)
CookedPostProcessor replaces all large images with their optimized
versions, but for GIF images the optimized version is limited to first
frame only. This caused animations it cooked posts to require a click
to show up the lightbox and start playing.
2020-11-12 21:47:30 +02:00
Roman Rizzi efb9fd6ac0
FIX: Make sure rel attributes are correctly set. (#10645)
We must guarantee that "rel=noopener" was set if "target=_blank" is present, which is not always the case for trusted users. Also, if the link contains the "nofollow" attribute, it has to have the "ugc" attribute as well.
2020-09-10 12:59:51 -03:00
David Taylor cb12a721c4
REFACTOR: Refactor pull_hotlinked_images job
This commit should cause no functional change
- Split into functions to avoid deep nesting
- Register custom field type, and remove manual json parse/serialize
- Recover from deleted upload records

Also adds a test to ensure pull_hotlinked_images redownloads secure images only once
2020-08-05 12:14:59 +01:00
Krzysztof Kotlarek e0d9232259
FIX: use allowlist and blocklist terminology (#10209)
This is a PR of the renaming whitelist to allowlist and blacklist to the blocklist.
2020-07-27 10:23:54 +10:00
Robin Ward 7045a2a87c FIX: Don't strip `noopener` from oneboxes 2020-07-13 16:54:42 -04:00
David Taylor e159fb06df
FEATURE: Download remote images even for old posts (#9925)
When a post is rebaked, the admins expect it to work the same regardless of how old the post is.
2020-05-29 17:13:55 +01:00
David Taylor 28f46c171c
FIX: Pull hotlinked images even when edited by system users (#9890)
Previously the pull hotlinked images job was skipped after system edits. This ensured that we never had an infinite loop of system-edit/pull-hotlinked/system-edit/pull-hotlinked etc.

A side effect was that edits made by system for any other reason (e.g. API, removing full quotes) would prevent pulling hotlinked images. This commit removes the system edit check, and replaces it with another method to avoid an infinite job scheduling loop.
2020-05-29 13:07:47 +01:00
David Taylor 956d15d13f
UX: Do not use small onebox images as post/topic images 2020-05-14 18:01:43 +01:00
Robin Ward f9608c0af5 DEV: Remove INLINE_ONEBOX_* constants
There were two constants here, `INLINE_ONEBOX_LOADING_CSS_CLASS` and
`INLINE_ONEBOX_CSS_CLASS` that were both longer than the strings they
were DRYing up: `inline-onebox-loading` and `inline-onebox`

I normally appreciate constants, but in this case it meant that we had
a lot of JS imports resulting in many more lines of code (and CPU cycles
spent figuring them out.)

It also meant we had an `.erb` file and had to invoke Ruby to create the
JS file, which meant the app was harder to port to Ember CLI.

I removed the constants. It's less DRY but faster and simpler, and
arguably the loss of DRYness is not significant as you can still search
for the `inline-onebox-loading` and `inline-onebox` strings easily if
you are refactoring.
2020-05-07 16:14:38 -04:00
David Taylor 03818e642a
FEATURE: Include optimized thumbnails for topics (#9215)
This introduces new APIs for obtaining optimized thumbnails for topics. There are a few building blocks required for this:

- Introduces new `image_upload_id` columns on the `posts` and `topics` table. This replaces the old `image_url` column, which means that thumbnails are now restricted to uploads. Hotlinked thumbnails are no longer possible. In normal use (with pull_hotlinked_images enabled), this has no noticeable impact

- A migration attempts to match existing urls to upload records. If a match cannot be found then the posts will be queued for rebake

- Optimized thumbnails are generated during post_process_cooked. If thumbnails are missing when serializing a topic list, then a sidekiq job is queued

- Topic lists and topics now include a `thumbnails` key, which includes all the available images:
   ```
   "thumbnails": [
   {
     "max_width": null,
     "max_height": null,
     "url": "//example.com/original-image.png",
     "width": 1380,
     "height": 1840
   },
   {
     "max_width": 1024,
     "max_height": 1024,
     "url": "//example.com/optimized-image.png",
     "width": 768,
     "height": 1024
   }
   ]
  ```

- Themes can request additional thumbnail sizes by using a modifier in their `about.json` file:
   ```
    "modifiers": {
      "topic_thumbnail_sizes": [
        [200, 200],
        [800, 800]
      ],
      ...
  ```
  Remember that these are generated asynchronously, so your theme should include logic to fallback to other available thumbnails if your requested size has not yet been generated

- Two new raw plugin outlets are introduced, to improve the customisability of the topic list. `topic-list-before-columns` and `topic-list-before-link`
2020-05-05 09:07:50 +01:00
Krzysztof Kotlarek 9bff0882c3
FEATURE: Nokogumbo (#9577)
* FEATURE: Nokogumbo

Use Nokogumbo HTML parser.
2020-05-05 13:46:57 +10:00
Martin Brennan cd1c7d7560
FIX: Copying image markdown for secure media loading full image (#9488)
* When copying the markdown for an image between posts, we were not adding the srcset and data-small-image attributes which are done by calling optimize_image! in cooked post processor
* Refactored the code which was confusing in its current state (the consider_for_reuse method was super confusing) and fixed the issue
2020-04-24 10:29:02 +10:00
Jarek Radosz ab52bed014
DEV: Remove the return value of disable_if_low_on_disk_space (#9469)
It was used only in specs.
2020-04-21 03:48:33 +02:00
Jarek Radosz 5a81e3999c
DEV: Remove `bypass_bump` from CookedPostProcessor (#9468)
It was only passing it along to `PullHotlinkedImages` and that class have not used that arg since April 2014 (c52ee665b4)
2020-04-21 03:48:19 +02:00
Bianca Nenciu 3914e9cb5c
FIX: get_size_from_image_sizes should return [width, height] or nil (#9298) 2020-03-28 20:20:51 +02:00
Bianca Nenciu 7952cbb9a2
FIX: Perform crop using user-specified image sizes (#9224)
* FIX: Perform crop using user-specified image sizes

It used to resize the images to max width and height first and then
perform the crop operation. This is wrong because it ignored the user
specified image sizes from the Markdown.

* DEV: Use real images in test
2020-03-26 16:40:00 +02:00
Dan Ungureanu 0754c7c404
FIX: Various fixes to support posts with no user (#8877)
* Do not grant badges for posts with no user
* Ensure instructions are correct in Change Owner modal
* Hide user-dependent actions from posts with no user
* Make PostRevisor work with posts with no user
* Ensure posts with no user can be deleted
* discourse-narrative-bot should ignore posts with no user
* Skip TopicLink creation for posts with no user
2020-03-11 14:03:20 +02:00
Sam Saffron 64b3512084
DEV: use DiskSpace module for all disk space calculations
This normalizes it so we only carry one place for grabbing disk space size

It also normalizes the command made so it uses Discourse.execute_command
which splits off params in a far cleaner way.
2020-02-18 15:13:19 +11:00
Robin Ward c2e58b6b85 FIX: Don't remove the topic image if posts don't have them 2020-02-13 14:00:30 -05:00
Dan Ungureanu ec40242b5c
FIX: Make inline oneboxes work with secured topics in secured contexts (#8895) 2020-02-12 12:11:28 +02:00
Penar Musaraj 0fd39cc511 FIX: Remove post/topic image_url on post edits
- resets image_url when image is removed from first post on edit
- excludes onebox icons from being featured as topic/post images
2020-02-06 11:23:08 -05:00
Sam Saffron 7f3a30d79f FIX: blank cooked markdown could raise an exception in logs
Previously if somehow a user created a blank markdown document using tag
tricks (eg `<p></p><p></p><p></p><p></p><p></p><p></p>`) and so on, we would
completely strip the document down to blank on post process due to onebox
hack.

Needs a followup cause I am still unclear about the reason for empty p stripping
and it can cause some unclear cases when we re-cook posts.
2020-01-29 11:37:25 +11:00
Martin Brennan ab3bda6cd0
FIX: Mitigate issue where legacy pre-secure hotlinked media would not be redownloaded (#8802)
Basically, say you had already downloaded a certain image from a certain URL
using pull_hotlinked_images and the onebox. The upload would be stored
by its sha as an upload record. Whenever you linked to the same URL again
in a post (e.g. in our case an og:image on review.discourse) we would
would reuse the original upload record because of the sha1.

However when you turned on secure media this could cause problems as
the first post that uses that upload after secure media is enabled
will set the access control post for the upload to the new post.
Then if the post is deleted every single onebox/link to that same image
URL will fail forever with 403 as the secure-media-uploads URL fails
if the access control post has been deleted.

To fix this when cooking posts and pulling hotlinked images, we only
allow using an original upload by URL if its access control post
matches the current post, and if the original_sha1 is filled in,
meaning it was uploaded AFTER secure media was enabled. otherwise
we just redownload the media again to be safe, as the URL will always
be new then.
2020-01-29 10:11:38 +10:00
Martin Brennan 45b37a8bd1
FIX: Resolve pull hotlinked image and broken link issues for secure media URLs (#8777)
When pull_hotlinked_images tried to run on posts with secure media (which had already been downloaded from external sources) we were getting a 404 when trying to download the image because the secure endpoint doesn't allow anon downloads.

Also, we were getting into an infinite loop of pull_hotlinked_images because the job didn't consider the secure media URLs as "downloaded" already so it kept trying to download them over and over.

In this PR I have also refactored secure-media-upload URL checks and mutations into single source of truth in Upload, adding a SECURE_MEDIA_ROUTE constant to check URLs against too.
2020-01-24 11:59:30 +10:00
Martin Brennan 4646a38ae6
FIX: Use presigned URL to avoid 403 when pulling hotlinked images for secure media (#8764)
When we were pulling hotlinked images for oneboxes in the CookedPostProcessor, we were using the direct S3 URL, which returned a 403 error and thus did not set widths and heights of the images. We now cook the URL first based on whether the upload is secure before handing off to FastImage.
2020-01-23 09:31:46 +10:00
Bianca Nenciu 1bccd8eca9
FIX: Remove full nested quotes on direct reply (#8581)
It used to check how many quotes were inside a post, without taking
considering that some quotes can contain other quotes. This commit
selects only top level quotes.

I had to use XPath because I could not find an equivalent CSS
selector.
2019-12-20 10:24:34 +02:00
Dan Ungureanu ebe6fa95be
FIX: Optimize images in Onebox (#8471)
This commit ensures that images in Onebox are being optimized, but not
converted to lightbox too.
2019-12-09 15:39:25 +02:00
Jarek Radosz 02ca6fa6c8 DEV: See if the store is external before checking disk space (#8480)
`available_disk_space` calls `df` which exits with an error if the `uploads` path doesn't exist. That's often the case when the `Discourse.store.external?` is true.

By doing the `external?` check first the `disable_if_low_on_disk_space` does less work and doesn't output any errors to the console.
2019-12-09 12:48:45 +11:00
Jarek Radosz d07f039468 FIX: Secure Upload URLs in lightbox (#8451)
This fixes the following issues:

* The link element on the lightbox which pops open the lightbox was linking to the S3 URL with a private ACL instead of the secure media URL for the image
* Change to use `@post.with_secure_media?` in `CookedPostProcessor` for URL cooking, as in some cases, like when a post is edited and an upload is added, `upload.secure?` can be false which resulted in `srcset` URLs not being cooked correctly to secure media upload urls.
2019-12-05 09:13:09 +10:00
Dan Ungureanu 1e0c2235a3
FIX: Optimize quoted images (#8427)
Only images that were part of a lightbox used to be optimized. This
patch ensures that quoted images are also optimized.
2019-11-29 15:18:42 +02:00
Penar Musaraj 102909edb3 FEATURE: Add support for secure media (#7888)
This PR introduces a new secure media setting. When enabled, it prevent unathorized access to media uploads (files of type image, video and audio). When the `login_required` setting is enabled, then all media uploads will be protected from unauthorized (anonymous) access. When `login_required`is disabled, only media in private messages will be protected from unauthorized access. 

A few notes: 

- the `prevent_anons_from_downloading_files` setting no longer applies to audio and video uploads
- the `secure_media` setting can only be enabled if S3 uploads are already enabled and configured
- upload records have a new column, `secure`, which is a boolean `true/false` of the upload's secure status
- when creating a public post with an upload that has already been uploaded and is marked as secure, the post creator will raise an error
- when enabling or disabling the setting on a site with existing uploads, the rake task `uploads:ensure_correct_acl` should be used to update all uploads' secure status and their ACL on S3
2019-11-18 11:25:42 +10:00
Penar Musaraj 067696df8f DEV: Apply Rubocop redundant return style 2019-11-14 15:10:51 -05:00
Joe ce0bac7a3d FEATURE: fallback to image alt before filename if there's no title in lightboxes (#8286)
* use image alt as a fallback when there's no title

* update spec

we used to check that the overlay information is added when the image has a titie. This adds 2 more scenarios. One where an image has both a title and an alt, in which case the title should be used and alt ignored.

The other is when there's only an alt, it should then be used to generate the overlay
2019-11-04 10:15:14 +11:00
Arpit Jalan 1e9d9d9346
FIX: respect `tl3 links no follow` setting (#8232) 2019-10-22 22:41:04 +05:30
Krzysztof Kotlarek 427d54b2b0 DEV: Upgrading Discourse to Zeitwerk (#8098)
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains. 

We no longer need to use Rails "require_dependency" anywhere and instead can just use standard 
Ruby patterns to require files.

This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
Bianca Nenciu 0d22beb81d
FIX: Improve Onebox detection (#8019)
Follow-up to 7c83d2eeb2.
2019-09-10 13:59:48 +03:00
Bianca Nenciu 7c83d2eeb2 FIX: Award 'First Onebox' badge just for Oneboxed URLs. (#7974) 2019-08-08 18:45:18 +02:00
Sam Saffron 67f5ad5ac0 FEATURE: allow post process mutex to be held longer
Previously we would only hold the post process mutex for 1 minute, that is
not enough when processing a post with lots of images. This raises the bar
to 10 minutes.

It also cleans up error reporting around distributed mutexes expiring. We
used to double report.
2019-08-05 11:57:35 +10:00
Osama Sayegh 65a6f3080e FIX: don't disable download_remote_images_to_local if site uses S3 (#7861) 2019-07-05 13:36:03 +10:00