Commit Graph

260 Commits

Author SHA1 Message Date
Sam 9e241e82e9
DEV: use HTML5 version of loofah (#21522)
https://meta.discourse.org/t/markdown-preview-and-result-differ/263878

The result of this markdown had different results in the composer preview and the post. This is solved by updating Loofah to the latest version and using html5 fragments like our user had reported. While the change was only needed in cooked_post_processor.rb for this fix, other areas also had to be updated due to various side effects.
2023-06-20 09:49:22 +08:00
Natalie Tay ccca2dbfe0
FIX: Quoted images should be lightboxed (#21332)
An older change about optimising images caused the selector that adds lightboxing not to apply on quoted images. This fixes that. The selector is now not applicable as optimisation occurs in a separate place.

This change allows quoted images to be opened in a lightbox.
2023-05-02 17:02:19 +08:00
Bianca Nenciu 024b8b2640
FIX: Show large image placeholder for image onebox (#21237)
Large or broken images are removed from oneboxes, but sometimes images
were removed when they were oneboxed too. The reason is that images can
be oneboxed by the AllowlistedGenericOnebox or ImageOnebox and only
AllowlistedGenericOnebox was handled correctly.
2023-04-26 20:05:22 +03:00
David Taylor e52f322cb5
UX: Use dominant color while loading onebox images (#21091)
When we "pull hotlinked images" on onebox images, they are added to the uploads table and their dominant color is calculated. This commit adds the data to the HTML so that it can be used by the client in the same way as non-onebox images. It also adds specific handling to the new `discourse-lazy-videos` plugin.
2023-04-13 12:04:46 +01:00
Daniel Waterworth 666536cbd1
DEV: Prefer \A and \z over ^ and $ in regexes (#19936) 2023-01-20 12:52:49 -06:00
David Taylor 6417173082
DEV: Apply syntax_tree formatting to `lib/*` 2023-01-09 12:10:19 +00:00
Sam 755ca0fcbb
PERF: stop downloading images from post processor and lean on uploads
Previously we would unconditionally fetch all images via HTTP to grab
original sizing from cooked post processor in 2 different spots.

This was wasteful as we already calculate and cache this info in upload records.

This also simplifies some specs and reduces use of mocks.
2022-11-25 12:40:31 +11:00
Martin Brennan 8ebd5edd1e
DEV: Rename secure_media to secure_uploads (#18376)
This commit renames all secure_media related settings to secure_uploads_* along with the associated functionality.

This is being done because "media" does not really cover it, we aren't just doing this for images and videos etc. but for all uploads in the site.

Additionally, in future we want to secure more types of uploads, and enable a kind of "mixed mode" where some uploads are secure and some are not, so keeping media in the name is just confusing.

This also keeps compatibility with the `secure-media-uploads` path, and changes new
secure URLs to be `secure-uploads`.

Deprecated settings:

* secure_media -> secure_uploads
* secure_media_allow_embed_images_in_emails -> secure_uploads_allow_embed_images_in_emails
* secure_media_max_email_embed_image_size_kb -> secure_uploads_max_email_embed_image_size_kb
2022-09-29 09:24:33 +10:00
David Taylor d0243f741e
UX: Use dominant color as image loading placeholder (#18248)
We previously had a system which would generate a 10x10px preview of images and add their URLs in a data-small-upload attribute. The client would then use that as the background-image of the `<img>` element. This works reasonably well on fast connections, but on slower connections it can take a few seconds for the placeholders to appear. The act of loading the placeholders can also break or delay the loading of the 'real' images.

This commit replaces the placeholder logic with a new approach. Instead of a 10x10px preview, we use imagemagick to calculate the average color of an image and store it in the database. The hex color value then added as a `data-dominant-color` attribute on the `<img>` element, and the client can use this as a `background-color` on the element while the real image is loading. That means no extra HTTP request is required, and so the placeholder color can appear instantly.

Dominant color will be calculated:
1. When a new upload is created
2. During a post rebake, if the dominant color is missing from an upload, it will be calculated and stored
3. Every 15 minutes, 25 old upload records are fetched and their dominant color calculated and stored. (part of the existing PeriodicalUpdates job)

Existing posts will continue to use the old 10x10px placeholder system until they are next rebaked
2022-09-20 10:28:17 +01:00
Bianca Nenciu c789c689c2
FIX: Remove dead and large images from oneboxes (#17868)
Dead and large images are replaced with a placeholder, either a broken
chain icon or a short text. This commit no longer applies this
transformation for images inside Oneboxes, but removes them instead.
2022-08-11 19:09:48 +03:00
David Taylor 5238f6788c
FEATURE: Allow hotlinked media to be blocked (#16940)
This commit introduces a new site setting: `block_hotlinked_media`. When enabled, all attempts to hotlink media (images, videos, and audio) will fail, and be replaced with a linked placeholder. Exceptions to the rule can be added via `block_hotlinked_media_exceptions`.

`download_remote_image_to_local` can be used alongside this feature. In that case, hotlinked images will be blocked immediately when the post is created, but will then be replaced with the downloaded version a few seconds later.

This implementation is purely server-side, and does not impact the composer preview.

Technically, there are two stages to this feature:

1. `PrettyText.sanitize_hotlinked_media` is called during `PrettyText.cook`, and whenever new images are introduced by Onebox. It will iterate over all src/srcset attributes in the post HTML and check if they're allowed. If not, the attributes will be removed and replaced with a `data-blocked-hotlinked-src(set)` attribute

2. In the `CookedPostProcessor`, we iterate over all `data-blocked-hotlinked-src(set)` attributes and check whether we have a downloaded version of the media. If yes, we update the src to use the downloaded version. If not, the entire media element is replaced with a placeholder. The placeholder is labelled 'external media', and is a link to the offsite media.
2022-06-07 15:23:04 +01:00
David Taylor bf6f8299a7 FEATURE: Pull hotlinked images immediately after posting
Previously, with the default `editing_grace_period`, hotlinked images were pulled 5 minutes after a post is created. This delay was added to reduce the chance of automated edits clashing with user edits.

This commit refactors things so that we can pull hotlinked images immediately. URLs are immediately updated in the post's `cooked` HTML. The post's raw markdown is updated later, after the `editing_grace_period`.

This involves a number of behind-the-scenes changes including:

- Schedule Jobs::PullHotlinkedImages immediately after Jobs::ProcessPost. Move scheduling to after the `update_column` call to avoid race conditions

- Move raw changes into a separate job, which is delayed until after the ninja-edit window

- Move disable_if_low_on_disk_space logic into the `pull_hotlinked_images` job

- Move raw-parsing/replacing logic into `InlineUpload` so it can be easily be shared between `UpdateHotlinkedRaw` and `PullUserProfileHotlinkedImages`
2022-05-23 14:28:02 +01:00
David Taylor 0baabafa9d DEV: Map already-downloaded hotlinked images in post_process_cooked
Previously this mapping of **cooked** images was only being run for oneboxes. Now it runs for all images, so we can transform hotlinked images without needing to immediately update `raw`
2022-05-23 14:28:02 +01:00
Isaac Janzen 20740f196c
FIX: handle quote rendering for external Discourse instance (#16722)
Gracefully handle quotes from an external discourse instance by stripping quote-controls and including username in the title
2022-05-12 10:07:43 -05:00
David Taylor c1db968740
DEV: Move hotlinked image information into a dedicated table (#16585)
This will make future changes to the 'pull hotlinked images' system easier. This commit should not introduce any functional change.

For now, the old post_custom_field data is kept in the database. This will be dropped in a future commit.
2022-05-03 13:53:32 +01:00
Bianca Nenciu f317783e65
DEV: Remove duplicated methods (#16178) 2022-03-14 19:35:01 +02:00
Martin Brennan 88a8584348
FIX: Cooking custom emojis should not use a secure URL (#15929)
When a site has secure media enabled and a post is with secure
media, we were incorrectly cooking custom emoji URLs and using the
secure URL for those emojis, even though they should not be
considered secure (their corresponding upload records in the
database are _not_ secure). Now instead of the blanket
post.with_secure_media? boolean for the secure: param, we also
want to make sure the image whose URL is being cooked is also
_not_ a custom emoji.
2022-02-14 13:02:42 +10:00
Natalie Tay 4c46c7e334
DEV: Remove xlink hrefs (#15059) 2021-11-25 15:22:43 +11:00
Mark VanLandingham 4da23e811b
DEV: Create CookedProcessMixin to process generic cooked (#15029) 2021-11-22 13:32:12 -06:00
Arpit Jalan d1fc759ac4
FIX: remove 'crawl_images' site setting (#14646) 2021-10-19 17:12:29 +05:30
Martin Brennan dba6a5eabf
FEATURE: Humanize file size error messages (#14398)
The file size error messages for max_image_size_kb and
max_attachment_size_kb are shown to the user in the KB
format, regardless of how large the limit is. Since we
are going to support uploading much larger files soon,
this KB-based limit soon becomes unfriendly to the end
user.

For example, if the max attachment size is set to 512000
KB, this is what the user sees:

> Sorry, the file you are trying to upload is too big (maximum
size is 512000KB)

This makes the user do math. In almost all file explorers that
a regular user would be familiar width, the file size is shown
in a format based on the maximum increment (e.g. KB, MB, GB).

This commit changes the behaviour to output a humanized file size
instead of the raw KB. For the above example, it would now say:

> Sorry, the file you are trying to upload is too big (maximum
size is 512 MB)

This humanization also handles decimals, e.g. 1536KB = 1.5 MB
2021-09-22 07:59:45 +10:00
Penar Musaraj 726500bc59
FEATURE: Enable pausing images from Giphy and Tenor (#13185) 2021-05-27 15:00:38 -04:00
Penar Musaraj 29f3621f45
FIX: Disable lightboxing of animated images (#13099) 2021-05-20 15:19:44 -04:00
Penar Musaraj c11d75da87
FEATURE: Allow pausing animated images in posts (#12795)
Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2021-04-22 11:28:35 -04:00
Bianca Nenciu 0c8d658ba8
SECURITY: Prefer Loofah for processing cooked HTML 2021-02-24 17:17:49 +02:00
David Taylor 04c75d417b
UX: Skip github commit avatars for topic/post thumbnails (#12157)
GitHub oneboxes use `.onebox-avatar-inline`, not `.onebox-avatar`
2021-02-22 10:40:40 +00:00
David Taylor b770c30391
FEATURE: Allow onebox images to be used as topic thumbnails (#12050)
Still excludes GitHub avatars. Those were the original reason for adding
this broad exclusion. Context at https://meta.discourse.org/t/165713/4

If we find more oneboxes which are unsuitable for thumbnails, we can add
them to this selector.
2021-02-11 17:50:42 +00:00
David Taylor 830797a9c3
FEATURE: Allow post/topic thumbnails to be prioritized via markdown (#12044)
Previously we would always take the first image in a post to use as the
thumbnail. On media-heavy sites, users may want to manually select a
specific image as the topic thumbnail. This commit allows this to be
done via a `|thumbnail` attribute in markdown.

For example, in this case, bbb would be chosen as the thumbnail:

```
![alttext|100x100](upload://aaa)
![alttext|100x100|thumbnail](upload://bbb)
```
2021-02-11 15:44:41 +00:00
Bianca Nenciu e98c7b15d6
FIX: Do not optimize animated images in cooked posts (#11214)
CookedPostProcessor replaces all large images with their optimized
versions, but for GIF images the optimized version is limited to first
frame only. This caused animations it cooked posts to require a click
to show up the lightbox and start playing.
2020-11-12 21:47:30 +02:00
Roman Rizzi efb9fd6ac0
FIX: Make sure rel attributes are correctly set. (#10645)
We must guarantee that "rel=noopener" was set if "target=_blank" is present, which is not always the case for trusted users. Also, if the link contains the "nofollow" attribute, it has to have the "ugc" attribute as well.
2020-09-10 12:59:51 -03:00
David Taylor cb12a721c4
REFACTOR: Refactor pull_hotlinked_images job
This commit should cause no functional change
- Split into functions to avoid deep nesting
- Register custom field type, and remove manual json parse/serialize
- Recover from deleted upload records

Also adds a test to ensure pull_hotlinked_images redownloads secure images only once
2020-08-05 12:14:59 +01:00
Krzysztof Kotlarek e0d9232259
FIX: use allowlist and blocklist terminology (#10209)
This is a PR of the renaming whitelist to allowlist and blacklist to the blocklist.
2020-07-27 10:23:54 +10:00
Robin Ward 7045a2a87c FIX: Don't strip `noopener` from oneboxes 2020-07-13 16:54:42 -04:00
David Taylor e159fb06df
FEATURE: Download remote images even for old posts (#9925)
When a post is rebaked, the admins expect it to work the same regardless of how old the post is.
2020-05-29 17:13:55 +01:00
David Taylor 28f46c171c
FIX: Pull hotlinked images even when edited by system users (#9890)
Previously the pull hotlinked images job was skipped after system edits. This ensured that we never had an infinite loop of system-edit/pull-hotlinked/system-edit/pull-hotlinked etc.

A side effect was that edits made by system for any other reason (e.g. API, removing full quotes) would prevent pulling hotlinked images. This commit removes the system edit check, and replaces it with another method to avoid an infinite job scheduling loop.
2020-05-29 13:07:47 +01:00
David Taylor 956d15d13f
UX: Do not use small onebox images as post/topic images 2020-05-14 18:01:43 +01:00
Robin Ward f9608c0af5 DEV: Remove INLINE_ONEBOX_* constants
There were two constants here, `INLINE_ONEBOX_LOADING_CSS_CLASS` and
`INLINE_ONEBOX_CSS_CLASS` that were both longer than the strings they
were DRYing up: `inline-onebox-loading` and `inline-onebox`

I normally appreciate constants, but in this case it meant that we had
a lot of JS imports resulting in many more lines of code (and CPU cycles
spent figuring them out.)

It also meant we had an `.erb` file and had to invoke Ruby to create the
JS file, which meant the app was harder to port to Ember CLI.

I removed the constants. It's less DRY but faster and simpler, and
arguably the loss of DRYness is not significant as you can still search
for the `inline-onebox-loading` and `inline-onebox` strings easily if
you are refactoring.
2020-05-07 16:14:38 -04:00
David Taylor 03818e642a
FEATURE: Include optimized thumbnails for topics (#9215)
This introduces new APIs for obtaining optimized thumbnails for topics. There are a few building blocks required for this:

- Introduces new `image_upload_id` columns on the `posts` and `topics` table. This replaces the old `image_url` column, which means that thumbnails are now restricted to uploads. Hotlinked thumbnails are no longer possible. In normal use (with pull_hotlinked_images enabled), this has no noticeable impact

- A migration attempts to match existing urls to upload records. If a match cannot be found then the posts will be queued for rebake

- Optimized thumbnails are generated during post_process_cooked. If thumbnails are missing when serializing a topic list, then a sidekiq job is queued

- Topic lists and topics now include a `thumbnails` key, which includes all the available images:
   ```
   "thumbnails": [
   {
     "max_width": null,
     "max_height": null,
     "url": "//example.com/original-image.png",
     "width": 1380,
     "height": 1840
   },
   {
     "max_width": 1024,
     "max_height": 1024,
     "url": "//example.com/optimized-image.png",
     "width": 768,
     "height": 1024
   }
   ]
  ```

- Themes can request additional thumbnail sizes by using a modifier in their `about.json` file:
   ```
    "modifiers": {
      "topic_thumbnail_sizes": [
        [200, 200],
        [800, 800]
      ],
      ...
  ```
  Remember that these are generated asynchronously, so your theme should include logic to fallback to other available thumbnails if your requested size has not yet been generated

- Two new raw plugin outlets are introduced, to improve the customisability of the topic list. `topic-list-before-columns` and `topic-list-before-link`
2020-05-05 09:07:50 +01:00
Krzysztof Kotlarek 9bff0882c3
FEATURE: Nokogumbo (#9577)
* FEATURE: Nokogumbo

Use Nokogumbo HTML parser.
2020-05-05 13:46:57 +10:00
Martin Brennan cd1c7d7560
FIX: Copying image markdown for secure media loading full image (#9488)
* When copying the markdown for an image between posts, we were not adding the srcset and data-small-image attributes which are done by calling optimize_image! in cooked post processor
* Refactored the code which was confusing in its current state (the consider_for_reuse method was super confusing) and fixed the issue
2020-04-24 10:29:02 +10:00
Jarek Radosz ab52bed014
DEV: Remove the return value of disable_if_low_on_disk_space (#9469)
It was used only in specs.
2020-04-21 03:48:33 +02:00
Jarek Radosz 5a81e3999c
DEV: Remove `bypass_bump` from CookedPostProcessor (#9468)
It was only passing it along to `PullHotlinkedImages` and that class have not used that arg since April 2014 (c52ee665b4)
2020-04-21 03:48:19 +02:00
Bianca Nenciu 3914e9cb5c
FIX: get_size_from_image_sizes should return [width, height] or nil (#9298) 2020-03-28 20:20:51 +02:00
Bianca Nenciu 7952cbb9a2
FIX: Perform crop using user-specified image sizes (#9224)
* FIX: Perform crop using user-specified image sizes

It used to resize the images to max width and height first and then
perform the crop operation. This is wrong because it ignored the user
specified image sizes from the Markdown.

* DEV: Use real images in test
2020-03-26 16:40:00 +02:00
Dan Ungureanu 0754c7c404
FIX: Various fixes to support posts with no user (#8877)
* Do not grant badges for posts with no user
* Ensure instructions are correct in Change Owner modal
* Hide user-dependent actions from posts with no user
* Make PostRevisor work with posts with no user
* Ensure posts with no user can be deleted
* discourse-narrative-bot should ignore posts with no user
* Skip TopicLink creation for posts with no user
2020-03-11 14:03:20 +02:00
Sam Saffron 64b3512084
DEV: use DiskSpace module for all disk space calculations
This normalizes it so we only carry one place for grabbing disk space size

It also normalizes the command made so it uses Discourse.execute_command
which splits off params in a far cleaner way.
2020-02-18 15:13:19 +11:00
Robin Ward c2e58b6b85 FIX: Don't remove the topic image if posts don't have them 2020-02-13 14:00:30 -05:00
Dan Ungureanu ec40242b5c
FIX: Make inline oneboxes work with secured topics in secured contexts (#8895) 2020-02-12 12:11:28 +02:00
Penar Musaraj 0fd39cc511 FIX: Remove post/topic image_url on post edits
- resets image_url when image is removed from first post on edit
- excludes onebox icons from being featured as topic/post images
2020-02-06 11:23:08 -05:00
Sam Saffron 7f3a30d79f FIX: blank cooked markdown could raise an exception in logs
Previously if somehow a user created a blank markdown document using tag
tricks (eg `<p></p><p></p><p></p><p></p><p></p><p></p>`) and so on, we would
completely strip the document down to blank on post process due to onebox
hack.

Needs a followup cause I am still unclear about the reason for empty p stripping
and it can cause some unclear cases when we re-cook posts.
2020-01-29 11:37:25 +11:00