Commit Graph

69 Commits

Author SHA1 Message Date
Bianca Nenciu 43e52a7dc1
DEV: Remove gifsicle dependency (#10357)
Dependency on gifsicle, allow_animated_avatars and allow_animated_thumbnails
site settings were all removed. Animated GIF images are still allowed, but
the generated optimized images are no longer animated for those (which were
used for avatars and thumbnails).

The added 'animated' is populated by extracting information using FastImage.
This field was used to selectively reoptimize old animations. This process
happens in the background.
2020-10-16 13:41:27 +03:00
Krzysztof Kotlarek e7c72cd1e4
FIX: deprecate whitelist constants (#10716)
Deprecation of:
WHITELISTED_REDIRECT_HOSTNAMES
CUSTOM_INTERPOLATION_KEYS_WHITELIST
WHITELISTED_SVG_ELEMENTS
2020-09-28 13:52:05 +10:00
Anthony Dillon 01afcf30cc
FIX: Whitelist marker elements when uploading SVGs (#10703) 2020-09-18 14:13:07 -04:00
Krzysztof Kotlarek 333ddd4011 SECURITY: return error on oversized images 2020-09-14 10:45:11 +10:00
Roman Rizzi 726bae8f0c
DEV: Pass the for_export flag to the before_upload_creation event (#10599) 2020-09-04 17:24:40 -03:00
Penar Musaraj 86cb5803ba
Convert HEIC uploads to JPG by default (#10395) 2020-08-07 11:17:50 -04:00
Krzysztof Kotlarek e0d9232259
FIX: use allowlist and blocklist terminology (#10209)
This is a PR of the renaming whitelist to allowlist and blacklist to the blocklist.
2020-07-27 10:23:54 +10:00
Penar Musaraj 7559758e10
FEATURE: Support converting HEIF images to JPEG (#10079) 2020-07-22 21:40:09 -04:00
Roman Rizzi 2abfd30d22
FIX: Trigger before upload event after saving and before uploading it, so we are sure that the upload is valid. (#10269) 2020-07-20 17:59:37 -03:00
Martin Brennan 31e31ef449
SECURITY: Add content-disposition: attachment for SVG uploads
* strip out the href and xlink:href attributes from use element that
  are _not_ anchors in svgs which can be used for XSS
* adding the content-disposition: attachment ensures that
  uploaded SVGs cannot be opened and executed using the XSS exploit.
  svgs embedded using an img tag do not suffer from the same exploit
2020-07-09 13:31:48 +10:00
Régis Hanol 48b4ed41f5 FIX: uploading an existing image as a site setting
The previous fix (f43c0a5d85) wasn't working for images that were already uploaded.
The "metadata" (eg. 'for_*' and 'secure' attributes) were not added to existing uploads.

Also used 'Upload.get_from_url' is the admin/site_setting controller to properly retrieve
an upload from its URL.

Fixed the Upload::URL_REGEX to use the \h (hexadecimal) for the SHA

Follow-up-to: f43c0a5d85
2020-07-03 19:16:54 +02:00
Roman Rizzi a4bfa35420
DEV: Pass the is_image flag when triggering the before_upload_creation event (#10031) 2020-06-11 11:06:48 -03:00
romanrizzi ac555610d9 DEV: Trigger an event before creating an upload 2020-06-10 16:30:53 -03:00
Vinoth Kannan b57d4586d7 FIX: use correct command line attribute for `gifsicle` while scale down the gif. 2020-04-10 18:16:47 +05:30
Dan Ungureanu 49395ec577
FIX: Fix image optimization pipeline (#9257)
* FIX: Do not use original filename to extract the original filename

Prefer extracting filename from the destination path, which is build
using extracted image information.

* UX: Show better error images
2020-03-25 12:59:16 +02:00
Martin Brennan 7c32411881
FEATURE: Secure media allowing duplicated uploads with category-level privacy and post-based access rules (#8664)
### General Changes and Duplication

* We now consider a post `with_secure_media?` if it is in a read-restricted category.
* When uploading we now set an upload's secure status straight away.
* When uploading if `SiteSetting.secure_media` is enabled, we do not check to see if the upload already exists using the `sha1` digest of the upload. The `sha1` column of the upload is filled with a `SecureRandom.hex(20)` value which is the same length as `Upload::SHA1_LENGTH`. The `original_sha1` column is filled with the _real_ sha1 digest of the file. 
* Whether an upload `should_be_secure?` is now determined by whether the `access_control_post` is `with_secure_media?` (if there is no access control post then we leave the secure status as is).
* When serializing the upload, we now cook the URL if the upload is secure. This is so it shows up correctly in the composer preview, because we set secure status on upload.

### Viewing Secure Media

* The secure-media-upload URL will take the post that the upload is attached to into account via `Guardian.can_see?` for access permissions
* If there is no `access_control_post` then we just deliver the media. This should be a rare occurrance and shouldn't cause issues as the `access_control_post` is set when `link_post_uploads` is called via `CookedPostProcessor`

### Removed

We no longer do any of these because we do not reuse uploads by sha1 if secure media is enabled.

* We no longer have a way to prevent cross-posting of a secure upload from a private context to a public context.
* We no longer have to set `secure: false` for uploads when uploading for a theme component.
2020-01-16 13:50:27 +10:00
Dan Ungureanu 4e130f1e03
SECURITY: Remove event handlers from SVG files 2019-12-11 17:04:50 +02:00
Penar Musaraj 102909edb3 FEATURE: Add support for secure media (#7888)
This PR introduces a new secure media setting. When enabled, it prevent unathorized access to media uploads (files of type image, video and audio). When the `login_required` setting is enabled, then all media uploads will be protected from unauthorized (anonymous) access. When `login_required`is disabled, only media in private messages will be protected from unauthorized access. 

A few notes: 

- the `prevent_anons_from_downloading_files` setting no longer applies to audio and video uploads
- the `secure_media` setting can only be enabled if S3 uploads are already enabled and configured
- upload records have a new column, `secure`, which is a boolean `true/false` of the upload's secure status
- when creating a public post with an upload that has already been uploaded and is marked as secure, the post creator will raise an error
- when enabling or disabling the setting on a site with existing uploads, the rake task `uploads:ensure_correct_acl` should be used to update all uploads' secure status and their ACL on S3
2019-11-18 11:25:42 +10:00
Régis Hanol 9a81cb9e55 FIX: ensure we remove tempfiles from disk when creating an upload
Follow-up to 46d12c5ad3
2019-10-11 11:13:10 +02:00
Régis Hanol 46d12c5ad3 FIX: properly downsize image on upload
Overwriting the same file with 'convert' is not always working as expected.
Adding a temporary file as the destination of the downsize makes this operation much more reliable.

Also switched to using (the more aggressive) 50% resize instead of halving the number of pixels.
2019-10-04 16:22:57 +02:00
Krzysztof Kotlarek 427d54b2b0 DEV: Upgrading Discourse to Zeitwerk (#8098)
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains. 

We no longer need to use Rails "require_dependency" anywhere and instead can just use standard 
Ruby patterns to require files.

This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
Guo Xiang Tan 7bd93eba3e FIX: Gravatar uploads being dependent on authorized_extensions. 2019-08-01 16:24:09 +08:00
Sam Saffron 30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Guo Xiang Tan ebca588fd0 DEV: Remove unused line of code. 2019-05-02 16:54:10 +08:00
Régis Hanol 2b721dff57 FIX: whitelist 'feGaussianBlur' and 'filter' svg elements 2019-04-01 12:56:47 +02:00
Penar Musaraj d6d4a5ba4a FEATURE: support custom icons in themes (#7155)
* First take

* Add support for sprites in themes

Automatically register any custom icons added via themes or plugins

* Fix theme sprite caching

* Simplify test

* Update lib/svg_sprite/svg_sprite.rb

Co-Authored-By: pmusaraj <pmusaraj@gmail.com>

* Fix /svg-sprite/search request
2019-03-15 17:16:15 +11:00
Penar Musaraj c50db76f5d FIX: do not treat TIFF, BMP, WEBP as images
Treating TIFF and BMP as images cause us to add them to IMG tags, this is very inconsistent across browsers.

You can still upload these files they will simply not be displayed in IMG tags.
2019-02-11 16:28:43 +11:00
Régis Hanol c256121833 FIX: add support for style element in SVGs 2019-02-07 12:09:06 +01:00
Régis Hanol 788719d271 DEV: speed up posts base imports 2019-01-04 15:30:17 +01:00
Vinoth Kannan 75dbb98cca FEATURE: Add S3 etag value to uploads table (#6795) 2019-01-04 14:16:22 +08:00
Sam 86255faa08 FEATURE: do not switch to JPEG unless you meet 75k byte savings
This also adjusts the algorithm to expect

- 30% saving for JPEG conversion

AND

- Minimum of 75K bytes saved

The reasoning for increase of saving requirements is cause PNG may have been
uploaded unoptimized, 30% saving on PNG is very possible
2018-11-21 11:01:08 +11:00
Guo Xiang Tan 44391ee8ab
FEATURE: Upload Site Settings. (#6573) 2018-11-14 15:03:02 +08:00
Sam 0a442e319c FIX: correct svg handling for images
We regressed and optimized images no longer worked with svg

The following adds the correct logic to simply copy file for svgs
and bypasses resizing for svg avatars
2018-11-07 15:29:26 +11:00
Sam df45e82377 SECURITY: only allow picking of avatars created by self (#6417)
* SECURITY: only allow picking of avatars created by self

Also adds origin tracking to all uploads including de-duplicated uploads
2018-09-19 22:33:10 -07:00
Guo Xiang Tan e1b16e445e Rename `FileHelper.is_image?` -> `FileHelper.is_supported_image?`. 2018-09-12 09:22:28 +08:00
Sam 9ab1fb7dfc FEATURE: correctly store width and height on uploads
Previously we used width and height for thumbnails, new code ensures

1. We auto correct width and height
2. We added extra columns for thumbnail_width and height, this is determined
 by actual upload and no longer passed in as a side effect
3. Optimized Image now stores filesize which can be used for analysis, decisions

Also

- fixes Android image manifest as a side effect
- fixes issue where a thumbnail generated that is smaller than the upload is no longer used
2018-08-28 12:59:22 +10:00
Régis Hanol d1607a387a FIX: only allow printable characters in uploads filename 2018-08-21 18:11:01 +02:00
Sam f5fe58384f correct regression around file renaming 2018-08-20 16:08:05 +10:00
Sam d7b1919ead correct specs 2018-08-20 12:46:14 +10:00
Sam 8b5e42ea16 FIX: always test and coerce to image on upload
In the past the filename of the origin was used as the source
for the extension of the file when optimizing on upload.

We now use the actual calculated extension based on upload data.
2018-08-20 12:18:59 +10:00
Guo Xiang Tan a9e502936f FIX: Converting PNG to JPEG does not set the correct extension. 2018-08-17 13:09:48 +08:00
Régis Hanol 98898ff4b3 FIX: raise an exception when 'downsize'git st fails 2018-08-16 19:20:07 +02:00
Guo Xiang Tan d10c9d7d75 FIX: Missing extensions for non-image uploads due to 2b57239389. 2018-08-13 10:58:55 +08:00
Guo Xiang Tan 2b57239389 FIX: Upload's content is the only source of truth for the file type. 2018-08-07 13:15:00 +08:00
Régis Hanol f94aeaf6cf SECURITY: force IM decoder based on file extension - part 3 2018-07-25 23:55:06 +02:00
Régis Hanol 800c57c6ab SECURITY: force IM decoder based on file extension - part 2 2018-07-25 23:08:02 +02:00
Régis Hanol 4bf3bf6786 SECURITY: force IM decoder based on file extension 2018-07-25 22:00:04 +02:00
Guo Xiang Tan 1d74ccaaf8 Add compatibility for ImageMagick7. 2018-07-17 15:50:58 +08:00
Guo Xiang Tan 72a3457379 Bump `discourse_image_optim` which uses a global timeout.
Our previous solution has the timeout set at the worker level
which means the total timeout would be X timeout secs * N number of
workers.
2018-07-09 10:30:18 +08:00
Arpit Jalan 91bf10bd12 FIX: create upload record for exported csv files 2018-04-20 00:27:49 +05:30