Commit Graph

50 Commits

Author SHA1 Message Date
Martin Brennan 3af2670bd5
FIX: Consider webp a supported image format for upload (#9015)
* Also fixes an issue where if webp was a downloaded hotlinked
  image and then secure + sent in an email, it was not being
  redacted because webp was not a supported media format in
  FileHelper
* Webp originally removed as an image format in
  https://github.com/discourse/discourse/pull/6377
  and there was a spec to make sure a .bin webp
  file did not get renamed from its type to webp.

  However we want to support webp images now to make
  sure they are properly redacted if secure media is
  on, so change the example in the spec to use tiff,
  another banned format, instead
2020-02-21 13:08:01 +10:00
Martin Brennan 7c32411881
FEATURE: Secure media allowing duplicated uploads with category-level privacy and post-based access rules (#8664)
### General Changes and Duplication

* We now consider a post `with_secure_media?` if it is in a read-restricted category.
* When uploading we now set an upload's secure status straight away.
* When uploading if `SiteSetting.secure_media` is enabled, we do not check to see if the upload already exists using the `sha1` digest of the upload. The `sha1` column of the upload is filled with a `SecureRandom.hex(20)` value which is the same length as `Upload::SHA1_LENGTH`. The `original_sha1` column is filled with the _real_ sha1 digest of the file. 
* Whether an upload `should_be_secure?` is now determined by whether the `access_control_post` is `with_secure_media?` (if there is no access control post then we leave the secure status as is).
* When serializing the upload, we now cook the URL if the upload is secure. This is so it shows up correctly in the composer preview, because we set secure status on upload.

### Viewing Secure Media

* The secure-media-upload URL will take the post that the upload is attached to into account via `Guardian.can_see?` for access permissions
* If there is no `access_control_post` then we just deliver the media. This should be a rare occurrance and shouldn't cause issues as the `access_control_post` is set when `link_post_uploads` is called via `CookedPostProcessor`

### Removed

We no longer do any of these because we do not reuse uploads by sha1 if secure media is enabled.

* We no longer have a way to prevent cross-posting of a secure upload from a private context to a public context.
* We no longer have to set `secure: false` for uploads when uploading for a theme component.
2020-01-16 13:50:27 +10:00
Penar Musaraj 102909edb3 FEATURE: Add support for secure media (#7888)
This PR introduces a new secure media setting. When enabled, it prevent unathorized access to media uploads (files of type image, video and audio). When the `login_required` setting is enabled, then all media uploads will be protected from unauthorized (anonymous) access. When `login_required`is disabled, only media in private messages will be protected from unauthorized access. 

A few notes: 

- the `prevent_anons_from_downloading_files` setting no longer applies to audio and video uploads
- the `secure_media` setting can only be enabled if S3 uploads are already enabled and configured
- upload records have a new column, `secure`, which is a boolean `true/false` of the upload's secure status
- when creating a public post with an upload that has already been uploaded and is marked as secure, the post creator will raise an error
- when enabling or disabling the setting on a site with existing uploads, the rake task `uploads:ensure_correct_acl` should be used to update all uploads' secure status and their ACL on S3
2019-11-18 11:25:42 +10:00
Guo Xiang Tan 7bd93eba3e FIX: Gravatar uploads being dependent on authorized_extensions. 2019-08-01 16:24:09 +08:00
Sam Saffron 7429700389 FIX: ensure we can download maxmind without redis or db config
This also corrects FileHelper.download so it supports "follow_redirect"
correctly (it used to always follow 1 redirect) and adds a `validate_url`
param that will bypass all uri validation if set to false (default is true)
2019-05-28 10:28:57 +10:00
Sam Saffron 30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Sam 4232d32699 PERF: reduce workload when optimizing images
Previously, we would initialize an ImageOptim object each time we resize.

This object init is mega expensive (170ms on a VERY fast machine):

```
[1] pry(main)> Benchmark.measure { FileHelper.image_optim   }
=> #<Benchmark::Tms:0x00007f55440c1de0
 @cstime=0.055742,
 @cutime=0.141031,
 @label="",
 @real=0.17165619300794788,
 @stime=0.0002750000000000252,
 @total=0.19890400000000008,
 @utime=0.0018560000000000798>

```

This happens cause during init it hunts for all the right binaries and sets
up internals.

We now memoize this object to avoid a huge amount of pointless work.
2019-01-09 12:28:18 +11:00
Sam 766e67ce57 FEATURE: introduce lossy color optimization on resized pngs
This feature ensures optimized images run via pngquant, this results extreme amounts of savings for resized images. Effectively the only impact is that the color palette on small resized images is reduced to 256.

To ensure safety we only apply this optimisation to images smaller than 500k.

This commit also makes a bunch of image specs less fragile.
2019-01-02 17:20:02 +11:00
Guo Xiang Tan 6c65718301 Include response body when raising an error in `FileHelper#download`. 2018-09-13 15:43:58 +08:00
Guo Xiang Tan 71caf7521d Drop `tif`, `tiff`, `webp` and `bmp` from supported images.
https://meta.discourse.org/t/cr2-raw-files-are-being-treated-as-tiff-files/96775/3?u=tgxworld
2018-09-12 09:29:54 +08:00
Guo Xiang Tan e1b16e445e Rename `FileHelper.is_image?` -> `FileHelper.is_supported_image?`. 2018-09-12 09:22:28 +08:00
Guo Xiang Tan d59e635a77 Rename `FileHelper.images` to `FileHelper.supported_images`. 2018-09-12 09:22:28 +08:00
Guo Xiang Tan 16c0ebe8a8 Fix the build. 2018-08-17 16:53:07 +08:00
Guo Xiang Tan a26ef7738f FIX: `FileHelper#download` should return nil if max size is exceeded. 2018-08-17 16:19:59 +08:00
Guo Xiang Tan b94633e844 FIX: `FileHelper` should prioritize response content-type.
Request to a URL with `.png` extension may return a jpg
instead causing us to attach the wrong extension to an
upload.
2018-07-30 10:54:36 +08:00
Guo Xiang Tan ad5082d969 Make rubocop happy again. 2018-06-07 13:28:18 +08:00
Régis Hanol 0559a4736a FIX: don't double request when downloading a file 2018-02-24 12:35:57 +01:00
Régis Hanol ca1fd774a1 Revert "WIP"
This reverts commit 2cf5479678.
2018-02-22 18:15:42 +01:00
Régis Hanol 2cf5479678 WIP 2018-02-22 17:56:56 +01:00
Gerhard Schlager d1f257d275 FinalDestination should only log when verbose is enabled 2017-10-31 17:16:59 +01:00
Gerhard Schlager 8c27f28dcb add more logging to FinalDestination 2017-10-31 12:26:35 +01:00
Sam 8ecf313a81 FIX: correctly raise errors when downloads fail
This corrects an issue where we are hitting Gravatar for 404 over and over

Also ensures file download properly reports errors
2017-09-28 16:35:43 +10:00
Sam b80320da5e no verbose logging for failed downloads of gravatars 2017-09-28 11:32:26 +10:00
Régis Hanol 165b388158 fix indentation 2017-09-28 01:07:43 +02:00
Régis Hanol 3a75242c38 add more logs to pull hotlinked images 2017-09-28 01:00:13 +02:00
Régis Hanol c7c93e7159 FEATURE: new 'strip image metadata' site setting 2017-07-25 11:48:39 +02:00
Guo Xiang Tan 02094517e9 Stop rescuing `nil` which is a really bad thing to do. 2017-07-05 10:22:31 +09:00
Régis Hanol ca539854c8 FIX: use 'jpg' instead of 'jpe' extension for 'image/jpeg' content type 2017-06-22 12:53:56 +02:00
Régis Hanol 5d63a7f4a6 FIX: pull hotlinked images even when they have no extension 2017-06-13 13:27:05 +02:00
Robin Ward 0a08c18a14 FIX: Don't rate limit gravatar downloads 2017-05-24 13:54:26 -04:00
Robin Ward cdbe027c1c Refactor `FileHelper` to use keyword arguments. 2017-05-24 13:54:26 -04:00
Robin Ward 3b0cbf7013 FIX: Always allow downloads from CDN 2017-05-23 16:32:54 -04:00
Robin Ward f101e052e6 FIX: Roll back file_helper changes, relative URLs were broken 2017-05-23 15:50:48 -04:00
Robin Ward 36e477750c FIX: Use same code path for downloading images 2017-05-23 14:51:30 -04:00
Robin Ward 31f27006e1 FIX: We can download protocol relative urls 2017-05-15 15:32:55 -04:00
Régis Hanol 3ce3abef8f FIX: add Content-Disposition and Content-Type headers when downloading attachments 2017-02-20 15:59:01 +01:00
Arpit Jalan da2f1fda15 FIX: increase read_timeout when downloading avatar 2016-03-24 17:40:03 +05:30
Régis Hanol bebeeffc30 FIX: ico are images too 2015-12-31 11:40:23 +01:00
Régis Hanol a3e76dc193 FIX: allow HTTP <-> HTTPS redirections when downloading images 2015-08-17 19:21:30 +02:00
Régis Hanol f0b35b2fe2 FIX: file helper to download file using chunks of 512KB 2015-05-19 12:39:46 +02:00
Régis Hanol 961f676b91 FIX: proper hack to support underscores in URLs 2014-11-12 18:15:50 +01:00
Régis Hanol 7244266174 FIX: support for underscores in URLs (when using S3 storage) 2014-11-12 17:08:11 +01:00
Jeff Atwood d926cc13ac FEATURE: add webp and svg to known image formats 2014-10-13 12:43:39 -07:00
Régis Hanol 488d4eebbe SECURITY: do not follow redirect by default when downloading hotlinked images 2014-07-27 17:21:47 +02:00
Sam a864f8aefd BUGFIX: sometimes we have stringio, it has no close 2014-05-27 10:08:03 +10:00
Régis Hanol 51750f7d0e TRIVIAL: slightly better information when logging download error 2014-05-12 16:57:52 +02:00
Régis Hanol 4371374ba6 FEATURE: support for enabling all upload file types
BUGFIX: authorized extensions is now case insensitive
2014-04-29 19:12:35 +02:00
Régis Hanol 76d373161d BUGIFX: generate avatars when using S3 store 2014-04-22 17:11:06 +02:00
Régis Hanol 542d54e6bf BUGFIX: uploads to S3 2014-04-15 13:04:14 +02:00
Régis Hanol 2505d18aa9 FEATURE: support email attachments 2014-04-14 22:55:57 +02:00