Commit Graph

104 Commits

Author SHA1 Message Date
Martin Brennan 7c32411881
FEATURE: Secure media allowing duplicated uploads with category-level privacy and post-based access rules (#8664)
### General Changes and Duplication

* We now consider a post `with_secure_media?` if it is in a read-restricted category.
* When uploading we now set an upload's secure status straight away.
* When uploading if `SiteSetting.secure_media` is enabled, we do not check to see if the upload already exists using the `sha1` digest of the upload. The `sha1` column of the upload is filled with a `SecureRandom.hex(20)` value which is the same length as `Upload::SHA1_LENGTH`. The `original_sha1` column is filled with the _real_ sha1 digest of the file. 
* Whether an upload `should_be_secure?` is now determined by whether the `access_control_post` is `with_secure_media?` (if there is no access control post then we leave the secure status as is).
* When serializing the upload, we now cook the URL if the upload is secure. This is so it shows up correctly in the composer preview, because we set secure status on upload.

### Viewing Secure Media

* The secure-media-upload URL will take the post that the upload is attached to into account via `Guardian.can_see?` for access permissions
* If there is no `access_control_post` then we just deliver the media. This should be a rare occurrance and shouldn't cause issues as the `access_control_post` is set when `link_post_uploads` is called via `CookedPostProcessor`

### Removed

We no longer do any of these because we do not reuse uploads by sha1 if secure media is enabled.

* We no longer have a way to prevent cross-posting of a secure upload from a private context to a public context.
* We no longer have to set `secure: false` for uploads when uploading for a theme component.
2020-01-16 13:50:27 +10:00
Martin Brennan 5b75b8c135 Still redirect to signed URL for secure uploads
if SiteSetting.secure_media is disabled we still want to
redirect to the signed url for uploads that are marked as
secure because their ACLs are probably still private
2020-01-07 14:02:17 +10:00
Martin Brennan abca91cc4d
FEATURE: Add rake task to disable secure media (#8669)
* Add a rake task to disable secure media. This sets all uploads to `secure: false`, changes the upload ACL to public, and rebakes all the posts using the uploads to make sure they point to the correct URLs. This is in a transaction for each upload with the upload being updated the last step, so if the task fails it can be resumed.
* Also allow viewing media via the secure url if secure media is disabled, redirecting to the normal CDN url, because otherwise media links will be broken while we go and rebake all the posts + update ACLs
2020-01-07 12:27:24 +10:00
Dan Ungureanu adfa793731
SECURITY: Ensure only image uploads can be inlined
This prevents malicious files (for example special crafted XMLs) to be
used in XSS attacks.
2019-12-11 17:04:49 +02:00
Martin Brennan af0cc89b11 Mark upload show paths as is_asset_path (#8365)
* this is to avoid excessive rate limiting, especially
  for secure media on media-heavy topics
2019-11-18 16:56:20 +11:00
Penar Musaraj 102909edb3 FEATURE: Add support for secure media (#7888)
This PR introduces a new secure media setting. When enabled, it prevent unathorized access to media uploads (files of type image, video and audio). When the `login_required` setting is enabled, then all media uploads will be protected from unauthorized (anonymous) access. When `login_required`is disabled, only media in private messages will be protected from unauthorized access. 

A few notes: 

- the `prevent_anons_from_downloading_files` setting no longer applies to audio and video uploads
- the `secure_media` setting can only be enabled if S3 uploads are already enabled and configured
- upload records have a new column, `secure`, which is a boolean `true/false` of the upload's secure status
- when creating a public post with an upload that has already been uploaded and is marked as secure, the post creator will raise an error
- when enabling or disabling the setting on a site with existing uploads, the rake task `uploads:ensure_correct_acl` should be used to update all uploads' secure status and their ACL on S3
2019-11-18 11:25:42 +10:00
Krzysztof Kotlarek 99086edf85
FIX: Allow themes to upload and serve js files (#8188)
If you set `config.public_file_server.enabled = false` when you try to get uploaded js file you will get an error:
`Security warning: an embedded <script> tag on another site requested protected JavaScript. If you know what you're doing, go ahead and disable forgery protection on this action to permit cross-origin JavaScript embedding.`

The reason is that content type is `application/javascript` and in Rails 5 guard looked like that:
https://github.com/rails/rails/blob/5-2-stable/actionpack/lib/action_controller/metal/request_forgery_protection.rb#L278-L280
However, in Rails 6 `application` was added to regex:
https://github.com/rails/rails/blob/master/actionpack/lib/action_controller/metal/request_forgery_protection.rb#L282-L284

This pull request is related to https://meta.discourse.org/t/uploaded-js-file-for-theme-causes-a-rejection/129753/8
2019-10-14 15:40:33 +11:00
Krzysztof Kotlarek 427d54b2b0 DEV: Upgrading Discourse to Zeitwerk (#8098)
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains. 

We no longer need to use Rails "require_dependency" anywhere and instead can just use standard 
Ruby patterns to require files.

This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
Penar Musaraj 03805e5a76
FIX: Ensure lightbox image download has correct content disposition in S3 (#7845) 2019-07-04 11:32:51 -04:00
Gerhard Schlager fe870a1e54 DEV: Respond with error 400 to uploads requested via XHR
follow-up to 13f38055
2019-06-27 11:15:35 +02:00
Gerhard Schlager 13f38055ac SECURITY: XSS in routes
Co-authored-by: Guo Xiang Tan <tgx_world@hotmail.com>
Co-authored-by: David Taylor <david@taylorhq.com>
2019-06-26 16:34:15 +02:00
Penar Musaraj f00275ded3 FEATURE: Support private attachments when using S3 storage (#7677)
* Support private uploads in S3
* Use localStore for local avatars
* Add job to update private upload ACL on S3
* Test multisite paths
* update ACL for private uploads in migrate_to_s3 task
2019-06-06 13:27:24 +10:00
Guo Xiang Tan a3938f98f8 Revert changes to `FileStore::S3Store#path_for` in f0620e7118.
There are some places in the code base that assumes the method should
return nil.
2019-05-29 18:39:07 +08:00
Guo Xiang Tan f0620e7118 FEATURE: Support `[description|attachment](upload://<short-sha>)` in MD take 2.
Previous attempt was missing `post_uploads` records.
2019-05-29 09:26:32 +08:00
Penar Musaraj 7c9fb95c15 Temporarily revert "FEATURE: Support `[description|attachment](upload://<short-sha>)` in MD. (#7603)"
This reverts commit b1d3c678ca.

We need to make sure post_upload records are correctly stored.
2019-05-28 16:37:01 -04:00
Guo Xiang Tan b1d3c678ca FEATURE: Support `[description|attachment](upload://<short-sha>)` in MD. (#7603) 2019-05-28 11:18:21 -04:00
Sam Saffron 7429700389 FIX: ensure we can download maxmind without redis or db config
This also corrects FileHelper.download so it supports "follow_redirect"
correctly (it used to always follow 1 redirect) and adds a `validate_url`
param that will bypass all uri validation if set to false (default is true)
2019-05-28 10:28:57 +10:00
Vinoth Kannan 42b10a646d FIX: return 404 only if upload url also not internal. 2019-05-15 02:06:54 +05:30
Sam Saffron 30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Sam Saffron 1be01f8dd4 DEV: Add support for Rails 6
Minor fixes to add Rails 6 support to Discourse, we now will boot
with RAILS_MASTER=1, all specs pass

Only one tiny deprecation left

Largest change was the way ActiveModel:Errors changed interface a
bit but there is a simple backwards compat way of working it
2019-05-02 16:23:25 +10:00
Guo Xiang Tan 58b0e945bd
UX: Lightbox support for image uploader. (#7034) 2019-02-21 10:13:37 +08:00
Robin Ward 6f656f6e7d FIX: Better error handling if a file cannot be sent
If for some reason `Discourse.store.path_for` returns `nil`, the
forum would throw an error rather than returning 404.

Why would it be `nil`? One cause could be changing the type of
file store and having the `url` field no longer be relative.
2019-01-29 16:47:25 -05:00
Bianca Nenciu 1a4f592749 FIX: Always allow admins upload selectable avatars. 2018-12-05 21:55:23 +01:00
Guo Xiang Tan 44391ee8ab
FEATURE: Upload Site Settings. (#6573) 2018-11-14 15:03:02 +08:00
Guo Xiang Tan e1b16e445e Rename `FileHelper.is_image?` -> `FileHelper.is_supported_image?`. 2018-09-12 09:22:28 +08:00
Guo Xiang Tan 2b57239389 FIX: Upload's content is the only source of truth for the file type. 2018-08-07 13:15:00 +08:00
Guo Xiang Tan 142571bba0 Remove use of `rescue nil`.
* `rescue nil` is a really bad pattern to use in our code base.
  We should rescue errors that we expect the code to throw and
  not rescue everything because we're unsure of what errors the
  code would throw. This would reduce the amount of pain we face
  when debugging why something isn't working as expexted. I've
  been bitten countless of times by errors being swallowed as a
  result during debugging sessions.
2018-04-02 13:52:51 +08:00
Sam 41986cdb2f Refactor requires login logic, reduce duplicate code
This also corrects the positioning in the chain of the check
and removes misuse of prepend_before_action
2018-02-01 15:17:59 +11:00
Sam f2e7b74d88 FIX: don't return 200s when login is required to paths
When running `ensure_login_required` it should always happen prior to
`check_xhr` cause check xhr will trigger a 200 response
2018-02-01 12:26:45 +11:00
Régis Hanol f5e170c6b5 FIX: catch all server-side error when uploading a file
UX: always show a message to the user whenever an error happens on the server when uploading a file
2017-12-27 16:33:25 +01:00
Sam eb428ef54d FEATURE: uploads are processed a faster
Also cleans up API to always return 422 on upload error. (previously returned 200)

Uploads are processed using new hijack pattern
2017-11-27 12:43:35 +11:00
Sam 49f97d75b7 FIX: make uploads safe for block that can run later 2017-11-23 17:28:18 +11:00
Sam a92f61e926 FIX: allow login required sites access to attachements 2017-11-22 10:50:55 +11:00
Guo Xiang Tan 77d4c4d8dc Fix all the errors to get our tests green on Rails 5.1. 2017-09-25 13:48:58 +08:00
Sam d7a2584c6e FEATURE: image uploads now have short urls
Shorten all image uploads to use short urls, this is the client
side implementation.
2017-08-22 16:40:08 -04:00
Régis Hanol a9c0df0b58 FIX: always try to convert PNG to JPG when pasting an image 2017-06-23 12:13:48 +02:00
Régis Hanol 5d63a7f4a6 FIX: pull hotlinked images even when they have no extension 2017-06-13 13:27:05 +02:00
Régis Hanol 54e8fb0d89 FEATURE: new 'allow_staff_to_upload_any_file_in_pm' site setting 2017-06-12 22:41:29 +02:00
Robin Ward cdbe027c1c Refactor `FileHelper` to use keyword arguments. 2017-05-24 13:54:26 -04:00
Arpit Jalan 1fd8e426f2 FIX: better uploads error page 2017-05-18 23:29:37 +05:30
Régis Hanol 13e489b4ca replace the upload type whitelist with a sanitizer 2017-05-18 12:13:13 +02:00
Régis Hanol 9641d2413d REFACTOR: upload workflow creation into UploadCreator
- Automatically convert large-ish PNG/BMP to JPEG
- Updated fast_image to latest version
2017-05-11 00:16:57 +02:00
Guo Xiang Tan e7c972ac89 FIX: Don't use backticks that take in inputs. 2017-03-17 15:33:51 +08:00
Régis Hanol 3ce3abef8f FIX: add Content-Disposition and Content-Type headers when downloading attachments 2017-02-20 15:59:01 +01:00
Guo Xiang Tan 18007ed34b FIX: Can't use an internal name here if `SiteSetting.convert_pasted_images_to_hq_jpg` is `false`. 2017-02-01 14:51:56 +08:00
Guo Xiang Tan f6d9745c5f Bye bye byebug. 2017-02-01 14:50:14 +08:00
Guo Xiang Tan 6c8c91dca4 UX: Change default filename for images that have been pasted. 2017-02-01 14:44:41 +08:00
Sam 15b5fddd49 SECURITY: protect upload params, only allow very strict filenames 2016-12-19 10:16:18 +11:00
Robin Ward af83c8dc14 Upload Logos Step 2016-09-22 09:52:19 -04:00
Régis Hanol 6dac9075dc new 'convert_pasted_images_quality' site setting 2016-07-27 19:59:44 +02:00