Commit Graph

112 Commits

Author SHA1 Message Date
Martin Brennan 154afa60eb
FIX: Skip upload extension validation when changing security (#16498)
When changing upload security using `Upload#update_secure_status`,
we may not have the context of how an upload is being created, because
this code path can be run through scheduled jobs. When calling
update_secure_status, the normal ActiveRecord validations are run,
and ours include validating extensions. In some cases the upload
is created in an automated way, such as user export zips, and the
security is applied later, with the extension prohibited from
use when normally uploading.

This caused the upload to fail validation on `update_secure_status`,
causing the security change to silently fail. This fixes the issue
by skipping the file extension validation when the upload security
is being changed.
2022-04-20 14:11:39 +10:00
Sam cedcdb0057
FEATURE: allow for local theme js assets (#16374)
Due to default CSP web workers instantiated from CDN based assets are still
treated as "same-origin" meaning that we had no way of safely instansiating
a web worker from a theme.

This limits the theme system and adds the arbitrary restriction that WASM
based components can not be safely used.

To resolve this limitation all js assets in about.json are also cached on
local domain.

{
  "name": "Header Icons",
  "assets" : {
    "worker" : "assets/worker.js"
  }
}

This can then be referenced in JS via:

settings.theme_uploads_local.worker

local_js_assets are unconditionally served from the site directly and
bypass the entire CDN, using the pre-existing JavascriptCache

Previous to this change this code was completely dormant on sites which
used s3 based uploads, this reuses the very well tested and cached asset
system on s3 based sites.

Note, when creating local_js_assets it is highly recommended to keep the
assets lean and keep all the heavy working in CDN based assets. For example
wasm files can still live on the CDN but the lean worker that loads it can
live on local.

This change unlocks wasm in theme components, so wasm is now also allowed
in `theme_authorized_extensions`

* more usages of upload.content

* add a specific test for upload.content

* Adjust logic to ensure that after upgrades we still get a cached local js
on save
2022-04-07 07:58:10 +10:00
David Taylor c9dab6fd08
DEV: Automatically require 'rails_helper' in all specs (#16077)
It's very easy to forget to add `require 'rails_helper'` at the top of every core/plugin spec file, and omissions can cause some very confusing/sporadic errors.

By setting this flag in `.rspec`, we can remove the need for `require 'rails_helper'` entirely.
2022-03-01 17:50:50 +00:00
Joffrey JAFFEUX 5eb6e9281a
FIX: manually adds frowning_face_with_open_mouth for apple (#13528) 2021-07-21 23:27:20 +02:00
Martin Brennan 9f275c12ab
FIX: Handle storage providers not implementing ACLs (#13675)
When secure media is enabled or when upload secure status
is updated, we also try and update the upload ACL. However
if the object storage provider does not implement this we
get an Aws::S3::Errors::NotImplemented error. This PR handles
this error so the update_secure_status method does not error
out and still returns whether the secure status changed.
2021-07-09 11:31:44 +10:00
Jarek Radosz 046a875222
DEV: Improve `script/downsize_uploads.rb` (#13508)
* Only shrink images that are used in Posts and no other models
* Don't save the upload if the size is the same
2021-06-24 00:09:40 +02:00
Gerhard Schlager 157f10db4c
FEATURE: Use path from existing URL of uploads and optimized images (#13177)
Discourse shouldn't dynamically calculate the path of uploads and optimized images after a file has been stored on disk or S3. Otherwise it might calculate the wrong path if the SHA1 or extension stored in the database doesn't match the actual file path.
2021-05-27 17:42:25 +02:00
Martin Brennan f49e3e5731
DEV: Add security_last_changed_at and security_last_changed_reason to uploads (#11860)
This PR adds security_last_changed_at and security_last_changed_reason to uploads. This has been done to make it easier to track down why an upload's secure column has changed and when. This necessitated a refactor of the UploadSecurity class to provide reasons why the upload security would have changed.

As well as this, a source is now provided from the location which called for the upload's security status to be updated as they are several (e.g. post creator, topic security updater, rake tasks, manual change).
2021-01-29 09:03:44 +10:00
Bianca Nenciu 25b8ed740b
DEV: Make site setting type uploaded_image_list use upload IDs (#10401)
It used to be a list of concatenated upload URLs which was prone to
break.
2020-10-13 16:17:06 +03:00
Martin Brennan 39b2fb8649
FIX: Invalid URLs could raise exceptions when calling UrlHelper.rails_route_from_url (#10782)
Upload.secure_media_url? raised an exceptions when the URL was invalid,
which was a issue in some situations where secure media URLs must be
removed.

For example, sending digests used PrettyText.strip_secure_media,
which used Upload.secure_media_url? to replace secure media with
placeholders. If the URL was invalid, then an exception would be raised
and left unhandled.

Now instead in UrlHelper.rails_route_from_url we return nil if there is something wrong with the URL.

Co-authored-by: Bianca Nenciu <nenciu.bianca@gmail.com>
2020-09-30 15:20:00 +10:00
Jarek Radosz e00abbe1b7 DEV: Clean up S3 specs, stubs, and helpers
Extracted commonly used spec helpers into spec/support/uploads_helpers.rb, removed unused stubs and let definitions. Makes it easier to write new S3-related specs without copy and pasting setup steps from other specs.
2020-09-28 12:02:25 +01:00
Krzysztof Kotlarek d83e3f9ce8 FIX: improvements after code review 2020-09-15 13:29:35 +08:00
Krzysztof Kotlarek 145814d29c FIX: spec for oversized images security fix
Spec to cover solution presented here - 333ddd4011
2020-09-15 13:29:35 +08:00
Krzysztof Kotlarek 333ddd4011 SECURITY: return error on oversized images 2020-09-14 10:45:11 +10:00
Martin Brennan 2352f4bfc7
DEV: Replace SECURE_MEDIA_ROUTE const with other methods (#10545)
This is so if the route changes this const won't be around to bite us, use the Rails route methods instead.
2020-08-28 11:28:11 +10:00
Sam Saffron 38e7b1a049
FIX: when destroying uploads clear card and profile background
There is an fk to user_profile that can make destroying uploads fail
if they happen to be set as user profile.

This ensures we clear this information when destroying uploads.

There are more relationships, but this makes some more progress.
2020-08-18 10:55:16 +10:00
Gerhard Schlager 957e851ffe Revert "FIX: Regularly reset unknown extension of uploads"
This reverts commit cc7b24b88b as it shouldn't be needed anymore for new uploads.
2020-08-03 13:37:32 +02:00
Martin Brennan cd1c7d7560
FIX: Copying image markdown for secure media loading full image (#9488)
* When copying the markdown for an image between posts, we were not adding the srcset and data-small-image attributes which are done by calling optimize_image! in cooked post processor
* Refactored the code which was confusing in its current state (the consider_for_reuse method was super confusing) and fixed the issue
2020-04-24 10:29:02 +10:00
Martin Brennan 097851c135
FIX: Change secure media to encompass attachments as well (#9271)
If the “secure media” site setting is enabled then ALL files uploaded to Discourse (images, video, audio, pdf, txt, zip etc. etc.) will follow the secure media rules. The “prevent anons from downloading files” setting will no longer have any bearing on upload security. Basically, the feature will more appropriately be called “secure uploads” instead of “secure media”.

This is being done because there are communities out there that would like all attachments and media to be secure based on category rules but still allow anonymous users to download attachments in public places, which is not possible in the current arrangement.
2020-03-26 07:16:02 +10:00
Martin Brennan 04df3bd46d
FIX: Only mark attachments as secure media if SiteSetting.secure_media? (#9009)
* Attachments (non media files) were being marked as secure if just
SiteSetting.prevent_anons_from_downloading_files was enabled. this
was not correct as nothing should be marked as actually "secure" in
the DB without that site setting enabled
* Also add a proper standalone spec file for the upload security class
2020-02-21 09:35:16 +10:00
Martin Brennan 5122826bde Rubocop lint 2020-02-18 15:13:34 +10:00
Martin Brennan 500185dc11 Try fix upload_spec flakys and remove logging from tasks/uploads_spec 2020-02-18 15:08:58 +10:00
Martin Brennan e8efdd60d4
FIX: Tweak upload security emoji check (#8981)
Further on from my earlier PR #8973 also reject upload as secure if its origin URL contains images/emoji. We still check Emoji.all first to try and be canonical.

This may be a little heavy handed (e.g. if an external URL followed this same path it would be a false positive), but there are a lot of emoji aliases where the actual Emoji url is something, but you can have another image that should not be secure that that thing is an alias for. For example slight_smile.png does not show up in Emoji.all BUT slightly_smiling_face does, and it aliases slight_smile e.g. /images/emoji/twitter/slight_smile.png?v=9 and /images/emoji/twitter/slightly_smiling_face.png?v=9 are equivalent.
2020-02-17 15:11:15 +10:00
Martin Brennan dac923379a
FIX: Never mark uploads based on regular emoji secure (#8973)
Sometimes PullHotlinkedImages pulls down a site emoji and creates a new upload record for it. In the cases where these happen the upload is not created via the normal path that custom emoji follows, so we need to check in UploadSecurity whether the origin of the upload is based on a regular site emoji. If it is we never want to mark it as secure (we don't want emoji not accessible from other posts because of secure media).

This only became apparent because the uploads:ensure_correct_acl rake task uses UploadSecurity to check whether an upload should be secure, which would have marked a whole bunch of regular-old-emojis as secure.
2020-02-17 12:30:47 +10:00
Martin Brennan 56b16bc68e
FIX: Never allow custom emoji to be marked secure (#8965)
* Because custom emoji count as post "uploads" we were
marking them as secure when updating the secure status for post uploads.
* We were also giving them an access control post id, which meant
broken image previews from 403 errors in the admin custom emoji list.
* We now check if an upload is used as a custom emoji and do not
assign the access control post + never mark as secure.
2020-02-14 11:17:09 +10:00
Martin Brennan ab3bda6cd0
FIX: Mitigate issue where legacy pre-secure hotlinked media would not be redownloaded (#8802)
Basically, say you had already downloaded a certain image from a certain URL
using pull_hotlinked_images and the onebox. The upload would be stored
by its sha as an upload record. Whenever you linked to the same URL again
in a post (e.g. in our case an og:image on review.discourse) we would
would reuse the original upload record because of the sha1.

However when you turned on secure media this could cause problems as
the first post that uses that upload after secure media is enabled
will set the access control post for the upload to the new post.
Then if the post is deleted every single onebox/link to that same image
URL will fail forever with 403 as the secure-media-uploads URL fails
if the access control post has been deleted.

To fix this when cooking posts and pulling hotlinked images, we only
allow using an original upload by URL if its access control post
matches the current post, and if the original_sha1 is filled in,
meaning it was uploaded AFTER secure media was enabled. otherwise
we just redownload the media again to be safe, as the URL will always
be new then.
2020-01-29 10:11:38 +10:00
Martin Brennan 7c32411881
FEATURE: Secure media allowing duplicated uploads with category-level privacy and post-based access rules (#8664)
### General Changes and Duplication

* We now consider a post `with_secure_media?` if it is in a read-restricted category.
* When uploading we now set an upload's secure status straight away.
* When uploading if `SiteSetting.secure_media` is enabled, we do not check to see if the upload already exists using the `sha1` digest of the upload. The `sha1` column of the upload is filled with a `SecureRandom.hex(20)` value which is the same length as `Upload::SHA1_LENGTH`. The `original_sha1` column is filled with the _real_ sha1 digest of the file. 
* Whether an upload `should_be_secure?` is now determined by whether the `access_control_post` is `with_secure_media?` (if there is no access control post then we leave the secure status as is).
* When serializing the upload, we now cook the URL if the upload is secure. This is so it shows up correctly in the composer preview, because we set secure status on upload.

### Viewing Secure Media

* The secure-media-upload URL will take the post that the upload is attached to into account via `Guardian.can_see?` for access permissions
* If there is no `access_control_post` then we just deliver the media. This should be a rare occurrance and shouldn't cause issues as the `access_control_post` is set when `link_post_uploads` is called via `CookedPostProcessor`

### Removed

We no longer do any of these because we do not reuse uploads by sha1 if secure media is enabled.

* We no longer have a way to prevent cross-posting of a secure upload from a private context to a public context.
* We no longer have to set `secure: false` for uploads when uploading for a theme component.
2020-01-16 13:50:27 +10:00
Mark VanLandingham 09d9baa6d7
FIX: Update S3 stubs for more aws-sdk API changes (#8534) 2019-12-11 11:26:52 -08:00
dependabot-preview[bot] b90a592146 DEV: Bump aws-sdk-sns from 1.13.0 to 1.21.0 (#8490)
Bumps [aws-sdk-sns](https://github.com/aws/aws-sdk-ruby) from 1.13.0 to 1.21.0.
- [Release notes](https://github.com/aws/aws-sdk-ruby/releases)
- [Changelog](https://github.com/aws/aws-sdk-ruby/blob/master/gems/aws-sdk-sns/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-ruby/compare/1.13.0...1.21.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2019-12-11 06:13:17 -08:00
Martin Brennan e7c7a05097
FIX: Mark secure media upload insecure automatically if used for theme component (#8413)
When uploading a file to a theme component, and that file is existing and has already been marked as secure, we now automatically mark the file as secure: false, change the ACL, and log the action as the user (also rebake the posts for the upload)
2019-11-28 07:32:17 +10:00
Penar Musaraj 102909edb3 FEATURE: Add support for secure media (#7888)
This PR introduces a new secure media setting. When enabled, it prevent unathorized access to media uploads (files of type image, video and audio). When the `login_required` setting is enabled, then all media uploads will be protected from unauthorized (anonymous) access. When `login_required`is disabled, only media in private messages will be protected from unauthorized access. 

A few notes: 

- the `prevent_anons_from_downloading_files` setting no longer applies to audio and video uploads
- the `secure_media` setting can only be enabled if S3 uploads are already enabled and configured
- upload records have a new column, `secure`, which is a boolean `true/false` of the upload's secure status
- when creating a public post with an upload that has already been uploaded and is marked as secure, the post creator will raise an error
- when enabling or disabling the setting on a site with existing uploads, the rake task `uploads:ensure_correct_acl` should be used to update all uploads' secure status and their ACL on S3
2019-11-18 11:25:42 +10:00
Gerhard Schlager cc7b24b88b FIX: Regularly reset unknown extension of uploads
Discourse will try to detect the extension the next time the upload is needed. Maybe there was just a transient error the last time.
2019-08-21 10:23:20 +02:00
Guo Xiang Tan 73a45048a0 FIX: `Upload#short_url` generates incorrect URL when extension is `nil`. 2019-06-19 09:10:50 +08:00
Guo Xiang Tan bb98785569 DEV: Fix broken `Upload#base62_sha1`. 2019-06-04 14:10:46 +08:00
Guo Xiang Tan f0620e7118 FEATURE: Support `[description|attachment](upload://<short-sha>)` in MD take 2.
Previous attempt was missing `post_uploads` records.
2019-05-29 09:26:32 +08:00
Sam Saffron 4ea21fa2d0 DEV: use #frozen_string_literal: true on all spec
This change both speeds up specs (less strings to allocate) and helps catch
cases where methods in Discourse are mutating inputs.

Overall we will be migrating everything to use #frozen_string_literal: true
it will take a while, but this is the first and safest move in this direction
2019-04-30 10:27:42 +10:00
Daniel Waterworth ad44243a57 Removed unused let blocks (#7446)
The bodies of these blocks were never evaluated.
2019-04-29 15:08:56 +08:00
Guo Xiang Tan 7bac70676e DEV: Improve `Upload.get_from_url` test case.
Follow up to a8baa6d9e7.
2019-04-24 10:20:42 +08:00
Guo Xiang Tan a8baa6d9e7 FIX: Old `Upload#url` scheme support for `Upload.get_from_url`.
This is for backwards compatibility purposes. Even if `Upload#url` has a
format that we don't recognize, we should still return the upload object
as long as the upload record is present.
2019-04-24 10:15:47 +08:00
Vinoth Kannan f5ac01253a FIX: use new changed method name and remove whitespaces 2019-04-09 02:25:26 +05:30
Guo Xiang Tan 40b03e717b FIX: `Upload#migrate_to_new_scheme` should not migrate system uploads. 2019-03-14 12:45:03 +08:00
Régis Hanol c256121833 FIX: add support for style element in SVGs 2019-02-07 12:09:06 +01:00
Sam Saffron 303a535dba PERF: automatic upload size calculation not persisted
Previously if upload had missing width and height we would calculate
on first use BUT we (me) forgot to save this to the database

This was particularly bad on home page cause category images (when old)
miss dimensions.
2018-12-26 17:19:29 +02:00
Robin Ward 6080e3a2c0 FIX: Broken spec 2018-12-18 14:55:09 -05:00
Penar Musaraj f8e6a37858 FIX: raise exception when getting dimensions of missing image
- follow-up on 0eacd45ab1
2018-12-03 10:19:49 -05:00
Guo Xiang Tan 44391ee8ab
FEATURE: Upload Site Settings. (#6573) 2018-11-14 15:03:02 +08:00
Guo Xiang Tan c3f6b4d966 DEV: Test against real `Upload#url` format. 2018-09-14 13:43:33 +08:00
Guo Xiang Tan 3884e99e88 Add extra protection in `Upload#get_from_url`.
In case the extension goes missing from the URL.
2018-09-12 00:12:14 -07:00
Guo Xiang Tan b3469bea2d FIX: Uploads not being linked correctly to posts.
Regression due to 1f636c445b.
2018-09-11 23:50:23 -07:00
Guo Xiang Tan 1f636c445b PERF: Add fast path to find uploads before resorting to `LIKE` query.
For a normal upload url

Before

```
Warming up --------------------------------------
                       264.000  i/100ms
Calculating -------------------------------------
                          2.754k (± 8.4%) i/s -     13.728k in   5.022066s
```

After

```
Warming up --------------------------------------
                       341.000  i/100ms
Calculating -------------------------------------
                          3.435k (±11.6%) i/s -     17.050k in   5.045676s
```
2018-09-06 14:44:24 +08:00