Commit Graph

42 Commits

Author SHA1 Message Date
David Taylor 13e39d8b9f
PERF: Improve cook_url performance for topic thumbnails (#11609)
- Only initialize the S3Helper when needed
- Skip initializing the S3Helper for S3Store#cdn_url
- Allow cook_url to be passed a `local` hint to skip unnecessary checks
2020-12-30 18:13:13 +00:00
Jarek Radosz e00abbe1b7 DEV: Clean up S3 specs, stubs, and helpers
Extracted commonly used spec helpers into spec/support/uploads_helpers.rb, removed unused stubs and let definitions. Makes it easier to write new S3-related specs without copy and pasting setup steps from other specs.
2020-09-28 12:02:25 +01:00
Sam Saffron 689568c216
FIX: invalid urls should not break store.has_been_uploaded?
Breaking this method has wide ramification including breaking
search indexing.
2020-06-25 15:00:15 +10:00
Martin Brennan e92909aa77
FIX: Use ActionDispatch::Http::ContentDisposition for uploads content-disposition (#10108)
See https://meta.discourse.org/t/broken-pipe-error-when-uploading-to-a-s3-clone-a-pdf-with-a-name-containing-e-i-etc/155414

When setting content-disposition for attachment, use the ContentDisposition class to format it. This handles filenames with weird characters and localization (accented characters) correctly.
2020-06-23 17:10:56 +10:00
Martin Brennan e5da2d24e5
FIX: Add attachment content-disposition for all non-image files (#10058)
This will make it so the original filename is used when downloading all non-image files, bringing S3Store into line with the to_s3 migration and local storage. Video and audio files will still stream correctly in HTML players as well.

See https://meta.discourse.org/t/cannot-download-non-image-media-files-original-filenames-lost-when-uploaded-to-s3/152797 for a lot of extra context.
2020-06-17 11:16:37 +10:00
Roman Rizzi b61a291cf3
FIX: returns false if the upload url is an invalid mailto link (#9877) 2020-05-26 10:32:48 -03:00
Michael Brown d9a02d1336
Revert "Revert "Merge branch 'master' of https://github.com/discourse/discourse""
This reverts commit 20780a1eee.

* SECURITY: re-adds accidentally reverted commit:
  03d26cd6: ensure embed_url contains valid http(s) uri
* when the merge commit e62a85cf was reverted, git chose the 2660c2e2 parent to land on
  instead of the 03d26cd6 parent (which contains security fixes)
2020-05-23 00:56:13 -04:00
Jeff Atwood 20780a1eee Revert "Merge branch 'master' of https://github.com/discourse/discourse"
This reverts commit e62a85cf6f, reversing
changes made to 2660c2e21d.
2020-05-22 20:25:56 -07:00
Martin Brennan 72f139191e
FIX: S3 store has_been_uploaded? was not taking into account s3 bucket path (#9810)
In some cases, between Discourse forums the hostname of a URL could match if they are hosting S3 files on the same bucket but the S3 bucket path might not. So e.g. https://testbucket.somesite.com/testpath/some/file/url.png vs https://testbucket.somesite.com/prodpath/some/file/url.png. So has_been_uploaded? was returning true for the second URL, even though it may have been uploaded on a different Discourse forum.

This is a very rare case but must be accounted for, because this impacts UrlHelper.is_local which mistakenly thinks the file has already been downloaded and thus allows the URL to be cooked, where we want to return the full URL to be downloaded using PullHotlinkedImages.
2020-05-20 10:40:38 +10:00
Martin Brennan 097851c135
FIX: Change secure media to encompass attachments as well (#9271)
If the “secure media” site setting is enabled then ALL files uploaded to Discourse (images, video, audio, pdf, txt, zip etc. etc.) will follow the secure media rules. The “prevent anons from downloading files” setting will no longer have any bearing on upload security. Basically, the feature will more appropriately be called “secure uploads” instead of “secure media”.

This is being done because there are communities out there that would like all attachments and media to be secure based on category rules but still allow anonymous users to download attachments in public places, which is not possible in the current arrangement.
2020-03-26 07:16:02 +10:00
Vinoth Kannan 3b7f5db5ba
FIX: parallel spec system needs a dedicated upload folder for each worker. (#8547) 2019-12-18 11:21:57 +05:30
Penar Musaraj 102909edb3 FEATURE: Add support for secure media (#7888)
This PR introduces a new secure media setting. When enabled, it prevent unathorized access to media uploads (files of type image, video and audio). When the `login_required` setting is enabled, then all media uploads will be protected from unauthorized (anonymous) access. When `login_required`is disabled, only media in private messages will be protected from unauthorized access. 

A few notes: 

- the `prevent_anons_from_downloading_files` setting no longer applies to audio and video uploads
- the `secure_media` setting can only be enabled if S3 uploads are already enabled and configured
- upload records have a new column, `secure`, which is a boolean `true/false` of the upload's secure status
- when creating a public post with an upload that has already been uploaded and is marked as secure, the post creator will raise an error
- when enabling or disabling the setting on a site with existing uploads, the rake task `uploads:ensure_correct_acl` should be used to update all uploads' secure status and their ACL on S3
2019-11-18 11:25:42 +10:00
Krzysztof Kotlarek 427d54b2b0 DEV: Upgrading Discourse to Zeitwerk (#8098)
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains. 

We no longer need to use Rails "require_dependency" anywhere and instead can just use standard 
Ruby patterns to require files.

This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
Vinoth Kannan b1ca64487a FIX: multisite upload urls must have either db name or the word 'short-url'. 2019-06-25 01:19:58 +05:30
Penar Musaraj f00275ded3 FEATURE: Support private attachments when using S3 storage (#7677)
* Support private uploads in S3
* Use localStore for local avatars
* Add job to update private upload ACL on S3
* Test multisite paths
* update ACL for private uploads in migrate_to_s3 task
2019-06-06 13:27:24 +10:00
Gerhard Schlager b788948985 FEATURE: English locale with international date formats
Makes en_US the new default locale
2019-05-20 13:47:20 +02:00
Sam Saffron 4ea21fa2d0 DEV: use #frozen_string_literal: true on all spec
This change both speeds up specs (less strings to allocate) and helps catch
cases where methods in Discourse are mutating inputs.

Overall we will be migrating everything to use #frozen_string_literal: true
it will take a while, but this is the first and safest move in this direction
2019-04-30 10:27:42 +10:00
Sam Saffron 45285f1477 DEV: remove update_attributes which is deprecated in Rails 6
See: https://github.com/rails/rails/pull/31998

update_attributes is a relic of the past, it should no longer be used.
2019-04-29 17:32:25 +10:00
Daniel Waterworth ad44243a57 Removed unused let blocks (#7446)
The bodies of these blocks were never evaluated.
2019-04-29 15:08:56 +08:00
Guo Xiang Tan bf21ebaecc DEV: Allow custom value when pausing sidekiq to aid in debugging.
Sometimes, it is useful to know what caused Sidekiq to be paused.
2019-02-19 10:55:53 +08:00
Sam 74d2d4f658 FEATURE: add APIS for unpausing all sites
This adjusts 53d592ad by @tgxworld

- Adds Sidekiq.upause_all! to unpause all sites
- Adds Sidekiq.paused_dbs to list dbs that are currently paused
- Handles some edge cases where unpause thread could extend expiry on
sites that were unpaused from a different process
- Ensures tests always terminates background thread used for pause
keepalive
2019-02-14 13:34:20 +11:00
Guo Xiang Tan 53d592ad3b FIX: Add multisite support to Sidekiq::Pausable. (#6960)
Having a global Sidekiq pause switch is problematic because a site in
the cluster can pause Sidekiq for the entire cluster.
2019-02-14 12:22:40 +11:00
Robin Ward 0f73026c21 FIX: Heisentest
These tests were failing for the same reason as:
bee68bba2e

Fix was the same.
2019-01-25 15:25:48 -05:00
Gerhard Schlager 0947fa2bad Fix specs
Follow-up to 7e9da812ea
2019-01-24 22:54:03 +01:00
Robin Ward 9ba8bfb1aa FIX: Multisite DB was leaving old data in test mode
This commit introduces a new helper to enable transactional fixtures
when testing multisite. This would show up as tests that passed the
first time then failed the second time due to stale data being leftover.
2019-01-09 15:20:37 -05:00
Vinoth Kannan 75dbb98cca FEATURE: Add S3 etag value to uploads table (#6795) 2019-01-04 14:16:22 +08:00
Rishabh cae5ba7356 FIX: Ensure that multisite s3 uploads are tombstoned correctly (#6769)
* FIX: Ensure that multisite uploads are tombstoned into the correct paths

* Move multisite specs to spec/multisite/s3_store_spec.rb
2018-12-19 13:32:32 +08:00
Rishabh 503ae1829f FIX: All multisite upload paths should start with /uploads/default/.. (#6707) 2018-12-03 12:04:14 +08:00
Rishabh 05a4f3fb51 FEATURE: Multisite support for S3 image stores (#6689)
* FEATURE: Multisite support for S3 image stores

* Use File.join to concatenate all paths & fix linting on multisite/s3_store_spec.rb
2018-11-29 12:11:48 +08:00
Guo Xiang Tan 85620abb71 DEV: Clear connections after multisite specs. 2018-09-11 10:15:06 +08:00
Guo Xiang Tan 9c7e029d01 DEV: Attempt to stablize multisite tests. 2018-08-30 17:31:17 +08:00
Sam f331d2603d DEV: improve design of site setting default provider
This refactors it so "Defaults provider" is only responsible for "defaults"

Locale handling and management of locale settings is moved back into
SiteSettingExtension

This eliminates complex state management using DistributedCache and makes
it way easier to test SiteSettingExtension
2018-06-07 14:33:41 +10:00
Guo Xiang Tan 54dc191a91 Update `rails_multisite` to 2.0.1. 2018-01-19 10:19:16 +08:00
Guo Xiang Tan 57d9830bd2 FIX: DistributedCache without namespace mode wasn't working. 2017-10-20 22:32:41 +08:00
Guo Xiang Tan 9dcb11f553 Fix the build. 2017-10-11 17:45:19 +08:00
Guo Xiang Tan 09721090a3 FIX: Ensure that we revert back to default connection after running jobs. 2017-10-11 17:17:03 +08:00
Robin Ward 00b190af75 Revert "A safe way to create class variables in a multisite environment."
The approach taken by this interface was flawed. We need a better
solution.
2017-09-29 11:06:12 -04:00
Robin Ward 4f0fee1ce7 FIX: Test failures 2017-09-27 17:02:36 -04:00
Robin Ward 3e13becf33 A safe way to create class variables in a multisite environment.
This should allow plugins to set class variables that will not
stomp on other plugins.
2017-09-27 13:00:47 -04:00
Guo Xiang Tan 77d4c4d8dc Fix all the errors to get our tests green on Rails 5.1. 2017-09-25 13:48:58 +08:00
Sam 04e3526096 Only remove default locale which is the only column we touch
see: https://meta.discourse.org/t/failing-tests-in-discourse-dev-due-to-multisite/68720/5
2017-08-29 14:02:24 -04:00
Guo Xiang Tan 2c39743d5d Introduce multisite tests for better coverage. 2017-08-08 12:58:22 +09:00