1998be3b27
DEV: Raise errors when cleaning the download cache, and fix for macOS ( #8319 )
...
POSIX's `head` specification states: "The application shall ensure that the number option-argument is a positive decimal integer"
Negative values are supported on GNU `head`, so this works in the discourse docker image. However, in some environments (e.g. macOS), the system `head` version fails with a negative `n` parameter.
This commit does two things:
Checks the status at each stage of the pipe, so it cannot fail silently
Flip the `ls` command to list in descending time order, and use `tail -n +501` instead of `head -n -500`.
The visible result is that macOS users no longer see head: illegal line count -- -500 printed throughout the test suite.
2019-11-08 15:34:03 +00:00
55a1394342
DEV: pluck_first
...
Doing .pluck(:column).first is a very common pattern in Discourse and in
most cases, a limit cause isn't being added. Instead of adding a limit
clause to all these callsites, this commit adds two new methods to
ActiveRecord::Relation:
pluck_first, equivalent to limit(1).pluck(*columns).first
and pluck_first! which, like other finder methods, raises an exception
when no record is found
2019-10-21 12:08:20 +01:00
24877a7b8c
FIX: Correctly encode non-ASCII filenames in HTTP header
...
Backport of fix from Rails 6: 890485cfce
2019-08-07 19:10:50 +02:00
606c0ed14d
FIX: S3 uploads were missing a cache-control header ( #7902 )
...
Admins still need to run the rake task to fix the files who where uploaded previously.
2019-08-06 14:55:17 -03:00
f2dc59d61f
FEATURE: Add hidden setting to include S3 uploads in backups
2019-07-09 14:04:16 +02:00
03805e5a76
FIX: Ensure lightbox image download has correct content disposition in S3 ( #7845 )
2019-07-04 11:32:51 -04:00
b7830680b6
DEV: use cdn url to download the external uploads to local.
2019-06-06 19:17:19 +05:30
f00275ded3
FEATURE: Support private attachments when using S3 storage ( #7677 )
...
* Support private uploads in S3
* Use localStore for local avatars
* Add job to update private upload ACL on S3
* Test multisite paths
* update ACL for private uploads in migrate_to_s3 task
2019-06-06 13:27:24 +10:00
a3938f98f8
Revert changes to `FileStore::S3Store#path_for` in f0620e7118.
...
There are some places in the code base that assumes the method should
return nil.
2019-05-29 18:39:07 +08:00
f0620e7118
FEATURE: Support `[description|attachment](upload://<short-sha>)` in MD take 2.
...
Previous attempt was missing `post_uploads` records.
2019-05-29 09:26:32 +08:00
7c9fb95c15
Temporarily revert "FEATURE: Support `[description|attachment](upload://<short-sha>)` in MD. ( #7603 )"
...
This reverts commit b1d3c678ca
2019-05-28 16:37:01 -04:00
b1d3c678ca
FEATURE: Support `[description|attachment](upload://<short-sha>)` in MD. ( #7603 )
2019-05-28 11:18:21 -04:00
ef660d5a3e
FIX: Return consistent character encodings when downloading S3 uploads
...
Net::HTTP always returns ASCII-8BIT encoding. File.read auto-detects the encoding. This leads to an encoding inconsistency between a fresh download, and a cached download. This commit ensures all downloaded files are treated equally, by always returning the cached version from the filesystem, even during initial download.
One symptom of this problem is during theme exports: https://meta.discourse.org/t/116907
Related ruby ticket: https://bugs.ruby-lang.org/issues/2567
2019-05-17 11:27:00 +01:00
30990006a9
DEV: enable frozen string literal on all files
...
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.
Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
5b934cb33d
FIX: Error when trying to move the same file to tombstone.
...
If an optimized image is destroyed when a previous similar optimized
image is already placed in the tombstone, `FileUtils.move` will blow up.
2019-04-24 16:47:36 +08:00
243fb8d9ad
Fix the build.
2019-03-13 17:39:07 +08:00
b0c8fdd7da
FIX: Properly support defaults for upload site settings.
2019-03-13 16:36:57 +08:00
664e90bd17
FIX: ensure local images use local CDN when uploads are stored on S3
...
When the S3 store was enabled, we were only applying the S3 CDN.
So all images stored locally, like the emojis, were never put on the local CDN.
Fixed a bunch of CookedPostProcessor test by adding a call to 'optimize_urls'
in order to get final URLs.
I also removed the unnecessary PrettyText.add_s3_cdn method since this is already
handled in the CookedPostProcessor.
2019-02-20 19:24:38 +01:00
563b953224
DEV: Add 'backfill_etags_' to the method name since it also backfilling the etags
2019-02-19 21:54:35 +05:30
0472bd4adc
FIX: Remove 'backfill_etags' keyword argument from 'uploads:missing' rake task
...
And etags backfilling code is optimized
2019-02-15 00:34:35 +05:30
7b5931013a
Update rake task to backfill etags from s3 inventory
2019-02-14 05:18:06 +05:30
b4f713ca52
FEATURE: Use amazon s3 inventory to manage upload stats ( #6867 )
2019-02-01 10:10:48 +05:30
f94c0283b2
FIX: Use correct version when generating file path for optimized image ( #6871 )
2019-01-11 18:35:38 +05:30
75dbb98cca
FEATURE: Add S3 etag value to uploads table ( #6795 )
2019-01-04 14:16:22 +08:00
5381096bfd
PERF: new 'migrate_to_s3' rake task
2018-12-26 17:34:49 +01:00
cae5ba7356
FIX: Ensure that multisite s3 uploads are tombstoned correctly ( #6769 )
...
* FIX: Ensure that multisite uploads are tombstoned into the correct paths
* Move multisite specs to spec/multisite/s3_store_spec.rb
2018-12-19 13:32:32 +08:00
503ae1829f
FIX: All multisite upload paths should start with /uploads/default/.. ( #6707 )
2018-12-03 12:04:14 +08:00
871d4543cc
FIX: Use File.join for relative_base_url, fix spec
2018-11-29 09:49:56 +05:30
05a4f3fb51
FEATURE: Multisite support for S3 image stores ( #6689 )
...
* FEATURE: Multisite support for S3 image stores
* Use File.join to concatenate all paths & fix linting on multisite/s3_store_spec.rb
2018-11-29 12:11:48 +08:00
bcdf5b2f47
DEV: improve missing uploads query and skip checking file size
2018-11-27 02:21:33 +05:30
4ccf9d28eb
Remove trailing whitespaces
2018-11-27 01:15:29 +05:30
fd272eee44
FEATURE: Make uploads:missing task compatible with s3 uploads
2018-11-27 00:54:51 +05:30
ce6a0a5e9e
FIX: Moving upload to tombstone should update modification time.
...
A upload created a long time ago will be nuked from the tombstone
immediately if it gets deleted.
2018-09-18 10:48:29 +08:00
e1b16e445e
Rename `FileHelper.is_image?` -> `FileHelper.is_supported_image?`.
2018-09-12 09:22:28 +08:00
8496537590
Add `RECOVER_FROM_S3` to `uploads:list_posts_with_broken_images` rake task.
2018-09-10 15:14:30 +08:00
5d96809abd
FIX: improve support for subfolder S3 CDN
2018-08-22 12:31:13 +10:00
f5142861e5
Revert "Revert "FIX: upload URLs from S3 on subfolder installs""
...
This reverts commit 26c96e97e5
2018-08-22 11:31:33 +10:00
26c96e97e5
Revert "FIX: upload URLs from S3 on subfolder installs"
...
This reverts commit 357df2ff4f
2018-08-22 10:51:40 +10:00
357df2ff4f
FIX: upload URLs from S3 on subfolder installs
2018-08-21 14:58:55 -04:00
aafff740d2
Add `FileStore::S3Store#copy_file`.
2018-08-08 11:30:34 +08:00
dba22bbde2
rollback changes
...
This reverts:
* 1baba84c438e "fix s3 subfolders harder"
* ea5e57938edf "fix test for absolute_base_url change"
2018-07-06 17:16:40 -05:00
52e9f49ec1
fix s3 subfolders harder
...
specifically, include the folder in absolute_base_url
2018-07-06 16:28:40 -05:00
448e2fe1a2
FIX: properly delete files in the download cache
2018-07-04 18:18:39 +02:00
4be0e31459
fix s3_cdn_url when the s3 bucket contains a folder
2018-05-23 15:51:02 -05:00
5f4f617689
FIX: cache_file storage cleanup logic was wrong
...
https://meta.discourse.org/t/68296
2018-01-18 17:00:04 +01:00
70bb2aa426
FEATURE: allow specifying s3 config via globals
...
This refactors handling of s3 so it can be specified via GlobalSetting
This means that in a multisite environment you can configure s3 uploads
without actual sites knowing credentials in s3
It is a critical setting for situations where assets are mirrored to s3.
2017-10-06 16:20:01 +11:00
8cc8010564
Maintain backwards compatibility before `Jobs::MigrateUploadExtensions` runs.
2017-08-03 11:56:55 +09:00
83011045c8
fix rubocop offenses
2017-07-31 11:59:16 -04:00
5d528f0d15
Merge pull request #4958 from dmacjam/search_posts_by_filetype
...
FEATURE: Search posts by filetype
2017-07-31 11:55:34 -04:00
5012d46cbd
Add rubocop to our build. ( #5004 )
2017-07-28 10:20:09 +09:00
d8c27e3871
Merge branch 'master' into search_posts_by_filetype
2017-07-25 14:41:20 -04:00
677267ae78
Add onceoff job for uploads migration of column extension. Simplify filetype search and related rspec tests.
2017-07-12 17:19:27 +02:00
8c445e9f17
Fix backend code for searching by a filetype as a combination of uploads and topic links. Add rspec test for extracting file extension in upload.
2017-07-06 19:19:31 +02:00
5d63a7f4a6
FIX: pull hotlinked images even when they have no extension
2017-06-13 13:27:05 +02:00
cdbe027c1c
Refactor `FileHelper` to use keyword arguments.
2017-05-24 13:54:26 -04:00
f534f041a0
FIX: Ensure directory exists.
2017-04-07 15:50:17 +08:00
da7a44064b
Fix purge_tombstone for the brave new world of secure command execution
2017-03-22 10:27:07 +11:00
e7c972ac89
FIX: Don't use backticks that take in inputs.
2017-03-17 15:33:51 +08:00
93dfc87b99
FIX: always set the 'content_type' when storing a file on S3
2016-10-17 19:16:29 +02:00
3141c179f7
REFACTOR: Get bucket name from S3Helper.
2016-08-19 14:08:37 +08:00
7ff1f6cb9d
Allow custom bucket name for `FileStore::S3Store`.
2016-08-16 15:25:42 +08:00
205be0d044
Remove unused require.
2016-08-15 21:58:55 +08:00
0433163866
FEATURE: Support subfolders in `SiteSetting.s3_backup_bucket`.
2016-08-15 16:14:51 +08:00
aa5de3c40a
FEATURE: Support subfolders in S3 bucket name.
...
This commit also fixes a bug where s3 uploads are not
moved to a tombstone folder when removed.
2016-08-15 13:07:41 +08:00
3378ee223f
FIX: Incorrect path being passed to `S3Store#remove_file`.
2016-08-15 11:35:30 +08:00
1779a9634a
FIX: Make sure we raise an error when method is not implemented.
2016-08-12 11:43:57 +08:00
f8a12d4940
Add support for AWS cn ( #4327 )
2016-07-14 16:56:09 +02:00
5169bcdb6e
FIX: httpshttps ultra secure URLs
2016-06-30 16:55:01 +02:00
1caaf5208f
move tombstone under 'uploads/' for easier deployment
2016-05-30 09:46:27 +02:00
26084397c1
FIX: Check if file exists upfront.
2016-05-23 10:25:53 +08:00
c2e0eeb089
Separate relative base_url and upload_path
...
This makes easier to reason about paths
2016-03-10 00:47:18 -03:00
4d01328bc2
FIX: Image Lightbox on Subfolder Install
...
Since LocalSotre.is_relative wasn't subfolder aware, images on subfolder install weren't getting lightboxed.
2016-03-02 18:44:15 -03:00
f155ff8270
FIX: Tests would fail if your test db's optimized image ids were high
2015-10-16 17:08:41 -04:00
acc05dd3a5
Fix LocalStore.remove_file to not raise if source doesn't exist
...
FileUtils.move actually ends up raising an "unknown file type" error if the file doesn't exist instead of Errno::ENOENT.
It's possible to rescue this, but in the end it's easier to just ask move not to throw an error, since we're going to throw it away anyway.
2015-07-12 14:14:12 +01:00
81a699e2b0
better support for mixed content
2015-06-01 17:49:58 +02:00
56f077db69
FIX: optimized images fail if source is remote and S3 is disabled
2015-06-01 11:13:56 +02:00
5a143c0c6e
storage engines refactor
2015-05-29 18:39:47 +02:00
8e7bfd0f29
FIX: automatically growing uploads tree
2015-05-28 01:03:24 +02:00
83d2b59fc3
FIX: s3 endpoint when using 'us-east-1' region
2015-05-27 17:50:49 +02:00
a988cd5abe
FIX: redirect to CDN avatar for s3 avatars
2015-05-27 12:02:57 +10:00
a5d93c6705
FIX: undefined method 'max_file_size_kb'
2015-05-26 16:39:41 +02:00
a797f7c664
FIX: properly handle images when using 's3_cdn_url'
2015-05-26 11:47:33 +02:00
e17f614771
FIX: fallback to local store when uploads are not on S3
2015-05-26 13:08:31 +10:00
bb0c2813ac
FEATURE: generate (avatar) thumbnails in a background task
...
FIX: keep the "uploading..." indicator until the server replies via the MessageBus
FIX: text was disapearing when uploading an avatar
PERF: always use a region for S3 (defaults to 'us-east-1')
FEATURE: ApplyCDN middleware when using S3
FIX: use the same pattern to store files on S3 and locally
PERF: keep a local cache of uploads when generating thumbnails
FEATURE: migrate_to_s3 rake task
2015-05-25 17:59:00 +02:00
f079dd63ed
PERF: remove "fog" dependency
2015-05-25 17:57:06 +10:00
9ded21e4c6
FIX: consistent and future-proof upload storage pattern
2015-05-19 12:31:12 +02:00
31e9cafe0e
FEATURE: use original filename when clicking the download link in the lightbox
2014-10-15 19:20:04 +02:00
652cc3efba
FEATURE: new rake task to clean up uploads & thumbnails
2014-09-29 18:31:53 +02:00
bfdbb70b3b
FIX: automatic backup uploads to S3 when using a region
2014-09-24 22:52:09 +02:00
80468037e9
Don't set content-disposition header for images
...
If this header is set for images, clicking the image link causes them to automatically download the file, rather than viewing it inside the browser. This commit forces this behavior only for uploads that aren't images
2014-09-09 19:43:02 +03:00
e1191a5fcf
Merge pull request #2511 from windhamg/s3_iam_role
...
support for EC2 IAM roles with Amazon S3 file store/backup
2014-08-18 16:14:14 +02:00
7ef61144e7
Avoid using to_s when performing String Interpolation
2014-08-14 23:55:27 +05:30
c7330ed73f
BUGFIX: errors when post-processing 'data images'
2014-07-18 17:54:18 +02:00
3f6bd0606a
support for EC2 IAM roles with Amazon S3 file store/backup
2014-07-14 09:14:16 -07:00
457e32d1b0
Optimistic fix for deploying errors due to missing files. Please review
...
@ZogStriP
2014-05-28 10:35:21 -04:00
542d54e6bf
BUGFIX: uploads to S3
2014-04-15 13:04:14 +02:00
2505d18aa9
FEATURE: support email attachments
2014-04-14 22:55:57 +02:00
1533a1163c
use_ssl is just confusing, it means use_https , fix name of setting
2014-01-09 10:51:38 +11:00
e732aa8a86
BUGFIX: we should not store absolute urls for locally uploaded avatar templates
...
Highly recommended to run: `RAILS_ENV=production bundle exec rake avatars:regenerate` to fix the avatar templates stored in the database.
2014-01-07 17:45:06 +01:00
c6fb60e0a0
FIX: S3 upload when using dots in bucket name
2013-12-16 11:44:59 +01:00
David Taylor
Daniel Waterworth
Gerhard Schlager
Rafael dos Santos Silva
Penar Musaraj
Vinoth Kannan
Guo Xiang Tan
Sam Saffron
Régis Hanol
Vinoth Kannan
Rishabh
Neil Lalonde
Andrew Schleifer
Guo Xiang Tan
Jakub Macina
Robin Ward
Matt Palmer
Hu Ming
Luke GB
Mesut Kayman
Akshay
Gary Windham