7b5931013a
Update rake task to backfill etags from s3 inventory
2019-02-14 05:18:06 +05:30
0cf2df3028
Fix remap in `migrate_to_s3` rake task.
...
The current way of doing the remap only allows to run the rake task
once. Running the rake task more than once will end up badly.
2019-01-23 15:50:44 +08:00
07850994d3
Add ENV to skip multisite prefix when migrating to s3.
2019-01-23 15:19:50 +08:00
979d03aa68
Remove s3 bucket check in `migrate_to_s3` task.
...
Bucket creation is expected to be handled by the user. If the bucket
does not exist, the script will fail anyway.
2019-01-23 15:04:51 +08:00
99cd3ff6ee
FIX: `migrate_to_s3` task not setting the right content_disposition.
2019-01-23 15:04:47 +08:00
97e17fe084
FIX: Use ENV values instead of 'S3Helper.s3_options' in migrate_to_s3
...
This commit makes the rake task operational for all regions for s3. If we declare s3_endpoint as https://s3.amazonaws.com while
creating an instance of Aws::S3::Client, head_bucket fails for all s3 regions apart from us-east-1. The commit manually defines all
parameters for Aws::S3::Client apart from s3_endpoint to bypass this problem make this task usable for AWS S3.
Removing s3_endpoint from the payload means that custom endpoints like Minio/DO Spaces for will not work in the meantime and we'll
have to add support for a custom `s3_endpoint` in the future.
This commit follows up on 60790eb0
2019-01-20 20:55:27 +05:30
60790eb006
FIX: Use GlobalSetting values instead of ENV variables in migrate_to_s3
...
TIL how GlobalSetting works in sync with environment variables
Also fixes a small bug where bucket value was being used when it could have been nil
2019-01-16 14:40:38 +05:30
ff8f9dc1c9
FIX: prefix should precede folder path (follow-up on 10fbb07e)
2019-01-15 15:58:19 +05:30
10fbb07e1a
FIX: include folder name in prefix for listing files on S3 (follow-up on 3ec38f5a)
...
Fix the destination url in remap since it's already a part of s3_base_url
2019-01-15 15:23:55 +05:30
3ec38f5a3b
Revert "FIX: migrate_to_s3 rake task with folder path"
...
This reverts commit 97fd12e8af
2019-01-08 19:44:31 +01:00
97fd12e8af
FIX: migrate_to_s3 rake task with folder path
2019-01-08 18:56:18 +01:00
f181e9cc08
FIX: Add compatibility for bucket folder paths in migrate_to_s3 task ( #6855 )
...
* FIX: Add compatibility for bucket folder paths in migrate_to_s3 task
* Refactor bucket_name split logic into S3Helper
2019-01-08 20:04:48 +05:30
efc481d9c0
DEV: Use puts instead of printing newline (follow up on c5b7bda1)
2019-01-05 01:20:00 +05:30
c5b7bda198
DEV: Show migrate_to_s3 output on a new line
2019-01-04 18:09:54 +05:30
9e50813252
FIX: Pass all necessary options in `migrate_to_s3` rake task.
2019-01-02 09:11:23 +08:00
5381096bfd
PERF: new 'migrate_to_s3' rake task
2018-12-26 17:34:49 +01:00
fd272eee44
FEATURE: Make uploads:missing task compatible with s3 uploads
2018-11-27 00:54:51 +05:30
bc41057949
minor copy edit
2018-11-20 12:07:56 +05:30
1a9a2bd5c1
DEV: Report the missing uploads count
2018-11-19 12:06:46 +05:30
4a12cfaecb
Remove trailing whitespace for Rubocop
2018-11-13 17:19:26 +05:30
ea94323766
FIX: 'migrate_from_s3' rake task should respect max sizes ( #6598 )
...
Rather than hardcode a maximum size of 20MB for uploads migrated from S3, the task should use site settings for this value.
2018-11-13 12:27:38 +01:00
14ff47f6f1
Fix typo.
2018-11-08 16:42:12 +08:00
7290145641
PERF: Speed up `migrate_to_s3` rake task.
...
* Prioritizes non-image uploads
* Does one remap per upload instead of 3 remaps previously
* Every 100 uploads migrated, do 2 remaps which fixes broken
URLs
* Exclude email_logs table from remap
2018-11-08 16:39:56 +08:00
0232a3b5e5
PERF: Exclude tables when remapping in `migrate_to_s3` rake task.
2018-11-08 12:37:36 +08:00
918633aa12
FIX: `upload:migrate_to_s3` rake task not remapping properly.
2018-10-10 15:09:21 +08:00
24c55bd613
Add dry run option to `UploadRecovery`.
2018-09-12 21:53:01 +08:00
c053f8ccf6
New rake task `uploads:recover`.
2018-09-12 01:52:30 -07:00
6d01e0aa04
DEV: Print the error class in `uploads:list_posts_with_broken_images`.
2018-09-12 01:06:51 -07:00
94ff428571
Pass the right value to rake task.
2018-09-10 20:07:28 +08:00
4a966c639d
DEV: Update `uploads:list_posts_with_broken_images` to recover from tombstone.
2018-09-10 17:01:41 +08:00
68572b8afc
Print error messages on why upload fails to save.
2018-09-10 16:02:13 +08:00
0aca80e92a
Fixes to `uploads:list_posts_with_broken_images`.
2018-09-10 15:16:29 +08:00
8496537590
Add `RECOVER_FROM_S3` to `uploads:list_posts_with_broken_images` rake task.
2018-09-10 15:14:30 +08:00
72834f19ff
DEV: Add rake tasks to list posts with broken images.
2018-09-05 16:54:15 +08:00
33f4aa2835
Fix `migrate_to_s3` rake task, `File#close!` is not a valid method.
2018-08-16 13:26:58 +08:00
1fc2597626
better error handling for upload extension fixer
2018-08-10 11:28:22 +10:00
0879610ffd
Add missing require in `uploads:fix_incorrect_extensions`.
2018-08-08 15:39:37 +08:00
17047806b9
Add a rake task to fix uploads with wrong extension.
2018-08-08 13:15:17 +08:00
14bbd5d167
FIX: use the right URL when downloading the file from S3
2018-08-02 10:04:41 +02:00
0f1137dafa
FIX: 'migrate_from_s3' rake task wasn't handling short urls
2018-08-01 22:58:46 +02:00
5f64fd0a21
DEV: remove exec_sql and replace with mini_sql
...
Introduce new patterns for direct sql that are safe and fast.
MiniSql is not prone to memory bloat that can happen with direct PG usage.
It also has an extremely fast materializer and very a convenient API
- DB.exec(sql, *params) => runs sql returns row count
- DB.query(sql, *params) => runs sql returns usable objects (not a hash)
- DB.query_hash(sql, *params) => runs sql returns an array of hashes
- DB.query_single(sql, *params) => runs sql and returns a flat one dimensional array
- DB.build(sql) => returns a sql builder
See more at: https://github.com/discourse/mini_sql
2018-06-19 16:13:36 +10:00
9e7d5a3cdf
FIX: 'uploads:recover_from_tombstone' rake task wasn't restoring attachments
2018-03-28 15:51:47 +02:00
5782a23a69
Allow `uploads:missing` to skip optimized image check.
2018-03-12 20:08:53 +08:00
cbf4204ecc
FIX: uploads:migrate_from_s3 wasn't destroying records before creating new ones
2018-01-24 13:22:07 +01:00
28d432263e
rework the uploads:migrate_from_s3 rake task
2018-01-23 18:48:06 +01:00
83a346cd68
FIX: migrate_from_s3 rake task not working
2018-01-22 11:42:09 +11:00
b6c9d7a9d2
move requires to top of file
2018-01-19 17:51:42 +01:00
2d49250107
FIX: support recovering images using 'upload://' scheme
2018-01-19 15:40:17 +01:00
70bb2aa426
FEATURE: allow specifying s3 config via globals
...
This refactors handling of s3 so it can be specified via GlobalSetting
This means that in a multisite environment you can configure s3 uploads
without actual sites knowing credentials in s3
It is a critical setting for situations where assets are mirrored to s3.
2017-10-06 16:20:01 +11:00
5012d46cbd
Add rubocop to our build. ( #5004 )
2017-07-28 10:20:09 +09:00
b7e6620109
FIX: 'uploads:migrate_from_s3' rake task
2017-06-23 10:15:11 +02:00
94cee37edd
FIX: uploads:recover_from_tombstone rake task
2017-06-22 17:00:05 +02:00
cdbe027c1c
Refactor `FileHelper` to use keyword arguments.
2017-05-24 13:54:26 -04:00
9641d2413d
REFACTOR: upload workflow creation into UploadCreator
...
- Automatically convert large-ish PNG/BMP to JPEG
- Updated fast_image to latest version
2017-05-11 00:16:57 +02:00
b5d6d3c326
FIX: Make recover from tombstone script work for multisite.
2016-09-22 12:28:36 +08:00
333d6f9f10
More improvements to restore from tombstone script.
2016-09-22 10:39:39 +08:00
1d4451db5d
Improve restore from tombstone script by not having to loop through all posts.
2016-09-21 18:01:59 +08:00
b62b6ba12c
Improve restore from tombstone rake task.
2016-09-21 17:47:27 +08:00
3af5b19002
Add rake task to restore missing uploads from tombstone.
2016-09-21 16:56:32 +08:00
e4b75f604c
FIX: Make clean up upload script a safer task to run.
2016-09-05 10:06:02 +08:00
df8d24734a
Regenerate missing optimized images for all dbs by default.
2016-09-02 13:06:31 +08:00
e4883725e6
Script to analyze uploads on a site.
2016-09-02 12:07:06 +08:00
82324db71a
Allow `uploads:missing` Rake task to be run for a single site.
2016-09-02 10:22:39 +08:00
2349ab5a51
`puts` not `putc`.
2016-08-29 10:30:10 +08:00
e57ed5223a
Fix rake task.
2016-08-26 15:46:42 +08:00
2448db4716
Prioritize user avatars when regenerating missing optimized images.
2016-08-25 18:29:52 +08:00
f19232646f
Rescue all errors when backfilling shas for uploads.
2016-08-23 15:05:37 +08:00
574b8fdf41
Allow backfilling of shas to complete.
2016-08-23 10:34:32 +08:00
5222022842
Use another rake task to regenerate optimized images for multisite.
2016-08-19 15:52:09 +08:00
9971e62254
FIX: migrate_from_s3 rake task
2016-07-25 12:12:10 +02:00
557e59c28e
check the uploads doesn't already exists before copying it
2016-04-13 16:33:00 +02:00
8e4113a468
add multisite support for the 'uploads:gather' rake task
2016-04-12 16:00:25 +02:00
71fe66c8c2
'uploads:gather' should only work on local uploads
2016-04-11 21:17:33 +02:00
90dfc96206
new 'uploads:gather' rake task
2016-04-11 20:42:40 +02:00
ca8c5f7382
migrate_to_s3: check for missing local files
...
If an image is already on S3, but not available locally (eg. if they're already migrated), `path` is nil. This crashes the task, because File.exists?(nil) is not valid.
2015-11-16 11:39:40 +01:00
189cb3ff12
FEATURE: move migrate_to_new_scheme into a background job
...
- new hidden site setting 'migrate_to_new_scheme' (defaults to false)
- new rake tasks to toggle migration to new scheme
- FIX: migrate_to_new_scheme also works with CDN
- PERF: improve perf of the DbHelper.remap method
- REFACTOR: UrlHelper is now a class
2015-06-12 12:07:57 +02:00
c1cf602de2
FIX: uploads:backfill_shas rake task
2015-06-10 17:19:58 +02:00
6000a37ff1
new 'uploads:migrate_to_new_pattern' task
2015-06-05 18:37:34 +02:00
a5d93c6705
FIX: undefined method 'max_file_size_kb'
2015-05-26 16:39:41 +02:00
bb0c2813ac
FEATURE: generate (avatar) thumbnails in a background task
...
FIX: keep the "uploading..." indicator until the server replies via the MessageBus
FIX: text was disapearing when uploading an avatar
PERF: always use a region for S3 (defaults to 'us-east-1')
FEATURE: ApplyCDN middleware when using S3
FIX: use the same pattern to store files on S3 and locally
PERF: keep a local cache of uploads when generating thumbnails
FEATURE: migrate_to_s3 rake task
2015-05-25 17:59:00 +02:00
5e527178b9
FEATURE: new 'uploads:migrate_to_new_pattern' rake task
2015-05-19 12:31:51 +02:00
54b63bfdbf
remove ALL useless 'order by'
2015-05-12 09:37:48 +02:00
232b5387d0
remove useless 'order by'
2015-05-12 09:36:46 +02:00
b489384b59
FIX: uploads rake tasks to support subfolders
2015-05-12 09:28:43 +02:00
95a57ce3e4
properly fix missing uploads by redownloading them
2015-05-11 19:07:39 +02:00
48f4589c6f
try to fix missing downloads by re-downloading them
2015-05-11 17:03:48 +02:00
6d83526bf1
add support for multisite to the 'uploads:regenerate_missing_optimized' rake task
2015-05-11 16:19:16 +02:00
27a9b5ee13
only deal with local uploads
2015-05-11 15:41:52 +02:00
ca89d16946
FEATURE: new 'uploads:regenerate_missing_optimized' images rake task
2015-05-11 12:59:50 +02:00
996ab1ae17
FEATURE: task to find all missing local images
2015-05-11 10:30:39 +10:00
6b85d5582c
FIX: 'uploads:migrate_from_s3' rake task
2015-03-18 18:23:55 +01:00
652cc3efba
FEATURE: new rake task to clean up uploads & thumbnails
2014-09-29 18:31:53 +02:00
87c0c5d6f9
TRIVIAL: we might run the task after disabling s3 storage
2014-06-25 23:16:04 +02:00
bdb6032c11
FEATURE: rake task to migrate uploads from S3 to local storage
2014-06-24 15:35:15 +02:00
f94e4ffdcb
added 'uploads:backfill_shas' rake task
2013-06-17 02:49:33 +02:00
Vinoth Kannan
Guo Xiang Tan
Rishabh
Régis Hanol
Rishabh
Brian Helba
Sam
Guo Xiang Tan
Robin Ward
uppfinnarn