2bfc0cf145
FIX: skip old scheme upload URLs while validating s3 uploads remap
2019-05-20 19:13:41 +05:30
41bc90dd3e
DEV: Add progress indicator for post rebake during s3 migration
...
Now that we run sidekiq jobs synchronously, this is important
2019-05-20 14:19:58 +01:00
77a06209cb
DEV: Skip S3 migration if all uploads are already migrated
...
This makes the task resumable in a multisite context
2019-05-20 14:17:37 +01:00
bc0c4b7253
FIX: should not migrate the system uploads to s3
2019-05-20 14:27:34 +05:30
be3fb85a04
DEV: add post migration checks and raise error if failed.
2019-05-20 14:18:28 +05:30
08371db0cc
FIX: ensure we don't queue any jobs during s3 migration
...
Previously we could flood sidekiq with jobs which is not ideal.
This ensures we are 100% done when the job is done.
2019-05-20 16:28:50 +10:00
30990006a9
DEV: enable frozen string literal on all files
...
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.
Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
20daa76829
DEV: Change priority to ultra_low for post-s3-migration rebake
2019-05-10 18:37:45 +08:00
2adbec1b3c
PERF: Speed up migrate_to_s3 rake task by deleting optimized image record.
...
No point moving all optimized image files to tombstone when the store is
changing. Also, `destroy_all` can easily blow memory since we are no
loading in batches.
2019-05-07 16:10:32 +08:00
71e431de9e
DEV: Fix hardcoded value introduced in cc2bac86e9.
2019-03-26 07:45:21 +08:00
cd4fd447ca
Make Rubocop happy
2019-03-25 17:04:45 +01:00
cc2bac86e9
FIX: Dry run broken for `uploads:migrate_to_s3`.
2019-03-25 22:38:24 +08:00
19c3c25db1
FIX: Handle BBCode in `migrate_to_s3` task as well.
2019-03-22 16:47:06 +08:00
2d34be24be
FIX: Rebake lightbox and use short upload urls in `migrate_to_s3`.
2019-03-22 13:09:59 +08:00
4e594f2b2b
FIX: Destroy optimized images in `upload:migrate_to_s3` rake task.
...
`OptimizedImage` are currently not renegerated when the image changes
store.
2019-03-21 16:50:15 +08:00
9dbca41152
FIX: don't check system uploads in migrate_to_s3 rake task
2019-03-18 11:01:18 +01:00
d82876896e
FIX: `uploads:migrate_to_s3` broken for GlobalSetting using file provider.
2019-03-11 14:21:35 +08:00
ad87b0d662
Make "uploads:recover_from_tombstone" call the "uploads:recover" rake task
2019-03-07 14:15:30 +01:00
433f07fcb3
Fix confusing ENV in `migrate_to_s3` rake task.
2019-02-22 14:44:46 +08:00
e69634ec3a
FIX: Use s3_endpoint in migrate_to_s3 when not using S3
...
- overrides :region and uses :endpoint when SiteSetting.s3_endpoint is provided
- Now, we can use the new rake task with DigitalOcean Spaces
- I've tested that it's compatible with/without bucket folder path
- I've tested that it's compatible with S3 and it doesn't break S3 for non-default regions
- follow-up on 97e17fe0
2019-02-21 15:42:48 +05:30
0472bd4adc
FIX: Remove 'backfill_etags' keyword argument from 'uploads:missing' rake task
...
And etags backfilling code is optimized
2019-02-15 00:34:35 +05:30
7b5931013a
Update rake task to backfill etags from s3 inventory
2019-02-14 05:18:06 +05:30
0cf2df3028
Fix remap in `migrate_to_s3` rake task.
...
The current way of doing the remap only allows to run the rake task
once. Running the rake task more than once will end up badly.
2019-01-23 15:50:44 +08:00
07850994d3
Add ENV to skip multisite prefix when migrating to s3.
2019-01-23 15:19:50 +08:00
979d03aa68
Remove s3 bucket check in `migrate_to_s3` task.
...
Bucket creation is expected to be handled by the user. If the bucket
does not exist, the script will fail anyway.
2019-01-23 15:04:51 +08:00
99cd3ff6ee
FIX: `migrate_to_s3` task not setting the right content_disposition.
2019-01-23 15:04:47 +08:00
97e17fe084
FIX: Use ENV values instead of 'S3Helper.s3_options' in migrate_to_s3
...
This commit makes the rake task operational for all regions for s3. If we declare s3_endpoint as https://s3.amazonaws.com while
creating an instance of Aws::S3::Client, head_bucket fails for all s3 regions apart from us-east-1. The commit manually defines all
parameters for Aws::S3::Client apart from s3_endpoint to bypass this problem make this task usable for AWS S3.
Removing s3_endpoint from the payload means that custom endpoints like Minio/DO Spaces for will not work in the meantime and we'll
have to add support for a custom `s3_endpoint` in the future.
This commit follows up on 60790eb0
2019-01-20 20:55:27 +05:30
60790eb006
FIX: Use GlobalSetting values instead of ENV variables in migrate_to_s3
...
TIL how GlobalSetting works in sync with environment variables
Also fixes a small bug where bucket value was being used when it could have been nil
2019-01-16 14:40:38 +05:30
ff8f9dc1c9
FIX: prefix should precede folder path (follow-up on 10fbb07e)
2019-01-15 15:58:19 +05:30
10fbb07e1a
FIX: include folder name in prefix for listing files on S3 (follow-up on 3ec38f5a)
...
Fix the destination url in remap since it's already a part of s3_base_url
2019-01-15 15:23:55 +05:30
3ec38f5a3b
Revert "FIX: migrate_to_s3 rake task with folder path"
...
This reverts commit 97fd12e8af
2019-01-08 19:44:31 +01:00
97fd12e8af
FIX: migrate_to_s3 rake task with folder path
2019-01-08 18:56:18 +01:00
f181e9cc08
FIX: Add compatibility for bucket folder paths in migrate_to_s3 task ( #6855 )
...
* FIX: Add compatibility for bucket folder paths in migrate_to_s3 task
* Refactor bucket_name split logic into S3Helper
2019-01-08 20:04:48 +05:30
efc481d9c0
DEV: Use puts instead of printing newline (follow up on c5b7bda1)
2019-01-05 01:20:00 +05:30
c5b7bda198
DEV: Show migrate_to_s3 output on a new line
2019-01-04 18:09:54 +05:30
9e50813252
FIX: Pass all necessary options in `migrate_to_s3` rake task.
2019-01-02 09:11:23 +08:00
5381096bfd
PERF: new 'migrate_to_s3' rake task
2018-12-26 17:34:49 +01:00
fd272eee44
FEATURE: Make uploads:missing task compatible with s3 uploads
2018-11-27 00:54:51 +05:30
bc41057949
minor copy edit
2018-11-20 12:07:56 +05:30
1a9a2bd5c1
DEV: Report the missing uploads count
2018-11-19 12:06:46 +05:30
4a12cfaecb
Remove trailing whitespace for Rubocop
2018-11-13 17:19:26 +05:30
ea94323766
FIX: 'migrate_from_s3' rake task should respect max sizes ( #6598 )
...
Rather than hardcode a maximum size of 20MB for uploads migrated from S3, the task should use site settings for this value.
2018-11-13 12:27:38 +01:00
14ff47f6f1
Fix typo.
2018-11-08 16:42:12 +08:00
7290145641
PERF: Speed up `migrate_to_s3` rake task.
...
* Prioritizes non-image uploads
* Does one remap per upload instead of 3 remaps previously
* Every 100 uploads migrated, do 2 remaps which fixes broken
URLs
* Exclude email_logs table from remap
2018-11-08 16:39:56 +08:00
0232a3b5e5
PERF: Exclude tables when remapping in `migrate_to_s3` rake task.
2018-11-08 12:37:36 +08:00
918633aa12
FIX: `upload:migrate_to_s3` rake task not remapping properly.
2018-10-10 15:09:21 +08:00
24c55bd613
Add dry run option to `UploadRecovery`.
2018-09-12 21:53:01 +08:00
c053f8ccf6
New rake task `uploads:recover`.
2018-09-12 01:52:30 -07:00
6d01e0aa04
DEV: Print the error class in `uploads:list_posts_with_broken_images`.
2018-09-12 01:06:51 -07:00
94ff428571
Pass the right value to rake task.
2018-09-10 20:07:28 +08:00
Vinoth Kannan
David Taylor
Sam Saffron
Guo Xiang Tan
Gerhard Schlager
Régis Hanol
Rishabh
Rishabh
Brian Helba