Sam Saffron
a5ce9cb470
FEATURE: fix_relative_upload_links now multisite safety
...
This also finds `<img src="/uploads/xyz` HTML images in raw and corrects
them. Also handles some cross multisite recovery and provides better output
2019-05-23 15:09:16 +10:00
Sam Saffron
e8799f0ba4
DEV: improve uploads:recover job so it stores a map of old to new sha
...
Previous attempt created broken images
2019-05-22 15:51:09 +10:00
Sam Saffron
ebcb571de7
FIX: allow upload recovery to recover uploads with sha mismatch
...
Filename on disk may mismatch sha of file in some old 1X setups. This will
attempt to recover file even if sha1 mismatches. We had an old bug that
caused this.
This also adds `uploads:fix_relative_upload_links` which attempts to replace
urls of the format `/upload/default/...` with `upload://`
2019-05-22 15:24:36 +10:00
Sam Saffron
f772ecc597
DEV: Correct missing output detailing failure
...
uploads:s3_migration_status was failing but not returning proper output
2019-05-22 12:58:54 +10:00
Guo Xiang Tan
5429c9b5e9
DEV: Fix incorrectly hardcoded value in rake task.
2019-05-22 09:01:25 +08:00
Sam Saffron
a2428bd862
FEATURE: warn about sidekiq overload prior to migrating
...
Also makes pre-flight check ensure there is no giant backlog of posts that
need to be cooked
2019-05-22 10:04:33 +10:00
Sam Saffron
5fdc7b7ca2
Correct 59012fc0
...
Logic was flipped here by mistake, oops
2019-05-22 09:48:03 +10:00
Sam Saffron
73f178a634
FEATURE: posts:rebake_uncooked_posts to look at mismatching baked_version
...
also amends flagging onebox updates to set baked_version to nil
2019-05-22 09:43:31 +10:00
Sam Saffron
4f296608da
FEATURE: add uploads:s3_migration_status for looking at current status
...
Also a few minor cleanups and better progress reporting
2019-05-22 09:00:32 +10:00
Sam Saffron
59012fc0f7
PERF: mark posts for rebake vs forcing a rebake inline when migrating to s3
...
Rebaking posts can be expensive instead of blocking here simply mark posts
for rebake.
We can then work through them faster in other jobs, plus this should not
hold of a datacenter migration.
2019-05-22 08:39:25 +10:00
Vinoth Kannan
7b82850f66
FIX: migrate_to_s3 task should remap attachment links too.
2019-05-21 21:58:11 +05:30
Sam Saffron
8360415453
FEATURE: big warning for uploads missing which can be very very slow on S3
2019-05-21 16:11:56 +10:00
Sam Saffron
cb86d8279a
FEATURE: add toggle for uploads:missing so it can skip external
...
Validating s3 uploads in uploads:missing can be very expensive, allow to
bypass.
2019-05-21 16:11:56 +10:00
Sam Saffron
f4d4f7871e
FEATURE: make posts:missing_uploads multisite friendly
...
Previously this rake job would only run on a single site which is a bit
misleading
This also adds `VERBOSE=1 rake posts:missing_uploads` that will provide a
full report of missing uploads
2019-05-21 12:45:51 +10:00
David Taylor
0fbff66d97
DEV: Correct rake task syntax error
2019-05-20 17:43:30 +01:00
David Taylor
31902159af
DEV: Allow skipping failed migrations when running S3 migration
...
Use the SKIP_FAILED environment variable to skip failed sites. Use with caution - make sure you go back and re-run the failed migrations afterwards.
2019-05-20 17:25:56 +01:00
David Taylor
a15cca9a0f
DEV: Improve error message for posts:missing_uploads during S3 migration
2019-05-20 16:09:22 +01:00
Vinoth Kannan
2bfc0cf145
FIX: skip old scheme upload URLs while validating s3 uploads remap
2019-05-20 19:13:41 +05:30
David Taylor
41bc90dd3e
DEV: Add progress indicator for post rebake during s3 migration
...
Now that we run sidekiq jobs synchronously, this is important
2019-05-20 14:19:58 +01:00
David Taylor
77a06209cb
DEV: Skip S3 migration if all uploads are already migrated
...
This makes the task resumable in a multisite context
2019-05-20 14:17:37 +01:00
Vinoth Kannan
bc0c4b7253
FIX: should not migrate the system uploads to s3
2019-05-20 14:27:34 +05:30
Vinoth Kannan
be3fb85a04
DEV: add post migration checks and raise error if failed.
2019-05-20 14:18:28 +05:30
Sam Saffron
08371db0cc
FIX: ensure we don't queue any jobs during s3 migration
...
Previously we could flood sidekiq with jobs which is not ideal.
This ensures we are 100% done when the job is done.
2019-05-20 16:28:50 +10:00
Sam Saffron
30990006a9
DEV: enable frozen string literal on all files
...
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.
Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
David Taylor
20daa76829
DEV: Change priority to ultra_low for post-s3-migration rebake
2019-05-10 18:37:45 +08:00
Guo Xiang Tan
2adbec1b3c
PERF: Speed up migrate_to_s3 rake task by deleting optimized image record.
...
No point moving all optimized image files to tombstone when the store is
changing. Also, `destroy_all` can easily blow memory since we are no
loading in batches.
2019-05-07 16:10:32 +08:00
Guo Xiang Tan
71e431de9e
DEV: Fix hardcoded value introduced in cc2bac86e9
.
2019-03-26 07:45:21 +08:00
Gerhard Schlager
cd4fd447ca
Make Rubocop happy
2019-03-25 17:04:45 +01:00
Guo Xiang Tan
cc2bac86e9
FIX: Dry run broken for `uploads:migrate_to_s3`.
2019-03-25 22:38:24 +08:00
Guo Xiang Tan
19c3c25db1
FIX: Handle BBCode in `migrate_to_s3` task as well.
2019-03-22 16:47:06 +08:00
Guo Xiang Tan
2d34be24be
FIX: Rebake lightbox and use short upload urls in `migrate_to_s3`.
2019-03-22 13:09:59 +08:00
Guo Xiang Tan
4e594f2b2b
FIX: Destroy optimized images in `upload:migrate_to_s3` rake task.
...
`OptimizedImage` are currently not renegerated when the image changes
store.
2019-03-21 16:50:15 +08:00
Régis Hanol
9dbca41152
FIX: don't check system uploads in migrate_to_s3 rake task
2019-03-18 11:01:18 +01:00
Guo Xiang Tan
d82876896e
FIX: `uploads:migrate_to_s3` broken for GlobalSetting using file provider.
2019-03-11 14:21:35 +08:00
Régis Hanol
ad87b0d662
Make "uploads:recover_from_tombstone" call the "uploads:recover" rake task
2019-03-07 14:15:30 +01:00
Guo Xiang Tan
433f07fcb3
Fix confusing ENV in `migrate_to_s3` rake task.
2019-02-22 14:44:46 +08:00
Rishabh
e69634ec3a
FIX: Use s3_endpoint in migrate_to_s3 when not using S3
...
- overrides :region and uses :endpoint when SiteSetting.s3_endpoint is provided
- Now, we can use the new rake task with DigitalOcean Spaces
- I've tested that it's compatible with/without bucket folder path
- I've tested that it's compatible with S3 and it doesn't break S3 for non-default regions
- follow-up on 97e17fe0
2019-02-21 15:42:48 +05:30
Vinoth Kannan
0472bd4adc
FIX: Remove 'backfill_etags' keyword argument from 'uploads:missing' rake task
...
And etags backfilling code is optimized
2019-02-15 00:34:35 +05:30
Vinoth Kannan
7b5931013a
Update rake task to backfill etags from s3 inventory
2019-02-14 05:18:06 +05:30
Guo Xiang Tan
0cf2df3028
Fix remap in `migrate_to_s3` rake task.
...
The current way of doing the remap only allows to run the rake task
once. Running the rake task more than once will end up badly.
2019-01-23 15:50:44 +08:00
Guo Xiang Tan
07850994d3
Add ENV to skip multisite prefix when migrating to s3.
2019-01-23 15:19:50 +08:00
Guo Xiang Tan
979d03aa68
Remove s3 bucket check in `migrate_to_s3` task.
...
Bucket creation is expected to be handled by the user. If the bucket
does not exist, the script will fail anyway.
2019-01-23 15:04:51 +08:00
Guo Xiang Tan
99cd3ff6ee
FIX: `migrate_to_s3` task not setting the right content_disposition.
2019-01-23 15:04:47 +08:00
Rishabh
97e17fe084
FIX: Use ENV values instead of 'S3Helper.s3_options' in migrate_to_s3
...
This commit makes the rake task operational for all regions for s3. If we declare s3_endpoint as https://s3.amazonaws.com while
creating an instance of Aws::S3::Client, head_bucket fails for all s3 regions apart from us-east-1. The commit manually defines all
parameters for Aws::S3::Client apart from s3_endpoint to bypass this problem make this task usable for AWS S3.
Removing s3_endpoint from the payload means that custom endpoints like Minio/DO Spaces for will not work in the meantime and we'll
have to add support for a custom `s3_endpoint` in the future.
This commit follows up on 60790eb0
.
2019-01-20 20:55:27 +05:30
Rishabh
60790eb006
FIX: Use GlobalSetting values instead of ENV variables in migrate_to_s3
...
TIL how GlobalSetting works in sync with environment variables
Also fixes a small bug where bucket value was being used when it could have been nil
2019-01-16 14:40:38 +05:30
Rishabh
ff8f9dc1c9
FIX: prefix should precede folder path (follow-up on 10fbb07e
)
2019-01-15 15:58:19 +05:30
Rishabh
10fbb07e1a
FIX: include folder name in prefix for listing files on S3 (follow-up on 3ec38f5a
)
...
Fix the destination url in remap since it's already a part of s3_base_url
2019-01-15 15:23:55 +05:30
Régis Hanol
3ec38f5a3b
Revert "FIX: migrate_to_s3 rake task with folder path"
...
This reverts commit 97fd12e8af
.
2019-01-08 19:44:31 +01:00
Régis Hanol
97fd12e8af
FIX: migrate_to_s3 rake task with folder path
2019-01-08 18:56:18 +01:00
Rishabh
f181e9cc08
FIX: Add compatibility for bucket folder paths in migrate_to_s3 task ( #6855 )
...
* FIX: Add compatibility for bucket folder paths in migrate_to_s3 task
* Refactor bucket_name split logic into S3Helper
2019-01-08 20:04:48 +05:30
Rishabh
efc481d9c0
DEV: Use puts instead of printing newline (follow up on c5b7bda1
)
2019-01-05 01:20:00 +05:30
Rishabh
c5b7bda198
DEV: Show migrate_to_s3 output on a new line
2019-01-04 18:09:54 +05:30
Guo Xiang Tan
9e50813252
FIX: Pass all necessary options in `migrate_to_s3` rake task.
2019-01-02 09:11:23 +08:00
Régis Hanol
5381096bfd
PERF: new 'migrate_to_s3' rake task
2018-12-26 17:34:49 +01:00
Vinoth Kannan
fd272eee44
FEATURE: Make uploads:missing task compatible with s3 uploads
2018-11-27 00:54:51 +05:30
Vinoth Kannan
bc41057949
minor copy edit
2018-11-20 12:07:56 +05:30
Vinoth Kannan
1a9a2bd5c1
DEV: Report the missing uploads count
2018-11-19 12:06:46 +05:30
Rishabh
4a12cfaecb
Remove trailing whitespace for Rubocop
2018-11-13 17:19:26 +05:30
Brian Helba
ea94323766
FIX: 'migrate_from_s3' rake task should respect max sizes ( #6598 )
...
Rather than hardcode a maximum size of 20MB for uploads migrated from S3, the task should use site settings for this value.
2018-11-13 12:27:38 +01:00
Guo Xiang Tan
14ff47f6f1
Fix typo.
2018-11-08 16:42:12 +08:00
Guo Xiang Tan
7290145641
PERF: Speed up `migrate_to_s3` rake task.
...
* Prioritizes non-image uploads
* Does one remap per upload instead of 3 remaps previously
* Every 100 uploads migrated, do 2 remaps which fixes broken
URLs
* Exclude email_logs table from remap
2018-11-08 16:39:56 +08:00
Guo Xiang Tan
0232a3b5e5
PERF: Exclude tables when remapping in `migrate_to_s3` rake task.
2018-11-08 12:37:36 +08:00
Guo Xiang Tan
918633aa12
FIX: `upload:migrate_to_s3` rake task not remapping properly.
2018-10-10 15:09:21 +08:00
Guo Xiang Tan
24c55bd613
Add dry run option to `UploadRecovery`.
2018-09-12 21:53:01 +08:00
Guo Xiang Tan
c053f8ccf6
New rake task `uploads:recover`.
2018-09-12 01:52:30 -07:00
Guo Xiang Tan
6d01e0aa04
DEV: Print the error class in `uploads:list_posts_with_broken_images`.
2018-09-12 01:06:51 -07:00
Guo Xiang Tan
94ff428571
Pass the right value to rake task.
2018-09-10 20:07:28 +08:00
Guo Xiang Tan
4a966c639d
DEV: Update `uploads:list_posts_with_broken_images` to recover from tombstone.
2018-09-10 17:01:41 +08:00
Guo Xiang Tan
68572b8afc
Print error messages on why upload fails to save.
2018-09-10 16:02:13 +08:00
Guo Xiang Tan
0aca80e92a
Fixes to `uploads:list_posts_with_broken_images`.
2018-09-10 15:16:29 +08:00
Guo Xiang Tan
8496537590
Add `RECOVER_FROM_S3` to `uploads:list_posts_with_broken_images` rake task.
2018-09-10 15:14:30 +08:00
Guo Xiang Tan
72834f19ff
DEV: Add rake tasks to list posts with broken images.
2018-09-05 16:54:15 +08:00
Guo Xiang Tan
33f4aa2835
Fix `migrate_to_s3` rake task, `File#close!` is not a valid method.
2018-08-16 13:26:58 +08:00
Sam
1fc2597626
better error handling for upload extension fixer
2018-08-10 11:28:22 +10:00
Guo Xiang Tan
0879610ffd
Add missing require in `uploads:fix_incorrect_extensions`.
2018-08-08 15:39:37 +08:00
Guo Xiang Tan
17047806b9
Add a rake task to fix uploads with wrong extension.
2018-08-08 13:15:17 +08:00
Régis Hanol
14bbd5d167
FIX: use the right URL when downloading the file from S3
2018-08-02 10:04:41 +02:00
Régis Hanol
0f1137dafa
FIX: 'migrate_from_s3' rake task wasn't handling short urls
2018-08-01 22:58:46 +02:00
Sam
5f64fd0a21
DEV: remove exec_sql and replace with mini_sql
...
Introduce new patterns for direct sql that are safe and fast.
MiniSql is not prone to memory bloat that can happen with direct PG usage.
It also has an extremely fast materializer and very a convenient API
- DB.exec(sql, *params) => runs sql returns row count
- DB.query(sql, *params) => runs sql returns usable objects (not a hash)
- DB.query_hash(sql, *params) => runs sql returns an array of hashes
- DB.query_single(sql, *params) => runs sql and returns a flat one dimensional array
- DB.build(sql) => returns a sql builder
See more at: https://github.com/discourse/mini_sql
2018-06-19 16:13:36 +10:00
Régis Hanol
9e7d5a3cdf
FIX: 'uploads:recover_from_tombstone' rake task wasn't restoring attachments
2018-03-28 15:51:47 +02:00
Guo Xiang Tan
5782a23a69
Allow `uploads:missing` to skip optimized image check.
2018-03-12 20:08:53 +08:00
Régis Hanol
cbf4204ecc
FIX: uploads:migrate_from_s3 wasn't destroying records before creating new ones
2018-01-24 13:22:07 +01:00
Régis Hanol
28d432263e
rework the uploads:migrate_from_s3 rake task
2018-01-23 18:48:06 +01:00
Sam
83a346cd68
FIX: migrate_from_s3 rake task not working
2018-01-22 11:42:09 +11:00
Régis Hanol
b6c9d7a9d2
move requires to top of file
2018-01-19 17:51:42 +01:00
Régis Hanol
2d49250107
FIX: support recovering images using 'upload://' scheme
2018-01-19 15:40:17 +01:00
Sam
70bb2aa426
FEATURE: allow specifying s3 config via globals
...
This refactors handling of s3 so it can be specified via GlobalSetting
This means that in a multisite environment you can configure s3 uploads
without actual sites knowing credentials in s3
It is a critical setting for situations where assets are mirrored to s3.
2017-10-06 16:20:01 +11:00
Guo Xiang Tan
5012d46cbd
Add rubocop to our build. ( #5004 )
2017-07-28 10:20:09 +09:00
Régis Hanol
b7e6620109
FIX: 'uploads:migrate_from_s3' rake task
2017-06-23 10:15:11 +02:00
Régis Hanol
94cee37edd
FIX: uploads:recover_from_tombstone rake task
2017-06-22 17:00:05 +02:00
Robin Ward
cdbe027c1c
Refactor `FileHelper` to use keyword arguments.
2017-05-24 13:54:26 -04:00
Régis Hanol
9641d2413d
REFACTOR: upload workflow creation into UploadCreator
...
- Automatically convert large-ish PNG/BMP to JPEG
- Updated fast_image to latest version
2017-05-11 00:16:57 +02:00
Guo Xiang Tan
b5d6d3c326
FIX: Make recover from tombstone script work for multisite.
2016-09-22 12:28:36 +08:00
Guo Xiang Tan
333d6f9f10
More improvements to restore from tombstone script.
2016-09-22 10:39:39 +08:00
Guo Xiang Tan
1d4451db5d
Improve restore from tombstone script by not having to loop through all posts.
2016-09-21 18:01:59 +08:00
Guo Xiang Tan
b62b6ba12c
Improve restore from tombstone rake task.
2016-09-21 17:47:27 +08:00
Guo Xiang Tan
3af5b19002
Add rake task to restore missing uploads from tombstone.
2016-09-21 16:56:32 +08:00
Guo Xiang Tan
e4b75f604c
FIX: Make clean up upload script a safer task to run.
2016-09-05 10:06:02 +08:00
Guo Xiang Tan
df8d24734a
Regenerate missing optimized images for all dbs by default.
2016-09-02 13:06:31 +08:00
Guo Xiang Tan
e4883725e6
Script to analyze uploads on a site.
2016-09-02 12:07:06 +08:00