Discource-C
/
discourse
mirror of https://github.com/discourse/discourse.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
31,542 Commits 181 Branches 498 Tags 917 MiB
Commit Graph

95 Commits

Author SHA1 Message Date
Vinoth Kannan 7b5931013a Update rake task to backfill etags from s3 inventory 2019-02-14 05:18:06 +05:30
Guo Xiang Tan 0cf2df3028 Fix remap in `migrate_to_s3` rake task. 
The current way of doing the remap only allows to run the rake task
once. Running the rake task more than once will end up badly.
 2019-01-23 15:50:44 +08:00
Guo Xiang Tan 07850994d3 Add ENV to skip multisite prefix when migrating to s3. 2019-01-23 15:19:50 +08:00
Guo Xiang Tan 979d03aa68 Remove s3 bucket check in `migrate_to_s3` task. 
Bucket creation is expected to be handled by the user. If the bucket
does not exist, the script will fail anyway.
 2019-01-23 15:04:51 +08:00
Guo Xiang Tan 99cd3ff6ee FIX: `migrate_to_s3` task not setting the right content_disposition. 2019-01-23 15:04:47 +08:00
Rishabh 97e17fe084 FIX: Use ENV values instead of 'S3Helper.s3_options' in migrate_to_s3 
This commit makes the rake task operational for all regions for s3. If we declare s3_endpoint as https://s3.amazonaws.com while
creating an instance of Aws::S3::Client, head_bucket fails for all s3 regions apart from us-east-1. The commit manually defines all
parameters for Aws::S3::Client apart from s3_endpoint to bypass this problem make this task usable for AWS S3.

Removing s3_endpoint from the payload means that custom endpoints like Minio/DO Spaces for will not work in the meantime and we'll
have to add support for a custom `s3_endpoint` in the future.

This commit follows up on 60790eb0.
 2019-01-20 20:55:27 +05:30
Rishabh 60790eb006 FIX: Use GlobalSetting values instead of ENV variables in migrate_to_s3 
TIL how GlobalSetting works in sync with environment variables
Also fixes a small bug where bucket value was being used when it could have been nil
 2019-01-16 14:40:38 +05:30
Rishabh ff8f9dc1c9 FIX: prefix should precede folder path (follow-up on 10fbb07e) 2019-01-15 15:58:19 +05:30
Rishabh 10fbb07e1a FIX: include folder name in prefix for listing files on S3 (follow-up on 3ec38f5a) 
Fix the destination url in remap since it's already a part of s3_base_url
 2019-01-15 15:23:55 +05:30
Régis Hanol 3ec38f5a3b Revert "FIX: migrate_to_s3 rake task with folder path" 
This reverts commit 97fd12e8af.
 2019-01-08 19:44:31 +01:00
Régis Hanol 97fd12e8af
FIX: migrate_to_s3 rake task with folder path 2019-01-08 18:56:18 +01:00
Rishabh f181e9cc08
FIX: Add compatibility for bucket folder paths in migrate_to_s3 task (#6855) 
* FIX: Add compatibility for bucket folder paths in migrate_to_s3 task
* Refactor bucket_name split logic into S3Helper
 2019-01-08 20:04:48 +05:30
Rishabh efc481d9c0 DEV: Use puts instead of printing newline (follow up on c5b7bda1) 2019-01-05 01:20:00 +05:30
Rishabh c5b7bda198 DEV: Show migrate_to_s3 output on a new line 2019-01-04 18:09:54 +05:30
Guo Xiang Tan 9e50813252 FIX: Pass all necessary options in `migrate_to_s3` rake task. 2019-01-02 09:11:23 +08:00
Régis Hanol 5381096bfd PERF: new 'migrate_to_s3' rake task 2018-12-26 17:34:49 +01:00
Vinoth Kannan fd272eee44 FEATURE: Make uploads:missing task compatible with s3 uploads 2018-11-27 00:54:51 +05:30
Vinoth Kannan bc41057949 minor copy edit 2018-11-20 12:07:56 +05:30
Vinoth Kannan 1a9a2bd5c1 DEV: Report the missing uploads count 2018-11-19 12:06:46 +05:30
Rishabh 4a12cfaecb Remove trailing whitespace for Rubocop 2018-11-13 17:19:26 +05:30
Brian Helba ea94323766 FIX: 'migrate_from_s3' rake task should respect max sizes (#6598) 
Rather than hardcode a maximum size of 20MB for uploads migrated from S3, the task should use site settings for this value.
 2018-11-13 12:27:38 +01:00
Guo Xiang Tan 14ff47f6f1 Fix typo. 2018-11-08 16:42:12 +08:00
Guo Xiang Tan 7290145641 PERF: Speed up `migrate_to_s3` rake task. 
* Prioritizes non-image uploads
* Does one remap per upload instead of 3 remaps previously
* Every 100 uploads migrated, do 2 remaps which fixes broken
  URLs
* Exclude email_logs table from remap
 2018-11-08 16:39:56 +08:00
Guo Xiang Tan 0232a3b5e5 PERF: Exclude tables when remapping in `migrate_to_s3` rake task. 2018-11-08 12:37:36 +08:00
Guo Xiang Tan 918633aa12 FIX: `upload:migrate_to_s3` rake task not remapping properly. 2018-10-10 15:09:21 +08:00
Guo Xiang Tan 24c55bd613 Add dry run option to `UploadRecovery`. 2018-09-12 21:53:01 +08:00
Guo Xiang Tan c053f8ccf6 New rake task `uploads:recover`. 2018-09-12 01:52:30 -07:00
Guo Xiang Tan 6d01e0aa04 DEV: Print the error class in `uploads:list_posts_with_broken_images`. 2018-09-12 01:06:51 -07:00
Guo Xiang Tan 94ff428571 Pass the right value to rake task. 2018-09-10 20:07:28 +08:00
Guo Xiang Tan 4a966c639d DEV: Update `uploads:list_posts_with_broken_images` to recover from tombstone. 2018-09-10 17:01:41 +08:00
Guo Xiang Tan 68572b8afc Print error messages on why upload fails to save. 2018-09-10 16:02:13 +08:00
Guo Xiang Tan 0aca80e92a Fixes to `uploads:list_posts_with_broken_images`. 2018-09-10 15:16:29 +08:00
Guo Xiang Tan 8496537590 Add `RECOVER_FROM_S3` to `uploads:list_posts_with_broken_images` rake task. 2018-09-10 15:14:30 +08:00
Guo Xiang Tan 72834f19ff DEV: Add rake tasks to list posts with broken images. 2018-09-05 16:54:15 +08:00
Guo Xiang Tan 33f4aa2835 Fix `migrate_to_s3` rake task, `File#close!` is not a valid method. 2018-08-16 13:26:58 +08:00
Sam 1fc2597626 better error handling for upload extension fixer 2018-08-10 11:28:22 +10:00
Guo Xiang Tan 0879610ffd Add missing require in `uploads:fix_incorrect_extensions`. 2018-08-08 15:39:37 +08:00
Guo Xiang Tan 17047806b9 Add a rake task to fix uploads with wrong extension. 2018-08-08 13:15:17 +08:00
Régis Hanol 14bbd5d167 FIX: use the right URL when downloading the file from S3 2018-08-02 10:04:41 +02:00
Régis Hanol 0f1137dafa FIX: 'migrate_from_s3' rake task wasn't handling short urls 2018-08-01 22:58:46 +02:00
Sam 5f64fd0a21 DEV: remove exec_sql and replace with mini_sql 
Introduce new patterns for direct sql that are safe and fast.

MiniSql is not prone to memory bloat that can happen with direct PG usage.
It also has an extremely fast materializer and very a convenient API

- DB.exec(sql, *params) => runs sql returns row count
- DB.query(sql, *params) => runs sql returns usable objects (not a hash)
- DB.query_hash(sql, *params) => runs sql returns an array of hashes
- DB.query_single(sql, *params) => runs sql and returns a flat one dimensional array
- DB.build(sql) => returns a sql builder

See more at: https://github.com/discourse/mini_sql
 2018-06-19 16:13:36 +10:00
Régis Hanol 9e7d5a3cdf FIX: 'uploads:recover_from_tombstone' rake task wasn't restoring attachments 2018-03-28 15:51:47 +02:00
Guo Xiang Tan 5782a23a69 Allow `uploads:missing` to skip optimized image check. 2018-03-12 20:08:53 +08:00
Régis Hanol cbf4204ecc FIX: uploads:migrate_from_s3 wasn't destroying records before creating new ones 2018-01-24 13:22:07 +01:00
Régis Hanol 28d432263e rework the uploads:migrate_from_s3 rake task 2018-01-23 18:48:06 +01:00
Sam 83a346cd68 FIX: migrate_from_s3 rake task not working 2018-01-22 11:42:09 +11:00
Régis Hanol b6c9d7a9d2 move requires to top of file 2018-01-19 17:51:42 +01:00
Régis Hanol 2d49250107 FIX: support recovering images using 'upload://' scheme 2018-01-19 15:40:17 +01:00
Sam 70bb2aa426 FEATURE: allow specifying s3 config via globals 
This refactors handling of s3 so it can be specified via GlobalSetting

This means that in a multisite environment you can configure s3 uploads
without actual sites knowing credentials in s3

It is a critical setting for situations where assets are mirrored to s3.
 2017-10-06 16:20:01 +11:00
Guo Xiang Tan 5012d46cbd Add rubocop to our build. (#5004) 2017-07-28 10:20:09 +09:00