9ce06d53d1
DEV: improve upload recovery rake task
...
- Better error handling
- Automatically generates index file if missing
- Improved output
2019-05-24 13:36:03 +10:00
791241e0b2
DEV: Fix rake task to delete old export topics.
2019-05-24 11:20:53 +08:00
76f7e90e23
FEATURE: rake posts:recover_uploads_from_index improvements
...
We now recover more smartly based on failed db remaps as well.
2019-05-24 12:38:38 +10:00
1059aafc55
FEATURE: rake task for recovery of uploads from an index file
...
`rake posts:recover_uploads_from_index`
Searches through all missing uploads in the cluster, if it finds one it
tries to find it in the "upload index file" and creates a new upload for
it.
2019-05-23 17:08:22 +10:00
a5ce9cb470
FEATURE: fix_relative_upload_links now multisite safety
...
This also finds `<img src="/uploads/xyz` HTML images in raw and corrects
them. Also handles some cross multisite recovery and provides better output
2019-05-23 15:09:16 +10:00
96e58125a7
FEATURE: rename task to posts:destroy_old_data_exports
...
Previously we were only catching one type of data export, the new job will
catch every csv export we have
Job is pretty safe as it filters on system user id / pm with a particular
slug
2019-05-23 15:02:56 +10:00
954293655f
FEATURE: rake posts:destroy_old_user_data_exports
...
Historically we would keep the user data export posts around but delete
the uploads.
This leaves a lot of broken uploads in the system.
This rake task allows us to clean up old mess.
2019-05-23 11:11:37 +10:00
a9af77d650
PERF: ensure priority for rake rebake tasks is ultra low
...
Flooding the default queue with lots of jobs is never a good thing. This
reduces the risk of sidekiq going into a bad state post rebake
2019-05-22 10:31:49 +10:00
d6daa60d0a
FEATURE: multisite support for posts:rebake_uncooked_posts
2019-05-22 10:04:54 +10:00
73f178a634
FEATURE: posts:rebake_uncooked_posts to look at mismatching baked_version
...
also amends flagging onebox updates to set baked_version to nil
2019-05-22 09:43:31 +10:00
4f296608da
FEATURE: add uploads:s3_migration_status for looking at current status
...
Also a few minor cleanups and better progress reporting
2019-05-22 09:00:32 +10:00
338b927f95
DEV: include local uploads while finding missing uploads
...
in rake task 'posts:missing_uploads'
2019-05-21 22:07:08 +05:30
f4d4f7871e
FEATURE: make posts:missing_uploads multisite friendly
...
Previously this rake job would only run on a single site which is a bit
misleading
This also adds `VERBOSE=1 rake posts:missing_uploads` that will provide a
full report of missing uploads
2019-05-21 12:45:51 +10:00
9f43852609
FIX: remove the unwanted code.
2019-05-20 19:10:21 +05:30
30990006a9
DEV: enable frozen string literal on all files
...
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.
Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
87cd4701b8
FEATURE: option to skip posts with ignored missing uploads
2019-05-09 05:11:15 +05:30
e8fafbc123
List and restore missing post uploads from S3 inventory.
2019-05-04 01:16:20 +05:30
c448e014fe
Recover missing old scheme uploads from tombstone directory too.
2019-04-29 11:33:57 +05:30
8698a10920
Don't raise error if a record already exists.
2019-04-25 21:48:39 +05:30
a65ce81e25
Don't check same upload url twice in the rake task
2019-04-25 20:06:10 +05:30
1724c27713
FIX: reload the 'post' model to retrive raw field value.
2019-04-25 02:09:27 +05:30
aed683390c
FIX: Don't treat 'upload_patterns' as constant to make the rake task compatible with multisite
2019-04-25 02:06:20 +05:30
b0bb1d95a5
trying to recover old scheme uploads on 'posts:missing_uploads' rake task
2019-04-25 01:16:18 +05:30
3dae12dec1
reduce unlinked count from missing post upload count
2019-04-19 21:53:00 +05:30
4fcefc3615
add unlinked missing post upload count in stat
2019-04-19 20:39:21 +05:30
ba6369edc5
Add more missing upload stats in rake task
2019-04-19 20:14:40 +05:30
48cedf3ebe
fix: retrieve posts in order by id.
...
so we can find starting and ending id of the batch.
2019-04-19 18:30:59 +05:30
ce8d30f667
PERF: use 'delete_all' method instead of 'destroy_all'
2019-04-19 18:00:01 +05:30
5de483a1b9
PERF: Very post upload's existence by preloaded upload sha1s array
2019-04-19 17:27:16 +05:30
8d5c900142
DEV: add unique missing uploads index in post custom fields
...
https://review.discourse.org/t/feature-send-missing-post-uploads-stat-to-prometheus/2609/6?u=vinothkannans
2019-04-10 18:09:35 +05:30
d0fe42e2ef
FIX: should look through posts for image markdown
...
Downloaded onebox images only included in the cooked HTML content. So we have to check 'post.cooked' instead of 'raw'. bfdd0fe64c
2019-04-10 13:52:35 +05:30
4878ee95e5
DEV: optimize sql query to narrow down the filtering of post with uploads
2019-04-10 12:58:10 +05:30
bfdd0fe64c
FIX: empty the missing list on each post loop
2019-04-10 11:20:26 +05:30
914ada1c74
DEV: convert scheduled job EnsurePostUploadsExistence into a rake task
2019-04-09 02:07:35 +05:30
341836eb42
Fix the rake task and importer instead
2018-10-17 16:48:09 +02:00
6588a45880
Make Rubocop happy
2018-08-23 15:09:20 +02:00
409ee66839
Add optional "ignore_case" parameter to posts:remap rake task
2018-08-23 14:49:17 +02:00
536cef86f4
PERF: do not carry post ids in memory when rebaking all posts
2018-07-11 14:34:33 +05:30
59a2767de8
Ensure that we restore the site setting in posts:rebake rake task.
2018-07-06 16:22:54 +08:00
cb824a6b33
DEV: remove all calls to SqlBuilder use DB.build instead
...
This is part of the migration to mini_sql, SqlBuilder.new is being
deprecated and replaced with DB.build
2018-06-20 17:53:49 +10:00
5f64fd0a21
DEV: remove exec_sql and replace with mini_sql
...
Introduce new patterns for direct sql that are safe and fast.
MiniSql is not prone to memory bloat that can happen with direct PG usage.
It also has an extremely fast materializer and very a convenient API
- DB.exec(sql, *params) => runs sql returns row count
- DB.query(sql, *params) => runs sql returns usable objects (not a hash)
- DB.query_hash(sql, *params) => runs sql returns an array of hashes
- DB.query_single(sql, *params) => runs sql and returns a flat one dimensional array
- DB.build(sql) => returns a sql builder
See more at: https://github.com/discourse/mini_sql
2018-06-19 16:13:36 +10:00
64f2086cca
rake task should allow reordering posts of single topic
2018-04-10 13:52:08 +02:00
adb93716ca
FIX: rake task should rebake posts in descending ID order
2018-04-10 00:18:49 +05:30
cf9965fd54
Add rake task for reordering posts by creation_date
2018-03-29 14:42:41 +02:00
81427e26ea
Ignore errors when remapping posts
2017-12-28 15:13:01 +01:00
b15059418b
FIX: rake task for updating posts received by email should not crash
2017-12-13 22:03:31 +01:00
82c18f6ca3
fix: undefined variable in `post:rebake_match`
2017-10-24 20:05:58 -04:00
1477a0e910
Adds a rake task for refreshing posts received via email
...
This is useful when the email_reply_trimmer gem was updated and you want to apply those changes to existing posts.
2017-10-06 14:29:11 +02:00
0342324b47
FEATURE: support regex in rake post:remap ( #5201 )
2017-10-04 11:47:53 +11:00
041ace5d3b
new rake task to defer all flags
2017-09-15 16:42:58 +05:30
Sam Saffron
Guo Xiang Tan
Vinoth Kannan
Gerhard Schlager
Arpit Jalan
Kyle Zhao