Commit Graph

31 Commits

Author SHA1 Message Date
Jarek Radosz 891987a284
DEV: Recover missing files of existing uploads (#10757)
UploadRecovery only worked on missing Upload records. Now it also works with existing ones that have an invalid_etag status.

The specs (first that test the S3 path) are a bit of stub-a-palooza, but that's how much this class interacts with the the outside world. 🤷‍♂️
2020-10-01 14:54:45 +02:00
Jarek Radosz 188d224303
DEV: Restore UploadRecovery specs (#10543)
Originally disabled in 0c0192e. Upload specs now use separate paths for each spec worker.

Fixes an issue in UploadRecovery#recover_from_local – it didn't take into account the testing infix (e.g. test_0) in the uploads/tombstone paths.
2020-08-27 15:57:10 +02:00
Krzysztof Kotlarek e0d9232259
FIX: use allowlist and blocklist terminology (#10209)
This is a PR of the renaming whitelist to allowlist and blacklist to the blocklist.
2020-07-27 10:23:54 +10:00
Michael Brown dbe0111822 FEATURE: allow UploadRecovery to be run on a single post (#8094) 2019-10-02 14:57:36 +10:00
Vinoth Kannan a1fc35fdf7 FIX: make uploads recovery compatible with multisite. 2019-08-02 07:08:21 +05:30
Vinoth Kannan e44d56e4d2 DEV: raise error only when 'STOP_ON_ERROR' env variable is available. 2019-08-01 23:54:06 +05:30
Sam Saffron e8799f0ba4 DEV: improve uploads:recover job so it stores a map of old to new sha
Previous attempt created broken images
2019-05-22 15:51:09 +10:00
Sam Saffron ebcb571de7 FIX: allow upload recovery to recover uploads with sha mismatch
Filename on disk may mismatch sha of file in some old 1X setups. This will
attempt to recover file even if sha1 mismatches. We had an old bug that
caused this.

This also adds `uploads:fix_relative_upload_links` which attempts to replace
urls of the format `/upload/default/...` with `upload://`
2019-05-22 15:24:36 +10:00
Sam Saffron 30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Guo Xiang Tan 24347ace10 FIX: Properly associate user_profiles background urls via upload id.
`Upload#url` is more likely and can change from time to time. When it
does changes, we don't want to have to look through multiple tables to
ensure that the URLs are all up to date. Instead, we simply associate
uploads properly to `UserProfile` so that it does not have to replicate
the URLs in the table.
2019-05-02 14:58:24 +08:00
Vinoth Kannan d0fe42e2ef FIX: should look through posts for image markdown
Downloaded onebox images only included in the cooked HTML content. So we have to check 'post.cooked' instead of 'raw'. bfdd0fe64c
2019-04-10 13:52:35 +05:30
Vinoth Kannan f5ac01253a FIX: use new changed method name and remove whitespaces 2019-04-09 02:25:26 +05:30
Guo Xiang Tan 6a95d3fded FIX: `UploadRecovery` should look through posts for img src and bbcode. 2019-04-02 11:41:00 +08:00
Guo Xiang Tan 1f9799c979 FIX: Handle more cases in `UploadRecovery`. 2019-04-02 11:29:26 +08:00
Guo Xiang Tan ac52d8d3c7 Fix `UploadRecovery` from S3 fails with bucket name containing sub-folder. 2018-10-01 19:03:02 +08:00
Guo Xiang Tan a3dbe3685e Fix the build. 2018-10-01 16:42:38 +08:00
Guo Xiang Tan de85bb0a39 FIX: Don't update user_profile URLs unless upload is persisted. 2018-10-01 14:21:39 +08:00
Guo Xiang Tan e262a08350 Add `UploadRecovery#recover_user_profile_backgrounds`. 2018-10-01 10:51:54 +08:00
Guo Xiang Tan 1a64b3a487 FIX: Don't try to recover an invalid sha1. 2018-09-20 14:21:57 +08:00
Guo Xiang Tan 293cf600f0 FIX: Use a tmp file in `UploadRecovery` for local store. 2018-09-19 15:47:15 +08:00
Guo Xiang Tan bc7f58191e FIX: `UploadRecovery` should look at links too. 2018-09-19 11:52:57 +08:00
Guo Xiang Tan 4a92c5b2d6 `UploadRecovery` should recover attachments too. 2018-09-19 10:44:36 +08:00
Guo Xiang Tan 529e4018f0 DEV: Avoid using `send` and make the method public instead. 2018-09-13 16:34:32 +08:00
Guo Xiang Tan 0a06b3d977 Accept custom AR relation for `UploadRecovery`. 2018-09-13 16:33:14 +08:00
Guo Xiang Tan 5eb65ad612 FIX: Do not try to recover invalid `Upload#short_url` in `UploadRecovery`. 2018-09-13 13:59:17 +08:00
Guo Xiang Tan d99dd840e4 Add basic test case for `UploadRecovery`. 2018-09-13 13:26:23 +08:00
Guo Xiang Tan a3b3b0810d Rescue errors when running dry run for `UploadRecovery`. 2018-09-13 11:57:51 +08:00
Guo Xiang Tan 4dc25ad201 Fix s3 recovery from tombstone in `UploadRecovery`. 2018-09-13 09:19:45 +08:00
Guo Xiang Tan 24c55bd613 Add dry run option to `UploadRecovery`. 2018-09-12 21:53:01 +08:00
Guo Xiang Tan 2cc48cfd06 Fix incorrect variable. 2018-09-12 02:06:14 -07:00
Guo Xiang Tan c053f8ccf6 New rake task `uploads:recover`. 2018-09-12 01:52:30 -07:00