Commit Graph

106 Commits

Author SHA1 Message Date
Michael Brown 31f583855a DEV: pull static check out of loop
* followup to 08b28680
* as per https://review.discourse.org/t/4713/2
2019-07-23 17:18:16 -04:00
Gerhard Schlager 68b082e1a4 FIX: Ensure that jobs don't run immediately after migrate_to_s3 2019-07-23 17:42:12 +02:00
Gerhard Schlager b73bd7fc1b FIX: Always backup local uploads in addition to files stored on S3 2019-07-19 15:13:05 +02:00
Blake Erickson b0c92bb0b9 REFACTOR: Clean up parameterized title
Follow up to [FIX: Empty backup names with unicode site titles][1]

- Use .presence - "It's cleaner"
- Update spec to use System.system_user so it is more readable

[1]: c8661674d4
2019-07-18 15:49:16 -06:00
Blake Erickson c8661674d4 FIX: Empty backup names with unicode site titles
If a site title contains unicode it may end up with an empty backup
filename because of the rails `parameterize` method we are calling.

This fix ensures that the backup filenames default to "discourse" if the
parameterized site title is empty.

Bug reported [here][1].

[1]: https://meta.discourse.org/t/backup-checksum-and-backup-name-missing-when-unicode-site-name/123192?u=blake
2019-07-17 17:07:10 -06:00
Michael Brown 08b286808a
FIX: backups taken by pg_dump >= 11 are nonportable (#7893) 2019-07-15 18:07:44 -04:00
Gerhard Schlager 5f0d38341e FIX: Remapping during restore was wrong for CDN URLs 2019-07-09 17:34:41 +02:00
Gerhard Schlager 4c1b8c7559 FIX: Remap differently when backup comes from multisite 2019-07-09 16:11:32 +02:00
Gerhard Schlager a65a9a85d5 FEATURE: Remap uploads during restore when S3 or CDN changes
In order for this to work the Backuper stores a couple of site settings
in the new backup_metadata table, because the old setting values might
not be available on restore anymore.
2019-07-09 14:04:16 +02:00
Gerhard Schlager f2dc59d61f FEATURE: Add hidden setting to include S3 uploads in backups 2019-07-09 14:04:16 +02:00
Penar Musaraj f00275ded3 FEATURE: Support private attachments when using S3 storage (#7677)
* Support private uploads in S3
* Use localStore for local avatars
* Add job to update private upload ACL on S3
* Test multisite paths
* update ACL for private uploads in migrate_to_s3 task
2019-06-06 13:27:24 +10:00
Gerhard Schlager f7a2648694 FEATURE: Migrate uploads to S3 during restore 2019-06-04 15:47:36 +02:00
Sam Saffron 30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Gerhard Schlager 1ddd4a44d5 FIX: Wrong color palette after backup restore 2019-05-07 17:02:57 +02:00
Gerhard Schlager 2487e01c73 FIX: Optimized site icons were missing after backup restore 2019-05-07 17:02:57 +02:00
Guo Xiang Tan ebca588fd0 DEV: Remove unused line of code. 2019-05-02 16:54:10 +08:00
Gerhard Schlager 3aca070311 FIX: Restoring backup shouldn't change disable_emails from "yes" to "non-staff" 2019-04-16 11:48:07 +02:00
Gerhard Schlager 78f8114989 FEATURE: Allow discourse script to skip disabling of emails after restore 2019-03-07 21:49:33 +01:00
Joffrey JAFFEUX 42df20e4f0
typo (#7065) 2019-02-25 16:36:22 +01:00
Gerhard Schlager dc961fecb9 FIX: Outgoing emails were not disabled after restoring backup 2019-02-25 16:07:24 +01:00
Gerhard Schlager 6a8007e5fb FEATURE: Improve handling of backup storage errors 2019-02-20 15:16:49 +01:00
Guo Xiang Tan 8cd4ceba49 DEV: Remove unnecessary `Sidekiq.unpause!` during backup. 2019-02-19 14:01:13 +08:00
Gerhard Schlager 99ad61afb7 FEATURE: Trigger an event after a backup restore 2019-02-18 11:48:03 +01:00
Gerhard Schlager b087719340 FEATURE: Setting for excluding optimized images from backups 2019-02-13 11:10:51 +01:00
Gerhard Schlager 9eb7dea0f1 FEATURE: Setting for compression level of upload in backups 2019-02-12 15:50:31 +01:00
Gerhard Schlager 220944a38a FIX: Unpause sidekiq before adding uploads to backup
tar exits with status 1 when uploads are modified or deleted by a sidekiq job, so we need to treat it like status 0.

According to the documentation it should be safe to ignore status 1 ("Some files differ"):

> If tar was given `--create', `--append' or `--update' option, this exit code means that some files were changed while being archived and so the resulting archive does not contain the exact copy of the file set.

Status 2 ("Fatal error") still results in an exception.
2019-02-12 13:50:50 +01:00
Gerhard Schlager 5bb955dcb7 FIX: Allow restore when latest migration is a post_migration 2019-02-08 17:37:05 +01:00
Gerhard Schlager bdbf77dc38 FIX: Unpause Sidekiq before uploading backup to S3
No need to pause Sidekiq longer than really needed. Uploads to S3 can take a long time.
2019-02-05 21:22:25 +01:00
Vinoth Kannan b4f713ca52
FEATURE: Use amazon s3 inventory to manage upload stats (#6867) 2019-02-01 10:10:48 +05:30
Gerhard Schlager 45b056b615 FIX: Do not show backups stored in subfolder of bucket 2019-01-24 22:28:03 +01:00
Gerhard Schlager c94a2bc69b FIX: Raise or log error when deleting of backup fails 2019-01-24 22:26:50 +01:00
Joffrey JAFFEUX f9648de897
DEV: upgrades from Ember 2.13 to Ember 3.5.1 (#6808)
Co-Authored-By: Bianca Nenciu <nbianca@users.noreply.github.com>
Co-Authored-By: David Taylor <david@taylorhq.com>
2019-01-10 11:06:01 +01:00
Gerhard Schlager 0bc1fa8aa4 FEATURE: Don't create PM for successful automatic backups 2018-12-20 13:34:24 +01:00
Gerhard Schlager 1a8ca68ea3 FEATURE: Improve backup stats on admin dashboard
* Dashboard doesn't timeout anymore when Amazon S3 is used for backups
* Storage stats are now a proper report with the same caching rules
* Changing the backup_location, s3_backup_bucket or creating and deleting backups removes the report from the cache
* It shows the number of backups and the backup location
* It shows the used space for the correct backup location instead of always showing used space on local storage
* It shows the date of the last backup as relative date
2018-12-17 11:35:11 +01:00
Gerhard Schlager 7e1f20b07f FIX: Create CORS rule on S3 only before a backup upload 2018-12-17 00:15:37 +01:00
Gerhard Schlager 99117d664c FEATURE: Multisite support for S3 backup store (#6700) 2018-12-05 10:10:39 +08:00
Sam 1b4f2029d7 FIX: clear theme cache when restoring
Previously old themes may be cached incorrectly, this also forces
a rebake of old themes to ensure version can compile cleanly
2018-11-20 13:37:58 +11:00
Guo Xiang Tan 84d4c81a26 FEATURE: Support backup uploads/downloads directly to/from S3.
This reverts commit 3c59106bac.
2018-10-15 09:43:31 +08:00
Guo Xiang Tan 3c59106bac Revert "FEATURE: Support backup uploads/downloads directly to/from S3."
This reverts commit c29a4dddc1.

We're doing a beta bump soon so un-revert this after that is done.
2018-10-11 11:08:23 +08:00
Gerhard Schlager c29a4dddc1 FEATURE: Support backup uploads/downloads directly to/from S3. 2018-10-11 10:38:43 +08:00
Gerhard Schlager 469a2c36ed FIX: Always unpause Sidekiq after backup and restore
* Logs exceptions during the cleanup phase, but doesn't stop executing subsequent cleanup tasks.
* Notifies the user at the end of the cleanup phase, so that the log contains possible errors during that phase.
2018-09-19 20:35:43 +02:00
Guo Xiang Tan 212ee15804 FIX: Create `BaseDropper` functions in a different schema.
https://meta.discourse.org/t/error-when-restore-db-backup/93145/25?u=tgxworld
2018-08-23 12:52:21 +08:00
Régis Hanol 8f1db615db FIX: don't break restore if function does not exist 2018-07-30 22:11:38 +02:00
Guo Xiang Tan 6740631fdb TEMPFIX: Fix broken restores. 2018-07-27 12:48:16 +08:00
Joffrey JAFFEUX 578c8e861b
FIX: refreshes disk_space on backup create/destroy (#6169) 2018-07-25 08:26:30 -04:00
Arpit Jalan 7590128d38 fix typo 2018-07-04 12:01:15 +05:30
Guo Xiang Tan 0af159546a FIX: `BackupRestore::Backuper#remove_tar_leftovers` not cleaning up files.
Wildcard is sanitized when passed to `system()`.
2018-07-04 13:58:39 +08:00
Sam 5f64fd0a21 DEV: remove exec_sql and replace with mini_sql
Introduce new patterns for direct sql that are safe and fast.

MiniSql is not prone to memory bloat that can happen with direct PG usage.
It also has an extremely fast materializer and very a convenient API

- DB.exec(sql, *params) => runs sql returns row count
- DB.query(sql, *params) => runs sql returns usable objects (not a hash)
- DB.query_hash(sql, *params) => runs sql returns an array of hashes
- DB.query_single(sql, *params) => runs sql and returns a flat one dimensional array
- DB.build(sql) => returns a sql builder

See more at: https://github.com/discourse/mini_sql
2018-06-19 16:13:36 +10:00
Guo Xiang Tan 5da7c2a4ad FIX: Restorer wasn't rolling back if restore fails.
* This only applies to backup file taken with
  pg_dump 10.3+ and pg_dump 9.5.12+.
2018-04-06 09:43:32 +08:00
Guo Xiang Tan 142571bba0 Remove use of `rescue nil`.
* `rescue nil` is a really bad pattern to use in our code base.
  We should rescue errors that we expect the code to throw and
  not rescue everything because we're unsure of what errors the
  code would throw. This would reduce the amount of pain we face
  when debugging why something isn't working as expexted. I've
  been bitten countless of times by errors being swallowed as a
  result during debugging sessions.
2018-04-02 13:52:51 +08:00