Commit Graph

79 Commits

Author SHA1 Message Date
Gerhard Schlager bdbf77dc38 FIX: Unpause Sidekiq before uploading backup to S3
No need to pause Sidekiq longer than really needed. Uploads to S3 can take a long time.
2019-02-05 21:22:25 +01:00
Vinoth Kannan b4f713ca52
FEATURE: Use amazon s3 inventory to manage upload stats (#6867) 2019-02-01 10:10:48 +05:30
Gerhard Schlager 45b056b615 FIX: Do not show backups stored in subfolder of bucket 2019-01-24 22:28:03 +01:00
Gerhard Schlager c94a2bc69b FIX: Raise or log error when deleting of backup fails 2019-01-24 22:26:50 +01:00
Joffrey JAFFEUX f9648de897
DEV: upgrades from Ember 2.13 to Ember 3.5.1 (#6808)
Co-Authored-By: Bianca Nenciu <nbianca@users.noreply.github.com>
Co-Authored-By: David Taylor <david@taylorhq.com>
2019-01-10 11:06:01 +01:00
Gerhard Schlager 0bc1fa8aa4 FEATURE: Don't create PM for successful automatic backups 2018-12-20 13:34:24 +01:00
Gerhard Schlager 1a8ca68ea3 FEATURE: Improve backup stats on admin dashboard
* Dashboard doesn't timeout anymore when Amazon S3 is used for backups
* Storage stats are now a proper report with the same caching rules
* Changing the backup_location, s3_backup_bucket or creating and deleting backups removes the report from the cache
* It shows the number of backups and the backup location
* It shows the used space for the correct backup location instead of always showing used space on local storage
* It shows the date of the last backup as relative date
2018-12-17 11:35:11 +01:00
Gerhard Schlager 7e1f20b07f FIX: Create CORS rule on S3 only before a backup upload 2018-12-17 00:15:37 +01:00
Gerhard Schlager 99117d664c FEATURE: Multisite support for S3 backup store (#6700) 2018-12-05 10:10:39 +08:00
Sam 1b4f2029d7 FIX: clear theme cache when restoring
Previously old themes may be cached incorrectly, this also forces
a rebake of old themes to ensure version can compile cleanly
2018-11-20 13:37:58 +11:00
Guo Xiang Tan 84d4c81a26 FEATURE: Support backup uploads/downloads directly to/from S3.
This reverts commit 3c59106bac.
2018-10-15 09:43:31 +08:00
Guo Xiang Tan 3c59106bac Revert "FEATURE: Support backup uploads/downloads directly to/from S3."
This reverts commit c29a4dddc1.

We're doing a beta bump soon so un-revert this after that is done.
2018-10-11 11:08:23 +08:00
Gerhard Schlager c29a4dddc1 FEATURE: Support backup uploads/downloads directly to/from S3. 2018-10-11 10:38:43 +08:00
Gerhard Schlager 469a2c36ed FIX: Always unpause Sidekiq after backup and restore
* Logs exceptions during the cleanup phase, but doesn't stop executing subsequent cleanup tasks.
* Notifies the user at the end of the cleanup phase, so that the log contains possible errors during that phase.
2018-09-19 20:35:43 +02:00
Guo Xiang Tan 212ee15804 FIX: Create `BaseDropper` functions in a different schema.
https://meta.discourse.org/t/error-when-restore-db-backup/93145/25?u=tgxworld
2018-08-23 12:52:21 +08:00
Régis Hanol 8f1db615db FIX: don't break restore if function does not exist 2018-07-30 22:11:38 +02:00
Guo Xiang Tan 6740631fdb TEMPFIX: Fix broken restores. 2018-07-27 12:48:16 +08:00
Joffrey JAFFEUX 578c8e861b
FIX: refreshes disk_space on backup create/destroy (#6169) 2018-07-25 08:26:30 -04:00
Arpit Jalan 7590128d38 fix typo 2018-07-04 12:01:15 +05:30
Guo Xiang Tan 0af159546a FIX: `BackupRestore::Backuper#remove_tar_leftovers` not cleaning up files.
Wildcard is sanitized when passed to `system()`.
2018-07-04 13:58:39 +08:00
Sam 5f64fd0a21 DEV: remove exec_sql and replace with mini_sql
Introduce new patterns for direct sql that are safe and fast.

MiniSql is not prone to memory bloat that can happen with direct PG usage.
It also has an extremely fast materializer and very a convenient API

- DB.exec(sql, *params) => runs sql returns row count
- DB.query(sql, *params) => runs sql returns usable objects (not a hash)
- DB.query_hash(sql, *params) => runs sql returns an array of hashes
- DB.query_single(sql, *params) => runs sql and returns a flat one dimensional array
- DB.build(sql) => returns a sql builder

See more at: https://github.com/discourse/mini_sql
2018-06-19 16:13:36 +10:00
Guo Xiang Tan 5da7c2a4ad FIX: Restorer wasn't rolling back if restore fails.
* This only applies to backup file taken with
  pg_dump 10.3+ and pg_dump 9.5.12+.
2018-04-06 09:43:32 +08:00
Guo Xiang Tan 142571bba0 Remove use of `rescue nil`.
* `rescue nil` is a really bad pattern to use in our code base.
  We should rescue errors that we expect the code to throw and
  not rescue everything because we're unsure of what errors the
  code would throw. This would reduce the amount of pain we face
  when debugging why something isn't working as expexted. I've
  been bitten countless of times by errors being swallowed as a
  result during debugging sessions.
2018-04-02 13:52:51 +08:00
Michael Brown 63a1e9b60a backup restorer: tidy pg_dump schema portability logic, add test 2018-03-20 10:32:39 +08:00
Guo Xiang Tan da8e15f954 FIX: Restorer was not extracting the patch version in dump file. 2018-03-16 11:09:56 +08:00
Michael Brown 90291318eb restorer: clarify logging 2018-03-15 12:14:08 -04:00
Guo Xiang Tan 5ef75c9c61 Improve grep pattern in restorer. 2018-03-09 15:48:12 +08:00
Guo Xiang Tan 766b41d9f4 Fix version check in restorer. 2018-03-09 15:01:10 +08:00
Guo Xiang Tan 8fd47314d9 FIX: Restore process for dump taken with `pg_dump` 10.3+.
* Since we can no longer restore into a different schema,
  we will move tables in the public schema into the backup schema
  first before restoring the dump file which goes into the public
  schema. The downside to this approach is that we will increase
  the downtime experienced during the restore process. Downtime
  would equal the duration of restoring the dump file.
2018-03-09 13:24:58 +08:00
Guo Xiang Tan a89f3160a5 Add new config to ensure backup/restore connects to PG directly.
* In `pg_dump` 10.3+ and 9.5.12+, in
  it does a `SELECT pg_catalog.set_config('search_path', '', false)`
  which changes the state of the current connection. This is known
  to be problematic with Pgbouncer which reuses connections. As such,
  we'll always try to connect directly to PG directly during
  the backup/restore process.
2018-03-09 10:28:03 +08:00
Will Jordan a41446a502 single quote password in restore command
> Followup to #3283. Quotes passwords passed to shell for backup restore.
2018-03-01 12:08:35 -08:00
Sam 88a4ec5f1b FIX: stop forking regular backup jobs 2017-12-21 09:00:48 +11:00
Guo Xiang Tan 5012d46cbd Add rubocop to our build. (#5004) 2017-07-28 10:20:09 +09:00
Leo McArdle d0b027d88d FEATURE: phase 1 of supporting multiple email addresses 2017-07-20 11:22:27 +09:00
Guo Xiang Tan b70d4da858 FIX: Only invite admins when automatic backup fails. 2017-06-15 14:04:22 +08:00
Guo Xiang Tan f6060bfbf6 Invite admins to automatic backups failure topic.
https://meta.discourse.org/t/if-automatic-backup-fails-there-should-be-a-warning/64461
2017-06-14 15:01:11 +09:00
Guo Xiang Tan 5ce8d7a8c5 Log all errors during clean up as well. 2017-06-14 11:03:50 +09:00
Jay Pfaffman 83110a1a81 FIX: allow tar to finish if files change during backup 2017-06-07 13:31:02 -07:00
Guo Xiang Tan a4deb0e47d Fix typo. 2017-03-24 20:59:34 +08:00
Guo Xiang Tan e7c972ac89 FIX: Don't use backticks that take in inputs. 2017-03-17 15:33:51 +08:00
Guo Xiang Tan b49bf889f6 SECURITY: Disallow symlinks when restoring uploads. 2017-03-17 14:27:01 +08:00
Guo Xiang Tan 7139538286 Fix typo. 2016-09-21 16:04:41 +08:00
Guo Xiang Tan 0bf7519a8a FIX: `tar --list` against a `.tar.gz` file takes too long.
This resulted in requests being blocked for an extended amount
of time when initializing the restorer.
2016-09-16 17:11:14 +08:00
Guo Xiang Tan 68637f2164 FIX: Uploads being restored into the wrong directory for multisite. 2016-09-16 14:26:06 +08:00
Guo Xiang Tan f63a797e39 SECUIRTY: Escape input made to system calls. 2016-09-16 11:58:14 +08:00
Guo Xiang Tan 8f36290c05 FIX: No need to list all the files. 2016-09-16 11:57:35 +08:00
Guo Xiang Tan 7e80810de1 FIX: Raise an error if metadata is not extracted correctly. 2016-08-25 17:20:32 +08:00
Guo Xiang Tan 3e4b02bbd4 FIX: Make sure constant reflects the right backup extenstion. 2016-08-24 10:28:23 +08:00
Guo Xiang Tan 8539f02b5e FIX: Backuper should return the full path. 2016-08-08 07:49:37 +08:00
Guo Xiang Tan adc8336949 Make sure we track restore/backlog success logs as well. 2016-08-03 16:23:47 +08:00