5357ab3324
SECURITY: Safely decompress backups when restoring. ( #8166 )
...
* SECURITY: Safely decompress backups when restoring.
* Fix tests and update theme_controller_spec to work with zip files instead of .tar.gz
2019-10-09 11:41:16 -03:00
427d54b2b0
DEV: Upgrading Discourse to Zeitwerk ( #8098 )
...
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains.
We no longer need to use Rails "require_dependency" anywhere and instead can just use standard
Ruby patterns to require files.
This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
1f118b1309
FEATURE: Allow plugins to manipulate site settings during backup restore
2019-08-22 22:41:26 +02:00
d686318133
FIX: Prevent failed remaps during restores
...
Additional changes:
* Verbose logging of remaps during restores
* Exclude the backup_metadata table from restores
2019-08-12 17:15:01 +02:00
7cb51d0e40
FIX: Create readonly functions during backup
...
Temporarily recreate already dropped functions in the discourse_functions schema in order to allow restoring of backups which still reference dropped functions.
2019-08-09 11:39:46 +02:00
31f583855a
DEV: pull static check out of loop
...
* followup to 08b28680https://review.discourse.org/t/4713/2
2019-07-23 17:18:16 -04:00
68b082e1a4
FIX: Ensure that jobs don't run immediately after migrate_to_s3
2019-07-23 17:42:12 +02:00
b73bd7fc1b
FIX: Always backup local uploads in addition to files stored on S3
2019-07-19 15:13:05 +02:00
b0c92bb0b9
REFACTOR: Clean up parameterized title
...
Follow up to [FIX: Empty backup names with unicode site titles][1]
- Use .presence - "It's cleaner"
- Update spec to use System.system_user so it is more readable
[1]: c8661674d4
2019-07-18 15:49:16 -06:00
c8661674d4
FIX: Empty backup names with unicode site titles
...
If a site title contains unicode it may end up with an empty backup
filename because of the rails `parameterize` method we are calling.
This fix ensures that the backup filenames default to "discourse" if the
parameterized site title is empty.
Bug reported [here][1].
[1]: https://meta.discourse.org/t/backup-checksum-and-backup-name-missing-when-unicode-site-name/123192?u=blake
2019-07-17 17:07:10 -06:00
08b286808a
FIX: backups taken by pg_dump >= 11 are nonportable ( #7893 )
2019-07-15 18:07:44 -04:00
5f0d38341e
FIX: Remapping during restore was wrong for CDN URLs
2019-07-09 17:34:41 +02:00
4c1b8c7559
FIX: Remap differently when backup comes from multisite
2019-07-09 16:11:32 +02:00
a65a9a85d5
FEATURE: Remap uploads during restore when S3 or CDN changes
...
In order for this to work the Backuper stores a couple of site settings
in the new backup_metadata table, because the old setting values might
not be available on restore anymore.
2019-07-09 14:04:16 +02:00
f2dc59d61f
FEATURE: Add hidden setting to include S3 uploads in backups
2019-07-09 14:04:16 +02:00
f00275ded3
FEATURE: Support private attachments when using S3 storage ( #7677 )
...
* Support private uploads in S3
* Use localStore for local avatars
* Add job to update private upload ACL on S3
* Test multisite paths
* update ACL for private uploads in migrate_to_s3 task
2019-06-06 13:27:24 +10:00
f7a2648694
FEATURE: Migrate uploads to S3 during restore
2019-06-04 15:47:36 +02:00
30990006a9
DEV: enable frozen string literal on all files
...
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.
Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
1ddd4a44d5
FIX: Wrong color palette after backup restore
2019-05-07 17:02:57 +02:00
2487e01c73
FIX: Optimized site icons were missing after backup restore
2019-05-07 17:02:57 +02:00
ebca588fd0
DEV: Remove unused line of code.
2019-05-02 16:54:10 +08:00
3aca070311
FIX: Restoring backup shouldn't change disable_emails from "yes" to "non-staff"
2019-04-16 11:48:07 +02:00
78f8114989
FEATURE: Allow discourse script to skip disabling of emails after restore
2019-03-07 21:49:33 +01:00
42df20e4f0
typo ( #7065 )
2019-02-25 16:36:22 +01:00
dc961fecb9
FIX: Outgoing emails were not disabled after restoring backup
2019-02-25 16:07:24 +01:00
6a8007e5fb
FEATURE: Improve handling of backup storage errors
2019-02-20 15:16:49 +01:00
8cd4ceba49
DEV: Remove unnecessary `Sidekiq.unpause!` during backup.
2019-02-19 14:01:13 +08:00
99ad61afb7
FEATURE: Trigger an event after a backup restore
2019-02-18 11:48:03 +01:00
b087719340
FEATURE: Setting for excluding optimized images from backups
2019-02-13 11:10:51 +01:00
9eb7dea0f1
FEATURE: Setting for compression level of upload in backups
2019-02-12 15:50:31 +01:00
220944a38a
FIX: Unpause sidekiq before adding uploads to backup
...
tar exits with status 1 when uploads are modified or deleted by a sidekiq job, so we need to treat it like status 0.
According to the documentation it should be safe to ignore status 1 ("Some files differ"):
> If tar was given `--create', `--append' or `--update' option, this exit code means that some files were changed while being archived and so the resulting archive does not contain the exact copy of the file set.
Status 2 ("Fatal error") still results in an exception.
2019-02-12 13:50:50 +01:00
5bb955dcb7
FIX: Allow restore when latest migration is a post_migration
2019-02-08 17:37:05 +01:00
bdbf77dc38
FIX: Unpause Sidekiq before uploading backup to S3
...
No need to pause Sidekiq longer than really needed. Uploads to S3 can take a long time.
2019-02-05 21:22:25 +01:00
b4f713ca52
FEATURE: Use amazon s3 inventory to manage upload stats ( #6867 )
2019-02-01 10:10:48 +05:30
45b056b615
FIX: Do not show backups stored in subfolder of bucket
2019-01-24 22:28:03 +01:00
c94a2bc69b
FIX: Raise or log error when deleting of backup fails
2019-01-24 22:26:50 +01:00
f9648de897
DEV: upgrades from Ember 2.13 to Ember 3.5.1 ( #6808 )
...
Co-Authored-By: Bianca Nenciu <nbianca@users.noreply.github.com>
Co-Authored-By: David Taylor <david@taylorhq.com>
2019-01-10 11:06:01 +01:00
0bc1fa8aa4
FEATURE: Don't create PM for successful automatic backups
2018-12-20 13:34:24 +01:00
1a8ca68ea3
FEATURE: Improve backup stats on admin dashboard
...
* Dashboard doesn't timeout anymore when Amazon S3 is used for backups
* Storage stats are now a proper report with the same caching rules
* Changing the backup_location, s3_backup_bucket or creating and deleting backups removes the report from the cache
* It shows the number of backups and the backup location
* It shows the used space for the correct backup location instead of always showing used space on local storage
* It shows the date of the last backup as relative date
2018-12-17 11:35:11 +01:00
7e1f20b07f
FIX: Create CORS rule on S3 only before a backup upload
2018-12-17 00:15:37 +01:00
99117d664c
FEATURE: Multisite support for S3 backup store ( #6700 )
2018-12-05 10:10:39 +08:00
1b4f2029d7
FIX: clear theme cache when restoring
...
Previously old themes may be cached incorrectly, this also forces
a rebake of old themes to ensure version can compile cleanly
2018-11-20 13:37:58 +11:00
84d4c81a26
FEATURE: Support backup uploads/downloads directly to/from S3.
...
This reverts commit 3c59106bac
2018-10-15 09:43:31 +08:00
3c59106bac
Revert "FEATURE: Support backup uploads/downloads directly to/from S3."
...
This reverts commit c29a4dddc1
2018-10-11 11:08:23 +08:00
c29a4dddc1
FEATURE: Support backup uploads/downloads directly to/from S3.
2018-10-11 10:38:43 +08:00
469a2c36ed
FIX: Always unpause Sidekiq after backup and restore
...
* Logs exceptions during the cleanup phase, but doesn't stop executing subsequent cleanup tasks.
* Notifies the user at the end of the cleanup phase, so that the log contains possible errors during that phase.
2018-09-19 20:35:43 +02:00
212ee15804
FIX: Create `BaseDropper` functions in a different schema.
...
https://meta.discourse.org/t/error-when-restore-db-backup/93145/25?u=tgxworld
2018-08-23 12:52:21 +08:00
8f1db615db
FIX: don't break restore if function does not exist
2018-07-30 22:11:38 +02:00
6740631fdb
TEMPFIX: Fix broken restores.
2018-07-27 12:48:16 +08:00
578c8e861b
FIX: refreshes disk_space on backup create/destroy ( #6169 )
2018-07-25 08:26:30 -04:00
7590128d38
fix typo
2018-07-04 12:01:15 +05:30
0af159546a
FIX: `BackupRestore::Backuper#remove_tar_leftovers` not cleaning up files.
...
Wildcard is sanitized when passed to `system()`.
2018-07-04 13:58:39 +08:00
5f64fd0a21
DEV: remove exec_sql and replace with mini_sql
...
Introduce new patterns for direct sql that are safe and fast.
MiniSql is not prone to memory bloat that can happen with direct PG usage.
It also has an extremely fast materializer and very a convenient API
- DB.exec(sql, *params) => runs sql returns row count
- DB.query(sql, *params) => runs sql returns usable objects (not a hash)
- DB.query_hash(sql, *params) => runs sql returns an array of hashes
- DB.query_single(sql, *params) => runs sql and returns a flat one dimensional array
- DB.build(sql) => returns a sql builder
See more at: https://github.com/discourse/mini_sql
2018-06-19 16:13:36 +10:00
5da7c2a4ad
FIX: Restorer wasn't rolling back if restore fails.
...
* This only applies to backup file taken with
pg_dump 10.3+ and pg_dump 9.5.12+.
2018-04-06 09:43:32 +08:00
142571bba0
Remove use of `rescue nil`.
...
* `rescue nil` is a really bad pattern to use in our code base.
We should rescue errors that we expect the code to throw and
not rescue everything because we're unsure of what errors the
code would throw. This would reduce the amount of pain we face
when debugging why something isn't working as expexted. I've
been bitten countless of times by errors being swallowed as a
result during debugging sessions.
2018-04-02 13:52:51 +08:00
63a1e9b60a
backup restorer: tidy pg_dump schema portability logic, add test
2018-03-20 10:32:39 +08:00
da8e15f954
FIX: Restorer was not extracting the patch version in dump file.
2018-03-16 11:09:56 +08:00
90291318eb
restorer: clarify logging
2018-03-15 12:14:08 -04:00
5ef75c9c61
Improve grep pattern in restorer.
2018-03-09 15:48:12 +08:00
766b41d9f4
Fix version check in restorer.
2018-03-09 15:01:10 +08:00
8fd47314d9
FIX: Restore process for dump taken with `pg_dump` 10.3+.
...
* Since we can no longer restore into a different schema,
we will move tables in the public schema into the backup schema
first before restoring the dump file which goes into the public
schema. The downside to this approach is that we will increase
the downtime experienced during the restore process. Downtime
would equal the duration of restoring the dump file.
2018-03-09 13:24:58 +08:00
a89f3160a5
Add new config to ensure backup/restore connects to PG directly.
...
* In `pg_dump` 10.3+ and 9.5.12+, in
it does a `SELECT pg_catalog.set_config('search_path', '', false)`
which changes the state of the current connection. This is known
to be problematic with Pgbouncer which reuses connections. As such,
we'll always try to connect directly to PG directly during
the backup/restore process.
2018-03-09 10:28:03 +08:00
a41446a502
single quote password in restore command
...
> Followup to #3283 . Quotes passwords passed to shell for backup restore.
2018-03-01 12:08:35 -08:00
88a4ec5f1b
FIX: stop forking regular backup jobs
2017-12-21 09:00:48 +11:00
5012d46cbd
Add rubocop to our build. ( #5004 )
2017-07-28 10:20:09 +09:00
d0b027d88d
FEATURE: phase 1 of supporting multiple email addresses
2017-07-20 11:22:27 +09:00
b70d4da858
FIX: Only invite admins when automatic backup fails.
2017-06-15 14:04:22 +08:00
f6060bfbf6
Invite admins to automatic backups failure topic.
...
https://meta.discourse.org/t/if-automatic-backup-fails-there-should-be-a-warning/64461
2017-06-14 15:01:11 +09:00
5ce8d7a8c5
Log all errors during clean up as well.
2017-06-14 11:03:50 +09:00
83110a1a81
FIX: allow tar to finish if files change during backup
2017-06-07 13:31:02 -07:00
a4deb0e47d
Fix typo.
2017-03-24 20:59:34 +08:00
e7c972ac89
FIX: Don't use backticks that take in inputs.
2017-03-17 15:33:51 +08:00
b49bf889f6
SECURITY: Disallow symlinks when restoring uploads.
2017-03-17 14:27:01 +08:00
7139538286
Fix typo.
2016-09-21 16:04:41 +08:00
0bf7519a8a
FIX: `tar --list` against a `.tar.gz` file takes too long.
...
This resulted in requests being blocked for an extended amount
of time when initializing the restorer.
2016-09-16 17:11:14 +08:00
68637f2164
FIX: Uploads being restored into the wrong directory for multisite.
2016-09-16 14:26:06 +08:00
f63a797e39
SECUIRTY: Escape input made to system calls.
2016-09-16 11:58:14 +08:00
8f36290c05
FIX: No need to list all the files.
2016-09-16 11:57:35 +08:00
7e80810de1
FIX: Raise an error if metadata is not extracted correctly.
2016-08-25 17:20:32 +08:00
3e4b02bbd4
FIX: Make sure constant reflects the right backup extenstion.
2016-08-24 10:28:23 +08:00
8539f02b5e
FIX: Backuper should return the full path.
2016-08-08 07:49:37 +08:00
adc8336949
Make sure we track restore/backlog success logs as well.
2016-08-03 16:23:47 +08:00
b860d1b254
FIX: Ensure uploads directory exists.
2016-08-03 16:23:47 +08:00
0a942dbc73
FEATURE: Avoid creating an archive for database only backups.
2016-08-03 16:23:46 +08:00
441b98579a
FIX: Ensure that our restorer is backwards compatible.
2016-08-02 09:19:56 +08:00
76e57ddef3
FIX: Log errors in `ensure` block of restorer.
2016-07-26 10:24:01 +08:00
03aa13b2bb
FEATURE: Work with compressed version of `pg_dump` during backup and restore.
2016-07-26 10:24:01 +08:00
1adfa0a4b5
FEATURE: Add SiteSetting to disable readonly mode during backup.
2016-07-19 17:44:04 +08:00
b981041f6f
Make sure we log failures in `ensure` block.
2016-07-15 11:36:47 +08:00
91e4af0d3d
FIX: restore of a backup from an older Discourse version can create new tables in the wrong schema, leading to UndefinedTable errors
2016-07-12 16:26:45 -04:00
166d753bd3
FIX: delete PostgreSQL dump before gzipping archive ( #4323 )
2016-07-12 14:23:26 +02:00
ed53a24dbe
FIX: backup was failing on large instances ( #4319 )
2016-07-11 08:36:20 +01:00
8c52508007
warn users it may take a while to un/zip backup
2016-05-23 09:33:29 +02:00
332a1ea87e
FIX: add support for 'space' in backup's filename
2016-04-04 17:24:02 +02:00
3939b9ec7d
FIX: restore in development mode connects to the wrong database
2016-02-12 17:20:38 -05:00
02d9ec5bde
oops, should not have been commited
2016-01-19 01:01:17 +01:00
3083657358
FEATURE: better email in support
...
FEATURE: new incoming_email model
FEATURE: infinite scrolling in emails admin
FEATURE: new 'emails:import' rake task
2016-01-19 00:57:55 +01:00
4d8eb25148
let's explicitly use gzip level 5 here
...
see http://tukaani.org/lzma/benchmarks.html
2016-01-11 03:03:14 -08:00
a726f5efea
FIX: ensure we always remove old backups
2015-09-21 10:52:03 +02:00
96c23d51a2
FIX: don't break the message bus when restoring a backup
2015-08-27 20:02:13 +02:00
Roman Rizzi
Krzysztof Kotlarek
Gerhard Schlager
Michael Brown
Blake Erickson
Michael Brown
Penar Musaraj
Sam Saffron
Guo Xiang Tan
Joffrey JAFFEUX
Vinoth Kannan
Régis Hanol
Arpit Jalan
Will Jordan
Guo Xiang Tan
Leo McArdle
Jay Pfaffman
Neil Lalonde
Jeff Atwood