d6aa92e98e
DEV: Add a verbose option to ./bin/turbo_rspec
2019-06-27 15:49:21 +01:00
b3ec3fac93
FIX: turbo_rspec doesn't accept these options
2019-06-26 17:05:27 +01:00
68d54d818a
DEV: Add USE_TURBO option to docker rake task ( #7811 )
2019-06-26 11:54:39 -04:00
d21594f4f7
Revert changes added by mistake in 2b19e2acc8.
2019-06-25 15:25:12 +08:00
2b19e2acc8
Fix typo in a0aeabbb94.
2019-06-25 15:18:57 +08:00
a0aeabbb94
DEV: Safer replacement for old scheme uploads for `posts:missing_uploads`.
2019-06-25 11:34:26 +08:00
f3c13d80c9
DEV: Don't copy file when trying to recover missing uploads.
2019-06-24 16:32:59 +08:00
2cc5561504
DEV: Correct description in `posts:missing_uploads` rake task.
2019-06-24 14:57:43 +08:00
5bc92296be
DEV: lint a bunch of files we missed
2019-06-21 11:33:41 +10:00
e18ce56f4b
DEV: Add a new way to run specs in parallel with better output ( #7778 )
...
* DEV: Add a new way to run specs in parallel with better output
This commit:
1. adds a new executable, `bin/interleaved_rspec` which works much like
`rspec`, but runs the tests in parallel.
2. adds a rake task, `rake interleaved:spec` which runs the whole test
suite.
3. makes autospec use this new wrapper by default. You can disable this
by running `PARALLEL_SPEC=0 rake autospec`.
It works much like the `parallel_tests` gem (and relies on it), but
makes each subprocess use a machine-readable formatter and parses this
output in order to provide a better overall summary.
(It's called interleaved, because parallel was taken and naming is
hard).
* Make popen3 invocation safer
* Use FileUtils instead of shelling out
* DRY up reporter
* Moved summary logic into Reporter
* s/interleaved/turbo/g
* Move Reporter into its own file
* Moved run into its own class
* Moved Runner into its own file
* Move JsonRowsFormatter under TurboTests
* Join on threads at the end
* Acted on feedback from eviltrout
2019-06-21 10:59:01 +10:00
4bbb43cb2b
DEV: pull plugin repos before linting them
2019-06-14 13:50:15 -04:00
ecb2fd8222
DEV: improve db:create to catch all cases
...
- No RAILS_ENV - create multisite / dev / test
- `test` RAILS_ENV - create multisite / test
2019-06-14 15:47:05 +10:00
a01488ae67
DEV: improve on rake db:create
...
Followup on 3af00a65
2019-06-14 15:06:07 +10:00
3af00a65e6
FIX: site settings loading default values when no db
...
This fixes a condition where an intermittent db connection could cause
invalid site settings to be stored
It also removes a catch all we had.
Somewhere around Rails 5 `db:create` started wanting full environment
this is a problem for Discourse since it needs to boot up data from the
db.
This removes the catch all and surgically adds a db / redis bypass to
db:create task.
2019-06-14 14:21:07 +10:00
eaa7527933
DEV: Switch `posts:inline_uploads` scope to be more generic.
2019-06-14 11:53:34 +08:00
66b15b9d87
DEV: support installing plugin gems in test images
...
- adds plugin:install_all_gems rake task
- adds UPDATE_ALL_PLUGINS env variable in docker:test
2019-06-13 09:58:57 -04:00
9bce3d5e65
DEV: Multisite support for `rake posts:inline_uploads`.
2019-06-13 16:13:45 +08:00
215216df48
FIX: stop re-downloading backup maxmind
...
We omitted updating the mmdb_time which changed causing a constant
re-download of the maxmind db
2019-06-13 17:42:26 +10:00
e47ca6633b
DEV: Fix plugin:update_all task
...
Reenable subtask so it loops through all plugins.
Only run for plugins that are git repositories.
2019-06-12 17:21:56 -04:00
d49c193289
FIX: `posts:inline_uploads` rake task needs to bypass bump.
2019-06-12 09:13:31 +08:00
864222372f
Remove incorrectly committed byebug.
...
Follow up to 49b921147a
2019-06-11 16:33:49 +08:00
49b921147a
FIX: Undefined method in `posts:inline_uploads`.
2019-06-11 16:28:41 +08:00
1cf3bf016c
Make rubocop happy.
2019-06-10 09:49:02 +08:00
29259b46ae
DEV: Verbose mode for `posts:inline_uploads` rake task.
2019-06-10 08:59:11 +08:00
8cd13b22a1
DEV: allows smoke test to run on macOS ( #7735 )
2019-06-07 18:20:48 +02:00
f63b8bb79d
FIX: Periodically ensure consistency of categories. ( #7663 )
2019-06-06 11:30:52 +02:00
f00275ded3
FEATURE: Support private attachments when using S3 storage ( #7677 )
...
* Support private uploads in S3
* Use localStore for local avatars
* Add job to update private upload ACL on S3
* Test multisite paths
* update ACL for private uploads in migrate_to_s3 task
2019-06-06 13:27:24 +10:00
f7a2648694
FEATURE: Migrate uploads to S3 during restore
2019-06-04 15:47:36 +02:00
fb121815a5
DEV: Allow rebaking oncooked posts of single site
2019-06-04 15:47:36 +02:00
1991af2abb
DEV: Switch `InlineUploads` to a regexp based implementation.
2019-06-04 15:54:25 +08:00
d93e5fb00d
DEV: Class that converts MD with old attachment links to new MD.
2019-06-04 15:54:25 +08:00
af20d616de
FEATURE: new rake task to invalidate broken images in post
2019-06-04 12:40:51 +05:30
81dcadf788
DEV: Stop compiling themes during DB migration. Recompile on cdn change. ( #7676 )
...
This is an improved implementation for bc8b7b13
2019-06-03 16:38:02 +01:00
725588f835
FIX: migrate_to_s3 wasn't IAM profile aware
2019-06-01 12:09:46 -03:00
388433ec06
DEV: remove unused method (rake task deleted in 6a274fd)
2019-05-31 11:16:10 +05:30
5c526e5abb
FEATURE: new rake task to watch all replied topics
2019-05-31 11:08:00 +05:30
6a274fd2cd
DEV: Remove redundant rake task.
...
Onceoff job created in 028121b95b
2019-05-31 10:05:13 +08:00
630e9814bc
datetime is not available at this point ( #7630 )
2019-05-29 14:06:32 +02:00
6439004161
DEV: do not use STDERR to print tests timestamps ( #7629 )
2019-05-29 13:28:02 +02:00
5d47d1833b
DEV: prints timestamp at every docker:test commands ( #7628 )
2019-05-29 12:20:33 +02:00
e12ae453e9
FIX: verify the exitence of s3_object properly without db name
2019-05-29 15:10:36 +05:30
9a9a06e34b
DEV: add option to skip etag verification on 'migrate_to_s3' rake task
2019-05-29 14:16:36 +05:30
b3779dc377
DEV: rename 'uploads:missing' rake task into 'uploads:missing_files'.
...
To improve the readability.
2019-05-28 23:30:43 +05:30
04abf18194
Fixed typo in S3 rake task
2019-05-28 16:57:55 +08:00
9a232e1a0a
FEATURE: use GIVE_UP=1 to inform rake posts:missing_uploads you are done
...
We need this give up for cases where uploads can not be recovered
This also improves the recovery routines
2019-05-28 14:44:59 +10:00
c780d36ddc
DEV: Correct posts:recover_uploads_from_index
...
Was creating a blank index file by mistake
2019-05-28 12:39:39 +10:00
3de8c58134
DEV: clean up backup maxmind path
...
This follows up on 6580025a
2019-05-28 11:47:58 +10:00
7429700389
FIX: ensure we can download maxmind without redis or db config
...
This also corrects FileHelper.download so it supports "follow_redirect"
correctly (it used to always follow 1 redirect) and adds a `validate_url`
param that will bypass all uri validation if set to false (default is true)
2019-05-28 10:28:57 +10:00
e4e2acf148
FIX: rake maxminddb:get broke cause it required environment
2019-05-28 08:59:28 +10:00
6580025af9
FEATURE: add backup directory for mmdb files
...
This new `DISCOURSE_MAXMIND_BACKUP_PATH` can be used a secondary location
for maxmind db. That way a build machine, for example can cache it on the
host and reuse between builds.
Also per 5bfeef77
2019-05-27 16:51:24 +10:00
2f5938c3bf
DEV: use FileHelper to download MaxMindDB
2019-05-24 18:11:39 +02:00
8c9ffee3af
TEMP: log more information when failing to download db
2019-05-24 17:33:52 +02:00
9ed2c54e27
TEMP: log information when failing to download db
2019-05-24 17:26:04 +02:00
0e24cb0f78
DEV: don't skip discourse-calendar plugin specs
2019-05-24 11:35:22 +02:00
9ce06d53d1
DEV: improve upload recovery rake task
...
- Better error handling
- Automatically generates index file if missing
- Improved output
2019-05-24 13:36:03 +10:00
791241e0b2
DEV: Fix rake task to delete old export topics.
2019-05-24 11:20:53 +08:00
76f7e90e23
FEATURE: rake posts:recover_uploads_from_index improvements
...
We now recover more smartly based on failed db remaps as well.
2019-05-24 12:38:38 +10:00
1059aafc55
FEATURE: rake task for recovery of uploads from an index file
...
`rake posts:recover_uploads_from_index`
Searches through all missing uploads in the cluster, if it finds one it
tries to find it in the "upload index file" and creates a new upload for
it.
2019-05-23 17:08:22 +10:00
7ce58df7bf
lint the file
...
somehow I did not notice this on save
2019-05-23 15:28:41 +10:00
a5ce9cb470
FEATURE: fix_relative_upload_links now multisite safety
...
This also finds `<img src="/uploads/xyz` HTML images in raw and corrects
them. Also handles some cross multisite recovery and provides better output
2019-05-23 15:09:16 +10:00
96e58125a7
FEATURE: rename task to posts:destroy_old_data_exports
...
Previously we were only catching one type of data export, the new job will
catch every csv export we have
Job is pretty safe as it filters on system user id / pm with a particular
slug
2019-05-23 15:02:56 +10:00
954293655f
FEATURE: rake posts:destroy_old_user_data_exports
...
Historically we would keep the user data export posts around but delete
the uploads.
This leaves a lot of broken uploads in the system.
This rake task allows us to clean up old mess.
2019-05-23 11:11:37 +10:00
e8799f0ba4
DEV: improve uploads:recover job so it stores a map of old to new sha
...
Previous attempt created broken images
2019-05-22 15:51:09 +10:00
ebcb571de7
FIX: allow upload recovery to recover uploads with sha mismatch
...
Filename on disk may mismatch sha of file in some old 1X setups. This will
attempt to recover file even if sha1 mismatches. We had an old bug that
caused this.
This also adds `uploads:fix_relative_upload_links` which attempts to replace
urls of the format `/upload/default/...` with `upload://`
2019-05-22 15:24:36 +10:00
f772ecc597
DEV: Correct missing output detailing failure
...
uploads:s3_migration_status was failing but not returning proper output
2019-05-22 12:58:54 +10:00
5429c9b5e9
DEV: Fix incorrectly hardcoded value in rake task.
2019-05-22 09:01:25 +08:00
a9af77d650
PERF: ensure priority for rake rebake tasks is ultra low
...
Flooding the default queue with lots of jobs is never a good thing. This
reduces the risk of sidekiq going into a bad state post rebake
2019-05-22 10:31:49 +10:00
d6daa60d0a
FEATURE: multisite support for posts:rebake_uncooked_posts
2019-05-22 10:04:54 +10:00
a2428bd862
FEATURE: warn about sidekiq overload prior to migrating
...
Also makes pre-flight check ensure there is no giant backlog of posts that
need to be cooked
2019-05-22 10:04:33 +10:00
5fdc7b7ca2
Correct 59012fc0
...
Logic was flipped here by mistake, oops
2019-05-22 09:48:03 +10:00
73f178a634
FEATURE: posts:rebake_uncooked_posts to look at mismatching baked_version
...
also amends flagging onebox updates to set baked_version to nil
2019-05-22 09:43:31 +10:00
4f296608da
FEATURE: add uploads:s3_migration_status for looking at current status
...
Also a few minor cleanups and better progress reporting
2019-05-22 09:00:32 +10:00
59012fc0f7
PERF: mark posts for rebake vs forcing a rebake inline when migrating to s3
...
Rebaking posts can be expensive instead of blocking here simply mark posts
for rebake.
We can then work through them faster in other jobs, plus this should not
hold of a datacenter migration.
2019-05-22 08:39:25 +10:00
338b927f95
DEV: include local uploads while finding missing uploads
...
in rake task 'posts:missing_uploads'
2019-05-21 22:07:08 +05:30
7b82850f66
FIX: migrate_to_s3 task should remap attachment links too.
2019-05-21 21:58:11 +05:30
8360415453
FEATURE: big warning for uploads missing which can be very very slow on S3
2019-05-21 16:11:56 +10:00
cb86d8279a
FEATURE: add toggle for uploads:missing so it can skip external
...
Validating s3 uploads in uploads:missing can be very expensive, allow to
bypass.
2019-05-21 16:11:56 +10:00
f4d4f7871e
FEATURE: make posts:missing_uploads multisite friendly
...
Previously this rake job would only run on a single site which is a bit
misleading
This also adds `VERBOSE=1 rake posts:missing_uploads` that will provide a
full report of missing uploads
2019-05-21 12:45:51 +10:00
0fbff66d97
DEV: Correct rake task syntax error
2019-05-20 17:43:30 +01:00
31902159af
DEV: Allow skipping failed migrations when running S3 migration
...
Use the SKIP_FAILED environment variable to skip failed sites. Use with caution - make sure you go back and re-run the failed migrations afterwards.
2019-05-20 17:25:56 +01:00
a15cca9a0f
DEV: Improve error message for posts:missing_uploads during S3 migration
2019-05-20 16:09:22 +01:00
2bfc0cf145
FIX: skip old scheme upload URLs while validating s3 uploads remap
2019-05-20 19:13:41 +05:30
9f43852609
FIX: remove the unwanted code.
2019-05-20 19:10:21 +05:30
41bc90dd3e
DEV: Add progress indicator for post rebake during s3 migration
...
Now that we run sidekiq jobs synchronously, this is important
2019-05-20 14:19:58 +01:00
77a06209cb
DEV: Skip S3 migration if all uploads are already migrated
...
This makes the task resumable in a multisite context
2019-05-20 14:17:37 +01:00
bc0c4b7253
FIX: should not migrate the system uploads to s3
2019-05-20 14:27:34 +05:30
be3fb85a04
DEV: add post migration checks and raise error if failed.
2019-05-20 14:18:28 +05:30
50e6ee55a9
DEV: add "WAIT_FOR_URL" env to smoke test
...
This allows you to wait up to N seconds for the smoke test url to come up
in some cases you want to kick off the smoke test prior to having the smoke
test env ready to accept connections
2019-05-20 17:23:09 +10:00
08371db0cc
FIX: ensure we don't queue any jobs during s3 migration
...
Previously we could flood sidekiq with jobs which is not ideal.
This ensures we are 100% done when the job is done.
2019-05-20 16:28:50 +10:00
9ce9d72e71
DEV: makes hidepassed default when running qunit ( #7558 )
...
Mostly useful when not running headless, but I endup doing it a lot when debugging, one less thing to check.
2019-05-16 14:37:01 +02:00
4a9756ff3f
DEV: sets rack server to test env when using rake qunit:test ( #7554 )
2019-05-16 10:44:29 +02:00
5605dba85c
DEV: Automatically annotate plugin models alongside core models
2019-05-13 16:37:47 +01:00
750c125707
DEV: Include foreign keys in model annotations
2019-05-13 15:53:42 +01:00
c00dab89e4
Fix the build take 2.
2019-05-13 11:22:48 +08:00
8c4caac2e4
Fix frozen string error in `plugin:install_all_official`.
2019-05-13 10:19:02 +08:00
8165ceb320
Make rubocop happy.
2019-05-13 09:55:44 +08:00
30990006a9
DEV: enable frozen string literal on all files
...
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.
Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
20daa76829
DEV: Change priority to ultra_low for post-s3-migration rebake
2019-05-10 18:37:45 +08:00
626808e100
Fix incorrect condition in `groups:grant_badge` rake task.
...
`#find` raises an error if the id given to it is invalid. As a result,
the conditional to check whether a `group` or `badge` is `present?` will
not be executed if any of the ids are invalid.
Follow up to
6ba914033c
2019-05-09 14:53:13 +08:00
87cd4701b8
FEATURE: option to skip posts with ignored missing uploads
2019-05-09 05:11:15 +05:30
Daniel Waterworth
Guo Xiang Tan
Sam Saffron
Penar Musaraj
Joffrey JAFFEUX
Bianca Nenciu
Gerhard Schlager
Arpit Jalan
David Taylor
Rafael dos Santos Silva
Vinoth Kannan
Ralph Rooding
Régis Hanol