Commit Graph

6225 Commits

Author SHA1 Message Date
Sam Saffron 73f178a634 FEATURE: posts:rebake_uncooked_posts to look at mismatching baked_version
also amends flagging onebox updates to set baked_version to nil
2019-05-22 09:43:31 +10:00
Sam Saffron 4f296608da FEATURE: add uploads:s3_migration_status for looking at current status
Also a few minor cleanups and better progress reporting
2019-05-22 09:00:32 +10:00
Sam Saffron 59012fc0f7 PERF: mark posts for rebake vs forcing a rebake inline when migrating to s3
Rebaking posts can be expensive instead of blocking here simply mark posts
for rebake.

We can then work through them faster in other jobs, plus this should not
hold of a datacenter migration.
2019-05-22 08:39:25 +10:00
Vinoth Kannan 338b927f95 DEV: include local uploads while finding missing uploads
in rake task 'posts:missing_uploads'
2019-05-21 22:07:08 +05:30
Vinoth Kannan 7b82850f66 FIX: migrate_to_s3 task should remap attachment links too. 2019-05-21 21:58:11 +05:30
Sam Saffron 8360415453 FEATURE: big warning for uploads missing which can be very very slow on S3 2019-05-21 16:11:56 +10:00
Sam Saffron cb86d8279a FEATURE: add toggle for uploads:missing so it can skip external
Validating s3 uploads in uploads:missing can be very expensive, allow to
bypass.
2019-05-21 16:11:56 +10:00
Sam Saffron f4d4f7871e FEATURE: make posts:missing_uploads multisite friendly
Previously this rake job would only run on a single site which is a bit
misleading

This also adds `VERBOSE=1 rake posts:missing_uploads` that will provide a
full report of missing uploads
2019-05-21 12:45:51 +10:00
Sam Saffron 0f76e3090c FIX: use tar -xzvf to extract maxmind vs Ruby wrappers
The tar/gz extractor in Ruby is somewhat buggy, instead rely on battle
tested `tar` and `gzip` Linux commands
2019-05-21 10:48:18 +10:00
Vinoth Kannan 2941c77abc FIX: skip upload recovery if file not found in s3 2019-05-21 00:06:36 +05:30
David Taylor 0fbff66d97 DEV: Correct rake task syntax error 2019-05-20 17:43:30 +01:00
David Taylor 31902159af DEV: Allow skipping failed migrations when running S3 migration
Use the SKIP_FAILED environment variable to skip failed sites. Use with caution - make sure you go back and re-run the failed migrations afterwards.
2019-05-20 17:25:56 +01:00
David Taylor a15cca9a0f DEV: Improve error message for posts:missing_uploads during S3 migration 2019-05-20 16:09:22 +01:00
Vinoth Kannan 2bfc0cf145 FIX: skip old scheme upload URLs while validating s3 uploads remap 2019-05-20 19:13:41 +05:30
Vinoth Kannan 9f43852609 FIX: remove the unwanted code. 2019-05-20 19:10:21 +05:30
David Taylor 41bc90dd3e DEV: Add progress indicator for post rebake during s3 migration
Now that we run sidekiq jobs synchronously, this is important
2019-05-20 14:19:58 +01:00
David Taylor 77a06209cb DEV: Skip S3 migration if all uploads are already migrated
This makes the task resumable in a multisite context
2019-05-20 14:17:37 +01:00
Gerhard Schlager b788948985 FEATURE: English locale with international date formats
Makes en_US the new default locale
2019-05-20 13:47:20 +02:00
Vinoth Kannan bc0c4b7253 FIX: should not migrate the system uploads to s3 2019-05-20 14:27:34 +05:30
Vinoth Kannan be3fb85a04 DEV: add post migration checks and raise error if failed. 2019-05-20 14:18:28 +05:30
Sam Saffron 50e6ee55a9 DEV: add "WAIT_FOR_URL" env to smoke test
This allows you to wait up to N seconds for the smoke test url to come up
in some cases you want to kick off the smoke test prior to having the smoke
test env ready to accept connections
2019-05-20 17:23:09 +10:00
Sam Saffron 08371db0cc FIX: ensure we don't queue any jobs during s3 migration
Previously we could flood sidekiq with jobs which is not ideal.

This ensures we are 100% done when the job is done.
2019-05-20 16:28:50 +10:00
Vinoth Kannan 2a7065c505 FIX: skip uploads without etag in s3 inventory check. 2019-05-20 00:09:52 +05:30
Régis Hanol 7696b92c8c FIX: handle clicks counters in full quotes 2019-05-17 14:17:29 +02:00
David Taylor ef660d5a3e FIX: Return consistent character encodings when downloading S3 uploads
Net::HTTP always returns ASCII-8BIT encoding. File.read auto-detects the encoding. This leads to an encoding inconsistency between a fresh download, and a cached download. This commit ensures all downloaded files are treated equally, by always returning the cached version from the filesystem, even during initial download.

One symptom of this problem is during theme exports: https://meta.discourse.org/t/116907

Related ruby ticket: https://bugs.ruby-lang.org/issues/2567
2019-05-17 11:27:00 +01:00
David Taylor 105359e54a FIX: Only delete theme folder if it has been created
If an exception is raised before the folder is created, then this `ensure` block is still run
2019-05-17 09:45:11 +01:00
Guo Xiang Tan 148bfc9be5 DEV: Simplify client and server side code to support removing tags.
Follow up to 834c86678f.
2019-05-17 16:39:20 +08:00
David Taylor 5666316285 FEATURE: Display 'last updated' on dashboard, improve release notes link (#7560) 2019-05-17 15:42:45 +10:00
David Taylor b5dd4478e5
FIX: Blocked watched words should apply to staff (#7547)
Having different behavior for staff and regular users can make it confusing for admins to understand how their configuration changes affect regular users
2019-05-16 15:19:41 +01:00
Joffrey JAFFEUX 9ce9d72e71
DEV: makes hidepassed default when running qunit (#7558)
Mostly useful when not running headless, but I endup doing it a lot when debugging, one less thing to check.
2019-05-16 14:37:01 +02:00
Vinoth Kannan 3172172b52 remove unused local variable
ec84c87ddb
2019-05-16 15:39:13 +05:30
Vinoth Kannan ec84c87ddb FIX: skip validation while recovering uploads from s3
TODO: add tests
2019-05-16 15:37:11 +05:30
Régis Hanol 5dd97d6d38
DEV: prevent already defined global warning 2019-05-16 11:32:10 +02:00
Joffrey JAFFEUX 4a9756ff3f
DEV: sets rack server to test env when using rake qunit:test (#7554) 2019-05-16 10:44:29 +02:00
Sam Saffron cabc203885 Followup to a8fbb19e
Correct edge condition where no available names are found

- increase search space
- search for 10 extra names
2019-05-16 18:15:56 +10:00
Sam Saffron a8fbb19e7c FEATURE: allow a huge number of users to share common prefix
Previously username suggester would give up after 100 attempts at getting
a username and fallback to random string.

This amends the logic so we do all the work of figuring out a good username
in SQL and avoids a large amount of queries in cases where a lot of usernames
were used up.

This corrects an issue on sites with large numbers of anon users
2019-05-16 17:15:16 +10:00
Vinoth Kannan 40328f055e FIX: retrieve original filename from s3 object's content disposition header 2019-05-16 09:47:22 +05:30
Guo Xiang Tan dd49be27d3 DEV: Fix undefined variable.
Follow up to e8fafbc123.
2019-05-16 11:28:48 +08:00
Régis Hanol fd5c5e326f FIX: remove full quote on direct replies when "typographed"
Use the cooked version of the post and the quote to compare their content in
order to take into account the "typographer" option of the markdown pipeline.
2019-05-15 17:49:29 +02:00
Vinoth Kannan be0555cc17 FIX: Add bucket folder path only if not exists 2019-05-15 15:37:40 +05:30
Neil Lalonde 6f747c6b71 FIX: don't allow username to be changed to same as password
We were blocking user registrations with same username and password,
but allowing usernames to be changed to be same as password later.
Also disallow names to be the same as password.
2019-05-13 16:43:40 -04:00
David Taylor 5605dba85c DEV: Automatically annotate plugin models alongside core models 2019-05-13 16:37:47 +01:00
David Taylor 750c125707 DEV: Include foreign keys in model annotations 2019-05-13 15:53:42 +01:00
Guo Xiang Tan 999fdb95db DEV: Mark `discourse-login-with-amazon` as official plugin. 2019-05-13 16:13:20 +08:00
Gerhard Schlager 427cf295d9 DEV: Always require %{count} in French pluralized strings 2019-05-13 09:36:05 +02:00
Gerhard Schlager 6d44be5142 FIX: Catch InvalidPluralizationData exception in fallback locales
It shouldn't raise an exception when a pluralized string in a fallback locale is only partially translated.
2019-05-13 09:36:05 +02:00
Gerhard Schlager a3e4f43a4d DEV: Remove patch for I18n.exists?
It was merged upstream a long time ago
2019-05-13 09:36:05 +02:00
Gerhard Schlager 73015521e2 FIX: Correctly calculate fallback locale list
* English shouldn't fallback to any other locale
* Calculate fallback for default locale if it isn't English (useful for en_US)
* Reuse the fallback locale list when outputting translations to JavaScript
2019-05-13 09:36:05 +02:00
Sam Saffron 3fbab595d9 FEATURE: use the exact name provided by auth provider
Previously we would attempt to titleize it causing names like McDonald's
not to work as expected
2019-05-13 16:45:24 +10:00
Guo Xiang Tan ba913a24db Add steam login to official plugin list. 2019-05-13 12:39:35 +08:00