Commit Graph

1010 Commits

Author SHA1 Message Date
Krzysztof Kotlarek 858cf5836c
FIX: update Redis gem to version 4.1.3
I run our benchmark on commit with hiredis and redis-4.1.3

Results:
type | hidredis | redis 4.1.3 | percent
--- | --- | --- | ---
Categories-50 | 49 | 50 | 102.04%
Categories-75 | 51 | 51 | 100.00%
Categories-90 | 63 | 64 | 101.59%
Categories-99 | 86 | 85 | 98.84%
Home-50 | 55 | 55 | 100.00%
Home-75 | 56 | 57 | 101.79%
Home-90 | 68 | 69 | 101.47%
Home-99 | 102 | 104 | 101.96%
Topic-50 | 36 | 37 | 102.78%
Topic-75 | 37 | 37 | 100.00%
Topic-90 | 47 | 48 | 102.13%
Topic-99 | 60 | 61 | 101.67%
Categories-admin-50 | 124 | 117 | 94.35%
Categories-admin-75 | 130 | 129 | 99.23%
Categories-admin-90 | 147 | 143 | 97.28%
Categories-admin-99 | 204 | 199 | 97.55%
Home-admin-50 | 146 | 148 | 101.37%
Home-admin-75 | 150 | 152 | 101.33%
Home-admin-90 | 169 | 168 | 99.41%
Home-admin-99 | 232 | 223 | 96.12%
Topic-admin-50 | 60 | 61 | 101.67%
Topic-admin-75 | 64 | 63 | 98.44%
Topic-admin-90 | 76 | 73 | 96.05%
Topic-admin-99 | 124 | 94 | 75.81%
Load rails | 2412 | 2360 | 97.84%
rss | 290204 | 295828 | 101.94%
pss | 277948 | 283624 | 102.04%

Redis gem is manipulating Redis config https://github.com/redis/redis-rb/blob/master/lib/redis/client.rb#L95
therefore we cannot pass the frozen config object.

Pass of the copy of the object is protecting original config
2019-10-21 09:59:24 +11:00
OsamaSayegh 1f6f118e52 DEV: Bump Logster version to 2.4.1
This version includes a few performance fixes, details here: 59f8cb0abf
2019-10-17 20:06:27 +00:00
Sam Saffron ae2a56999e Revert "FIX: update Redis gem to version 4.1.3 (#8197)"
This reverts commit ab74a50d85.

We really want to upgrade redis, but discovered some edge cases
around failover we need to test.

Holding off on the upgrade till a bit more testing happens
2019-10-17 11:41:46 +11:00
Krzysztof Kotlarek ab74a50d85 FIX: update Redis gem to version 4.1.3 (#8197)
* FIX: update Redis gem to version 4.1.3

I run our benchmark on commit with hiredis and redis-4.1.3

Results:
type | hidredis | redis 4.1.3 | percent
--- | --- | --- | ---
Categories-50 | 49 | 50 | 102.04%
Categories-75 | 51 | 51 | 100.00%
Categories-90 | 63 | 64 | 101.59%
Categories-99 | 86 | 85 | 98.84%
Home-50 | 55 | 55 | 100.00%
Home-75 | 56 | 57 | 101.79%
Home-90 | 68 | 69 | 101.47%
Home-99 | 102 | 104 | 101.96%
Topic-50 | 36 | 37 | 102.78%
Topic-75 | 37 | 37 | 100.00%
Topic-90 | 47 | 48 | 102.13%
Topic-99 | 60 | 61 | 101.67%
Categories-admin-50 | 124 | 117 | 94.35%
Categories-admin-75 | 130 | 129 | 99.23%
Categories-admin-90 | 147 | 143 | 97.28%
Categories-admin-99 | 204 | 199 | 97.55%
Home-admin-50 | 146 | 148 | 101.37%
Home-admin-75 | 150 | 152 | 101.33%
Home-admin-90 | 169 | 168 | 99.41%
Home-admin-99 | 232 | 223 | 96.12%
Topic-admin-50 | 60 | 61 | 101.67%
Topic-admin-75 | 64 | 63 | 98.44%
Topic-admin-90 | 76 | 73 | 96.05%
Topic-admin-99 | 124 | 94 | 75.81%
Load rails | 2412 | 2360 | 97.84%
rss | 290204 | 295828 | 101.94%
pss | 277948 | 283624 | 102.04%

* FIX: get rid of redis freedom patch
2019-10-17 08:49:23 +11:00
David Taylor 061c8874f5 FIX: Correct line count link in GitHub commit onebox
Bump onebox version
2019-10-15 23:52:59 +01:00
Sam Saffron c3cc96084c FIX: remove hiredis gem which is no longer needed
Previously some local micro-benchmarks revealed it was not giving any perf
benefits.

Now that we upgraded to 2.6.5 we are seeing some segfaults.

No need to carry this dependency around anymore.

We can re-evaluate in future if it improves perf and fix the segfaults.
2019-10-15 18:17:14 +11:00
romanrizzi 9845963105 FEATURE: Use the 'ugc' rel attribute alongside 'nofollow' 2019-10-14 15:21:48 -03:00
David Taylor 939a746dcd UX: Use theme colors for GitHub issue labels
Bump onebox version to pull tag rendering bug fix
2019-10-09 12:28:48 +01:00
David Taylor 3edd514c72 FEATURE: Redesigned GitHub oneboxes
Bump onebox version, and add new styling

Commit, PR and Issue oneboxes are updated with a new design. Timestamps are now localized using local-dates (if installed).
2019-10-09 11:47:58 +01:00
OsamaSayegh 061b98bc75 DEV: Bump Logster version to 2.3.3
This new version of Logster has a new feature that keeps track of
message timestamp when it's merged into other similar messages.
2019-10-08 16:39:52 +00:00
David Taylor e7cc7def8b UX: Stop using fixed-width font to render github issue description
Bump onebox version
2019-10-08 11:48:05 +01:00
Joffrey JAFFEUX 67a90a7d97 FIX: updates discourse-ember-source gem (#8167)
This is related to fix made to prevent a crash in iOS 9.5
2019-10-08 11:39:20 +11:00
David Taylor 615039f228 FEATURE: Improve GitHub commit, PR and issue onebox rendering
Bump onebox version to include new github rendering, and add relevant CSS

Avatars are reduced in size significantly, and icons are added to easily differentiate PRs and commits. The 'Issue:' prefix is removed from issue oneboxes, to make them consistent with commits and PRs.
2019-10-07 19:26:10 +01:00
Sam Saffron 8d5f47dded PREF: optimise preloading application
We preload to ensure as much memory as possible is reused from unicorn master
to various workers using copy-on-write (sidekiq, unicorn)

This migrates the preloading code into the Discourse module for easier
reuse and adds 3 notable preloading changes

1. We attempt to localize a string on each site, ensuring we warmup
the i18n

2. We preload all our templates (compiling .erb to class)

3. We warm-up our search tokenizer which uses cppjieba which is a large
memory consumer, this will only cause a warmup on CJK sites or sites with
the special site setting enabled.
2019-10-07 00:33:37 -04:00
Martin Brennan 68d35b14f4 FEATURE: Webauthn authenticator management with 2FA login (Security Keys) (#8099)
Adds 2 factor authentication method via second factor security keys over [web authn](https://developer.mozilla.org/en-US/docs/Web/API/Web_Authentication_API).

Allows a user to authenticate a second factor on login, login-via-email, admin-login, and change password routes. Adds registration area within existing user second factor preferences to register multiple security keys. Supports both external (yubikey) and built-in (macOS/android fingerprint readers).
2019-10-01 19:08:41 -07:00
Sam Saffron 0420e8145e SECURITY: update rubyzip dependency
This updates rubyzip library so that callers can trust entries when
extracting files avoiding situations where a rogues zip imported by a rogue
admin could cause a disk space issue.
2019-10-01 17:11:20 +10:00
Sam Saffron ba0114a6ff SECURITY: update rack-mini-profiler to latest to correct XSS
This corrects an XSS in ?pp=help.

Also removes the jQuery dependency from rack-mini-profiler and restricts
memory sensitive profiling methods development only.
2019-10-01 16:55:58 +10:00
Krzysztof Kotlarek 32b8a2ccff DEV: Upgrade Discourse to Rails 6 (#8083)
* Adjustments to pass specs on Rails 6.0.0
* Use classic autoloader instead of Zeitwerk
* Update Rails 6.0.0 deprecated methods
* Rails 6.0.0 not allowing column with integer name
* Drop freedom_patches/rails6.rb
* Default value for trigger_transactional_callbacks? is true
* Bump rspec-rails version to 4.0.0.beta2
2019-09-12 10:41:50 +10:00
Arpit Jalan 4195548a17 Bump onebox version.
- indicate and link to Flickr Album
2019-09-11 23:23:11 +05:30
Sam Saffron 5da7ffd46c FEATURE: update mini_scheduler to support history filtering
New version of mini scheduler allows you to select the name of a schedule
in the history page in `/sidekiq/scheduler/history`.

This is handy for quickly looking up timing trends.
2019-09-11 18:43:02 +10:00
Sam Saffron ed00f35306 FEATURE: improve performance of anonymous cache
This commit introduces 2 features:

1. DISCOURSE_COMPRESS_ANON_CACHE (true|false, default false): this allows
you to optionally compress the anon cache body entries in Redis, can be
useful for high load sites with Redis that lives on a separate server to
to webs

2. DISCOURSE_ANON_CACHE_STORE_THRESHOLD (default 2), only pop entries into
redis if we observe them more than N times. This avoids situations where
a crawler can walk a big pile of topics and store them all in Redis never
to be used. Our default anon cache time for topics is only 60 seconds. Anon
cache is in place to avoid the "slashdot" effect where a single topic is
hit by 100s of people in one minute.
2019-09-04 17:18:32 +10:00
Sam Saffron 5db204f370 FIX: broken scheduler when changing per_host <-> global
Prior to this commit if we ever checked in a per_host directive and scheduled
job was previously global, total chaos could ensue as a feedback loop would
start

We very rarely used per_host to date, but just started making use of it
for heartbeats
2019-08-30 23:26:44 +10:00
Sam Saffron 098f9e8b5b PERF: Run multiple threads for regular job schedules
Under extreme load on large databases certain regular jobs can take quite
a while to run. We need to ensure we never starve a sidekiq from running
mini scheduler, cause without it we are unable to queue stuff such as
heartbeat jobs.
2019-08-29 15:34:36 +10:00
Arpit Jalan e9c971ba77 Bump onebox version.
- allow oneboxing for `www.amazon.com.mx`
2019-08-26 16:44:10 +05:30
Arpit Jalan 038bf02e33 Bump onebox version.
- strip whitespace from Twitter onebox
2019-08-21 10:19:54 +05:30
Régis Hanol 727430aacf SECURITY: bump nokogiri
We're not using the vulnerable method but there's no harm in upgrading.
2019-08-20 10:58:18 +02:00
Sam Saffron 8db38de9d7 SECURITY: add rate limiting to anon JS error reporting
This adds a 1 minute rate limit to all JS error reporting per IP. Previously
we would only use the global rate limit.

This also introduces DISCOURSE_ENABLE_JS_ERROR_REPORTING, if it is set to
false then no JS error reporting will be allowed on the site.
2019-08-20 11:29:11 +10:00
Arpit Jalan f27564a0a0 Bump onebox version.
- normalize and decode html entities for image URL
2019-08-19 19:01:23 +05:30
OsamaSayegh 50368940f7 DEV: Bump Logster version to 2.3.1
Commits since last version bump: 5cdcb76...c5bcb8e
2019-08-15 07:21:34 +00:00
Arpit Jalan 44f4801087 Bump onebox version.
- do not double encode percentage in url
- support hashbang in url
2019-08-12 08:42:50 +05:30
Sam Saffron 1f47ed1ea3 PERF: message_bus will be deferred by server when flooded
The message_bus performs a fair amount of work prior to hijacking requests
this change ensures that if there is a situation where the server is flooded
message_bus will inform client to back off for 30 seconds + random(120 secs)

This back-off is ultra cheap and happens very early in the middleware.

It corrects a situation where a flood to message bus could cause the app
to become unresponsive

MessageBus update is here to ensure message_bus gem properly respects
Retry-After header and status 429.

Under normal state this code should never trigger, to disable raise the
value of DISCOURSE_REJECT_MESSAGE_BUS_QUEUE_SECONDS, default is to tell
message bus to go away if we are queueing for 100ms or longer
2019-08-09 17:48:01 +10:00
Arpit Jalan bc5daa1466 Bump onebox version.
- prioritize `card_html` over `article_html`
2019-08-05 11:04:58 +05:30
Neil Lalonde 9656a21fdb
FEATURE: customization of html emails (#7934)
This feature adds the ability to customize the HTML part of all emails using a custom HTML template and optionally some CSS to style it. The CSS will be parsed and converted into inline styles because CSS is poorly supported by email clients. When writing the custom HTML and CSS, be aware of what email clients support. Keep customizations very simple.

Customizations can be added and edited in Admin > Customize > Email Style.

Since the summary email is already heavily styled, there is a setting to disable custom styles for summary emails called "apply custom styles to digest" found in Admin > Settings > Email.

As part of this work, RTL locales are now rendered correctly for all emails.
2019-07-30 15:05:08 -04:00
Arpit Jalan 6a0787445c Bump onebox version.
- Deprioritize Twitter card in generic onebox
2019-07-25 17:13:23 +05:30
Roman Rizzi f5c707c97a
FEATURE: Gz to zip for exports (#7889)
* Revert "Revert "FEATURE: admin/user exports are compressed using the zip format (#7784)""

This reverts commit f89bd55576.

* Replace .tar.zip with .zip
2019-07-18 09:34:48 -03:00
Joffrey JAFFEUX b3eb67976d
DEV: Upgrades to Ember 3.10 (#7871)
Co-Authored-By: majakomel <maja.komel@gmail.com>
2019-07-16 12:45:15 +02:00
Arpit Jalan 25830c73be Bump onebox version.
- use custom placeholder HTML for generic whitelisted oneboxes
- optimize usage of custom placeholder HTML
2019-07-11 18:31:51 +05:30
romanrizzi f89bd55576 Revert "FEATURE: admin/user exports are compressed using the zip format (#7784)"
This reverts commit 8b2580e20f.
2019-07-10 11:38:51 -03:00
Roman Rizzi 8b2580e20f
FEATURE: admin/user exports are compressed using the zip format (#7784)
* FEATURE: admin/user exports are compressed using the zip format

* Update translations. Theme exporter now exports .zip file. Theme importer supports .zip and .gz files

* Fix controller test, updated locale and skip saving the csv export to disk
2019-07-10 11:13:03 -03:00
Arpit Jalan f0f271cd5f Bump onebox version.
- remove additional whitespace from Twitter onebox
2019-07-09 13:12:03 +05:30
Arpit Jalan bb8cf81089 Bump onebox version.
- better placeholders for audio/video/trello/typeform oneboxes
- added CSS for audio/video/trello/typeform onebox placeholders
2019-07-08 21:40:33 +05:30
Arpit Jalan feb828172b Bump onebox version.
- improved spacing for quoted twitter onebox
2019-07-06 09:41:01 +05:30
Arpit Jalan 5bc1fd23b0 Bump onebox version.
- update HTML for twitter quoted onebox
- updated CSS for twitter quoted onebox
2019-07-05 19:35:36 +05:30
Arpit Jalan 1ebc3cce4a Bump onebox version.
- twitter oneboxing using API was broken
2019-07-04 11:41:07 +05:30
Arpit Jalan 32edaa3843 Bump onebox version
- support for Twitter quoted tweets
2019-07-04 11:01:30 +05:30
Abroskin Alexander 0872a1182d DEV: Replace Overcommit with Lefthook (#7826)
Overcommit uses prebuilt hooks and require global installation.
To avoid this issues replace it with Lefthook.
Lefthook will be installed with npm packages. New contributors
will have fully consistent git hooks.
2019-07-02 11:29:52 +02:00
Arpit Jalan 7f14e185cc Bump onebox version.
- prevent double escaping of URL
2019-06-27 23:35:35 +05:30
Arpit Jalan 168a38dc29 Bump onebox version.
- better Twitch placeholder
- CSS for said placeholder
2019-06-26 23:22:29 +05:30
Arpit Jalan 2af4002817 Bump onebox version.
- Add a placeholder icon for Twitch onebox preview
- Add CSS for showing fontawesome play icon for placeholder class
2019-06-26 13:40:14 +05:30
Sam Saffron 4c89f7f881 DEV: bump version on mini_scheduler
This corrects a catastrophic state that can ensue if redis becomes readonly

It also adds support for multiple queues and minor cleanup
2019-06-24 16:56:16 +10:00