Commit Graph

1032 Commits

Author SHA1 Message Date
Sam Saffron ed6d3b493c DEV: update dependencies
This updates some low risk dependencies. Mostly minor changes.

Public suffix now requires Ruby 2.3 and up which is not a problem for us.
2019-12-09 19:20:45 +11:00
dependabot-preview[bot] 9b30922109 Build(deps): Bump rotp from 3.3.1 to 5.1.0 (#8477)
* Build(deps): Bump rotp from 3.3.1 to 5.1.0

Bumps [rotp](https://github.com/mdp/rotp) from 3.3.1 to 5.1.0.
- [Release notes](https://github.com/mdp/rotp/releases)
- [Changelog](https://github.com/mdp/rotp/blob/master/CHANGELOG.md)
- [Commits](https://github.com/mdp/rotp/compare/v3.3.1...v5.1.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Fix breaking ROTP changes
2019-12-09 14:29:48 +10:00
dependabot-preview[bot] fca727fd92 DEV: Bump tilt from 2.0.9 to 2.0.10 (#8479)
Minor update, removes a Ruby 2.7 deprecation.
2019-12-09 14:22:55 +11:00
dependabot-preview[bot] c853051edb DEV: Bump mustache from 1.1.0 to 1.1.1 (#8478)
Minor update, removes one warning which we have not seen yet.
2019-12-09 14:21:55 +11:00
dependabot-preview[bot] 7aa3233bcb DEV: Bump puma from 3.12.2 to 4.3.1 (#8475)
The downside here is that we are now going to require nio4r which requires some native code.

It is reasonably low risk though cause puma is very well tested. We use puma mostly in test environments, not in production.
2019-12-09 12:51:41 +11:00
dependabot-preview[bot] d198e824ce DEV: Bump fastimage from 2.1.5 to 2.1.7
This library is used to detect size of images, upgrading a minor version here should be low risk, we have integration tests to detect issues.
2019-12-09 12:49:53 +11:00
Sam Saffron 8a6421565e DEV: upgrade low risk gems
This updates a bunch of gems that have very low risk of breaking anything
or are mostly used in dev. Overall our goal is to be on latest gems with
latest bug fixes so this is us working in that direction.
2019-12-06 17:12:43 +11:00
dependabot[bot] cc92aa9e71 SECURITY: Bump puma from 3.12.1 to 3.12.2 (#8464)
Bumps [puma](https://github.com/puma/puma) from 3.12.1 to 3.12.2.
- [Release notes](https://github.com/puma/puma/releases)
- [Changelog](https://github.com/puma/puma/blob/master/History.md)
- [Commits](https://github.com/puma/puma/compare/v3.12.1...v3.12.2)

Signed-off-by: dependabot[bot] <support@github.com>
2019-12-06 14:09:29 +11:00
Sam Saffron a06fccae1b DEV: update dependencies and add notes about exceptions
Previously it was unclear why certain gems are being held back cause Gemfile
had no comment explaining it.

I tried to add some explanation from memory and remove some exceptions that
seemed to be superfluous.

This upgrades shoulda to latest, it appears to work once a couple of assertions
are removed

Also update http accept language used to auto detect language from http header
this is tested

Zeitwerk small update seems fine
2019-12-06 13:00:28 +11:00
Arpit Jalan cab9c7c77e Bump onebox version.
- FIX: use dedicated Vimeo onebox for all video types
2019-11-27 16:22:25 +05:30
Arpit Jalan 7543db086a Bump onebox version.
- FIX: Amazon video oneboxes were not working.
2019-11-20 14:47:59 +05:30
David Taylor eaf6096890 DEV: Use rubocop-discourse gem to add custom chdir cop
Followup to b27e009655
2019-11-18 15:39:41 +00:00
Gerhard Schlager c04369ed8f
DEV: Revert to Bundler 1.17
Follow-up to a2d6169a5e
2019-11-11 15:37:13 +01:00
Sam Saffron a2d6169a5e DEV: upgrade mini_racer
This is done to fix a compatibility issue with Ruby 2.7, it ensures we clean
up fds for a pipe we use to track timeouts in mini racer.
2019-11-11 15:36:16 +11:00
Sam Saffron 652b6363a2 DEV: upgrade bootsnap
This fixes Ruby 2.7 support which we are starting to test
2019-11-08 17:07:58 +11:00
Sam Saffron 26c0199c01 DEV: update Rails to version 6.0.1
This version of Rails eliminates a monkey patch that is no longer needed!

Additionally it preps us for Ruby 2.7 support.
2019-11-08 16:56:30 +11:00
Mark VanLandingham f79796fcac DEV: Bump loofah version due to vulnerability 2019-11-07 10:02:02 -05:00
Arpit Jalan c5df853dea Bump onebox version.
- fix for gfycat onebox in email
2019-11-07 10:03:12 +05:30
Arpit Jalan cb9702bf7a Bump onebox version.
- Remove native caching
- FIX: dropbox videos were not loading
2019-11-04 10:46:20 +05:30
Sam Saffron af841fa883 DEV: update rack-mini-profiler
This includes an important new feature, we pre-compile templates so CSPs
that disable eval can still apply to our sites.
2019-10-28 16:46:13 +11:00
Sam Saffron c9714fcbf8 FIX: update rack-mini-profiler
1.1.0 had regressions where rack mini profiler would break the site for IE11
users cause the payload had errors.

1.1.2 fixes that.
2019-10-25 11:17:44 +11:00
Arpit Jalan 12409f63a0 Bump onebox version.
- FIX: Follow redirect returns url if response code is 200
- FIX: do not resize xkcd image
2019-10-22 12:26:01 +05:30
Krzysztof Kotlarek 858cf5836c
FIX: update Redis gem to version 4.1.3
I run our benchmark on commit with hiredis and redis-4.1.3

Results:
type | hidredis | redis 4.1.3 | percent
--- | --- | --- | ---
Categories-50 | 49 | 50 | 102.04%
Categories-75 | 51 | 51 | 100.00%
Categories-90 | 63 | 64 | 101.59%
Categories-99 | 86 | 85 | 98.84%
Home-50 | 55 | 55 | 100.00%
Home-75 | 56 | 57 | 101.79%
Home-90 | 68 | 69 | 101.47%
Home-99 | 102 | 104 | 101.96%
Topic-50 | 36 | 37 | 102.78%
Topic-75 | 37 | 37 | 100.00%
Topic-90 | 47 | 48 | 102.13%
Topic-99 | 60 | 61 | 101.67%
Categories-admin-50 | 124 | 117 | 94.35%
Categories-admin-75 | 130 | 129 | 99.23%
Categories-admin-90 | 147 | 143 | 97.28%
Categories-admin-99 | 204 | 199 | 97.55%
Home-admin-50 | 146 | 148 | 101.37%
Home-admin-75 | 150 | 152 | 101.33%
Home-admin-90 | 169 | 168 | 99.41%
Home-admin-99 | 232 | 223 | 96.12%
Topic-admin-50 | 60 | 61 | 101.67%
Topic-admin-75 | 64 | 63 | 98.44%
Topic-admin-90 | 76 | 73 | 96.05%
Topic-admin-99 | 124 | 94 | 75.81%
Load rails | 2412 | 2360 | 97.84%
rss | 290204 | 295828 | 101.94%
pss | 277948 | 283624 | 102.04%

Redis gem is manipulating Redis config https://github.com/redis/redis-rb/blob/master/lib/redis/client.rb#L95
therefore we cannot pass the frozen config object.

Pass of the copy of the object is protecting original config
2019-10-21 09:59:24 +11:00
OsamaSayegh 1f6f118e52 DEV: Bump Logster version to 2.4.1
This version includes a few performance fixes, details here: 59f8cb0abf
2019-10-17 20:06:27 +00:00
Sam Saffron ae2a56999e Revert "FIX: update Redis gem to version 4.1.3 (#8197)"
This reverts commit ab74a50d85.

We really want to upgrade redis, but discovered some edge cases
around failover we need to test.

Holding off on the upgrade till a bit more testing happens
2019-10-17 11:41:46 +11:00
Krzysztof Kotlarek ab74a50d85 FIX: update Redis gem to version 4.1.3 (#8197)
* FIX: update Redis gem to version 4.1.3

I run our benchmark on commit with hiredis and redis-4.1.3

Results:
type | hidredis | redis 4.1.3 | percent
--- | --- | --- | ---
Categories-50 | 49 | 50 | 102.04%
Categories-75 | 51 | 51 | 100.00%
Categories-90 | 63 | 64 | 101.59%
Categories-99 | 86 | 85 | 98.84%
Home-50 | 55 | 55 | 100.00%
Home-75 | 56 | 57 | 101.79%
Home-90 | 68 | 69 | 101.47%
Home-99 | 102 | 104 | 101.96%
Topic-50 | 36 | 37 | 102.78%
Topic-75 | 37 | 37 | 100.00%
Topic-90 | 47 | 48 | 102.13%
Topic-99 | 60 | 61 | 101.67%
Categories-admin-50 | 124 | 117 | 94.35%
Categories-admin-75 | 130 | 129 | 99.23%
Categories-admin-90 | 147 | 143 | 97.28%
Categories-admin-99 | 204 | 199 | 97.55%
Home-admin-50 | 146 | 148 | 101.37%
Home-admin-75 | 150 | 152 | 101.33%
Home-admin-90 | 169 | 168 | 99.41%
Home-admin-99 | 232 | 223 | 96.12%
Topic-admin-50 | 60 | 61 | 101.67%
Topic-admin-75 | 64 | 63 | 98.44%
Topic-admin-90 | 76 | 73 | 96.05%
Topic-admin-99 | 124 | 94 | 75.81%
Load rails | 2412 | 2360 | 97.84%
rss | 290204 | 295828 | 101.94%
pss | 277948 | 283624 | 102.04%

* FIX: get rid of redis freedom patch
2019-10-17 08:49:23 +11:00
David Taylor 061c8874f5 FIX: Correct line count link in GitHub commit onebox
Bump onebox version
2019-10-15 23:52:59 +01:00
Sam Saffron c3cc96084c FIX: remove hiredis gem which is no longer needed
Previously some local micro-benchmarks revealed it was not giving any perf
benefits.

Now that we upgraded to 2.6.5 we are seeing some segfaults.

No need to carry this dependency around anymore.

We can re-evaluate in future if it improves perf and fix the segfaults.
2019-10-15 18:17:14 +11:00
romanrizzi 9845963105 FEATURE: Use the 'ugc' rel attribute alongside 'nofollow' 2019-10-14 15:21:48 -03:00
David Taylor 939a746dcd UX: Use theme colors for GitHub issue labels
Bump onebox version to pull tag rendering bug fix
2019-10-09 12:28:48 +01:00
David Taylor 3edd514c72 FEATURE: Redesigned GitHub oneboxes
Bump onebox version, and add new styling

Commit, PR and Issue oneboxes are updated with a new design. Timestamps are now localized using local-dates (if installed).
2019-10-09 11:47:58 +01:00
OsamaSayegh 061b98bc75 DEV: Bump Logster version to 2.3.3
This new version of Logster has a new feature that keeps track of
message timestamp when it's merged into other similar messages.
2019-10-08 16:39:52 +00:00
David Taylor e7cc7def8b UX: Stop using fixed-width font to render github issue description
Bump onebox version
2019-10-08 11:48:05 +01:00
Joffrey JAFFEUX 67a90a7d97 FIX: updates discourse-ember-source gem (#8167)
This is related to fix made to prevent a crash in iOS 9.5
2019-10-08 11:39:20 +11:00
David Taylor 615039f228 FEATURE: Improve GitHub commit, PR and issue onebox rendering
Bump onebox version to include new github rendering, and add relevant CSS

Avatars are reduced in size significantly, and icons are added to easily differentiate PRs and commits. The 'Issue:' prefix is removed from issue oneboxes, to make them consistent with commits and PRs.
2019-10-07 19:26:10 +01:00
Sam Saffron 8d5f47dded PREF: optimise preloading application
We preload to ensure as much memory as possible is reused from unicorn master
to various workers using copy-on-write (sidekiq, unicorn)

This migrates the preloading code into the Discourse module for easier
reuse and adds 3 notable preloading changes

1. We attempt to localize a string on each site, ensuring we warmup
the i18n

2. We preload all our templates (compiling .erb to class)

3. We warm-up our search tokenizer which uses cppjieba which is a large
memory consumer, this will only cause a warmup on CJK sites or sites with
the special site setting enabled.
2019-10-07 00:33:37 -04:00
Martin Brennan 68d35b14f4 FEATURE: Webauthn authenticator management with 2FA login (Security Keys) (#8099)
Adds 2 factor authentication method via second factor security keys over [web authn](https://developer.mozilla.org/en-US/docs/Web/API/Web_Authentication_API).

Allows a user to authenticate a second factor on login, login-via-email, admin-login, and change password routes. Adds registration area within existing user second factor preferences to register multiple security keys. Supports both external (yubikey) and built-in (macOS/android fingerprint readers).
2019-10-01 19:08:41 -07:00
Sam Saffron 0420e8145e SECURITY: update rubyzip dependency
This updates rubyzip library so that callers can trust entries when
extracting files avoiding situations where a rogues zip imported by a rogue
admin could cause a disk space issue.
2019-10-01 17:11:20 +10:00
Sam Saffron ba0114a6ff SECURITY: update rack-mini-profiler to latest to correct XSS
This corrects an XSS in ?pp=help.

Also removes the jQuery dependency from rack-mini-profiler and restricts
memory sensitive profiling methods development only.
2019-10-01 16:55:58 +10:00
Krzysztof Kotlarek 32b8a2ccff DEV: Upgrade Discourse to Rails 6 (#8083)
* Adjustments to pass specs on Rails 6.0.0
* Use classic autoloader instead of Zeitwerk
* Update Rails 6.0.0 deprecated methods
* Rails 6.0.0 not allowing column with integer name
* Drop freedom_patches/rails6.rb
* Default value for trigger_transactional_callbacks? is true
* Bump rspec-rails version to 4.0.0.beta2
2019-09-12 10:41:50 +10:00
Arpit Jalan 4195548a17 Bump onebox version.
- indicate and link to Flickr Album
2019-09-11 23:23:11 +05:30
Sam Saffron 5da7ffd46c FEATURE: update mini_scheduler to support history filtering
New version of mini scheduler allows you to select the name of a schedule
in the history page in `/sidekiq/scheduler/history`.

This is handy for quickly looking up timing trends.
2019-09-11 18:43:02 +10:00
Sam Saffron ed00f35306 FEATURE: improve performance of anonymous cache
This commit introduces 2 features:

1. DISCOURSE_COMPRESS_ANON_CACHE (true|false, default false): this allows
you to optionally compress the anon cache body entries in Redis, can be
useful for high load sites with Redis that lives on a separate server to
to webs

2. DISCOURSE_ANON_CACHE_STORE_THRESHOLD (default 2), only pop entries into
redis if we observe them more than N times. This avoids situations where
a crawler can walk a big pile of topics and store them all in Redis never
to be used. Our default anon cache time for topics is only 60 seconds. Anon
cache is in place to avoid the "slashdot" effect where a single topic is
hit by 100s of people in one minute.
2019-09-04 17:18:32 +10:00
Sam Saffron 5db204f370 FIX: broken scheduler when changing per_host <-> global
Prior to this commit if we ever checked in a per_host directive and scheduled
job was previously global, total chaos could ensue as a feedback loop would
start

We very rarely used per_host to date, but just started making use of it
for heartbeats
2019-08-30 23:26:44 +10:00
Sam Saffron 098f9e8b5b PERF: Run multiple threads for regular job schedules
Under extreme load on large databases certain regular jobs can take quite
a while to run. We need to ensure we never starve a sidekiq from running
mini scheduler, cause without it we are unable to queue stuff such as
heartbeat jobs.
2019-08-29 15:34:36 +10:00
Arpit Jalan e9c971ba77 Bump onebox version.
- allow oneboxing for `www.amazon.com.mx`
2019-08-26 16:44:10 +05:30
Arpit Jalan 038bf02e33 Bump onebox version.
- strip whitespace from Twitter onebox
2019-08-21 10:19:54 +05:30
Régis Hanol 727430aacf SECURITY: bump nokogiri
We're not using the vulnerable method but there's no harm in upgrading.
2019-08-20 10:58:18 +02:00
Sam Saffron 8db38de9d7 SECURITY: add rate limiting to anon JS error reporting
This adds a 1 minute rate limit to all JS error reporting per IP. Previously
we would only use the global rate limit.

This also introduces DISCOURSE_ENABLE_JS_ERROR_REPORTING, if it is set to
false then no JS error reporting will be allowed on the site.
2019-08-20 11:29:11 +10:00
Arpit Jalan f27564a0a0 Bump onebox version.
- normalize and decode html entities for image URL
2019-08-19 19:01:23 +05:30