Commit Graph

716 Commits

Author SHA1 Message Date
Jarek Radosz fb25333b7f
DEV: Update oj gem (#15713)
https://github.com/ohler55/oj/issues/699 was fixed back in September 2021.
2022-02-01 09:45:52 +11:00
Vinoth Kannan 44b3b614bd
Revert "DEV: Avoid duplication of gems in gemfile." (#14784)
This reverts commit 2818ebd19f.
2021-11-01 17:58:24 +05:30
Osama Sayegh 228264d17c
Revert "DEV: add routes_lazy_route to boost boot-up time (#14545)" (#14581)
This reverts commit f5cf647e57.

The gem breaks usage of Rails URL helpers when used outside views and
controllers, for example in
88ecb83382/app/models/upload.rb (L239-L242)
the `upload_short_path` method call fails with an undefined method
exception when this gem is enabled.
2021-10-12 17:30:38 +03:00
Sam f5cf647e57
DEV: add routes_lazy_route to boost boot-up time (#14545)
The lazy route initialization cuts down boot time of rails.

On my local system it cuts out 200ms of boot time taking me from 3.2 to 3 seconds.

This is not a radically enormous amount of time, but paper cuts add up, and a faster boot in dev will make everyone happy.

TBD if we want to also include this in production.

Gem is heavily maintained by @amatsuda, last commit 3 days ago.
2021-10-11 13:22:13 +11:00
Yasuo Honda a29a6522d0
FIX: rss gem is bundled gem since Ruby 3 (#14350)
* FIX: rss gem is bundled gem since Ruby 3

Refer https://github.com/ruby/ruby/pull/2832
2021-09-20 11:51:47 +08:00
Alan Guo Xiang Tan 6a5b44be53
DEV: Pin oj gem to 3.13.2. (#14219)
Follow-up to 0183d51070
2021-09-02 11:01:54 +08:00
Alan Guo Xiang Tan 16cfbed89d
DEV: Upgrade Rails to 6.1.4.1. (#14098)
This pulls in a fix for CVE-2021-22942 but we're not tagging it as a
security fix because Discourse is not affected by it in production.
2021-08-20 10:20:08 +08:00
Alan Guo Xiang Tan 2818ebd19f DEV: Avoid duplication of gems in gemfile.
Follow-up to 08c6a7679d
2021-07-28 21:23:15 +08:00
Vinoth Kannan 08c6a7679d
DEV: allow `dev:populate` rake task in production environment. (#13777)
To use this rake task in production `ALLOW_DEV_POPULATE` environment variable should be set to "1".
2021-07-20 12:25:59 +05:30
David Taylor 8c370c3fe3 DEV: Add `annotate` rake tasks, and enforce via GitHub actions
`bin/rake annotate` is an alias of `bin/annotate --models`
`bin/rake annotate:clean` generates annotations by using a temporary, freshly migrated database. This should help us to produce more consistent annotations, even if development databases have been polluted by plugin migrations.

A GitHub actions task is also added which generates annotations on a clean database, and raises an error if they differ from the committed annotations.
2021-07-06 10:11:06 +01:00
Vinoth Kannan 6abc45e57b
DEV: move `discourse_dev` gem to the core. (#13360)
And get avatar images from `discourse_dev_assets` gem.
2021-06-14 20:34:44 +05:30
Arpit Jalan 283b08d45f
DEV: Absorb onebox gem into core (#12979)
* Move onebox gem in core library

* Update template file path

* Remove warning for onebox gem caching

* Remove onebox version file

* Remove onebox gem

* Add sanitize gem

* Require onebox library in lazy-yt plugin

* Remove onebox web specific code

This code was used in standalone onebox Sinatra application

* Merge Discourse specific AllowlistedGenericOnebox engine in core

* Fix onebox engine filenames to match class name casing

* Move onebox specs from gem into core

* DEV: Rename `response` helper to `onebox_response`

Fixes a naming collision.

* Require rails_helper

* Don't use `before/after(:all)`

* Whitespace

* Remove fakeweb

* Remove poor unit tests

* DEV: Re-add fakeweb, plugins are using it

* Move onebox helpers

* Stub Instagram API

* FIX: Follow additional redirect status codes (#476)

Don’t throw errors if we encounter 303, 307 or 308 HTTP status codes in responses

* Remove an empty file

* DEV: Update the license file

Using the copy from https://choosealicense.com/licenses/gpl-2.0/#

Hopefully this will enable GitHub to show the license UI?

* DEV: Update embedded copyrights

* DEV: Add Onebox copyright notice

* DEV: Add MIT license, convert COPYRIGHT.txt to md

* DEV: Remove an incorrect copyright claim

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
Co-authored-by: jbrw <jamie@goatforce5.org>
2021-05-26 15:11:35 +05:30
Blake Erickson fccf4d4375
DEV: Switch off of our image_optim fork (#13124)
The main image_optim gem now includes the timeout feature
that we had in our fork. So it is now safe to switch off of our fork and
back to the image_optim gem.

This is the link to the commit in the image_optim repo that adds the
timeout option:

ec3767dde0

One difference with the new timeout implementation is that image_optim
now handles the timeout exceptions instead of bubbling them up:

1ed0328587/lib/image_optim.rb (L128-L129)

```
 rescue Errors::TimeoutExceeded
   handler.result
```

So a timeout will just return `nil`, which is the same response if it
couldn't optimize an image. I don't think we were really watching for
or doing anything about these timeout warnings in our logs so I think
this is an okay change to have and we will have less warnings in our
logs now too.
2021-05-24 19:27:20 -06:00
David Taylor c456e31d26
SECURITY: Bump Rails to 6.1.3.2 (#12963)
Includes fixes for

- CVE-2021-22902
- CVE-2021-22903
- CVE-2021-22904
- CVE-2021-22885

https://github.com/rails/rails/blob/v6.1.3.2/actionpack/CHANGELOG.md
2021-05-06 12:41:45 +01:00
Osama Sayegh 45ccadeeeb
DEV: Upgrade Rails to 6.1.3.1 (#12688)
Rails 6.1.3.1 deprecates a few API and has some internal changes that break our tests suite, so this commit fixes all the deprecations and errors and now Discourse should be fully compatible with Rails 6.1.3.1. We also have a new release of the rails_failover gem that's compatible with Rails 6.1.3.1.
2021-04-21 12:36:32 +03:00
Roman Rizzi 5e4c0e2caa
FEATURE: Treat site settings as plain text and add a new HTML type. (#12618)
To add an extra layer of security, we sanitize settings before shipping them to the client. We don't sanitize those that have the "html" type.

The CookedPostProcessor already uses Loofah for sanitization, so I chose to also use it for this. I added it to our gemfile since we installed it as a transitive dependency.
2021-04-07 12:51:19 -03:00
Penar Musaraj 0cd4da51c7
FIX: Include `json_schemer` in prod (#12351) 2021-03-10 21:32:23 -05:00
Sam 9e79f8321a
DEV: Update Rails to latest patch level (#12281)
This contains 3 security fixes that do not impact us, see:

https://discuss.rubyonrails.org/c/security-announcements/9
2021-03-05 12:07:55 +11:00
Vinoth Kannan 8d96713aa0
DEV: add `discourse_dev` gem in development dependencies. (#12285)
For more details visit https://github.com/discourse/discourse_dev
2021-03-04 23:04:51 +05:30
Bianca Nenciu 3246c3cc92
DEV: Update mail and use fork (#10639)
Version 2.8 brings some changes to how address fields are handled and
this commits updates that and should also include a fix which handles
encoded attachment filenames.

The fork contains a bugfix to correctly decode mail attachments.
2021-02-18 20:15:02 +02:00
Blake Erickson 7434116933
DEV: Add schema checking to api doc testing (#11721)
* DEV: Add schema checking to api doc testing

This commit improves upon rswag which lacks schema checking. rswag
really only checks that the https status matches, but this change adds
in the json-schema_builder gem which also has schema validation.

Now we can define schemas for each of our requests/responses in the
`spec/requests/api/schemas` directory which will make our documentation
specs a lot cleaner.

If we update a serializer by either adding or removing an attribute the
tests will now fail (this is a good thing!). Also if you change the type
of an attribute say from an array to a string the tests will now fail.
This will help significantly with keeping the docs in sync with actual
code changes! Now if you change how an endpoint will respond you will
have to update the docs too in order for the tests to pass. :D

This PR is inspired by:

 https://www.tealhq.com/post/how-teal-keeps-their-api-tests-and-documentation-in-sync

* Swap out json schema validator gem

Swapped out the outdated json-schema_builder gem with the json_schemer
gem.

* Add validation fields to schema

In order to have "strict" validation we need to add
`additionalProperties: false` to the schema, and we need to specify
which attributes are required.

Updated the debugging test output to print out the error details if
there are any.
2021-01-21 16:28:08 -07:00
Sam c3fedae73a
DEV: update rack-mini-profiler (#11597)
Included support for flamegraphs using speedscope!
2020-12-29 13:54:41 +11:00
Vinoth Kannan 347423007a
DEV: remove instagram login site settings and auth classes. (#11073)
Instagram removed the support for login and should use Facebook login instead.
2020-10-30 09:09:56 +05:30
Guo Xiang Tan 122cf8d3fb
Update Rails to 6.0.3.3. 2020-09-10 15:41:06 +08:00
Blake Erickson b16915d166 DEV: Switch over to a fork of ember-rails
We are switching over to a fork because we are currently on a pinned
version of ember-rails 0.18.5 which is pretty old. Upgrading to the
latest version causes many things to break which isn't really worth the
time to debug while we plan to completely switch over to ember-cli
somewhat soonish. Our fork contains a single cherry-pick commit

https://github.com/emberjs/ember-rails/pull/534

which will fix an issue when running the `rails g migration` command and
it spits out a bunch of deprecation warnings.
2020-09-04 14:47:56 -06:00
Blake Erickson d79f50a3c6 DEV: Remove rb-inotify gem from Gemfile
We are no longer directly referencing the rb-inotify gem directly in
code. This was just a spec level dependency anyways.

Using `git log -S "Inotify"` resulted in these two commits as usages of
`Inotify`:

- b56b11d96a
- 9cf03b352c

both from 2013, but we no longer are using inotify in

https://github.com/discourse/discourse/blob/master/lib/tasks/autospec.rake

which appears to be the only file that was using it.

Based on this info we can safely remove rb-inotify from the Gemfile.

Just as a side note we still do have a couple of gems that do have
rb-inotify as a dependency: listen, and lru_redux.
2020-09-02 11:21:40 +08:00
Blake Erickson 43ffd4d28f
DEV: Switch our fast_xor gem for xorcist (#10565)
* DEV: Switch our fast_xor gem for xorcist

We use the `xor` function as part of password hashing and we want to use
a faster version than the native ruby xor'ing feature so we use a gem
for this.

fast_xor has been abandoned, and xorcist fixed our initial holdup for
switching in https://github.com/fny/xorcist/issues/4

xorcist also has jruby support so we can remove our jruby fallback
logic.

* Move using statement inside of class
2020-08-31 13:20:44 -06:00
Bianca Nenciu f2e14a3946
FEATURE: Add site setting and wizard step to set base font (#10250)
Co-authored-by: Neil Lalonde <neillalonde@gmail.com>
2020-08-31 13:14:09 +03:00
romanrizzi 0f63a61fa4
Revert "DEV: upgrades sassc to 2.4.0 (#10447)"
We suspect it could be the cause of some issues with themes.

This reverts commit 2fac77cc48.
2020-08-21 13:23:01 -03:00
Joffrey JAFFEUX 2fac77cc48
DEV: upgrades sassc to 2.4.0 (#10447)
We were stuck on 2.0.1 but multiple reports on the original issue shows that the issue has been fixed since 2.3 and 2.4:

https://github.com/sass/sassc-ruby/issues/146#issuecomment-654522157
https://github.com/sass/sassc-ruby/issues/146#issuecomment-654522157

There's one report show it was not fixed in 2.3.0, but that's the only one, it's probably worth trying: https://github.com/sass/sassc-ruby/issues/146#issuecomment-639073736
2020-08-21 13:46:46 +02:00
Blake Erickson ea2e58e622
DEV: Bump rotp gem to latest version (#10472)
The rotp gem is currently pinned to version 5.1.0 and this will bump it
up to version 6.0.1.

Follow up to: 85d4370f79

because this issue we were waiting on is now closed:

https://github.com/mdp/rotp/issues/98

Because version 6 is now encoding the params I needed to update the
tests as well.
2020-08-19 09:16:33 -06:00
Blake Erickson c767fb1ed3
DEV: Upgrade highline gem to latest (#10471)
Currently we have pinned highline to version 1.7.0. This is the gem that
we use to have an interactive command line for tasks like `rake
admin:create`.

Upgrading to the latest version 2.0.3 will remove ruby 2.7 deprecation
warnings.

I'm not sure why *this* gem was pinned. I manually executed a couple of
our rake tasks that use this and everything seems fine.
2020-08-19 09:15:23 -06:00
Sam Saffron 620c223d50
DEV: change platform mri to platform ruby on some gems
This assists with truffleruby testing which needs to be able to install
these gems
2020-07-29 17:10:38 +10:00
Sam Saffron 85d4370f79
DEV: pin the rotp gem
Not ready for an upgrade due to: https://github.com/mdp/rotp/issues/98

The policy here is that for cases like this we pin the version and add
a comment explaining why it is pinned.

We can revisit in a few months depending on upstream.
2020-07-14 17:05:12 +10:00
Sam Saffron bac25e6dd7
DEV: upgrade rack to version 2.2.3
This is very minor, see: https://github.com/advisories/GHSA-j6w9-fv6q-3q52

An attacker can elevate own cookie usage to bypass server cookie restrictions

Technically this is a security commit, but the surface area is extremely
low, we do not expect any real world impact.
2020-07-08 16:42:31 +10:00
Sam Saffron 8af5194e39
DEV: upgrade rails to version 6.0.3.2
This includes a fix for CVE-2020-8185 we are not vulnerable as we do not use
the impacted middleware. However it still makes sense to stay upgraded, other
small fixes exist in this release.
2020-07-08 16:34:29 +10:00
Guo Xiang Tan 58e52c0e4f
DEV: Use rails_failover gem for ActiveRecord and Redis failover handling 2020-06-15 15:47:07 +08:00
Guo Xiang Tan 2e2dcb7d93
DEV: Disable better_errors by default.
It adds a somewhat unnecessary middleware before `ActionDispatch::DebugExceptions` and totally bypasses it. Apps that register exception interceptors with `ActionDispatch::DebugExceptions` would therefore stop working if better_errors is used.
2020-06-12 09:22:40 +08:00
Martin Brennan e84ffb4861
DEV: Add pry-byebug and optionally require pry behind a ENV var for better spec debugging (#9984)
We removed pry-nav a while back because it is not up to date with pry but it is super useful. Luckily pry-byebug is here to save us all from Satan's power.

To get this to work you need to add the following to your $HOME/.pryrc file.

```
if defined?(PryByebug)
  Pry.commands.alias_command 'c', 'continue'
  Pry.commands.alias_command 's', 'step'
  Pry.commands.alias_command 'n', 'next'
  Pry.commands.alias_command 'f', 'finish'
end

Pry::Commands.command /^$/, "repeat last command" do
  pry_instance.run_command Pry.history.to_a.last
end
```

The require-ing of pry, pry-rails, and pry-byebug in specs is controlled by the IMPROVED_SPEC_DEBUGGING flag (disabled by default).
2020-06-05 12:30:34 +10:00
Robin Ward cae839ca96
DEV: Update gems to latest versions (#9909) 2020-05-28 13:13:10 -04:00
Guo Xiang Tan 8c86a109bb
DEV: Add ENV flag to test out `ActiveRecord::Failover`. 2020-05-28 16:24:22 +08:00
Michael Brown d9a02d1336
Revert "Revert "Merge branch 'master' of https://github.com/discourse/discourse""
This reverts commit 20780a1eee.

* SECURITY: re-adds accidentally reverted commit:
  03d26cd6: ensure embed_url contains valid http(s) uri
* when the merge commit e62a85cf was reverted, git chose the 2660c2e2 parent to land on
  instead of the 03d26cd6 parent (which contains security fixes)
2020-05-23 00:56:13 -04:00
Jeff Atwood 20780a1eee Revert "Merge branch 'master' of https://github.com/discourse/discourse"
This reverts commit e62a85cf6f, reversing
changes made to 2660c2e21d.
2020-05-22 20:25:56 -07:00
Guo Xiang Tan f7f436e536
DEV: Install `rails_failover` gem to test our Redis changes. 2020-05-20 15:40:27 +08:00
Sam Saffron 30bbc485d0
DEV: remove superfluous comment 2020-05-20 16:53:48 +10:00
Krzysztof Kotlarek 16f6240782
FIX: Bump rack version from 2.0.8 to 2.2.2 (#9811)
Version 2.1.1 was not working with our Sidekiq but version 2.2.2 is fine
2020-05-19 08:43:45 +10:00
Sam Saffron 609e929186
Revert "Revert "DEV: upgrade to Rails 6.0.3""
This reverts commit 2ff8b4f5d9.

Attempt #2 at a Rails update this time we also update the
rails_multisite gem to allow for cleaner reordering
2020-05-08 11:49:22 +10:00
David Taylor 2ff8b4f5d9
Revert "DEV: upgrade to Rails 6.0.3"
This was causing issues during multisite:migrate

https://meta.discourse.org/t/multisite-migrate-broken-since-rails-6-0-3-update/150691

This reverts commit 136a545653.
2020-05-07 11:44:39 +01:00
Sam Saffron 136a545653
DEV: upgrade to Rails 6.0.3
Upgrades Rails to latest, this version has better compatibility
with Ruby 2.7

During the upgrade we needed a new cleaner mechanism for configuring
message bus.

All tests are green.

If anything weird pops up please revert.
2020-05-07 15:53:40 +10:00
Jarek Radosz 666823d4b7 Revert "Revert "DEV: Move rubocop config to rubocop-discourse (#9616)""
This reverts commit 2d31a14789.

Should be good now - all the plugins are using the updated rubocop config.
2020-05-06 18:41:15 +02:00
Jarek Radosz 2d31a14789 Revert "DEV: Move rubocop config to rubocop-discourse (#9616)"
This reverts commit e23f1a9071.

Reverting as this currently breaks our plugin linting job in GithHub Action and Jenkins. Will re-revert after all the plugins get the latest rubocop config and/or a (potential) rubocop issue is fixed.
2020-05-06 17:22:25 +02:00
Jarek Radosz e23f1a9071
DEV: Move rubocop config to rubocop-discourse (#9616) 2020-05-06 15:03:06 +02:00
Sam Saffron 57fcea7709
DEV: update rspec dependencies
rspec-rails 4.0 was released so we no longer need to depend on a
beta version. Also updates minor on a bunch of rspec gems.

Thanks to @ryanwi for raising this.
2020-05-04 15:21:34 +10:00
Régis Hanol 501b19b6e0
FIX: server-side HtmlToMarkdown improvements (#9586)
TLDR; this commit vastly improves how whitespaces are handled when converting from HTML to Markdown.
It also adds support for converting HTML <tables> to markdown tables.

The previous 'remove_whitespaces!' method was traversing the whole HTML tree and used a heuristic to remove
leading and trailing whitespaces whenever it was appropriate (ie. mostly before and after HTML block elements)

It was a good idea, but it was very limited and leaded to bad conversion when the html had leading whitespaces on several lines for example.
One such example can be found [here](https://meta.discourse.org/t/86782).

For various reasons, most of the whitespaces in a HTML file is ignored when the page is being displayed in a browser.
The rules that the browsers follow are the [CSS' White Space Processing Rules](https://www.w3.org/TR/css-text-3/#white-space-rules).
They can be quite complicated when you take into account RTL languages and other various tidbits but they boils down to the following:

- Collapse whitespaces down to one space (0x20) inside an inline context (ie. nodes/tags that are being displaying on the same line)
- Remove any leading/trailing whitespaces inside an inline context

One quick & dirty way of getting this 90% solved would be to do 'HTML.gsub!(/[[:space:]]+/, " ")'.
We would also need to hoist <pre> elements in order to not mess with their whitespaces.
Unfortunately, this solution let some whitespaces creep around HTML tags which leads to more '.strip!' calls than I can bear.

I decided to "emulate" the browser's handling of whitespaces and came up with a solution in 4 parts

1. remove_not_allowed!

The HtmlToMarkdown library is recursively "visiting" all the nodes in the HTML in order to convert them to Markdown.
All the nodes that aren't handled by the library (eg. <script>, <style> or any non-textual HTML tags) are "swallowed".
In order to reduce the number of nodes visited, the method 'remove_not_allowed!' will automatically delete all the nodes
that have no "visitor" (eg. a 'visit_<tag>' method) defined.

2. remove_hidden!

Similar purpose as the previous method (eg. reducing number of nodes visited), there's no point trying to convert something that is hidden.
The 'remove_hidden!' method removes any nodes that was hidden using the "hidden" HTML attribute, some CSS or with a width or height equal to 0.

3. hoist_line_breaks!

The 'hoist_line_breaks!' method is there to handle <br> tags. I know those tiny <br> don't do much but they can be quite annoying.
The <br> tags are inline elements but they visually work like a block element (ie. they create a new line).
If you have the following HTML "<i>Foo<br>Bar</i>", it ends up visually similar to "<i>Foo</i><br><i>Bar</i>".
The latter being much more easy to process than the former, so that's what this method is doing.
The "hoist_line_breaks" will hoist <br> tags out of inline tags until their parent is a block element.

4. remove_whitespaces!

The "remove_whitespaces!" is where all the whitespace removal is happening. It's broken down into 4 methods as well

- remove_whitespaces!
- is_inline?
- collapse_spaces!
- remove_trailing_space!

The 'remove_whitespace!' method is recursively walking the HTML tree (skipping <pre> tags).
If a node has any children, they will be chunked into groups of inline elements vs block elements.
For each chunks of inline elements, it will call the "collapse_space!" and "remove_trailing_space!" methods.
For each chunks of block elements, it will call "remote_whitespace!" to keep walking the HTML tree recursively.

The "is_inline?" method determines whether a node is part of a inline context.
A node is inline iif it's a text node or it's an inline tag, but not <br>, and all its children are also inline.

The "collapse_spaces!" method will collapse any kind of (white) space into a single space (" ") character, even accros tags.
For example, if we have "  Foo \n<i> Bar </i>\t42", it will return "Foo <i>Bar </i>42".

Finally, the "remove_trailing_space!" method is there to remove any trailing space that might creep in at the end of the inline chunk.

This solution is not 100% bullet-proof.
It does not support RTL languages at all and has some caveats that I felt were not worth the work to get properly fixed.

FIX: better detection of hidden elements when converting HTML to Markdown
FIX: take into account the 'allowed_href_schemes' site setting when converting HTML <a> to Markdown
FIX: added support for 'mailto:' scheme when converting <a> from HTML to Markdown
FIX: added support for <img> dimensions when converting from HTML to Markdown
FIX: added support for <dl>, <dd> and <dt> when converting from HTML to Markdown
FIX: added support for multilines emphases, strongs and strikes when converting from HTML to Markdown
FIX: added support for <acronym> when converting from HTML to Markdown
DEV: remove unused 'sanitize' gem

Wow, did you just read all that?! Congratz, here's a cookie: 🍪.
2020-04-30 12:21:25 +02:00
Sam Saffron 4f5ed8e781
DEV: pry-nav was holding back on pry upgrades
pry-nav is not yet supported on latest pry, this holds off on
upgrading pry, which in turn holds off on upgrading deps

Stripping pry-nav for now till it works with latest pry
2020-04-30 09:40:50 +10:00
David Taylor 6a9a7b56df
DEV: Bump Hashie and Faraday (#9583)
These were previously pinned due to a dependency in the zendesk plugin. That has now been resolved.
2020-04-29 12:55:30 +01:00
Blake Erickson a93ef2926d
DEV: Add rswag to aid in api documention (#9546)
Adding in rswag will allow us to write spec files to document and test
our api.
2020-04-27 16:40:07 -06:00
Jarek Radosz 07e0490fe4
DEV: Update mocha (#9490)
The spec that was blocking the update was fixed in c08753dc34.
2020-04-21 18:32:42 +02:00
Daniel Waterworth 7876ee2d67 DEV: upgrade Rails
Latest version of Rails contains compatibility fixes for Ruby 2.7 and some
minor security fixes we would like to have

It also broke some of the multisite tests.

Rails tries to use the same connection for reading from a replica as writing
to the leader during tests, because, with everything happening in a
transaction, changes to the DB wouldn't otherwise be reflected in the
replica connection.

The difference now is that Rails tries to do this for connections opened
after the test has started which affected rails multisite connections.

The upshot of this is that, as things stand, you are likely to
experience problems if you try to connect to a different multisite DB in
a test when the `current_db` is not 'default'.
2020-04-20 12:55:53 +01:00
Jarek Radosz 7ff889574d
DEV: Add rubocop-rspec (#9288)
This adds rubocop-rspec, and enables some cops that were either already passing or are passing now, after fixing them in this commit.

Some new cops are disabled for now, with annotation: "TODO" or "To be decided". Those either need to be discussed first, or require manual changes, or the number of found and fixed offenses is too large to bundle them up in a single PR.

Includes:

* DEV: Update rubocop's `TargetRubyVersion` to 2.6
* DEV: Enable RSpec/VoidExpect
* DEV: Enable RSpec/SharedContext
* DEV: Enable RSpec/EmptyExampleGroup (Removed an obsolete empty spec file)
* DEV: Enable RSpec/ItBehavesLike
* DEV: Remove RSpec/ScatteredLet (It's too strict, as it doesn't recognize fab! as a let-like)
* DEV: Remove RSpec/MultipleExpectations
2020-03-27 17:35:40 +01:00
Sam Saffron c7151f0fd6
Revert "DEV: upgrade Rails"
This reverts commit 5b3bb4b2f0.

This erratically breaks multisite operation, we need more debugging
2020-03-24 17:11:13 +11:00
Sam Saffron 5b3bb4b2f0
DEV: upgrade Rails
Latest version of Rails contains compatibility fixes for Ruby 2.7 and some
minor security fixes we would like to have
2020-03-24 16:47:40 +11:00
Sam Saffron 9726a0e0b4
DEV: upgrade json gem and add explicit dependency
json is shipped out of sync with Ruby. Even though we use OJ for many things
we still use the json gem sometimes, this ensures we use the latest

b8b29e79ad/config/initializers/100-oj.rb (L9-L9)
2020-03-24 15:21:50 +11:00
David Taylor e9a3639b10
DEV: Pin hashie and faraday versions for zendesk api compatibility (#9214) 2020-03-19 19:52:31 +00:00
dependabot-preview[bot] 1b2019e7eb
Build(deps): Bump rack-mini-profiler from 1.1.6 to 2.0.1 (#9222)
* Build(deps): Bump rack-mini-profiler from 1.1.6 to 2.0.1

Bumps [rack-mini-profiler](https://github.com/MiniProfiler/rack-mini-profiler) from 1.1.6 to 2.0.1.
- [Release notes](https://github.com/MiniProfiler/rack-mini-profiler/releases)
- [Changelog](https://github.com/MiniProfiler/rack-mini-profiler/blob/master/CHANGELOG.md)
- [Commits](https://github.com/MiniProfiler/rack-mini-profiler/compare/v1.1.6...v2.0.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Enable rails patches

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: OsamaSayegh <asooomaasoooma90@gmail.com>
2020-03-17 14:09:45 +03:00
OsamaSayegh b23c2437ae DEV: Revert rack-mini-profiler version bump
New version breaks site deploys. Will investigate and fix.
2020-03-11 22:16:15 +03:00
OsamaSayegh c928287e0c DEV: Mini Profiler shouldn't be loaded in test environment 2020-03-11 21:31:57 +03:00
dependabot-preview[bot] a4929661af
Build(deps): Bump rack-mini-profiler from 1.1.6 to 2.0.0 (#9168)
* Build(deps): Bump rack-mini-profiler from 1.1.6 to 2.0.0

Bumps [rack-mini-profiler](https://github.com/MiniProfiler/rack-mini-profiler) from 1.1.6 to 2.0.0.
- [Release notes](https://github.com/MiniProfiler/rack-mini-profiler/releases)
- [Changelog](https://github.com/MiniProfiler/rack-mini-profiler/blob/master/CHANGELOG.md)
- [Commits](https://github.com/MiniProfiler/rack-mini-profiler/compare/v1.1.6...v2.0.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Enable rails patches

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: OsamaSayegh <asooomaasoooma90@gmail.com>
2020-03-11 20:11:12 +03:00
Robin Ward a3f0543f99
Support for transpiling `.js` files (#9160)
* Remove some `.es6` from comments where it does not matter

* Use a post processor for transpilation

This will allow us to eventually use the directory structure to
transpile rather than the extension.

* FIX: Some errors and clean up in confirm-new-email

It would throw an error if the webauthn element wasn't present.
Also I changed things so that no-module is not explicitly
referenced.

* Remove `no-module`

Instead we allow a magic comment: `// discourse-skip-module` to prevent
the asset pipeline from creating a module.

* DEV: Enable babel transpilation based on directory

If it's in `app/assets/javascripts/dicourse` it will be transpiled
even without the `.es6` extension.

* REFACTOR: Remove Tilt/ES6ModuleTranspiler
2020-03-11 09:43:55 -04:00
Sam Saffron 59a7afbde9
DEV: flag MRI specific gems
byebug, ruby-prof, better_errors and rbtrace are very MRI specific, flag
them as such

This helps move forward on potential jruby and truffleruby experiments
2020-02-18 11:04:56 +11:00
David Taylor 5919618a87
DEV: Drop legacy OpenID 2.0 support (#8894)
This is not used in core or official plugins, and has been printing a deprecation notice since v2.3.0beta4. All OpenID 2.0 code and dependencies have been dropped. The user_open_ids table remains for now, in case anyone has missed the deprecation notice, and needs to migrate their data.

Context at https://meta.discourse.org/t/-/113249
2020-02-07 17:32:35 +00:00
Jarek Radosz 53529a3427
DEV: Upgrade Ember to version 3.12.2 (#8753)
* DEV: Use Ember 3.12.2
* Add Ember version to ThemeField's DEPENDENT_CONSTANTS
* DEV: Use `id` instead of `elementId` (See: https://github.com/emberjs/ember.js/issues/18147)
* FIX: Don't leak event listeners (bug introduced in 999e2ff)
2020-02-05 14:51:00 +01:00
Sam Saffron eb105ba79d DEV: revert upgrade of rack to version 2.0.8
We can not upgrade rack cause it breaks Sidekiq web.

I can not find a trivial fix short of disabling sessions in Sidekiq which
is a security concern.

We need to figure out how to reuse sessions with our Rails application in
Sidekiq.

This gets extra complex cause we use a special cookie store for sessions.

9e399b42b9/lib/discourse_cookie_store.rb (L3-L21)
2020-01-13 18:07:16 +11:00
Martin Brennan beb91e7eff
FIX: require: false for rotp gem (#8540)
The ROTP gem is only used in a very small amount of places in the app, we don't need to globally require it.

Also set the Addressable gem to not have a specific version range, as it has not been a problem yet.

Some slight refactoring of UserSecondFactor here too to use SecondFactorManager to avoid code repetition
2019-12-17 10:33:51 +10:00
Martin Brennan edbc356593
FIX: Replace deprecated URI.encode, URI.escape, URI.unescape and URI.unencode (#8528)
The following methods have long been deprecated in ruby due to flaws in their implementation per http://blade.nagaokaut.ac.jp/cgi-bin/vframe.rb/ruby/ruby-core/29293?29179-31097:

URI.escape
URI.unescape
URI.encode
URI.unencode
escape/encode are just aliases for one another. This PR uses the Addressable gem to replace these methods with its own encode, unencode, and encode_component methods where appropriate.

I have put all references to Addressable::URI here into the UrlHelper to keep them corralled in one place to make changes to this implementation easier.

Addressable is now also an explicit gem dependency.
2019-12-12 12:49:21 +10:00
Sam Saffron b6acfb7847 DEV: upgrade redis-namespace gem
New release has a few extra commands namespaced, nothing we use.

Also added a comment about why this is explicitly required.
2019-12-12 13:36:08 +11:00
Mark VanLandingham 06c6062ed2
DEV: Lock sassc gem at version 2.0.1 with note (#8523) 2019-12-11 06:22:39 -08:00
Sam Saffron 4de39f6596 DEV: hold back mocha upgrade
This breaks our test suite and we want to properly document this.
2019-12-11 12:43:07 +11:00
Sam Saffron 3e0454c97b DEV: add a note about sprockets being held back
We want to upgrade to version 4, but it does not work atm.
2019-12-10 12:31:16 +11:00
Sam Saffron a06fccae1b DEV: update dependencies and add notes about exceptions
Previously it was unclear why certain gems are being held back cause Gemfile
had no comment explaining it.

I tried to add some explanation from memory and remove some exceptions that
seemed to be superfluous.

This upgrades shoulda to latest, it appears to work once a couple of assertions
are removed

Also update http accept language used to auto detect language from http header
this is tested

Zeitwerk small update seems fine
2019-12-06 13:00:28 +11:00
Arpit Jalan cab9c7c77e Bump onebox version.
- FIX: use dedicated Vimeo onebox for all video types
2019-11-27 16:22:25 +05:30
Arpit Jalan 7543db086a Bump onebox version.
- FIX: Amazon video oneboxes were not working.
2019-11-20 14:47:59 +05:30
David Taylor eaf6096890 DEV: Use rubocop-discourse gem to add custom chdir cop
Followup to b27e009655
2019-11-18 15:39:41 +00:00
Sam Saffron 26c0199c01 DEV: update Rails to version 6.0.1
This version of Rails eliminates a monkey patch that is no longer needed!

Additionally it preps us for Ruby 2.7 support.
2019-11-08 16:56:30 +11:00
Arpit Jalan c5df853dea Bump onebox version.
- fix for gfycat onebox in email
2019-11-07 10:03:12 +05:30
Arpit Jalan cb9702bf7a Bump onebox version.
- Remove native caching
- FIX: dropbox videos were not loading
2019-11-04 10:46:20 +05:30
Arpit Jalan 12409f63a0 Bump onebox version.
- FIX: Follow redirect returns url if response code is 200
- FIX: do not resize xkcd image
2019-10-22 12:26:01 +05:30
Krzysztof Kotlarek 858cf5836c
FIX: update Redis gem to version 4.1.3
I run our benchmark on commit with hiredis and redis-4.1.3

Results:
type | hidredis | redis 4.1.3 | percent
--- | --- | --- | ---
Categories-50 | 49 | 50 | 102.04%
Categories-75 | 51 | 51 | 100.00%
Categories-90 | 63 | 64 | 101.59%
Categories-99 | 86 | 85 | 98.84%
Home-50 | 55 | 55 | 100.00%
Home-75 | 56 | 57 | 101.79%
Home-90 | 68 | 69 | 101.47%
Home-99 | 102 | 104 | 101.96%
Topic-50 | 36 | 37 | 102.78%
Topic-75 | 37 | 37 | 100.00%
Topic-90 | 47 | 48 | 102.13%
Topic-99 | 60 | 61 | 101.67%
Categories-admin-50 | 124 | 117 | 94.35%
Categories-admin-75 | 130 | 129 | 99.23%
Categories-admin-90 | 147 | 143 | 97.28%
Categories-admin-99 | 204 | 199 | 97.55%
Home-admin-50 | 146 | 148 | 101.37%
Home-admin-75 | 150 | 152 | 101.33%
Home-admin-90 | 169 | 168 | 99.41%
Home-admin-99 | 232 | 223 | 96.12%
Topic-admin-50 | 60 | 61 | 101.67%
Topic-admin-75 | 64 | 63 | 98.44%
Topic-admin-90 | 76 | 73 | 96.05%
Topic-admin-99 | 124 | 94 | 75.81%
Load rails | 2412 | 2360 | 97.84%
rss | 290204 | 295828 | 101.94%
pss | 277948 | 283624 | 102.04%

Redis gem is manipulating Redis config https://github.com/redis/redis-rb/blob/master/lib/redis/client.rb#L95
therefore we cannot pass the frozen config object.

Pass of the copy of the object is protecting original config
2019-10-21 09:59:24 +11:00
Sam Saffron ae2a56999e Revert "FIX: update Redis gem to version 4.1.3 (#8197)"
This reverts commit ab74a50d85.

We really want to upgrade redis, but discovered some edge cases
around failover we need to test.

Holding off on the upgrade till a bit more testing happens
2019-10-17 11:41:46 +11:00
Krzysztof Kotlarek ab74a50d85 FIX: update Redis gem to version 4.1.3 (#8197)
* FIX: update Redis gem to version 4.1.3

I run our benchmark on commit with hiredis and redis-4.1.3

Results:
type | hidredis | redis 4.1.3 | percent
--- | --- | --- | ---
Categories-50 | 49 | 50 | 102.04%
Categories-75 | 51 | 51 | 100.00%
Categories-90 | 63 | 64 | 101.59%
Categories-99 | 86 | 85 | 98.84%
Home-50 | 55 | 55 | 100.00%
Home-75 | 56 | 57 | 101.79%
Home-90 | 68 | 69 | 101.47%
Home-99 | 102 | 104 | 101.96%
Topic-50 | 36 | 37 | 102.78%
Topic-75 | 37 | 37 | 100.00%
Topic-90 | 47 | 48 | 102.13%
Topic-99 | 60 | 61 | 101.67%
Categories-admin-50 | 124 | 117 | 94.35%
Categories-admin-75 | 130 | 129 | 99.23%
Categories-admin-90 | 147 | 143 | 97.28%
Categories-admin-99 | 204 | 199 | 97.55%
Home-admin-50 | 146 | 148 | 101.37%
Home-admin-75 | 150 | 152 | 101.33%
Home-admin-90 | 169 | 168 | 99.41%
Home-admin-99 | 232 | 223 | 96.12%
Topic-admin-50 | 60 | 61 | 101.67%
Topic-admin-75 | 64 | 63 | 98.44%
Topic-admin-90 | 76 | 73 | 96.05%
Topic-admin-99 | 124 | 94 | 75.81%
Load rails | 2412 | 2360 | 97.84%
rss | 290204 | 295828 | 101.94%
pss | 277948 | 283624 | 102.04%

* FIX: get rid of redis freedom patch
2019-10-17 08:49:23 +11:00
David Taylor 061c8874f5 FIX: Correct line count link in GitHub commit onebox
Bump onebox version
2019-10-15 23:52:59 +01:00
Sam Saffron c3cc96084c FIX: remove hiredis gem which is no longer needed
Previously some local micro-benchmarks revealed it was not giving any perf
benefits.

Now that we upgraded to 2.6.5 we are seeing some segfaults.

No need to carry this dependency around anymore.

We can re-evaluate in future if it improves perf and fix the segfaults.
2019-10-15 18:17:14 +11:00
romanrizzi 9845963105 FEATURE: Use the 'ugc' rel attribute alongside 'nofollow' 2019-10-14 15:21:48 -03:00
David Taylor 939a746dcd UX: Use theme colors for GitHub issue labels
Bump onebox version to pull tag rendering bug fix
2019-10-09 12:28:48 +01:00
David Taylor 3edd514c72 FEATURE: Redesigned GitHub oneboxes
Bump onebox version, and add new styling

Commit, PR and Issue oneboxes are updated with a new design. Timestamps are now localized using local-dates (if installed).
2019-10-09 11:47:58 +01:00
David Taylor e7cc7def8b UX: Stop using fixed-width font to render github issue description
Bump onebox version
2019-10-08 11:48:05 +01:00
David Taylor 615039f228 FEATURE: Improve GitHub commit, PR and issue onebox rendering
Bump onebox version to include new github rendering, and add relevant CSS

Avatars are reduced in size significantly, and icons are added to easily differentiate PRs and commits. The 'Issue:' prefix is removed from issue oneboxes, to make them consistent with commits and PRs.
2019-10-07 19:26:10 +01:00
Sam Saffron 8d5f47dded PREF: optimise preloading application
We preload to ensure as much memory as possible is reused from unicorn master
to various workers using copy-on-write (sidekiq, unicorn)

This migrates the preloading code into the Discourse module for easier
reuse and adds 3 notable preloading changes

1. We attempt to localize a string on each site, ensuring we warmup
the i18n

2. We preload all our templates (compiling .erb to class)

3. We warm-up our search tokenizer which uses cppjieba which is a large
memory consumer, this will only cause a warmup on CJK sites or sites with
the special site setting enabled.
2019-10-07 00:33:37 -04:00
Martin Brennan 68d35b14f4 FEATURE: Webauthn authenticator management with 2FA login (Security Keys) (#8099)
Adds 2 factor authentication method via second factor security keys over [web authn](https://developer.mozilla.org/en-US/docs/Web/API/Web_Authentication_API).

Allows a user to authenticate a second factor on login, login-via-email, admin-login, and change password routes. Adds registration area within existing user second factor preferences to register multiple security keys. Supports both external (yubikey) and built-in (macOS/android fingerprint readers).
2019-10-01 19:08:41 -07:00
Krzysztof Kotlarek 32b8a2ccff DEV: Upgrade Discourse to Rails 6 (#8083)
* Adjustments to pass specs on Rails 6.0.0
* Use classic autoloader instead of Zeitwerk
* Update Rails 6.0.0 deprecated methods
* Rails 6.0.0 not allowing column with integer name
* Drop freedom_patches/rails6.rb
* Default value for trigger_transactional_callbacks? is true
* Bump rspec-rails version to 4.0.0.beta2
2019-09-12 10:41:50 +10:00