Commit Graph

9666 Commits

Author SHA1 Message Date
Alan Guo Xiang Tan b876ff6281
FIX: Update user stat counts when post/topic visibility changes. (#15883)
Breakdown of fixes in this commit:

* `UserStat#topic_count` was not updated when visibility of
the topic changed.

* `UserStat#post_count` was not updated when post was hidden or
unhidden.

* `TopicConverter` was only incrementing or decrementing the counts by 1
even if a user has multiple posts in the topic.

* The commit turns off the verbose logging by default as it is just
noise to normal users who are not debugging this problem.
2022-02-11 09:00:58 +08:00
David Taylor c81d369ab6 DEV: Improve theme error handling UX
- Update UI to improve contrast
- Make it clear that the message is only shown to administrators
- Add theme name and id to the console output
- Parse the error backtrace to identify the theme-id for post-decoration errors
- Improve console output to include the theme name / URL
- Add `?safe_mode=no_custom` to the admin panel link, so that it will work even if the theme is causing the site to break
2022-02-10 22:56:11 +00:00
Blake Erickson e66241af7f
DEV: Update api docs to update username and to get post replies (#15881)
Added api docs for:

 - update username
 - get post replies
2022-02-09 15:04:36 -07:00
Dan Ungureanu 2d8ebe989a
FEATURE: Redeem invites for existent users (#15866)
This adds logic to increase an `InvitedUser` record, increase
`redemption_count` and create a `:invitee_accepted` to let the inviter
know that the invitee used the invite.

Initial support for this was implemented in commit 9969631.
2022-02-09 17:22:30 +02:00
Alan Guo Xiang Tan ae0625323a
FIX: Avoid errors when updating post and topic count user stats. (#15876)
In ab5361d69a, we rescue from the PG error
but the transaction is already aborted causing any DB query after to
fail. As such, we avoid triggering the error in the first place by
checking that we would not be insertin a negative number into the
counter cache.

Follow-up to ab5361d69a
2022-02-09 21:48:18 +08:00
Dan Ungureanu 1fb97f8bba
FIX: Replace Twitter handles one at a time (#15870)
Previously, all handles and hashtags were replaced in one go which could
result in a wrong result if a handle was a substring of another one.
2022-02-09 13:54:02 +02:00
Ayke Halder 5ff3a9c4bb
DEV: add native lazy loading for emojis (#15830) 2022-02-09 12:18:59 +01:00
Joffrey JAFFEUX b3ecf00c98
DEV: adds initial support for custom blocks using code fencing (#15743)
Allows to write custom code blocks:

```
   ```mermaid height=200,foo=bar
   test
   ```
```

Which will then get converted to:

```
<pre data-code-wrap="mermaid" data-code-height="200" data-code-foo="bar">
  <code class="lang-nohighlight">
    test
  </code>
</pre>
```
2022-02-09 11:23:44 +01:00
Bianca Nenciu c38114f0c6
FIX: Sort group owners and members together (#15708)
Sorting group members worked always kept the group owners at the top of
the list. This commit keeps the group owners at the top of the list only
when no order exists.
2022-02-09 11:43:58 +02:00
Bianca Nenciu f704deca17
FIX: Clear drafts only when post is created by real user (#15720)
This commits adds a new advance_draft to PostCreator that controls if
the draft sequence will be advanced or not. If the draft sequence is
advanced then the old drafts will be cleared. This used to happen for
posts created by plugins or through the API and cleared user drafts
by mistake.
2022-02-09 10:37:38 +02:00
Alan Guo Xiang Tan ab5361d69a
FIX: Avoid raising error when updating post and topic count user stats. (#15873)
There are still spots in the code base which results in us trying to turn the post and topic count negative. However,
we have a job that runs on a daily basis which will correct the count. Therefore, avoid raising an error for now
and log the exception instead.
2022-02-09 12:51:13 +08:00
Blake Erickson 71f7f7ed49
FEATURE: Add external_id to topics (#15825)
* FEATURE: Add external_id to topics

This commit allows for topics to be created and fetched by an
external_id. These changes are API only for now as there aren't any
front changes.

* add annotations

* add external_id to this spec

* Several PR feedback changes

- Add guardian to find topic
- 403 is returned for not found as well now
- add `include_external_id?`
- external_id is now case insensitive
- added test for posts_controller
- added test for topic creator
- created constant for max length
- check that it redirects to the correct path
- restrain external id in routes file

* remove puts

* fix tests

* only check for external_id in webhook if exists

* Update index to exclude external_id if null

* annotate

* Update app/controllers/topics_controller.rb

We need to check whether the topic is present first before passing it to the guardian.

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>

* Apply suggestions from code review

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2022-02-08 20:55:32 -07:00
Martin Brennan 59343c3057
DEV: Add Windows Hello webauthn authentication spec (#15871)
Follow up to 6f7364e48b to add a spec
that tests the full authentication of a Windows Hello algorithm (-257)
webauthn verification. The test added in that commit only tested that
we know about that algorithm, not whether it was actually usable.
2022-02-09 12:47:47 +10:00
Dan Ungureanu 6f7364e48b
FEATURE: RS512, RS384 and RS256 COSE algorithms (#15804)
* FEATURE: RS512, RS384 and RS256 COSE algorithms

These algorithms are not implemented by cose-ruby, but used in the web
authentication API and were marked as supported.

* FEATURE: Use all algorithms supported by cose-ruby

Previously only a subset of the algorithms were allowed.
2022-02-08 14:07:47 +02:00
Alan Guo Xiang Tan 7afe768d60
DEV: Add tests for wistia onebox. (#15860)
Follow-up to 4ef56b0ca4
2022-02-08 13:04:32 +08:00
Jarek Radosz 6758835387
FIX: Overridden MessageFormat fallbacks (#15855)
…were missing pluralization rules

This resulted in errors like `MessageFormat.locale.en is not a function`
2022-02-08 12:31:08 +11:00
David Taylor c8c23ba557
DEV: Introduce deprecation warning for non-json Job arguments (#15842)
This commit introduces our own handling and warning for Sidekiq's new 'non-json-serializable' warning. This decouples us from Sidekiq's own deprecation cycle, and allows us to use our own deprecation system. It also means that the dump/parse happens in test mode, which will help us to catch occurrences before they reach production.
2022-02-07 17:59:55 +00:00
David Taylor 64be371749
DEV: Improve handling of invalid requests (#15841)
Our discourse_public_exceptions middleware is designed to catch bubbled exceptions from lower in the stack, and then use `ApplicationController.rescue_with_handler` to render an appropriate error response.

When the request itself is invalid, we had an escape-hatch to skip re-dispatching the request to ApplicationController. However, it was possible to work around this by 'layering' the errors. For example, if you made a request which resulted in a 404, but **also** had some other invalidity, the escape hatch would not be triggered.

This commit ensures that these kind of 'layered' errors are properly handled, without logging warnings. It also adds detection for invalid JSON bodies and badly-formed multipart requests.

The user-facing behavior is unchanged. This commit simply prevents warnings being logged for invalid requests.
2022-02-07 13:16:57 +00:00
Andrei Prigorshnev 404f882e5c
FEATURE: in result.rb don't use email for username suggestions until enabled in settings (#15586) 2022-02-07 16:02:26 +04:00
Martin Brennan 0a738bd5bc
FEATURE: Allow sending group SMTP emails with from alias (#15687)
This commit allows group SMTP emails to be sent with a
different from email address that has been set up as an
alias in the email provider. Emails from the alias will
be grouped correctly using Message-IDs in the mail client,
and replies to the alias go into the correct group inbox.
2022-02-07 13:52:01 +10:00
Alan Guo Xiang Tan 454d3740b4
DEV: Remove deprecated site settings. (#15725) 2022-02-07 11:26:06 +08:00
Alan Guo Xiang Tan b7eacaed21
FIX: Handle addressable error when parsing an invalid URL. (#15836)
Passing in an invalid URL would result in an `Addressable::URI::InvalidURIError` which we were not catching.
2022-02-07 11:25:42 +08:00
Alan Guo Xiang Tan 5bd55acf83
FIX: Add DB constraints for post & topic counter cache for `UserStat` (#15626)
Ensures that `UserStat#post_count` and `UserStat#topic_count` does not
go below 0. When it does like it did now, we tend to have bugs in our
code since we're usually coding with the assumption that the count isn't
negative.

In order to support the constraints, our post and topic fabricators in
tests will now automatically increment the count for the respective
user's `UserStat` as well. We have to do this because our fabricators
bypasss `PostCreator` which holds the responsibility of updating `UserStat#post_count` and
`UserStat#topic_count`.
2022-02-07 11:23:34 +08:00
Ayke Halder 81e175e6ba
FIX: allow native lazy loading attribute for quoted avatar image (#15834) 2022-02-07 09:28:26 +08:00
Alan Guo Xiang Tan 65f46ad4ed FIX: Better param guards for `wiki` and `post_type` posts controller. 2022-02-07 09:21:24 +08:00
Alan Guo Xiang Tan 930f51e175 FEATURE: Split up text segmentation for Chinese and Japanese.
* Chinese segmenetation will continue to rely on cppjieba
* Japanese segmentation will use our port of TinySegmenter
* Korean currently does not rely on segmentation which was dropped in c677877e4f
* SiteSetting.search_tokenize_chinese_japanese_korean has been split
into SiteSetting.search_tokenize_chinese and
SiteSetting.search_tokenize_japanese respectively
2022-02-07 09:21:14 +08:00
janzenisaac cffc2836cb
DEV: Don't allow users to immediately reinvite (#15722)
- Limit bulk re-invite to 1 time per day
- Move bulk invite by csv behind a site setting (hidden by default)
- Bump invite expiry from 30 -> 90 days

## Updates to rate_limiter
When limiting reinvites I found that **staff** are never limited in any way. So I updated the **rate_limiter** model to allow for a few things:
- add an optional param of `staff_limit`, which (when included and passed values, and the user passes `.staff?`) will override the default `max` & `secs` values and apply them to the user.
- in the case you **do** pass values to `staff_limit` but the user **does not** pass `staff?` the standard `max` & `secs` values will be applied to the user.

This should give us enough flexibility to 
1. continue to apply a strict rate limit to a standard user
2. but also apply a secondary (less strict) limit to staff
2022-02-03 13:07:40 -06:00
Rafael dos Santos Silva 5b5cbbfe5c
FEATURE: Onebox for news.ycombinator.com (#15781) 2022-02-03 13:39:21 -03:00
Alan Guo Xiang Tan 4aa9a813ec
FIX: Liking whispers should not contribute to `Topic#like_count`. (#15703)
Non-staff users are not allowed to see whisper so this change prevents
non-staff user from seeing a like count that does not make sense to
them. In the future, we might consider adding another like count column
for staff user.

Follow-up to 4492718864
2022-02-03 16:24:33 +11:00
Natalie Tay 23a8341b28
FEATURE: Validate domain settings for blocked_onebox_domain only (#15754)
We want to prevent the user from adding ? or * minimally
when setting domains in sitesettings
2022-02-03 11:26:53 +08:00
Martin Brennan 82cb67e67b
FIX: Canonical Message-ID was incorrect for some cases (#15701)
When creating a direct message to a group with group SMTP
set up, and adding another person to that message in the OP,
we send an email to the second person in the OP via the group_smtp
job. This in turn creates an IncomingEmail record to guard against
IMAP double sync.

The issue with this was that this IncomingEmail (which is essentialy
a placeholder/dummy one) was having its Message-ID used as the canonical
References Message-ID for subsequent emails sent out to user_private_message
recipients (such as members of the group), causing threading issues in
the mail client. The canonical <topic/ID@HOST> format should be used
instead for these cases.

This commit fixes the issue by only using the IncomingEmail for the
OP's Message-ID if the OP was created via our handle_mail email receiver
pipeline. It does not make sense to use it in other cases.
2022-02-03 10:36:32 +10:00
Krzysztof Kotlarek bb01563682
FIX: post mover validation color and message (#15688)
When the record is not saved, we should display a proper message.
One potential reason can be plugins for example discourse-calendar is specifying that only first post can contain event
2022-02-02 16:22:52 +11:00
Rafael dos Santos Silva 2e2fee9da3
FIX: Remove svg icons from webmanifest shortcuts (#15765)
* FIX: Remove svg icons from webmanifest shortcuts

While SVGs are valid in the webmanifest, Chromium has not implemented
support for it in this specific manifest member.

Revert when https://bugs.chromium.org/p/chromium/issues/detail?id=1091612
lands.

* fix test
2022-02-01 15:26:58 -03:00
Natalie Tay aac9f43038
Only block domains at the final destination (#15689)
In an earlier PR, we decided that we only want to block a domain if 
the blocked domain in the SiteSetting is the final destination (/t/59305). That 
PR used `FinalDestination#get`. `resolve` however is used several places
 but blocks domains along the redirect chain when certain options are provided.

This commit changes the default options for `resolve` to not do that. Existing
users of `FinalDestination#resolve` are
- `Oneboxer#external_onebox`
- our onebox helper `fetch_html_doc`, which is used in amazon, standard embed 
and youtube
  - these folks already go through `Oneboxer#external_onebox` which already
  blocks correctly
2022-01-31 15:35:12 +08:00
Martin Brennan 0b8d0a14d7
DEV: Add markdown_additional_options to Site (#15738)
Sometimes plugins need to have additional data or options available
when rendering custom markdown features/rules that are not available
on the default opts.discourse object. These additional options should
be namespaced to the plugin adding them.

```
Site.markdown_additional_options["chat"] = { limited_pretty_text_markdown_rules: [] }
```

These are passed down to markdown rules on opts.discourse.additionalOptions.

The main motivation for adding this is the chat plugin, which currently stores
chat_pretty_text_features and chat_pretty_text_markdown_rules on
the Site object via additions to the serializer, and the Site object is
not accessible to import via markdown rules (either through
Site.current() or through container.lookup). So, to have this working
for both front + backend code, we need to attach these additional options
from the Site object onto the markdown options object.
2022-01-28 13:02:02 +10:00
Gerhard Schlager 1fef96a2e7
FIX: Prevent "integer out of range" when merging post timings (#15723) 2022-01-26 23:34:28 +01:00
Dan Ungureanu f5b94f152f
FIX: Allow staff to reset passwords by username (#15709)
When staff visits the user profile of another user, the `email` field
in the model is empty. In this case, staff cannot send the reset email
password because nothing is passed in the `login` field.

This commit changes the behavior for staff users to allow resetting
password by username instead.
2022-01-26 10:39:58 +02:00
Alan Guo Xiang Tan 77137c5d29 FIX: Single line emojis has emoji metadata indexed twice.
This commit fixes a bug where we our `HTMLScrubber` was only searching
for emoji img tags which contains only the "emoji" class. However, our emoji image tags
may contain more than just the "emoji" class like "only-emoji" when an
emoji exists by itself on a single line.
2022-01-24 14:03:17 +08:00
Bianca Nenciu 48e5d1af03
FIX: Improve top links section from user summary (#15675)
* Do not extract links for hotlinked images
* Include only links that have been clicked at least once in user
summary
2022-01-24 11:33:23 +11:00
Bianca Nenciu a0c040060a
FIX: Select best link from Atom feed (#15663)
Some Atom feeds can contain more than one link and it used to return
only the first link.
2022-01-21 17:54:18 +02:00
Martin Brennan 70af45055a
DEV: Clear custom field preload proxy on preload_custom_fields (#15671)
If a model class calls preload_custom_fields twice then
we have to clear this otherwise the fields are cached inside the
already existing proxy and no new ones are added, so when we check
for custom_fields[KEY] an error is likely to occur
2022-01-21 14:29:51 +10:00
Alan Guo Xiang Tan e4e37257cc FIX: Handle malformed URLs in `TopicEmbed.absolutize_urls`. 2022-01-21 11:18:54 +08:00
David Taylor 820564826e
FIX: Ensure that login does not fail for users with invite records (#15647)
In the unlikely, but possible, scenario where a user has no email_tokens, and has an invite record for their email address, login would fail. This commit fixes the `Invite` `user_doesnt_already_exist` validation so that it only applies to new invites, or when changing the email address.

This regressed in d8fe0f4199 (based on `git bisect`)
2022-01-20 10:54:38 +00:00
Dan Ungureanu 5b7bddf966
FIX: Prefer email when resetting password (#15650)
The UI used to request a password reset by username when the user was
logged in. This did not work when hide_email_already_taken site setting
was enabled, which disables the lookup-by-username functionality.

This commit also introduces a check to ensure that the parameter is an
email when hide_email_already_taken is enabled as the single allowed
type is email (no usernames are allowed).
2022-01-20 10:04:45 +02:00
Natalie Tay f5ea00c73f
FIX: Respect blocked domains list when redirecting (#15656)
Our previous implementation used a simple `blocked_domain_array.include?(hostname)`
so some values were not matching. Additionally, in some configurations like ours, we'd used
"cat.*.dog.com" with the assumption we'd support globbing.

This change implicitly allows globbing by blocking "http://a.b.com" if "b.com" is a blocked 
domain but does not actively do anything for "*".

An upcoming change might include frontend validation for values that can be inserted.
2022-01-20 14:12:34 +08:00
Bianca Nenciu 376799b1a4
FIX: Hide excerpt of binary files in GitHub onebox (#15639)
Oneboxer did not know if a file is binary or not and always tried to
show an excerpt of the file.
2022-01-19 14:45:36 +02:00
Blake Erickson 12f041de5d
FIX: Tag watching for everyone tag groups (#15622)
* FIX: Tag watching for everyone tag groups

Tags in tag groups that have permissions set to everyone were not able
to be saved correctly. A user on their preferences page would mark the
tags that they wanted to save, but the watched_tags in the response
would be empty. This did not apply to admins, just regular users. Even
though the watched tags were being saved in the db, the user serializer
response was filtering them out. When a user refreshed their preferences
pages it would show zero watched tags.

This appears to be a regression introduced by:

0f598ca51e

The issue that needed to be fixed is that we don't track the "everyone"
group (which has an id of 0) in the group_users table. This is because
everyone has access to it, so why fill a row for every single user, that
would be a lot. The fix was to update the query to include tag groups
that had permissions set to the "everyone" group (group_id 0).

I also added another check to the existing spec for updating
watched tags for tags that aren't in a tag group so that it checks the
response body. I then added a new spec which updates watched tags for
tags in a tag group which has permissions set to everyone.

* Resolve failing tests

Improve SQL query syntax for including the "everyone" group with the id
of 0.

This commit also fixes a few failing tests that were introduced. It
turns out that the Fabrication of the Tag Group Permissions was faulty.
What happens when creating the tag groups without any permissions is
that it sets the permission to "everyone". If we then follow up with
fabricating a tag group permission on the tag group instead of having a
single permission it will have 2 (everyone + the group specified)! We
don't want this. To fix it I removed the fabrication of tag group
permissions and just set the permissions directly when creating the tag
group.

* Use response.parsed_body instead of JSON.parse
2022-01-18 15:02:29 -07:00
Mark VanLandingham fcc80137ce
DEV: Add chat_group_mention notification type (#15616) 2022-01-18 08:26:27 -06:00
Roman Rizzi 5ee31cbf7d
FIX: Mark invites flash messages as HTML safe. (#15539)
* FIX: Mark invites flash messages as HTML safe.
This change should be safe as all user inputs included in the errors are sanitized before sending it back to the client.

Context: https://meta.discourse.org/t/html-tags-are-explicit-after-latest-update/214220

* If somebody adds a new error message that includes user input and doesn't sanitize it, using html-safe suddenly becomes unsafe again. As an extra layer of protection, we make the client sanitize the error message received from the backend.

* Escape user input instead of sanitizing
2022-01-18 09:38:31 -03:00
Rafael dos Santos Silva 3f91c8835b
FEATURE: Export topics to markdown (#15615)
* FEATURE: Export topics to markdown

The route `/raw/TOPIC_ID` will now export whole topics (paginated to 100
posts) in a markdown format.

See https://meta.discourse.org/t/-/152185/12
2022-01-17 18:05:14 -03:00
jbrw 2909b8b820
FIX: origins_to_regexes should always return an array (#15589)
If the SiteSetting `allowed_onebox_iframes` contains a value of `*`, it will use the values of `all_iframe_origins` during the Oneboxing process. If `all_iframe_origins` itself contains a value of `*`, `origins_to_regexes` will try to return a "catch-all" regex.

Other code assumes `origins_to_regexes`will return an array, so this change ensures the `*` case will return an array containing only the catch-all regex.
2022-01-17 12:48:41 -05:00
Jarek Radosz 31b27b3712
FIX: Broken GitHub folder onebox logic (#15612)
1. `html_doc.css('.Box.md')` always returns a truthy value (e.g. `[]`) so the second branch of the if-elsif never ran
2. `node&.css('text()')` was invalid code that would raise an error
3. Matching on h3 elements is no longer correct with the current html structure returned by GitHub
2022-01-17 18:32:07 +01:00
Roman Rizzi 8b3d50713d
FIX: Pass category and tag IDs to the emit webhook event job. (#15568)
* FIX: Pass category and tag IDs to the emit webhook event job.

Like webhooks won't fire when they're scoped to specific categories or tags because we're not passing the data to the job that emits it.

* Update config/initializers/012-web_hook_events.rb

Co-authored-by: Dan Ungureanu <dan@ungureanu.me>

Co-authored-by: Dan Ungureanu <dan@ungureanu.me>
2022-01-14 11:17:38 -03:00
Martin Brennan 5d0c2cba07
DEV: Fix typo for email encoded (#15577) 2022-01-14 09:33:15 +10:00
Robin Ward 6272edd121 DEV: Support for running theme test with Ember CLI (third attempt)
The second attempt fixed issues with smoke test.

This one makes sure minification only happens in production mode.
2022-01-13 16:02:07 -05:00
Martin Brennan 107239a442
Revert "DEV: Support for running theme test with Ember CLI (second attempt)" (#15559)
This reverts commit 2c7906999a.

The changes break some things in local development (putting JS files
into minified files, not allowing debugger, and others)
2022-01-13 10:05:35 +10:00
Robin Ward 2c7906999a DEV: Support for running theme test with Ember CLI (second attempt)
This PR includes support for running theme tests in legacy ember
production envrionments.
2022-01-12 15:43:29 -05:00
Dan Ungureanu 584c6a2e8b
SECURITY: Do not sign in unapproved users (#15552) 2022-01-12 22:24:54 +02:00
David Taylor 252bb87ab3
Revert "DEV: Support for running theme test with Ember CLI" (#15547)
This reverts commit ea84a82f77.

This is causing problems with `/theme-qunit` on legacy, non-ember-cli production sites. Reverting while we work on a fix
2022-01-11 23:38:59 +00:00
Robin Ward ea84a82f77 DEV: Support for running theme test with Ember CLI
This is quite complex as it means that in production we have to build
Ember CLI test files and allow them to be used by our Rails application.

There is a fair bit of glue we can remove in the future once we move to
Ember CLI completely.
2022-01-11 15:42:13 -05:00
Bianca Nenciu 5d35c38db2
FEATURE: Search screened IP address in blocks (#15461)
An admin could search for all screened ip addresses in a block by
using wildcards. 192.168.* returned all IPs in range 192.168.0.0/16.
This feature allows admins to search for a single IP address in all
screened IP blocks. 192.168.0.1 returns all IP blocks that match it,
for example 192.168.0.0/16.

* FEATURE: Remove roll up button for screened IPs

* FIX: Match more specific screened IP address first
2022-01-11 09:16:51 +02:00
Bianca Nenciu 5a8b8f6f1e
FEATURE: Show warning if user won't be mentioned (#15339)
The new warnings cover more cases and more accurate. Most of the
warnings will be visible only to staff members because otherwise they
would leak information about user's preferences.
2022-01-11 09:16:20 +02:00
Vinoth Kannan 6626089034
UX: change text of public_topic action code in login required sites. (#14764)
The wording "made this topic public" made confusion in login required forums.
2022-01-11 11:35:16 +05:30
Alan Guo Xiang Tan 2ee9a09c8c DEV: Fix failing test. 2022-01-11 11:45:22 +08:00
Alan Guo Xiang Tan c2afc3915b FEATURE: Customizable rules and plugins for `PrettyText.markdown`.
This commit extends the options which can be passed to
`PrettyText.markdown` so that which Markdown-it rules and Discourse
Markdown plugins to be used when rendering a text can be customizable.
Currently, this extension is mainly used by plugins.
2022-01-11 10:39:40 +08:00
Jarek Radosz 0d9ab0beb3
DEV: Fix a flaky spec (#15529)
It's very much possible for there to be a topic with id 999.
2022-01-11 01:51:57 +01:00
Vinoth Kannan f614b30032
FIX: don't trigger `topic_created` event for reply posts via email. (#15485)
Previously, we incorrectly triggered `topic_created` events when the posts are created via email since we didn't check the post number.
2022-01-10 13:54:10 +05:30
Alan Guo Xiang Tan fff8b98485 SECURITY: Advanced group search did not respect visiblity of groups. 2022-01-10 13:49:26 +08:00
Jarek Radosz 19fcb0b5ea
DEV: Prevent extraneous log message in specs (#15504)
Yo dawg, I put `silence_stdout` in your `silence_stdout` so you can still write to stdout? 🤔
2022-01-09 20:26:52 +01:00
Jarek Radosz 910796c4b4
DEV: Fix git deprecation warnings in specs (#15503)
The warnings on git 2.28+ are:

```
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: 	git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: 	git branch -m <name>
```
2022-01-09 20:26:19 +01:00
Jarek Radosz 5a50f18c0c
DEV: Avoid `$` globals (#15453)
Also:
* Remove an unused method (#fill_email)
* Replace a method that was used just once (#generate_username) with `SecureRandom.alphanumeric`
* Remove an obsolete dev puma `tmp/restart` file logic
2022-01-08 23:39:46 +01:00
Blake Erickson a850568a52
DEV: Document delete post API endpoint (#15495)
Adding a spec for documenting the delete post API endpoint for our api
docs. As part of this added detailed info for the `force_destroy`
parameter for permanently deleting a post.
2022-01-07 16:09:32 -07:00
janzenisaac 30388512ec
DEV: Retroactively add test for ignoring flagged responses (#15491) 2022-01-07 11:53:31 -06:00
Arpit Jalan 554ff07786
FIX: when composing a message do not suggest deleted posts users (#15488) 2022-01-07 17:06:11 +05:30
Martin Brennan 04c7776650
DEV: Rolling back bookmarkable column changes (#15482)
It is too close to release of 2.8 for incomplete
feature shenanigans. Ignores and drops the columns and drops
the trigger/function introduced in
e21c640a3c.
Will pick this feature back up post-release.
2022-01-07 12:16:43 +10:00
David Taylor 78d0ec35a5
DEV: Deprecate OAuth2Authenticator and OAuth2UserInfo (#15427)
These have been superseded by ManagedAuthenticator and UserAssociatedAccount. For more information, see https://meta.discourse.org/t/106695
2022-01-06 16:50:18 +00:00
Andrei Prigorshnev 16d9b2755c
DEV: rename single_sign_on classes to discourse_connect (#15332) 2022-01-06 16:28:46 +04:00
Martin Brennan e21c640a3c
DEV: Add polymorphic bookmarkable columns (#15454)
We are planning on attaching bookmarks to more and
more other models, so it makes sense to make a polymorphic
relationship to handle this. This commit adds the new
columns and backfills them in the bookmark table, and
makes sure that any new bookmark changes fill in the columns
via DB triggers.

This way we can gradually change the frontend and backend
to use these new columns, and eventually delete the
old post_id and for_topic columns in `bookmarks`.
2022-01-06 08:56:05 +10:00
Peter Zhu ab33d44bf6 DEV: Fix tests broken in 3.0 due to separation of keyword arguments
Tests fail in Ruby 3.0 and later due to separation of positional and
keyword arguments. RSpec treats the hash at the end of include_examples
as keyword arguments when it should be passed as a positional argument.
2022-01-05 13:37:52 -05:00
Peter Zhu c5fd8c42db
DEV: Fix methods removed in Ruby 3.2 (#15459)
* File.exists? is deprecated and removed in Ruby 3.2 in favor of
File.exist?
* Dir.exists? is deprecated and removed in Ruby 3.2 in favor of
Dir.exist?
2022-01-05 18:45:08 +01:00
Bianca Nenciu 5e2e178fcf FIX: Hide user's bio if profile is restricted
The bio was sometimes visible in the meta tags even though it it should
not have been.
2022-01-05 10:23:36 +08:00
Martin Brennan 099b679fc5
DEV: Clean up old bookmark code (#15455)
The rake task deleted here was added back in Feb 2020
when bookmarks were first converted from PostAction
records, it is no longer needed. The ignored columns
were removed in ed83d7573e.
2022-01-05 10:02:02 +10:00
Martin Brennan 20fe5eceb8
FEATURE: Scheduled group email credential problem check (#15396)
This commit adds a check that runs regularly as per
2d68e5d942 which tests the
credentials of groups with SMTP or IMAP enabled. If any issues
are found with those credentials a high priority problem is added to the
admin dashboard.

This commit also formats the admin dashboard differently if
there are high priority problems, bringing them to the top of
the list and highlighting them.

The problem will be cleared if the issue is fixed before the next
problem check, or if the group's settings are updated with a valid
credential.
2022-01-04 10:14:33 +10:00
Daniel Waterworth 4a0f73a337
PERF: Turn common test prefixes into contexts in topics_controller_spec (#15439) 2022-01-03 11:00:03 -06:00
Arpit Jalan 702685b6a0
SECURITY: only show user suggestions with regular post (#15436) 2022-01-03 13:37:40 +05:30
Roman Rizzi 23b75d8a2b
FEATURE: Before consolidation callbacks. (#15428)
You can add callbacks that get called before updating an already consolidated notification or creating a consolidated one.

Instances of this rule can add callbacks to access the old notifications about to be destroyed or the consolidated one and add additional data inside the data hash versus having to execute extra queries when adding this logic inside the `set_mutations` block.
2021-12-30 15:40:16 -03:00
Osama Sayegh 8e6988163f
FIX: Don't allow NULL values for `notification_level` in `category_users` (#15407) 2021-12-29 09:19:39 +11:00
Roman Rizzi e005e3f153
DEV: Create post actions without creating a notification and store custom data. (#15397)
I plan to use this in an upcoming discourse-reactions PR, where I want to like a post without notifying the user, so I can instead create a reaction notification.

Additionally, we decouple the a11y attributes from the icon itself, which will let us extend the widget's icon without losing them.
2021-12-27 11:25:37 -03:00
Daniel Waterworth 9e18868c31
DEV: Re-organize TopicsController#reset_new tests (#15399) 2021-12-23 12:29:51 -06:00
David Taylor cdf4d7156e
DEV: Introduce Auth::Result API for overrides_* (#15378)
This allows authenticators to instruct the Auth::Result to override attributes without using the general site settings. This provides an easy migration path for auth plugins which offer their own "overrides email", "overrides username" or "overrides name" settings. With this new api, they can set `overrides_*` on the result object, and the attribute will be overriden regardless of the general site setting.

ManagedAuthenticator is updated to use this new API. Plugins which consume ManagedAuthenticator will instantly take advantage of this change.
2021-12-23 10:53:17 +00:00
Blake Erickson b705971d42
DEV: Allow for taller images in posts and oneboxes (#15395)
The previous default aspect ratio for cropping tall images was a little
too strict and was cutting off images. This new setting should allow for
a larger range of image sizes before cropping them.
2021-12-22 16:01:34 -07:00
Martin Brennan 19089f21d2
DEV: Add API docs for uploads and API doc watcher (#15387)
This commit adds API documentation for the new upload
endpoints related to direct + multipart external uploads.

Also included is a rake task which watches the files in
the spec/requests/api directory and calls a script file
(spec/regenerate_swagger_docs) whenever one changes. This
script runs rake rswag:specs:swaggerize and then copies
the openapi.yml file over to the discourse_api_docs repo
directory, and hits a script there to convert the YML to
JSON so the API docs are refreshed while the server is
still running. This makes the loop of making a doc change
and seeing it in the local server much faster.

The rake task is rake autospec:swagger
2021-12-23 08:40:15 +10:00
Daniel Waterworth 5ff0b86b57
PERF: Make tests faster by prefabricating more things (#15392) 2021-12-22 11:09:43 -06:00
Jarek Radosz 72ad5bf8bd
FIX: SiteSetting.title was being polluted in StaticController (#15385)
Regressed in #15324
2021-12-21 20:51:18 +01:00
Daniel Waterworth 80591724cf
DEV: Avoid shadowing variables in topics_controller_spec (#15383) 2021-12-21 12:28:12 -06:00
Andrei Prigorshnev c202252190
FEATURE: when suggesting usernames skip input that consist entirely of disallowed characters (#15368) 2021-12-21 21:13:05 +04:00
Andrei Prigorshnev 952bebc5a1
FIX: fallback to `anonymous` not `user` when suggesting usernames for anonymous users (#15354) 2021-12-21 21:09:55 +04:00
Jarek Radosz c209be09f1
DEV: Fix a flaky test (#15374)
See: https://github.com/discourse/discourse/runs/4589134998?check_suite_focus=true
2021-12-21 02:21:47 +01:00
Roman Rizzi 491f020771
FIX: Use correct argument name in pending reviewables translation (#15371) 2021-12-20 17:35:51 -03:00
Daniel Waterworth 7e0c1fb039
PERF: Make tests faster by prefabricating more things (#15370) 2021-12-20 12:59:10 -06:00
Sam b6c3e9aa03
FEATURE: hide_email_address_taken forces use of email in forgot password form (#15362)
* FEATURE: hide_email_address_taken forces use of email in forgot password form

This strengthens this site setting which is meant to be used to harden sites
that are experiencing abuse on forgot password routes.

Previously we would only deny letting people know if forgot password worked on not
New change also bans usage of username for forgot password when enabled
2021-12-20 12:54:10 +11:00
Martin Brennan 2d68e5d942
FEATURE: Scheduled problem checks for admin dashboard (#15327)
This commit introduces scheduled problem checks for the admin dashboard, which are long running or otherwise cumbersome problem checks that will be run every 10 minutes rather than every time the dashboard is loaded. If these scheduled checks add a problem, the problem will remain until it is cleared or until the scheduled job runs again.

An example of a check that should be scheduled is validating credentials against an external provider.

This commit also introduces the concept of a `priority` to the problems generated by `AdminDashboardData` and the scheduled checks. This is `low` by default, and can be set to `high`, but this commit does not change any part of the UI with this information, only adds a CSS class.

I will be making a follow up PR to check group SMTP credentials.
2021-12-20 09:59:11 +10:00
jbrw 6e925fee6f
FIX: Use basic meta description if other description tags are missing (#15356)
When attempting to Onebox a page if there is no `meta property="og:description"` tag but there is a  `meta name="description"` tag, Onebox should try to use that value.
2021-12-17 19:36:54 -05:00
Gerhard Schlager 769388b8ba FIX: Translation overrides from fallback locale didn't work on client
Discourse sent only translation overrides for the current language to the client instead of sending overrides from fallback locales as well. This especially impacted en_GB -> en since most overrides would be done in English instead of English (UK).

This also adds lots of tests for previously untested code.

There's a small caveat: The client currently doesn't handle fallback locales for MessageFormat strings. That is why overrides for those strings always have a higher priority than regular translations. So, as an example, the lookup order for MessageFormat strings in German is:
1. override for de
2. override for en
3. value from de
4. value from en
2021-12-17 14:03:35 +01:00
Gerhard Schlager 4cd5158974 FIX: "Customize Text" showed compiled MessageFormat string for overridden `_MF` translations 2021-12-17 14:03:35 +01:00
Gerhard Schlager e19a7a7c8d FIX: translation precedence was different on client and server
As an example, the lookup order for German was:

1. override for de
2. override for en
3. value from de
4. value from en

After this change the lookup order is the same as on the client:
1. override for de
2. value from de
3. override for en
4. value from en

see /t/16381
2021-12-17 14:03:35 +01:00
Daniel Waterworth ca0eb8041d
PERF: Prefabricate more parts of users_controller_spec (#15335) 2021-12-16 13:36:49 -06:00
Andrei Prigorshnev 9365c4b364
DEV: make sure we handle staged users correctly in DiscourseConnect (#15320)
Some time ago, we made this fix to external authentication –  https://github.com/discourse/discourse/pull/13706. We didn't address Discourse Connect (https://meta.discourse.org/t/discourseconnect-official-single-sign-on-for-discourse-sso/13045) at that moment, so I wanted to fix it for Discourse Connect as well.

Turned out though that Discourse Connect doesn't contain this problem and already handles staged users correctly. This PR adds tests that confirm it. Also, I've extracted two functions in Discourse Connect implementation along the way and decided to merge this refactoring too (the refactoring is supported with tests).
2021-12-16 19:44:07 +04:00
Jarek Radosz c46b351888
DEV: Make i18n spec ignore these test-only settings (#15329)
Fixes flakyness introduced in #15324
2021-12-16 15:49:20 +01:00
Jarek Radosz 2a4df93b8e
FEATURE: Allow to modify topic-backed static pages (#15324)
A plugin API that allows customizing existing topic-backed static pages, like:
faq, tos, privacy (see: StaticController) The block passed to this
method has to return a SiteSetting name that contains a topic id.

```
add_topic_static_page("faq") do |controller|
  current_user&.locale == "pl" ? "polish_faq_topic_id" : "faq_topic_id"
end
```

You can also add new pages in a plugin, but remember to add a route,
for example:

```
get "contact" => "static#show", id: "contact"
```
2021-12-16 04:24:11 +01:00
Daniel Waterworth 102fa71ef3
PERF: Speed up the tests by pre-fabricating more things (#15318) 2021-12-15 11:41:14 -06:00
Natalie Tay 9fd1a00eef
DEV: Remove customer flair from being an official plugin (#15315) 2021-12-15 22:30:50 +08:00
Roman Rizzi 0719531bd3
FIX: Notify group members when someone quotes or replies to their post. (#15295)
When a member set a group PM notification level to Normal, we didn't notify group members if someone quoted or replied to one of their posts.
2021-12-15 09:07:39 -03:00
Alan Guo Xiang Tan 6fe4c8ae58
DEV: Fix intermittent failing spec. (#15308) 2021-12-15 09:51:46 +01:00
Daniel Waterworth 61e449379c
PERF: Prefabricate posters in topics_controller_spec (#15297)
It would be clearer to prefabricate posts, but that changes redis and
enabling snapshotting for all the tests in topics_controller_spec is
expensive.
2021-12-14 12:09:07 -06:00
Mark VanLandingham 175ced5096
DEV: Add chat invitation notification type (#15288) 2021-12-14 09:01:53 -06:00
Daniel Waterworth 9334abe249
PERF: Prefabricate more of topics_controller_spec.rb (#15281) 2021-12-13 13:44:55 -06:00
David Taylor 08f4edc032
DEV: Reset OmniAuth test mode between tests (#15278)
OmniAuth test mode is disabled by default, so that we can integration-test the omniauth strategies. Sometimes, we manually enable test mode for specific specs. This commit ensures that test_mode is always disabled again after each spec.
2021-12-13 17:05:26 +00:00
Jarek Radosz 42015fe8b2
DEV: Drop env-based SiteSetting deprecation errors (#15273)
These were deprecated ~4 years ago. No need to keep the errors anymore.
2021-12-13 17:36:29 +01:00
David Taylor 6e9bb84d12
FIX: Ensure theme names are escaped in HTML attributes (#15272)
If a theme name contained a double-quote, this problem could lead to invalid/unexpected HTML in the `<head>`

Note that this is not considered a security issue because themes can only be installed/named by administrators, and themes/administrators already have the ability to run arbitrary javascript.
2021-12-13 10:50:09 +00:00
Dan Ungureanu adb6202c94
FIX: Check if invite domain is valid (#15238)
* FIX: Check if invite domain is valid

Previous regex checked for generic hostname, which is too generic for
this case.
2021-12-13 16:39:14 +11:00
Dan Ungureanu 3d4aee1487
DEV: Drop unused column email_tokens.token (#15203) 2021-12-13 16:29:47 +11:00
Daniel Waterworth 02245ce41f
PERF: Redis snapshotting during tests (#15260)
We can fake redis transactions so that `fab!` works for redis and PG
data, but it's too slow to be used indiscriminately. Instead, you can
opt into it with the `use_redis_snapshotting` helper.

Insofar as snapshotting allows us to `fab!` more things, it provides a
speedup.
2021-12-10 14:25:26 -06:00
Blake Erickson b93b6c4299
FIX: Blurry onebox favicon images (#15258)
This is a fix to address blurry onebox favicon images if the site you
are linking to happens to have a favicon.ico file that contains multiple
images.

This fix detects of we are trying to create an upload for a favicon.ico
file. We then convert it to a png and not a jpeg like we were doing. We
want a png because it will preserve transparency, otherwise if we
convert it to a jpeg we lose that and it looks bad on dark themed sites.

This fix also addresses the fact that .ico files can include multiple
images. The blurry images we were producing was caused by the
ImageMagick `-flatten` option when the .ico file had multiple images
which then squishes them all together. So for .ico files we are no
longer flattening them and instead we are grabbing the last image in the
.ico bundle and converting that single image to a png.
2021-12-10 12:25:50 -07:00
Roman Rizzi b7b61d4b56
FEATURE: A notification consolidation plan for keeping the latest one. (#15249)
We previously used ConsolidateNotifications with a threshold of 1 to re-use an existing notification and bump it to the top instead of creating a new one. It produces some jumpiness in the user notification list, and it relies on updating the `created_at` attribute, which is a bit hacky.

As a better alternative, we're introducing a new plan that deletes all the previous versions of the notification, then creates a new one.
2021-12-10 10:32:15 -03:00
Roman Rizzi 3602f83cf4
FEATURE: Delete previous reviewable reminders. (#15250)
We send the reminder using the GroupMessage class, which supports removing previous messages. We can't match them by raw because they could mention different moderators. Also, I had to change the subject to remove dynamically generated values, which is necessary for finding them.
2021-12-10 10:17:39 -03:00
Daniel Waterworth ec36cddd2f
FIX: Defer topic/post created events until emails have been added (#15252) 2021-12-09 14:45:07 -06:00
Angus McLeod df3886d6e5
FEATURE: Experimental support for group membership via google auth (#14835)
This commit introduces a new site setting "google_oauth2_hd_groups". If enabled, group information will be fetched from Google during authentication, and stored in the Discourse database. These 'associated groups' can be connected to a Discourse group via the "Membership" tab of the group preferences UI. 

The majority of the implementation is generic, so we will be able to add support to more authentication methods in the near future.

https://meta.discourse.org/t/managing-group-membership-via-authentication/175950
2021-12-09 12:30:27 +00:00
Daniel Waterworth 3b0d46c659
FIX: make tests work with higher starting upload id (#15237) 2021-12-08 13:13:59 -06:00
Andrei Prigorshnev 4e8983036a
DEV: do not return no_result_help from the server (#15220)
We don't need it anymore. Actually, I removed using of it on the client side a long time ago, when I was working on improving blank page syndrome on user activity pages (see https://github.com/discourse/discourse/pull/14311).

This PR also removes some old resource strings that we don't use anymore. We have new strings for blank pages.
2021-12-08 21:46:54 +04:00
Dan Ungureanu d8fe0f4199
FEATURE: Restrict link invites to email domain (#15211)
Allow multiple emails to redeem a link invite only if the email domain
name matches the one specified in the link invite.
2021-12-08 17:06:57 +02:00
Bianca Nenciu b1c11d5787
FIX: Select correct topic draft for user (#15234)
The old query could return multiple rows.
2021-12-08 15:23:44 +02:00
Bianca Nenciu 049bc33838
FIX: Update has_topic_draft when draft is updated (#15219)
Current user state regarding the new topic draft was not updated when
the draft was created or destroyed.
2021-12-08 14:40:35 +02:00
David Taylor f799b8bfb1
FIX: Ensure MessageIdService can handle hostname changes and multisite (#15231) 2021-12-08 11:17:20 +00:00
Martin Brennan f26b8b448d
FIX: References header leading to broken email threading (#15206)
Since 3b13f1146b the email threading
in mail clients has been broken, because the random suffix meant
that the References header would always be different for non-group
SMTP email notifications sent out.

This commit fixes the issue by always using the "canonical" topic
reference ID inside the References header in the format:

topic/TOPIC_ID@HOST

Which was the old format. We also add the References header to
notifications sent for the first post arriving, so the threading
works for subsequent emails. The Message-ID header is still random
as per the previous change.
2021-12-08 08:14:48 +10:00
Daniel Waterworth 6cae6aadf4
FIX: Make thumbnail tests start with a clean slate (#15216)
* FIX: Make thumbnail tests start with a clean slate

Unfortunately, this exposes the fact that they don't actually work.
Marking as pending for now.
2021-12-07 13:07:45 -06:00
Andrei Prigorshnev 77cca39c2d
DEV: dry up omni_auth_callback_controller_spec (#15213) 2021-12-07 23:06:44 +04:00
Daniel Waterworth 8371c96b7e
PERF: Speed up tests (#15214) 2021-12-07 12:45:58 -06:00
Daniel Waterworth 95c75d3f4f
DEV: Set uploads_id_seq starting point before tests (#15215)
Some tests don't pass when this is elevated. They should be fixed,
since, at some point, we may create enough uploads during tests that
they fail naturally.
2021-12-07 12:26:35 -06:00
Loïc Guitaut d0888c190e FIX: Display pending posts in a moderated category
Currently we display pending posts in topics (both for author and staff
members) but the feature is only enabled when there’s an enabled global site
setting related to moderation.

This patch allows to have the same behavior for a site where there’s
nothing enabled globally but where a moderated category exists. So when
browsing a topic of a moderated category, the presence of pending posts
will be checked whereas nothing will happen in a normal category.
2021-12-07 10:14:45 +01:00
Alan Guo Xiang Tan 4e67297a7c FIX: Missing allowed urls when displaying granualar API key scopes.
Follow-up to 3791fbd919
2021-12-07 10:17:17 +08:00
Alan Guo Xiang Tan 44588255fc FEATURE: Introduce API scopes for badges. 2021-12-07 10:17:17 +08:00
Andrei Prigorshnev f3508065a3
FIX: auth incorrectly handles duplicate usernames (#15197) 2021-12-06 20:49:04 +04:00
Roman Rizzi 43903f8dfe
FIX: Updating a consolidated notification should bump it to the top. (#15199)
In the future, it would be better to have a consolidated_at timestamp instead of updating created_at.
2021-12-06 12:31:44 -03:00
Daniel Waterworth 3ebce550fe
DEV: Make add_api_parameter_route parameter deprecations errors (#15198)
Since we said we would remove support in 2.7, this is overdue.
2021-12-06 09:10:14 -06:00
Andrei Prigorshnev 10cc082560
FIX: when using external auth disallowed characters weren't removed from username (#15185) 2021-12-06 15:06:35 +04:00
Jarek Radosz 28bf9599f5
FEATURE: Pre-setting user locale via bulk invite (#15195) 2021-12-06 02:08:21 +01:00
Martin Brennan 3b13f1146b
FIX: Add random suffix to outbound Message-ID for email (#15179)
Currently the Message-IDs we send out for outbound email
are not unique; for a post they look like:

topic/TOPIC_ID/POST_ID@HOST

And for a topic they look like:

topic/TOPIC_ID@HOST

This commit changes the outbound Message-IDs to also have
a random suffix before the host, so the new format is
like this:

topic/TOPIC_ID/POST_ID.RANDOM_SUFFIX@HOST

Or:

topic/TOPIC_ID.RANDOM_SUFFIX@HOST

This should help with email deliverability. This change
is backwards-compatible, the old Message-ID format will
still be recognized in the mail receiver flow, so people
will still be able to reply using Message-IDs, In-Reply-To,
and References headers that have already been sent.

This commit also refactors Message-ID related logic
to a central location, and adds judicious amounts of
tests and documentation.
2021-12-06 10:34:39 +10:00
Daniel Waterworth 959923d3cf
FIX: Match for indeterminate depth in URL during upload tests (#15186)
Since the uploads id sequence counter isn't reset before test runs, the
URL might not be /1X/
2021-12-03 16:05:27 -06:00
Daniel Waterworth 63112f89a3
PERF: Shave off some test-suite time (#15183) 2021-12-03 14:54:07 -06:00
Blake Erickson 643f82d8d6
DEV: Update email responses in api docs (#15178)
Documenting the `/u/:username:/emails.json` endpoint.

Also removing some email fields from user api responses because they
aren't actually included in the response unless you are querying
yourself.
2021-12-03 08:03:58 -07:00
Michelle Bueno Saquetim Vendrame 9b5836aa1d
Add three reports (#14338)
* Add report top_users_by_received_likes

* Add report top_users_by_received_likes_from_inferior_trust_level

* Add report top_users_by_likes_received_from_a_variety_of_people

* Add test to report_top_users_by_received_likes

* add top_users_by_likes_received_from_a_variety_of_people report test

* add top_users_by_likes_received_from_inferior_trust_level report tests
2021-12-02 22:41:55 +05:30
Daniel Waterworth 2f04a9b9fb
DEV: Remove site_setting_saved event (#15164)
We said we would drop it from 2.4, so this is long overdue

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2021-12-02 09:33:03 -06:00
David Taylor cfb6199a95
FIX: Don't redirect XHR/JSON requests when login is required (#15093)
When redirecting to login, we store a destination_url cookie, which the user is then redirected to after login. We never want the user to be redirected to a JSON URL. Instead, we should return a 403 in these situations.

This should also be much less confusing for API consumers - a 403 is a better representation than a 302.
2021-12-02 15:12:25 +00:00
Andrei Prigorshnev 1c0022c195
FIX: extract and fix overriding of usernames by external auth (#14637) 2021-12-02 17:42:23 +04:00
Osama Sayegh 1d69261bc0
FIX: Set `auto_update` to false for non-git themes/components (#15157)
Related to: 20f736aa11.

`auto_update` is true by default at the database level, but it doesn't make sense for `auto_update` to be true on themes that are not imported from a Git repository.
2021-12-01 19:58:13 +03:00
David Taylor 1fa7a87f86
SECURITY: Remove ember-cli specific response from application routes (#15155)
Under some conditions, these varied responses could lead to cache poisoning, hence the 'security' label.

Previously the Rails application would serve JSON data in place of HTML whenever Ember CLI requested an `application.html.erb`-rendered page. This commit removes that logic, and instead parses the HTML out of the standard response. This means that Rails doesn't need to customize its response for Ember CLI.
2021-12-01 16:10:40 +00:00
Natalie Tay 0f598ca51e
SECURITY: Only show tags to users with permission (#15148) 2021-12-01 10:26:56 +08:00
Krzysztof Kotlarek 9cabd3721b
FEATURE: ability to add description to tags (#15125)
Ability to add description to tags, which will be displayed on hover.
2021-12-01 09:18:56 +11:00
Jarek Radosz 78723345c0
DEV: Fix a flaky bookmarks test (#15129)
* DEV: Specify bookmarks order

It's better to order by id than to have a semi-random order. Fixes a flaky test:

```
 1) TopicView with a few sample posts #bookmarks gets the first post bookmark reminder at for the user
59
     Failure/Error: expect(first[:post_id]).to eq(bookmark1.post_id)
60

61
       expected: 1901
62
            got: 1902
63

64
       (compared using ==)
65
     # ./spec/components/topic_view_spec.rb:420:in `block (4 levels) in <main>'
66
     # ./spec/rails_helper.rb:284:in `block (2 levels) in <top (required)>'
67
     # ./vendor/bundle/ruby/2.7.0/gems/webmock-3.14.0/lib/webmock/rspec.rb:37:in `block (2 levels) in <top (required)>'
68
```

* Change test

* Revert "DEV: Specify bookmarks order"

This reverts commit 1f50026231.
2021-12-01 08:56:00 +11:00
Roman Rizzi 1fc06520bd
REFACTOR: Improve support for consolidating notifications. (#14904)
* REFACTOR: Improve support for consolidating notifications.

Before this commit, we didn't have a single way of consolidating notifications. For notifications like group summaries, we manually removed old ones before creating a new one. On the other hand, we used an after_create callback for likes and group membership requests, which caused unnecessary work, as we need to delete the record we created to replace it with a consolidated one.

We now have all the consolidation rules centralized in a single place: the consolidation planner class. Other parts of the app looking to create a consolidable notification can do so by calling Notification#consolidate_or_save!, instead of the default Notification#create! method.

Finally, we added two more rules: one for re-using existing group summaries and another for deleting duplicated dashboard problems PMs notifications when the user is tracking the moderator's inbox. Setting the threshold to one forces the planner to apply this rule every time.

I plan to add plugin support for adding custom rules in another PR to keep this one relatively small.

* DEV: Introduces a plugin API for consolidating notifications.

This commit removes the `Notification#filter_by_consolidation_data` scope since plugins could have to define their criteria. The Plan class now receives two blocks, one to query for an already consolidated notification, which we'll try to update, and another to query for existing ones to consolidate.

It also receives a consolidation window, which accepts an ActiveSupport::Duration object, and filter notifications created since that value.
2021-11-30 13:36:14 -03:00
Andrei Prigorshnev 88f9bb3dc9
FIX: do not suggest "user1" as a username to invited users (#15031)
Recently, the wrong new behavior appeared – we started to suggest to invited users usernames like "user1".

To reproduce:
1. Create an invitation with default settings, do not restrict it to email
2. Copy an invitation link and follow it in incognito mode
See username already filled, with eg “user1”. See screenshot. Should be empty.

This bug was very likely introduced by my recent changes to UserNameSuggester.
2021-11-30 16:59:37 +04:00
Osama Sayegh 7bd3986b21
FEATURE: Replace `Crawl-delay` directive with proper rate limiting (#15131)
We have a couple of site setting, `slow_down_crawler_user_agents` and `slow_down_crawler_rate`, that are meant to allow site owners to signal to specific crawlers that they're crawling the site too aggressively and that they should slow down.

When a crawler is added to the `slow_down_crawler_user_agents` setting, Discourse currently adds a `Crawl-delay` directive for that crawler in `/robots.txt`. Unfortunately, many crawlers don't support the `Crawl-delay` directive in `/robots.txt` which leaves the site owners no options if a crawler is crawling the site too aggressively.

This PR replaces the `Crawl-delay` directive with proper rate limiting for crawlers added to the `slow_down_crawler_user_agents` list. On every request made by a non-logged in user, Discourse will check the User Agent string and if it contains one of the values of the `slow_down_crawler_user_agents` list, Discourse will only allow 1 request every N seconds for that User Agent (N is the value of the `slow_down_crawler_rate` setting) and the rest of requests made within the same interval will get a 429 response. 

The `slow_down_crawler_user_agents` setting becomes quite dangerous with this PR since it could rate limit lots if not all of anonymous traffic if the setting is not used appropriately. So to protect against this scenario, we've added a couple of new validations to the setting when it's changed:

1) each value added to setting must 3 characters or longer
2) each value cannot be a substring of tokens found in popular browser User Agent. The current list of prohibited values is: apple, windows, linux, ubuntu, gecko, firefox, chrome, safari, applewebkit, webkit, mozilla, macintosh, khtml, intel, osx, os x, iphone, ipad and mac.
2021-11-30 12:55:25 +03:00
Alan Guo Xiang Tan 0edacbd8f7
DEV: Load fabricators for plugins automatically. (#15106) 2021-11-30 15:55:45 +11:00
Loïc Guitaut a5fbb90df4 FEATURE: Display pending posts on user’s page
Currently when a user creates posts that are moderated (for whatever
reason), a popup is displayed saying the post needs approval and the
total number of the user’s pending posts. But then this piece of
information is kind of lost and there is nowhere for the user to know
what are their pending posts or how many there are.

This patch solves this issue by adding a new “Pending” section to the
user’s activity page when there are some pending posts to display. When
there are none, then the “Pending” section isn’t displayed at all.
2021-11-29 10:26:33 +01:00
Penar Musaraj d99deaf1ab
FEATURE: show recent searches in quick search panel (#15024) 2021-11-25 15:44:15 -05:00
Rafael dos Santos Silva 5647819de4
FEATURE: Send a 'noindex' header in non-canonical responses (#15026)
* FEATURE: Optionally send a 'noindex' header in non-canonical responses

This will be used in a SEO experiment.

Co-authored-by: David Taylor <david@taylorhq.com>
2021-11-25 16:58:39 -03:00
David Taylor 03998e0a29
FIX: Use CDN URL for internal onebox avatars (#15077)
This commit will also trigger a background rebake for all existing posts with internal oneboxes
2021-11-25 12:07:34 +00:00
Dan Ungureanu fa8cd629f1
DEV: Hash tokens stored from email_tokens (#14493)
This commit adds token_hash and scopes columns to email_tokens table.
token_hash is a replacement for the token column to avoid storing email
tokens in plaintext as it can pose a security risk. The new scope column
ensures that email tokens cannot be used to perform a different action
than the one intended.

To sum up, this commit:

* Adds token_hash and scope to email_tokens

* Reuses code that schedules critical_user_email

* Refactors EmailToken.confirm and EmailToken.atomic_confirm methods

* Periodically cleans old, unconfirmed or expired email tokens
2021-11-25 09:34:39 +02:00
Natalie Tay 4c46c7e334
DEV: Remove xlink hrefs (#15059) 2021-11-25 15:22:43 +11:00
janzenisaac 8c7cc426b7
FEATURE: Notify responders of post removal (#15049)
- Notify users whose posts were cascade deleted due to a flagged post
2021-11-24 09:28:20 -06:00
Bianca Nenciu 3ea8937157
FEATURE: Add email normalization rules setting (#14593)
When this setting is turned on, it will check that normalized emails
are unique. Normalized emails are emails without any dots or plus
aliases.

This setting can be used to block use of aliases of the same email
address.
2021-11-24 11:30:06 +02:00
Alan Guo Xiang Tan a6aff40e4b
DEV: Allow default scope to be configurable per topic. (#15018)
Not exposing this as a plugin API yet as we're testing it out with a
plugin.
2021-11-24 16:40:58 +08:00
Martin Brennan 44be79f095
FIX: Strip Auto-Submitted email header from group SMTP emails (#15057)
Remove Auto-Submitted header for group private message emails, it does
not make sense there and may hurt deliverability.

From https://www.iana.org/assignments/auto-submitted-keywords/auto-submitted-keywords.xhtml:

> Indicates that a message was generated by an automatic process, and is not a direct response to another message.
2021-11-24 10:54:01 +10:00
Bianca Nenciu 73760c77d9
FEATURE: Mention @here to notify users in topic (#14900)
Use @here to mention all users that were allowed to topic directly or
through group, who liked topics or read the topic. Only first 10 users
will be notified.
2021-11-23 22:25:54 +02:00
Dan Ungureanu ff7acc9828
FIX: Git should not prompt for credentials (#15062)
When cloning a public remote repository (no key), git should not prompt
for credentials.
2021-11-23 13:54:51 +02:00
Dan Ungureanu 948a1523ea
FIX: Keep existent tags when editing tag topics (#15050)
Allow current user to keep existent tags when adding or removing a tag.
For example, a user could not remove a tag from a topic if the topic
had another tag that was restricted to a different category.
2021-11-23 13:00:45 +02:00
Alan Guo Xiang Tan c0f278d358
DEV: Fix flaky specs due to 8226ab1099. (#15060)
The users all shared the same `User#last_seen_at` column so depending on
how the database returned the records, the user that we're interested in
may be excluded from the update query.

Follow-up to 8226ab1099
2021-11-23 15:26:55 +08:00
Alan Guo Xiang Tan 8226ab1099
PERF: Updating first unread PM for user not respecting limits. (#15056)
In b8c8909a9d, we introduced a regression
where users may have had their `UserStat.first_unread_pm_at` set
incorrectly. This commit introduces a migration to reset `UserStat.first_unread_pm_at` back to
`User#created_at`.

Follow-up to b8c8909a9d.
2021-11-23 12:51:54 +08:00
RogerBW fd66df5997
FEATURE: adds uploads scope for API keys (#14941)
* FEATURE: adds uploads scope for API keys

* Add basic test, change "image" to "file"
2021-11-22 10:49:08 -07:00
Rafael dos Santos Silva c891c44775
UX: Lower digest suppress period from a year to 6 months (#15023) 2021-11-22 14:37:32 -03:00
Jarek Radosz d91d67a442
FEATURE: Allow theme settings to request refresh (#15037)
Similar to site settings, adds support for `refresh` option to theme settings.

```yaml
super_feature_enabled:
  type: bool
  default: false
  refresh: true
```
2021-11-22 13:16:56 +01:00
Dan Ungureanu f43c433d50
FIX: Allow invites if must_approve_users is true (#15043)
This was implemented in 36e0e6a, but then regressed when the guardian
for invites was refactored in 76a7b75.
2021-11-22 14:04:35 +02:00
Martin Brennan 8a3ab1cc43
DEV: Fix spec (#15036)
Broken by 9f8ee8f137
2021-11-22 15:59:10 +11:00
Martin Brennan 9f8ee8f137
FIX: Don't publish notifications to MessageBus for inactive users (#15035)
We are pushing /notification-alert/#{user_id} and /notification/#{user_id}
messages to MessageBus from both PostAlerter and User#publish_notification_state.
This can cause memory issues on large sites with many users. This commit
stems the bleeding by only sending these alert messages if the user
in question has been seen in the last 30 days, which eliminates a large
chunk of users on some sites.
2021-11-22 14:38:49 +11:00
Martin Brennan fa6b87a1bf
SECURITY: Strip unrendered unicode bidirectional chars in code blocks (#15032)
When rendering the markdown code blocks we replace the
offending characters in the output string with spans highlighting a textual
representation of the character, along with a title attribute with
information about why the character was highlighted.

The list of characters stripped by this fix, which are the bidirectional
characters considered relevant, are:

U+202A
U+202B
U+202C
U+202D
U+202E
U+2066
U+2067
U+2068
U+2069
2021-11-22 10:43:03 +10:00
Vinoth Kannan fc1c76cfcc
FIX: exclude moderator_action post for reply count in user summary. (#14991)
Previously, incorrect reply counts are displayed in the "top categories" section of the user summary page since we included the `moderator_action` and `small_action` post types.

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2021-11-18 13:42:03 +05:30
Penar Musaraj 20f5474be9
FEATURE: Log only topic/post search queries in search log (#14994) 2021-11-18 09:21:12 +08:00
Martin Brennan b96c10a903
DEV: Extract shared external upload routes into controller helper (#14984)
This commit refactors the direct external upload routes (get presigned
put, complete external, create/abort/complete multipart) into a
helper which is then included in both BackupController and the
UploadController. This is done so UploadController doesn't need
strange backup logic added to it, and so each controller implementing
this helper can do their own validation/error handling nicely.

This is a follow up to e4350bb966
2021-11-18 09:17:23 +10:00
Osama Sayegh b86127ad12
FEATURE: Apply rate limits per user instead of IP for trusted users (#14706)
Currently, Discourse rate limits all incoming requests by the IP address they
originate from regardless of the user making the request. This can be
frustrating if there are multiple users using Discourse simultaneously while
sharing the same IP address (e.g. employees in an office).

This commit implements a new feature to make Discourse apply rate limits by
user id rather than IP address for users at or higher than the configured trust
level (1 is the default).

For example, let's say a Discourse instance is configured to allow 200 requests
per minute per IP address, and we have 10 users at trust level 4 using
Discourse simultaneously from the same IP address. Before this feature, the 10
users could only make a total of 200 requests per minute before they got rate
limited. But with the new feature, each user is allowed to make 200 requests
per minute because the rate limits are applied on user id rather than the IP
address.

The minimum trust level for applying user-id-based rate limits can be
configured by the `skip_per_ip_rate_limit_trust_level` global setting. The
default is 1, but it can be changed by either adding the
`DISCOURSE_SKIP_PER_IP_RATE_LIMIT_TRUST_LEVEL` environment variable with the
desired value to your `app.yml`, or changing the setting's value in the
`discourse.conf` file.

Requests made with API keys are still rate limited by IP address and the
relevant global settings that control API keys rate limits.

Before this commit, Discourse's auth cookie (`_t`) was simply a 32 characters
string that Discourse used to lookup the current user from the database and the
cookie contained no additional information about the user. However, we had to
change the cookie content in this commit so we could identify the user from the
cookie without making a database query before the rate limits logic and avoid
introducing a bottleneck on busy sites.

Besides the 32 characters auth token, the cookie now includes the user id,
trust level and the cookie's generation date, and we encrypt/sign the cookie to
prevent tampering.

Internal ticket number: t54739.
2021-11-17 23:27:30 +03:00
Bianca Nenciu cc1b45f58b
FIX: Convert URLs embedded topics to absolute form (#14975)
Sometimes the expanded post contained broken relative URLs because they
were not converted to their absolute form.
2021-11-17 16:39:49 +11:00
Martin Brennan 515acb8fc4
FIX: Error when SMTP enabled in PostAlerter with no incoming email (#14981)
When 31035010af
was done it failed to take into account the case where the smtp_enabled
site setting was true, but the topic had no allowed groups / no
incoming email record, which caused errors for topics even with
nothing to do with group SMTP.
2021-11-17 09:24:17 +10:00
Martin Brennan e7a4742490
FIX: Clean up emoji name which is file name (#14980)
Uppy adds the file name as the "name" parameter in the
payload by default, which means that for things like the
emoji uploader which have a name param used by the controller,
that param will be passed as the file name. We already use
the existing file name if the name param is null, so this
commit just does further cleanup of the name param, removing
the extension if it is a filename so we don't end up with
emoji names like blah_png.
2021-11-17 09:20:44 +10:00
Akshay Birajdar 6b5e8be25a Support parsing array in #param_to_integer_list
Co-authored-by: Akshay Birajdar <akshay.birajdar@coupa.com>
2021-11-16 10:27:00 -05:00
Vinoth Kannan c47a526371
FIX: exclude topics from muted tag in category featured list. (#14925)
Topics from muted tags were visible in the categories page's featured topics section since we didn't filter it before.
2021-11-16 12:10:50 +05:30
Martin Brennan 31035010af
FIX: Use correct group out of multiple for SMTP sender (#14957)
When there are multiple groups on a topic, we were selecting
the first from the topic allowed groups to act as the sender
email address when sending group SMTP replies via PostAlerter.
However, this was not ordered, and since there is no created_at
column on TopicAllowedGroup we cannot order this nicely, which
caused just a random group to be used (based on whatever postgres
decided it felt like that morning).

This commit changes the group used for SMTP sending to be the
group using the email_username of the to address of the first
incoming email for the topic, if there are more than one allowed
groups on the topic. Otherwise it just uses the only SMTP enabled
group.
2021-11-16 10:21:49 +10:00
Martin Brennan eabe2df8d2
FIX: Drop malformed CC addresses in GroupSmtpEmail job (#14934)
Sometimes, a user may have a malformed email such as
`test@test.com<mailto:test@test.com` their email address,
and as a topic participant will be included as a CC email
when sending a GroupSmtpEmail. This causes the CC parsing to
fail and further down the line in Email::Sender the code
to check the CC addresses expects an array but gets a string
instead because of the parse failure.

Instead, we can just check if the CC addresses are valid
and drop them if they are not in the GroupSmtpEmail job.
2021-11-16 08:15:11 +10:00
David Taylor f45853676f
SECURITY: Ensure _forum_session cookies cannot be reused between sites (#14950)
This only affects multisite Discourse instances (where multiple forums are served from a single application server). The vast majority of self-hosted Discourse forums do not fall into this category.

On affected instances, this vulnerability could allow encrypted session cookies to be re-used between sites served by the same application instance.
2021-11-15 15:50:12 +00:00
Martin Brennan 08e625c446
FIX: Use previous chunk to check if local backup chunk upload complete (#14896)
Uppy and Resumable slice up their chunks differently, which causes a difference
in this algorithm. Let's take a 131.6MB file (137951695 bytes) with a 5MB (5242880 bytes)
chunk size. For resumable, there are 26 chunks, and uppy there are 27. This is
controlled by forceChunkSize in resumable which is false by default. The final
chunk size is 6879695 (chunk size + remainder) whereas in uppy it is 1636815 (just remainder).

This means that the current condition of uploaded_file_size + current_chunk_size >= total_size
is hit twice by uppy, because it uses a more correct number of chunks. This
can be solved for both uppy and resumable by checking the _previous_ chunk
number * chunk_size as the uploaded_file_size.

An example of what is happening before that change, using the current
chunk number to calculate uploaded_file_size.

chunk 26: resumable: uploaded_file_size (26 * 5242880) + current_chunk_size (6879695) = 143194575 >= total_size (137951695) ? YES
chunk 26: uppy: uploaded_file_size (26 * 5242880) + current_chunk_size (5242880) = 141557760 >= total_size (137951695) ? YES
chunk 27: uppy: uploaded_file_size (27 * 5242880) + current_chunk_size (1636815) = 143194575 >= total_size (137951695) ? YES

An example of what this looks like after the change, using the previous
chunk number to calculate uploaded_file_size:

chunk 26: resumable: uploaded_file_size (25 * 5242880) + current_chunk_size (6879695) = 137951695 >= total_size (137951695) ? YES
chunk 26: uppy: uploaded_file_size (25 * 5242880) + current_chunk_size (5242880) = 136314880 >= total_size (137951695) ? NO
chunk 27: uppy: uploaded_file_size (26 * 5242880) + current_chunk_size (1636815) = 137951695 >= total_size (137951695) ? YES
2021-11-15 15:08:21 +10:00
Jarek Radosz 043e0dcad7
DEV: Don't try to load admin locales in tests (#14917)
It always fails with:

```
Failed to load resource: the server responded with a status of 403 (Forbidden), url: http://localhost:60099/extra-locales/admin?v=[…]
```
2021-11-13 15:31:55 +01:00
Rafael dos Santos Silva 6645243a26
SECURITY: Disallow caching of MIME/Content-Type errors (#14907)
This will sign intermediary proxies and/or misconfigured CDNs to not
cache those error responses.
2021-11-12 15:52:25 -03:00
Martin Brennan e0be6ce1ee
DEV: Catch Mocha::ExpectationError in request specs (#14897)
Same issue as 28b00dc6fc, the
Mocha::ExpectationError inherits from Exception instead
of StandardError so RspecErrorTracker does not show the
actual failed expectation in request specs, the status of
the response is just 500 with no further detail.
2021-11-12 13:10:16 +10:00
David Taylor 13fdc979a8
DEV: Improve multisite testing (#14884)
This commit adds the RailsMultisite middleware in test mode when Rails.configuration.multisite is true. This allows for much more realistic integration testing. The `multisite_spec.rb` file is rewritten to avoid needing to simulate a middleware stack.
2021-11-11 16:44:58 +00:00
Roman Rizzi a3814b1e56
FIX: Display top posts from private categories if the user has access. (#14878)
Users viewing the top topics from the categories page should see those belonging to a private category if they have access to it.
2021-11-11 13:35:03 -03:00
Martin Brennan e4350bb966
FEATURE: Direct S3 multipart uploads for backups (#14736)
This PR introduces a new `enable_experimental_backup_uploads` site setting (default false and hidden), which when enabled alongside `enable_direct_s3_uploads` will allow for direct S3 multipart uploads of backup .tar.gz files.

To make multipart external uploads work with both the S3BackupStore and the S3Store, I've had to move several methods out of S3Store and into S3Helper, including:

* presigned_url
* create_multipart
* abort_multipart
* complete_multipart
* presign_multipart_part
* list_multipart_parts

Then, S3Store and S3BackupStore either delegate directly to S3Helper or have their own special methods to call S3Helper for these methods. FileStore.temporary_upload_path has also removed its dependence on upload_path, and can now be used interchangeably between the stores. A similar change was made in the frontend as well, moving the multipart related JS code out of ComposerUppyUpload and into a mixin of its own, so it can also be used by UppyUploadMixin.

Some changes to ExternalUploadManager had to be made here as well. The backup direct uploads do not need an Upload record made for them in the database, so they can be moved to their final S3 resting place when completing the multipart upload.

This changeset is not perfect; it introduces some special cases in UploadController to handle backups that was previously in BackupController, because UploadController is where the multipart routes are located. A subsequent pull request will pull these routes into a module or some other sharing pattern, along with hooks, so the backup controller and the upload controller (and any future controllers that may need them) can include these routes in a nicer way.
2021-11-11 08:25:31 +10:00
Bianca Nenciu 3791fbd919
FEATURE: Add read-only scope to API keys (#14856)
This commit adds a global read-only scope that can be used to create
new API keys.
2021-11-10 17:48:00 +02:00
Martin Brennan 6a68bd4825
DEV: Limit list multipart parts to 1 (#14853)
We are only using list_multipart_parts right now in the
uploads controller for multipart uploads to check if the
upload exists; thus we don't need up to 1000 parts.

Also adding a note for future explorers that list_multipart_parts
only gets 1000 parts max, and adding params for max parts
and starting parts.
2021-11-10 08:01:28 +10:00
David Taylor 65a389c3ac
FIX: Allow bulk invites to be used with DiscourseConnect (#14862)
Support for invites alongside DiscourseConnect was added in 355d51af. This commit fixes the guardian method so that the bulk invite button functionality also works.
2021-11-09 17:43:23 +00:00
David Taylor 5ac10e2e79
DEV: Update DiscourseConnect nonce errors to be more descriptive (#14858) 2021-11-09 17:39:05 +00:00
Dan Ungureanu ec3758b573
FIX: Make PostRevisor more consistent (#14841)
* FIX: Preserve field types when updating revision

When a post was edited quickly twice by the same user, the old post
revision was updated with the newest changes. To check if the change
was reverted (i.e. rename topic A to B and then back to A) a comparison
of the initial value and last value is performed. If the check passes
then the intermediary value is dismissed and only the initial value and
the last ones are preserved. Otherwise, the modification is dismissed
because the field returned to its initial value.

This used to work well for most fields, but failed for "tags" because
the field is an array and the values were transformed to strings to
perform the comparison.

* FIX: Reset last_editor_id if revision is reverted

If a post was revised and then the same revision was reverted,
last_editor_id was still set to the ID of the user who last edited the
post. This was a problem because the same person could then edit the
same post again and because it was the same user and same post, the
system attempted to update the last one (that did not exist anymore).
2021-11-09 16:29:37 +02:00
Bianca Nenciu b203e316ac
FEATURE: Add pagination to API keys page (#14777) 2021-11-09 12:18:23 +02:00
Dan Ungureanu 46fed1303f
FIX: Regression introduced in #14715 (#14842)
* FIX: Regression introduced in #14715

Wrong method names were used to get Redis keys.

* DEV: Remove more stubs
2021-11-09 17:20:09 +11:00
Martin Brennan fc98d1edfa
DEV: Improve s3:ensure_cors_rules logging (#14832) 2021-11-08 11:44:12 +10:00
Osama Sayegh e2afa1b799
FIX: Skip post validations for system revisions when author deletes post (#14824) 2021-11-08 09:33:41 +08:00
Martin Brennan 9a72a0945f
FIX: Ensure CORS rules exist for S3 using rake task (#14802)
This commit introduces a new s3:ensure_cors_rules rake task
that is run as a prerequisite to s3:upload_assets. This rake
task calls out to the S3CorsRulesets class to ensure that
the 3 relevant sets of CORS rules are applied, depending on
site settings:

* assets
* direct S3 backups
* direct S3 uploads

This works for both Global S3 settings and Database S3 settings
(the latter set directly via SiteSetting).

As it is, only one rule can be applied, which is generally
the assets rule as it is called first. This commit changes
the ensure_cors! method to be able to apply new rules as
well as the existing ones.

This commit also slightly changes the existing rules to cover
direct S3 uploads via uppy, especially multipart, which requires
some more headers.
2021-11-08 09:16:38 +10:00
Roman Rizzi 53abcd825d
FIX: Canonical URLs may be relative (#14825)
FinalDestination's follow_canonical mode used for embedded topics should work when canonical URLs are relative, as specified in [RFC 6596](https://datatracker.ietf.org/doc/html/rfc6596)
2021-11-05 14:20:14 -03:00
Blake Erickson 892e33fd93
Add `embed_url` to the api docs (#14813)
When creating a topic via the api you can pass in the `embed_url` param,
so adding this to the api docs.

See: https://github.com/discourse/discourse_api_docs/pull/26
2021-11-03 19:22:55 -06:00
Mark VanLandingham 67265a5045
DEV: Plugin instance method for push_notification_filters (#14787) 2021-11-03 12:21:33 -05:00
Joe 836c0f5ffe fix spec 2021-11-03 11:41:13 -04:00
Vinoth Kannan 53b7220638
FIX: exclude suppressed category topics in digest even if unmuted. (#14793)
Previously, suppressed category topics are included in the digest emails if the user visited that topic before and the `TopicUser` record is created with any notification level except 'muted'.
2021-11-03 12:47:09 +05:30
jbrw aec125b617
FIX: Display Instagram Oneboxes in an iframe (#14789)
We are no longer able to display the image returned by Instagram directly within a Discourse site (either in the composer, or within a cooked post within a topic), so:

- Display an image placeholder in the composer preview
- A cooked post should use an iframe to display the Instagram 'embed' content
2021-11-02 14:34:51 -04:00
Jean 8d73730c44
FEATURE: Add setting to disable notifications for topic tags edits (#14794) 2021-11-02 13:53:21 -04:00
Martin Brennan 28b00dc6fc
DEV: Output webmock errors in request specs (#14782)
* DEV: Output webmock errors in request specs

In request specs, if you had not properly mocked an external
HTTP call, you would end up with a 500 error with no further
information instead of your expected response code, with an
rspec output like this:

```
Failures:

  1) UploadsController#generate_presigned_put when the store is external generates a presigned URL and creates an external upload stub
     Failure/Error: expect(response.status).to eq(200)

       expected: 200
            got: 500

       (compared using ==)
     # ./spec/requests/uploads_controller_spec.rb:727:in `block (4 levels) in <top (required)>'
     # ./spec/rails_helper.rb:280:in `block (2 levels) in <top (required)>'
```

This is not helpful at all when you want to find what you actually
failed to mock, which is shown straight away in non-request specs.

This commit introduces a rescue_from block in the application
controller to log this error, so we have a much nicer output that
helps the developer find the issue:

```
Failures:

  1) UploadsController#generate_presigned_put when the store is external generates a presigned URL and creates an external upload stub
     Failure/Error: expect(response.status).to eq(200)

       expected: 200
            got: 500

       (compared using ==)
     # ./spec/requests/uploads_controller_spec.rb:727:in `block (4 levels) in <top (required)>'
     # ./spec/rails_helper.rb:280:in `block (2 levels) in <top (required)>'
     # ------------------
     # --- Caused by: ---
     # WebMock::NetConnectNotAllowedError:
     #   Real HTTP connections are disabled. Unregistered request: GET https://s3-upload-bucket.s3.us-west-1.amazonaws.com/?cors with headers {'Accept'=>'*/*', 'Accept-Encoding'=>'', 'Authorization'=>'AWS4-HMAC-SHA256 Credential=some key/20211101/us-west-1/s3/aws4_request, SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date, Signature=test', 'Host'=>'s3-upload-bucket.s3.us-west-1.amazonaws.com', 'User-Agent'=>'aws-sdk-ruby3/3.121.2 ruby/2.7.1 x86_64-linux aws-sdk-s3/1.96.1', 'X-Amz-Content-Sha256'=>'test', 'X-Amz-Date'=>'20211101T035113Z'}
     #
     #   You can stub this request with the following snippet:
     #
     #   stub_request(:get, "https://s3-upload-bucket.s3.us-west-1.amazonaws.com/?cors").
     #     with(
     #       headers: {
     #   	  'Accept'=>'*/*',
     #   	  'Accept-Encoding'=>'',
     #   	  'Authorization'=>'AWS4-HMAC-SHA256 Credential=some key/20211101/us-west-1/s3/aws4_request, SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date, Signature=test',
     #   	  'Host'=>'s3-upload-bucket.s3.us-west-1.amazonaws.com',
     #   	  'User-Agent'=>'aws-sdk-ruby3/3.121.2 ruby/2.7.1 x86_64-linux aws-sdk-s3/1.96.1',
     #   	  'X-Amz-Content-Sha256'=>'test',
     #   	  'X-Amz-Date'=>'20211101T035113Z'
     #       }).
     #     to_return(status: 200, body: "", headers: {})
     #
     #   registered request stubs:
     #
     #   stub_request(:head, "https://s3-upload-bucket.s3.us-west-1.amazonaws.com/")
     #
     #   ============================================================
```

* DEV: Require webmock in application controller if rails.env.test

* DEV: Rescue from StandardError and NetConnectNotAllowedError
2021-11-01 14:38:41 +08:00
Alan Guo Xiang Tan a03c48b720
FIX: Use the same mode for chinese search when indexing and querying. (#14780)
The `白名单` term becomes `名单 白名单` after it is processed by
cppjieba in :query mode. However, `白名单` is not tokenized as such by cppjieba when it
appears in a string of text. Therefore, this may lead to failed matches as
the search data generated while indexing may not contain all of the
terms generated by :query mode. We've decided to maintain parity for now
such that both indexing and querying uses the same :mix mode. This may
lead to less accurate search but our plan is to properly support CJK
search in the future.
2021-11-01 10:14:47 +08:00
Martin Brennan a059c7251f
DEV: Add tests to S3Helper.ensure_cors and move rules to class (#14767)
In preparation for adding automatic CORS rules creation
for direct S3 uploads, I am adding tests here and moving the
CORS rule definitions into a dedicated class so they are all
in the one place.

There is a problem with ensure_cors! as well -- if there is
already a CORS rule defined (presumably the asset one) then
we do nothing and do not apply the new rule. This means that
the S3BackupStore.ensure_cors method does nothing right now
if the assets rule is already defined, and it will mean the
same for any direct S3 upload rules I add for uppy. We need
to be able to add more rules, not just one.

This is not a problem on our hosting because we define the
rules at an infra level.
2021-11-01 08:23:13 +10:00
Jarek Radosz ab374fff72
DEV: Move imap_helper to spec/support directory (#14776) 2021-10-29 20:46:25 +02:00
Jarek Radosz d760c86031
DEV: Load rails_helper in pbkdf2_spec (#14775)
Fixes failing tests in CI, where this spec file was first in a batch.
2021-10-29 20:15:10 +02:00
jbrw cfc62dbace
FIX: allowed_theme_ids should not be persisted in GlobalSettings (#14756)
* FIX: allowed_theme_ids should not be persisted in GlobalSettings

It was observed that the memoized value of `GlobalSetting.allowed_theme_ids` would be persisted across requests, which could lead to unpredictable/undesired behaviours in a multisite environment.

This change moves that logic out of GlobalSettings so that the returned theme IDs are correct for the current site.

Uses get_set_cache, which ultimately uses DistributedCache, which will take care of multisite issues for us.
2021-10-29 11:46:52 -04:00
Bianca Nenciu 1c3c0f04d9
FEATURE: Pull hotlinked images in user bios (#14726) 2021-10-29 17:58:05 +03:00
Bianca Nenciu 19ef6995a8
FIX: Do not replace words in hashtags and mentions (#14760)
Watched words were replaced inside mentions and hashtags when watched
word regular expressions were enabled.
2021-10-29 17:53:09 +03:00
Bianca Nenciu cb0958fcea
FIX: Hide links to muted topics and in categories list (#14761)
* FIX: Hide links to muted topics

* FIX: Hide muted topics in categories list on mobile
2021-10-29 17:52:23 +03:00
Vinoth Kannan deee715a2c
FIX: update flair group of all members if primary group setting changed. (#14762)
Previously, if we enable the `primary_group` setting on a group then the `flair_group_id` of its' members are not affected.
2021-10-28 22:56:44 +05:30
Vinoth Kannan c62242c6b3
FIX: should not receive topic invites from ignored users. (#14746)
Previously, ignored users can send notifications by inviting the ignorer to topics or PMs.
2021-10-28 21:03:00 +05:30
Vinoth Kannan a2851b5d4c
FIX: include new tags in validation if user can create one. (#14744)
Previously, users who have enough trust level are unable to create topics with new tags if the selected category required a minimum number of tags.
2021-10-28 11:59:46 +05:30
Martin Brennan b659e94a8e
DEV: Delete vacate_legacy_prefix_backups code (#14735)
Introduced in 3037617327, we no
longer need this code, as all of the backups have been
migrated.
2021-10-28 07:53:21 +10:00
Jean 92f4cdd330
FEATURE: bypass topic bump when disable_category_edit_notifications is enabled (#14754) 2021-10-27 17:05:10 -04:00
Roman Rizzi df3eb93973
DEV: Sanitize HTML admin inputs (#14681)
* DEV: Sanitize HTML admin inputs

This PR adds on-save HTML sanitization for:

Client site settings
translation overrides
badges descriptions
user fields descriptions

I used Rails's SafeListSanitizer, which [accepts the following HTML tags and attributes](018cf54073/lib/rails/html/sanitizer.rb (L108))

* Make sure that the sanitization logic doesn't corrupt settings with special characters
2021-10-27 11:33:07 -03:00
Andrei Prigorshnev 19d95c64af
DEV: simplify username suggester (#14531)
This PR doesn't change any behavior, but just removes code that wasn't in use. This is a pretty dangerous place to change, since it gets called during user's registration. At the same time the refactoring is very straightforward, it's clear that this code wasn't doing any work (it still needs to be double-checked during review though). Also, the test coverage of UserNameSuggester is good.
2021-10-27 14:41:24 +04:00
Dan Ungureanu 69f0f48dc0
DEV: Fix rubocop issues (#14715) 2021-10-27 11:39:28 +03:00
Bianca Nenciu 0c6f9d7c67
FIX: Show right message when permanently deleting topic (#14717) 2021-10-26 18:31:15 +03:00
David Taylor 9ac6f1d3bb
FIX: Include the Vary:Accept header on all Accept-based responses (#14647)
By default, Rails only includes the Vary:Accept header in responses when the Accept: header is included in the request. This means that proxies/browsers may cache a response to a request with a missing Accept header, and then later serve that cached version for a request which **does** supply the Accept header. This can lead to some very unexpected behavior in browsers.

This commit adds the Vary:Accept header for all requests, even if the Accept header is not present in the request. If a format parameter (e.g. `.json` suffix) is included in the path, then the Accept header is still omitted. (The format parameter takes precedence over any Accept: header, so the response is no longer varies based on the Accept header)
2021-10-25 12:53:50 +01:00
Joffrey JAFFEUX b18c01e3c6
DEV: prevents flakky spec when deleting plugin (#14701)
Not reseting the registry could lead to assets still being registered for example.

This flakky spec was reprdocible with this call: `bundle exec rspec --seed 9472 spec/components/discourse_plugin_registry_spec.rb spec/components/svg_sprite/svg_sprite_spec.rb`

Which would trigger the following error:

```
Failures:

  1) DiscoursePluginRegistry#register_asset registers vendored_core_pretty_text properly
     Failure/Error: expect(registry.javascripts.count).to eq(0)

       expected: 0
            got: 1

       (compared using ==)
     # ./spec/components/discourse_plugin_registry_spec.rb:248:in `block (3 levels) in <top (required)>'
     # ./spec/rails_helper.rb:280:in `block (2 levels) in <top (required)>'
     # /Users/joffreyjaffeux/.gem/ruby/2.7.3/gems/webmock-3.14.0/lib/webmock/rspec.rb:37:in `block (2 levels) in <top (required)>'
```
2021-10-25 10:24:21 +02:00
Martin Brennan 2b40049abb
FIX: Remove superfluous topic allowed users on group invite (#14656)
When inviting a group to a topic, there may be members of
the group already in the topic as topic allowed users. These
can be safely removed from the topic, because they are implicitly
allowed in the topic based on their group membership.

Also, this prevents issues with group SMTP emails, which rely
on the topic_allowed_users of the topic to send to and cc's
for emails, and if there are members of the group as topic_allowed_users
then that complicates things and causes odd behaviour.

We also ensure that the OP of the topic is not removed from
the topic_allowed_users when a group they belong to is added,
as it will make it harder to add them back later.
2021-10-22 08:57:51 +10:00
Osama Sayegh 70fa67a9e1
FIX: Don't leak unhashed user API keys to redis (#14682)
User API keys (not the same thing as admin API keys) are currently
leaked to redis when rate limits are applied to them since redis is the
backend for rate limits in Discourse and the API keys are included in
the redis keys that are used to track usage of user API keys in the last
24 hours.

This commit stops the leak by using a SHA-256 representation of the user
API key instead of the key itself to form the redis key.

We don't need to manually delete the existing redis keys that contain
unhashed user API keys because they're not long-lived and will be
automatically deleted within 48 hours after this commit is deployed to
your Discourse instance.
2021-10-21 19:43:26 +03:00
David Taylor aac3547cc2
DEV: Update AWS API stub following gem version bump (#14673)
The latest version of the gem doesn't send whitespace in this request body, so we need to update the test stub accordingly
2021-10-20 23:04:08 +01:00
Arpit Jalan d1fc759ac4
FIX: remove 'crawl_images' site setting (#14646) 2021-10-19 17:12:29 +05:30
Andrei Prigorshnev b9164d22ee
FEATURE: add a hidden setting that enables using email as a source for username suggestions (#14623)
This is a follow-up to https://github.com/discourse/discourse/pull/14541. This adds a hidden setting for restoring the old behavior for those users who rely on it. We'll likely deprecate this setting at some point in the future.
2021-10-19 14:37:39 +04:00
Martin Brennan d3678f6930
FIX: Do not show recipient user in email participants list (#14642)
This commit removes the recipient's username from the
respond to / participants list that is shown at the bottom
of user notification emails. For example if the recipient's
username was jsmith, and there were participants ljones and
bmiller, we currently show this:

> "reply to this email to respond to jsmith, ljones, bmiller"

or

> "Participants: jsmith, ljones, bmiller"

However this is a bit redundant, as you are not replying to
yourself here if you are the recipient user. So we omit the
recipient user's username from this list, which is only used
in the text of the email and not elsewhere.
2021-10-19 15:26:22 +10:00
Jean 6275658e3d
FEATURE: Add setting to disable notifications for topic category edits (#14632) 2021-10-18 09:04:01 -04:00
Alan Guo Xiang Tan fb5a062b1f
DEV: Remove `TopicView#first_post_id`. (#14631)
The method was only used for mega topics but it was redundant as the
first post can be determined from using the condition where
`Post#post_number` equal to one.
2021-10-18 14:47:47 +08:00