Commit Graph

1350 Commits

Author SHA1 Message Date
David Taylor 010309d108
SECURITY: Improve validation of SNS subscription confirm (#14671)
An upstream validation bug in the aws-sdk-sns library could enable RCE under certain circumstances. This commit updates the upstream gem, and adds additional validation to provide defense-in-depth.
2021-10-20 22:20:52 +01:00
Osama Sayegh 228264d17c
Revert "DEV: add routes_lazy_route to boost boot-up time (#14545)" (#14581)
This reverts commit f5cf647e57.

The gem breaks usage of Rails URL helpers when used outside views and
controllers, for example in
88ecb83382/app/models/upload.rb (L239-L242)
the `upload_short_path` method call fails with an undefined method
exception when this gem is enabled.
2021-10-12 17:30:38 +03:00
Sam f5cf647e57
DEV: add routes_lazy_route to boost boot-up time (#14545)
The lazy route initialization cuts down boot time of rails.

On my local system it cuts out 200ms of boot time taking me from 3.2 to 3 seconds.

This is not a radically enormous amount of time, but paper cuts add up, and a faster boot in dev will make everyone happy.

TBD if we want to also include this in production.

Gem is heavily maintained by @amatsuda, last commit 3 days ago.
2021-10-11 13:22:13 +11:00
Vinoth Kannan a6de4a5ce9
DEV: use upload id to save in theme setting instead of URL. (#14341)
When we use URL instead it creates the problem while changing the CDN hostname.
2021-09-16 07:58:53 +05:30
Martin Brennan 41e19adb0d
DEV: Ignore reminder_type for bookmarks (#14349)
We don't actually use the reminder_type for bookmarks anywhere;
we are just storing it. It has no bearing on the UI. It used
to be relevant with the at_desktop bookmark reminders (see
fa572d3a7a)

This commit marks the column as readonly, ignores it, and removes
the index, and it will be dropped in a later PR. Some plugins
are relying on reminder_type partially so some stubs have been
left in place to avoid errors.
2021-09-16 09:56:54 +10:00
Martin Brennan 22208836c5
DEV: Ignore bookmarks.topic_id column and remove references to it in code (#14289)
We don't need no stinkin' denormalization! This commit ignores
the topic_id column on bookmarks, to be deleted at a later date.
We don't really need this column and it's better to rely on the
post.topic_id as the canonical topic_id for bookmarks, then we
don't need to remember to update both columns if the bookmarked
post moves to another topic.
2021-09-15 10:16:54 +10:00
Krzysztof Kotlarek d99735e24d
FEATURE: remove duplicated messages about new advices (#14319)
Discourse is sending regularly message to admins when potential problems are persisted. Most of the time they have exactly the same content. In that case, when there are no replies, the old one should be trashed before a new one is created.
2021-09-15 08:59:25 +10:00
Mark VanLandingham f4cca4af75
DEV: Do not clean up chat message uploads (#14267) 2021-09-07 13:33:48 -05:00
David Taylor 31db83527b DEV: Introduce PresenceChannel API for core and plugin use
PresenceChannel aims to be a generic system for allow the server, and end-users, to track the number and identity of users performing a specific task on the site. For example, it might be used to track who is currently 'replying' to a specific topic, editing a specific wiki post, etc.

A few key pieces of information about the system:
- PresenceChannels are identified by a name of the format `/prefix/blah`, where `prefix` has been configured by some core/plugin implementation, and `blah` can be any string the implementation wants to use.
- Presence is a boolean thing - each user is either present, or not present. If a user has multiple clients 'present' in a channel, they will be deduplicated so that the user is only counted once
- Developers can configure the existence and configuration of channels 'just in time' using a callback. The result of this is cached for 2 minutes.
- Configuration of a channel can specify permissions in a similar way to MessageBus (public boolean, a list of allowed_user_ids, and a list of allowed_group_ids). A channel can also be placed in 'count_only' mode, where the identity of present users is not revealed to end-users.
- The backend implementation uses redis lua scripts, and is designed to scale well. In the future, hard limits may be introduced on the maximum number of users that can be present in a channel.
- Clients can enter/leave at will. If a client has not marked itself 'present' in the last 60 seconds, they will automatically 'leave' the channel. The JS implementation takes care of this regular check-in.
- On the client-side, PresenceChannel instances can be fetched from the `presence` ember service. Each PresenceChannel can be used entered/left/subscribed/unsubscribed, and the service will automatically deduplicate information before interacting with the server.
- When a client joins a PresenceChannel, the JS implementation will automatically make a GET request for the current channel state. To avoid this, the channel state can be serialized into one of your existing endpoints, and then passed to the `subscribe` method on the channel.
- The PresenceChannel JS object is an ember object. The `users` and `count` property can be used directly in ember templates, and in computed properties.
- It is important to make sure that you `unsubscribe()` and `leave()` any PresenceChannel objects after use

An example implementation may look something like this. On the server:

```ruby
register_presence_channel_prefix("site") do |channel|
  next nil unless channel == "/site/online"
  PresenceChannel::Config.new(public: true)
end
```

And on the client, a component could be implemented like this:

```javascript
import Component from "@ember/component";
import { inject as service } from "@ember/service";

export default Component.extend({
  presence: service(),
  init() {
    this._super(...arguments);
    this.set("presenceChannel", this.presence.getChannel("/site/online"));
  },
  didInsertElement() {
    this.presenceChannel.enter();
    this.presenceChannel.subscribe();
  },
  willDestroyElement() {
    this.presenceChannel.leave();
    this.presenceChannel.unsubscribe();
  },
});
```

With this template:

```handlebars
Online: {{presenceChannel.count}}
<ul>
  {{#each presenceChannel.users as |user|}} 
    <li>{{avatar user imageSize="tiny"}} {{user.username}}</li>
  {{/each}}
</ul>
```
2021-08-27 16:26:06 +01:00
Martin Brennan e43a8af3bd
FIX: Do not send emails to mailing_list_mode subscribers for PMs (#14159)
This bug was introduced by f66007ec83.

In PostJobsEnqueuer we previously did not fire the after_post_create
event and after_topic_create event for private message topics. This was
changed in the above commit in order to publish message bus messages
for topic tracking state updates. Unfortunately this caused the
NotifyMailingListSubscribers job to be enqueued for all posts including
private messages, and admins and the users involved in the PMs got
emailed the contents of the PMs if they had mailing list mode enabled.

Luckily the impact of this was mitigated by a Guardian#can_see? check
for each mailing list mode user in the NotifyMailingListSubscribers job.
We never want to notify mailing list mode subscribers for private messages
so an early return has been added there, plus the logic in PostJobsEnqueuer
has been fixed, and tests have been added to that class where there were
none before.
2021-08-26 15:16:35 +10:00
Alan Guo Xiang Tan f66007ec83
FEATURE: Display unread and new counts for messages. (#14059)
There are certain design decisions that were made in this commit.

Private messages implements its own version of topic tracking state because there are significant differences between regular and private_message topics. Regular topics have to track categories and tags while private messages do not. It is much easier to design the new topic tracking state if we maintain two different classes, instead of trying to mash this two worlds together.

One MessageBus channel per user and one MessageBus channel per group. This allows each user and each group to have their own channel backlog instead of having one global channel which requires the client to filter away unrelated messages.
2021-08-25 11:17:56 +08:00
Vinoth Kannan c75500b871
DEV: decrease webhook timeout and move to 'low' priority queue. (#14038)
Previously, webhooks has 60 seconds default timeout of Excon and were added in the 'default' queue.
2021-08-13 10:41:02 +05:30
Alan Guo Xiang Tan 849827841f Revert "DEV: Move private message topic tracking state publish into sidekiq."
This reverts commit c51c80466c.

There is a conditional in post jobs enqueuer that only enqueues the job
for non-PM posts.
2021-07-30 11:36:01 +08:00
Alan Guo Xiang Tan c51c80466c DEV: Move private message topic tracking state publish into sidekiq.
Same intend as b1f32f2f57.
2021-07-29 14:57:19 +08:00
Martin Brennan b500949ef6
FEATURE: Initial implementation of direct S3 uploads with uppy and stubs (#13787)
This adds a few different things to allow for direct S3 uploads using uppy. **These changes are still not the default.** There are hidden `enable_experimental_image_uploader` and `enable_direct_s3_uploads`  settings that must be turned on for any of this code to be used, and even if they are turned on only the User Card Background for the user profile actually uses uppy-image-uploader.

A new `ExternalUploadStub` model and database table is introduced in this pull request. This is used to keep track of uploads that are uploaded to a temporary location in S3 with the direct to S3 code, and they are eventually deleted a) when the direct upload is completed and b) after a certain time period of not being used. 

### Starting a direct S3 upload

When an S3 direct upload is initiated with uppy, we first request a presigned PUT URL from the new `generate-presigned-put` endpoint in `UploadsController`. This generates an S3 key in the `temp` folder inside the correct bucket path, along with any metadata from the clientside (e.g. the SHA1 checksum described below). This will also create an `ExternalUploadStub` and store the details of the temp object key and the file being uploaded.

Once the clientside has this URL, uppy will upload the file direct to S3 using the presigned URL. Once the upload is complete we go to the next stage.

### Completing a direct S3 upload

Once the upload to S3 is done we call the new `complete-external-upload` route with the unique identifier of the `ExternalUploadStub` created earlier. Only the user who made the stub can complete the external upload. One of two paths is followed via the `ExternalUploadManager`.

1. If the object in S3 is too large (currently 100mb defined by `ExternalUploadManager::DOWNLOAD_LIMIT`) we do not download and generate the SHA1 for that file. Instead we create the `Upload` record via `UploadCreator` and simply copy it to its final destination on S3 then delete the initial temp file. Several modifications to `UploadCreator` have been made to accommodate this.

2. If the object in S3 is small enough, we download it. When the temporary S3 file is downloaded, we compare the SHA1 checksum generated by the browser with the actual SHA1 checksum of the file generated by ruby. The browser SHA1 checksum is stored on the object in S3 with metadata, and is generated via the `UppyChecksum` plugin. Keep in mind that some browsers will not generate this due to compatibility or other issues.

    We then follow the normal `UploadCreator` path with one exception. To cut down on having to re-upload the file again, if there are no changes (such as resizing etc) to the file in `UploadCreator` we follow the same copy + delete temp path that we do for files that are too large.

3. Finally we return the serialized upload record back to the client

There are several errors that could happen that are handled by `UploadsController` as well.

Also in this PR is some refactoring of `displayErrorForUpload` to handle both uppy and jquery file uploader errors.
2021-07-28 08:42:25 +10:00
Alan Guo Xiang Tan 0d8144b62b DEV: Improve logging of errors in `Jobs::ProcessUserNotificationSchedules`
Gives us the actual error and backtrace to work with. Otherwise, the
logging of the error is not useful at all.
2021-07-21 12:20:44 +08:00
Dan Ungureanu 079d2af55f
FIX: Clear stale status of reloaded reviewables (#13750)
* FIX: Clear stale status of reloaded reviewables

Navigating away from and back to the reviewables reloaded Reviewable
records, but did not clear the "stale" attribute.

* FEATURE: Show user who last acted on reviewable

When a user acts on a reviewable, all other clients are notified and a
generic "reviewable was resolved by someone" notice was shown instead of
the buttons. There is no need to keep secret the username of the acting
user.
2021-07-16 19:57:12 +03:00
Penar Musaraj 361c8be547
PERF: Add scheduled job to delete old stylesheet cache rows (#13747) 2021-07-16 10:58:01 -04:00
Osama Sayegh 31aa701518
FEATURE: Add option to grant badge multiple times to users using Bulk Award (#13571)
Currently when bulk-awarding a badge that can be granted multiple times, users in the CSV file are granted the badge once no matter how many times they're listed in the file and only if they don't have the badge already.

This PR adds a new option to the Badge Bulk Award feature so that it's possible to grant users a badge even if they already have the badge and as many times as they appear in the CSV file.
2021-07-15 05:53:26 +03:00
Martin Brennan 068889cb5f
FIX: Email threads sometimes not grouping for group SMTP (#13727)
This PR fixes a couple of issues related to group SMTP:

1. When running the group SMTP job, we were exiting early if the email was for the OP because of an IMAP race condition. However this causes issues when replying as a new topic for an existing SMTP topic, as the recipient does not get the OP email which can cause threading problems.
2. When sending emails for a new topic spun out like the issue in 1., we are not maintaining the original subject/topic title because that is based on the incoming email record, which we were not doing because the group SMTP email was never sent because of issue 1.
2021-07-14 14:23:14 +10:00
Martin Brennan c6f2459cc4
FIX: Do not prevent other topic timers running on error (#13665)
There was an issue with the TopicTimerEnqueuer where any timer
that failed to enqueue_typed_job with an error would prevent
all other pending timers after the one that errored from running.

To mitigate this we just capture the error and log it (so we can
still fix it if needed for bug crushing) and proceed with the
rest of the timer enqueues.

The commit https://github.com/discourse/discourse/pull/13544 highlighted
this issue originally in hosted sites.

<!-- NOTE: All pull requests should have tests (rspec in Ruby, qunit in JavaScript). If your code does not include test coverage, please include an explanation of why it was omitted. -->
2021-07-08 12:49:58 +10:00
Martin Brennan b3d3ad250b
FIX: Handle SMTPServerBusy for group smtp email (#13632)
Use the `sidekiq_retry_in` code from Jobs::UserEmail in group SMTP. Also we don't need to keep `seconds_to_delay` -- sidekiq uses the default delay calculation if you return 0 or nil from the block. See 3330df0ee3/lib/sidekiq/job_retry.rb (L216-L234) for sidekiq default retry delay logic.

I experimented with extracting this into a concern or a module, but `sidekiq_retry_in` is quite magic and it would not allow me to abstract away into a module that calls some method specificall in the child job class.

I would love to write tests for this, but it does not seem possible (not sure if its because of our test
setup) to write tests that test sidekiq's retry capability, and I am not sure if we should be anyway. Initial addition
to UserEmail did not test this functionality 
d224966a0e
2021-07-06 13:37:52 +10:00
Martin Brennan 0f688f45bd
FIX: Handle edge cases for group SMTP email job (#13631)
Skip group SMTP email (and add log) if:

* topic is deleted
* post is deleted
* smtp has been disabled for the group

Skip without log if:

* enable_smtp site setting is false
* disable_emails site setting is yes

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2021-07-05 14:56:32 +10:00
Martin Brennan 03338f9086
FIX: Remove legacy topic timer code (#13544)
The new topic timer backend code introduced six months ago
in 0034cbd is now used instead of this legacy code. It can be safely removed
now.
2021-06-29 09:16:25 +10:00
Martin Brennan 87684f7c5e
FEATURE: Use group SMTP job and mailer instead of UserNotifications change (#13489)
This PR backtracks a fair bit on this one https://github.com/discourse/discourse/pull/13220/files.

Instead of sending the group SMTP email for each user via `UserNotifications`, we are changing to send only one email with the existing `Jobs::GroupSmtpEmail` job and `GroupSmtpMailer`. We are changing this job and mailer along with `PostAlerter` to make the first topic allowed user the `to_address` for the email and any other `topic_allowed_users` to be the CC address on the email. This is to cut down on emails sent via SMTP, which is subject to daily limits from providers such as Gmail. We log these details in the `EmailLog` table now.

In addition to this, we have changed `PostAlerter` to no longer rely on incoming email email addresses for sending the `GroupSmtpEmail` job. This was unreliable as a user's email could have changed in the meantime. Also it was a little overcomplicated to use the incoming email records -- it is far simpler to reason about to just use topic allowed users.

This also adds a fix to include cc_addresses in the EmailLog.addressed_to_user scope.
2021-06-28 08:55:13 +10:00
Roman Rizzi 2c918a3161
FEATURE: Staff can receive pending user reminders more frequently. (#13422)
* FEATURE: Staff can receive pending user reminders more frequently.

We now express the "pending_users_reminder_delay"  in minutes instead of hours so staff can have finer control over the delay.

We need to keep in mind that the reminders could still take up to 20 minutes, even when using a lower value. We send them from a scheduled job.

* Migrate to a new site setting for the reminders delay
2021-06-24 10:02:56 -03:00
Joffrey JAFFEUX 2654a6685c
DEV: adds support for bannered until (#13417)
ATM it only implements server side of it, as my need is for automation purposes. However it should probably be added in the UI too as it's unexpected to have pinned_until and no bannered_until.
2021-06-24 11:35:36 +02:00
Jarek Radosz 046a875222
DEV: Improve `script/downsize_uploads.rb` (#13508)
* Only shrink images that are used in Posts and no other models
* Don't save the upload if the size is the same
2021-06-24 00:09:40 +02:00
Bianca Nenciu d1b2e9db3b
PERF: Update post uploads secure status in a job (#13459)
When secure uploads are enabled, editing a post with many uploads can
cause a timeout because the store has to be contacted for each upload.
2021-06-21 19:15:24 +03:00
Dan Ungureanu c893b20298
FIX: Destroy invites of anonymized emails (#13404)
Anonymizing a user changed their email address, destroyed all
associated InvitedUser records, but did not destroy the invites
associated to user's email.
2021-06-17 10:45:40 +03:00
Martin Brennan c659e3e95b
FIX: Make sure topic_user.bookmarked is synced in more places (#13383)
When we call Bookmark.cleanup! we want to make sure that
topic_user.bookmarked is updated for topics linked to the
bookmarks that were deleted. Also when PostDestroyer calls
destroy and recover. We have a job for this already --
SyncTopicUserBookmarked -- so we just utilize that.
2021-06-16 08:30:40 +10:00
Roman Rizzi 4dc8c3c409
FEATURE: Blocking is optional when deleting a user from the review queue. (#13375)
Subclasses must call #delete_user_actions inside build_actions to support user deletion. The method adds a delete user bundle, which has a delete and a delete + block option. Every subclass is responsible for implementing these actions.
2021-06-15 12:35:45 -03:00
Dan Ungureanu fa02775095
PERF: Perform user filtering in SQL (#13358)
Notifying about a tag change sometimes resulted in loading a large
number of users in memory just to perform an exclusion. This commit
prefers to do inclusion (i.e. instead of exclude users X, do include
users in groups Y) and does it in SQL to avoid fetching unnecessary
data that is later discarded.
2021-06-11 10:55:50 +10:00
Sam 023ff9a282
DEV: ensure user export ordering is predictable (#13340)
Flaky spec due to random ordering for the post_actions table.

Introduces consistent ordering.
2021-06-09 15:55:52 +10:00
Dan Ungureanu da2889a7a8
DEV: Add more verbose logging for image uploads (#13270)
Image optimization fails randomly (very rare) without a trace and it is
near impossible to find culprit image, reproduce the issue and attempt
to fix.
2021-06-04 15:13:58 +03:00
Dan Ungureanu 197e3f24ce
FEATURE: Show stale reviewable to other clients (#13114)
The previous commits removed reviewables leading to a bad user
experience. This commit updates the status, replaces actions with a
message and greys out the reviewable.
2021-05-26 09:47:35 +10:00
Josh Soref 59097b207f
DEV: Correct typos and spelling mistakes (#12812)
Over the years we accrued many spelling mistakes in the code base. 

This PR attempts to fix spelling mistakes and typos in all areas of the code that are extremely safe to change 

- comments
- test descriptions
- other low risk areas
2021-05-21 11:43:47 +10:00
Dan Ungureanu d903d4dc5a
DEV: Periodically delete old email change requests (#13054)
Email change requests are never deleted no matter if they completed
successfully or not. The abandoned requests have the disadvantage of
showing up as unconfirmed emails in user's preferences page.
2021-05-14 10:34:56 +03:00
Roman Rizzi d4b5a81b05
FIX: Recalculate scores only when approving or transitioning to pending. (#13009)
Recalculating a ReviewableFlaggedPost's score after rejecting or ignoring it sets the score as 0, which means that we can't find them after reviewing. They don't surpass the minimum priority threshold and are hidden.

Additionally, we only want to use agreed flags when calculating the different priority thresholds.
2021-05-10 14:09:04 -03:00
Jeff Wong 75e159f0ed
FEATURE: add support for like webhooks (#12917)
* FEATURE: add support for like webhooks

Add support for like webhooks. Webhook events only send on user membership
in the defined webhook group filters.

This also fixes group webhook events, as before this was never used, and
the logic was not correct.
2021-04-30 17:08:38 -07:00
Roman Rizzi 60059a7190
FEATURE: A low priority filter for the review queue. (#12822)
This filter hides reviewables with a score lower than the "reviewable_low_priority_threshold" setting. We only use reviewables that already met this threshold to calculate the Medium and High priority filters.
2021-04-23 15:34:24 -03:00
Bianca Nenciu 8c4a11c006
DEV: Move autotag to topic creator (#12790)
This move was necessary to automatically tag the topic with the right
tags from creation time. The process post job may be delayed for a
short time.
2021-04-23 16:55:34 +03:00
Dan Ungureanu 31d3990986
FIX: Send a different message if export fails (#12799)
It used to check if an upload record exists, which is wrong because an
invalid upload record exists even if the upload was not created.

The other improvement is a better log message.
2021-04-22 20:21:31 +03:00
Sam e4f1760bab
FEATURE: watch title for automatic tagging (#12782)
Previously watched words ignored topic titles when applying auto tagging rules.

Also copy has been improved to reflect how the system behaves.

The text hints that we are only watching first post now
2021-04-21 18:16:25 +03:00
Sam 5c49009c6c
PERF: properly preload emails to speed up user exports (#12778)
scopes are incredibly annoying to preload, simply adding :user_emails is not
enough.

Instead of relying on scopes simply iterate through user_emails which is
properly preloaded.

This removes 2 * N+1 when generating user reports.
2021-04-21 10:42:07 +10:00
Gerhard Schlager 92e222d246
FIX: POP3 polling shouldn't stop after exception or old email (#12742) 2021-04-19 10:27:29 +02:00
Dan Ungureanu 99dadb2129
PERF: Async notify users after inviting group (#12697)
Inviting a group generates a notification for each member. If this
happens synchronously it may take a while, leading to a poor user
experience.
2021-04-14 19:30:51 +03:00
Martin Brennan 66d17fdd6b
FIX: Topic user bookmarked column is out of sync after post moves (#12612)
When posts are moved from one topic to another, the `topic_user.bookmarked` column for all users in the new and the old topic needs to be resynced, for example because a user bookmarks post 12 in topic 1, then it is moved to topic 2, the topic_user record for topic 1 should no longer be bookmarked. A background job has been added to sync the column for a specified topic, or for no topic at all, which does it for all topics like the migration.

Also includes a migration that we have run in the past to fix bad data.

----

This has been addressed in other places in the past:

https://github.com/discourse/discourse/pull/10211
https://github.com/discourse/discourse/pull/10188
2021-04-14 09:10:53 +10:00
Dan Ungureanu e704f0a541
FIX: Autocorrect values for dropdown imported user fields (#12572)
When bulk inviting, the uploaded CSV file may contain wrong values for
the user fields. This tries to automatically correct them by finding
the most similar option (by ignoring the case).
2021-03-31 19:19:57 +03:00
Dan Ungureanu e8c576cca9
FIX: User fields are case insensitive in bulk CSV (#12559)
The CSV column title had to be case sensitive match with the name of
the user field which was unnecessary complex.
2021-03-31 13:42:53 +03:00
Dan Ungureanu 8335c8dc1a
FEATURE: Allow admins to pre-populate user fields (#12361)
Admins can use bulk invites to pre-populate user fields. The imported
CSV file must have a header with "email" column (first position) and
names of the user fields (exact match).

Under the hood, the bulk invite will create staged users and populate
the user fields of those.
2021-03-29 14:03:19 +03:00
Bianca Nenciu 97623f5351
FIX: Do not raise if post no longer exists (#12428) 2021-03-17 19:22:05 +02:00
Bianca Nenciu 16b5fa030b
DEV: Set disable_mailing_list_mode automatically (#12402)
The user mailing list mode continued to be silently enabled and
UserEmail job checked just that ignoring site setting
disable_mailing_list_mode.

An additional migrate was added to set disable_mailing_list_mode
to false if any users enabled the mailing list mode already.
2021-03-17 17:39:10 +02:00
Osama Sayegh a23d0f9961
UX: Add image uploader widget for uploading badge images (#12377)
Currently the process of adding a custom image to badge is quite clunky; you have to upload your image to a topic, and then copy the image URL and pasting it in a text field. Besides being clucky, if the topic or post that contains the image is deleted, the image will be garbage-collected in a few days and the badge will lose the image because the application is not that the image is referenced by a badge.

This commit improves that by adding a proper image uploader widget for badge images.
2021-03-17 08:55:23 +03:00
Dan Ungureanu fb19ee9eee
FIX: Correctly use invite to topic email templates (#12411)
It was used both when inviting from a topic page and when creating
invites with "Send to topic on first login", while it should be used
only in the former case.
2021-03-16 17:08:54 +02:00
David Taylor bce837db0c
FIX: Update excerpt after process_post (#12340)
Onebox content may only be resolved during the process_post job. Onebox content could change the content of the excerpt, so we need to make sure the excerpt is updated accordingly.
2021-03-10 17:07:13 +00:00
David Taylor 4430bc153d
FIX: Do not clean up uploads when they're used by theme settings (#12326)
We intend to move ThemeSetting to use an upload_id column, rather than storing the URL. So this is a short-term solution.
2021-03-09 19:16:45 +00:00
Blake Erickson 2123561125
FIX: Mobile app notification urls w/ subfolder (#12282)
The urls that we generate for mobile post notifications don't take into
account the subfolder url if a site happens to have one configured. When
this happens when you tap on a new mobile notification it takes you to
a url that doesn't work because it is missing the subfolder portion.

I honestly think this should be handled in the Post model like we do
with the Topic model. `Post.url` should know how to handle subfolder
installs, but that seemed like a very risky change because there are
lots of other places in the codebase where we tack on the base_path and
I didn't want to risk duplicating it.

I also found a small typo in the topics controller spec.
2021-03-04 07:07:37 -07:00
Dan Ungureanu c047640ad4
FEATURE: Various improvements to invite system (#12023)
The user interface has been reorganized to show email and link invites
in the same screen. Staff has more control over creating and updating
invites. Bulk invite has also been improved with better explanations.

On the server side, many code paths for email and link invites have
been merged to avoid duplicated logic. The API returns better responses
with more appropriate HTTP status codes.
2021-03-03 11:45:29 +02:00
Bianca Nenciu b49b455e47
FEATURE: Autotag watched words (#12244)
New topics with be matched against a set of watched words and be
tagged accordingly.
2021-03-03 10:53:38 +02:00
Rafael dos Santos Silva 83f332b5a5
FEATURE: Add a site setting to allow emojis to come from an external URL (#12180) 2021-03-02 16:04:16 -03:00
Gerhard Schlager a96a5db0fb
DEV: Add option to send system message to groups (#12256) 2021-03-02 18:51:50 +01:00
Roman Rizzi 4e716e9ce5
FIX: Reduce the time_read threshold to one minute. (#12159)
* FIX: Reduce the time_read threshold to one minute.

Five minutes is too much and could fill the queue with false positives.

* Update spec/jobs/enqueue_suspect_users_spec.rb

Co-authored-by: Arpit Jalan <arpit@techapj.com>

Co-authored-by: Arpit Jalan <arpit@techapj.com>
2021-02-20 08:25:32 -03:00
Roman Rizzi 95fb363c2a
FEATURE: Use the "time_read" stat to flag users as suspicious. (#12145)
Completing the discobot tutorial gives you ~3m of reading time, so we set the limit at 5m. Additionally, we use an "OR" clause to cover the case when you just scroll through a single topic.
2021-02-19 13:10:19 -03:00
Krzysztof Kotlarek ad3ec5809f
FIX: Dismiss new with better migration (#12062)
Original PR was reverted because of broken migration https://github.com/discourse/discourse/pull/12058

I fixed it by adding this line
```
          AND topics.id IN(SELECT id FROM topics ORDER BY created_at DESC LIMIT :max_new_topics)
```

This time it is left joining a limited amount of topics. I tested it on few databases and it worked quite smooth
2021-02-15 08:50:33 +11:00
Penar Musaraj 900d4187ef
DEV: Prevents rate limits for new feature checks on multisite (#12053) 2021-02-12 08:52:59 -05:00
Krzysztof Kotlarek a696cc07d2
Revert "FEATURE: Ability to dismiss all new topics (#12018)" (#12058)
This reverts commits 7426764af4 and f5b18e2a31
2021-02-12 08:50:25 +11:00
Krzysztof Kotlarek f5b18e2a31
FEATURE: Ability to dismiss all new topics (#12018)
Follow up https://github.com/discourse/discourse/pull/11968

Dismiss all new topics using the same DismissTopicService. In addition, MessageBus receives exact topic ids which should be marked as `seen`.
2021-02-11 13:35:09 +11:00
Krzysztof Kotlarek a1aa37758c
FIX: do not notify when the hidden tag is added or removed (#12025)
The bug was mentioned on meta https://meta.discourse.org/t/users-are-seeing-handling-of-unhandled-tag-again/155367

It was related to users who are watching a specific topic. In that case, when the hidden tag was added or removed to the topic they were notified by `NotifyTagChangeJob`.

That job should take hidden tags into consideration. If all changed tags are in a hidden group, it should exclude user not belong to that group.

At the same time, if visible to anyone tag is added or removed users watching topic should be notified.
2021-02-11 10:03:45 +11:00
Bianca Nenciu f4db1675f3
FIX: Check if post.topic exists before publishing topic updates (#11900) 2021-02-09 16:41:22 +11:00
David Taylor 821bb1e8cb
FEATURE: Rename 'Discourse SSO' to DiscourseConnect (#11978)
The 'Discourse SSO' protocol is being rebranded to DiscourseConnect. This should help to reduce confusion when 'SSO' is used in the generic sense.

This commit aims to:
- Rename `sso_` site settings. DiscourseConnect specific ones are prefixed `discourse_connect_`. Generic settings are prefixed `auth_`
- Add (server-side-only) backwards compatibility for the old setting names, with deprecation notices
- Copy `site_settings` database records to the new names
- Rename relevant translation keys
- Update relevant translations

This commit does **not** aim to:
- Rename any Ruby classes or methods. This might be done in a future commit
- Change any URLs. This would break existing integrations
- Make any changes to the protocol. This would break existing integrations
- Change any functionality. Further normalization across DiscourseConnect and other auth methods will be done separately

The risks are:
- There is no backwards compatibility for site settings on the client-side. Accessing auth-related site settings in Javascript is fairly rare, and an error on the client side would not be security-critical.
- If a plugin is monkey-patching parts of the auth process, changes to locale keys could cause broken error messages. This should also be unlikely. The old site setting names remain functional, so security-related overrides will remain working.

A follow-up commit will be made with a post-deploy migration to delete the old `site_settings` rows.
2021-02-08 10:04:33 +00:00
Martin Brennan 4af77f1e38
FEATURE: Allow durations < 1 hour and < 1 day for topic timers where duration is specified (auto delete replies, close based on last post) (#11961)
This PR allows entering a float value for topic timers e.g. 0.5 for 30 minutes when entering hours, 0.5 for 12 hours when entering days. This is achieved by adding a new column to store the duration of a topic timer in minutes instead of the ambiguous both hours and days that it could be before.

This PR has ommitted the post migration to delete the duration column in topic timers; it will be done in a subsequent PR to ensure that no data is lost if the UPDATE query to set duration_mintues fails.

I have to keep the old keyword of duration in set_or_create_topic_timer for backwards compat, will remove at a later date after plugins are updated.
2021-02-05 10:12:56 +10:00
Krzysztof Kotlarek f39e7fe81d
FEATURE: New way to dismiss new topics (#11927)
This is a try to simplify logic around dismiss new topics to have one solution to work in all places - dismiss all-new, dismiss new in a specific category or even in a specific tag.
2021-02-04 11:27:34 +11:00
Robin Ward 93c25070fa
FIX: In FastImage 2.2.2 an error is raised with a `nil` path (#11954)
* FIX: In FastImage 2.2.2 an error is raised with a `nil` path

Sometimes Discourse.store.path_for would return `nil`, which the job
handled gracefully before, but raises an error with the new version of
the gem.

Note the logic of this job is a bit awkward since it depends on `nil`
being a string, but at least now it's no longer filling logs with
errors.

* Update app/jobs/onceoff/fix_invalid_gravatar_uploads.rb

Co-authored-by: Bianca Nenciu <nbianca@users.noreply.github.com>

Co-authored-by: Bianca Nenciu <nbianca@users.noreply.github.com>
2021-02-03 11:45:12 -05:00
Alan Guo Xiang Tan 0cc178d58b
FIX: Avoid pulling hotlinked images of post that have been deleted. (#11913) 2021-02-03 16:45:07 +11:00
Martin Brennan 6d72c8ab19
FEATURE: Topic timer UI revamp (#11912)
This PR revamps the topic timer UI, using the time shortcut selector from the bookmark modal.

* Fixes an issue where the duration of hours/days after last reply or auto delete replies was not enforced to be > 0
* Fixed an issue where the timer dropdown options were not reloaded correctly if the topic status changes in the background (use `MessageBus` to publish topic state in the open/close timer jobs)
* Moved the duration input and the "based on last post" option from the `future-date-input` component, as it was only used for topic timers. Also moved out the notice that is displayed which was also only relevant for topic timers.
2021-02-03 10:13:32 +10:00
Penar Musaraj 49e97279c7 FEATURE: Add daily job to check for new features 2021-02-01 10:31:44 +08:00
Mark VanLandingham 809274fe0d
DEV: Replace 'processed' column on notifications with new table (#11864) 2021-01-27 10:29:24 -06:00
Régis Hanol cd3d24ed8c
FIX: move post_search_data migration into onceoff job (#11851)
And reduce the size of the batches to 100k.

That should hopefully make the migrations run smoother...
2021-01-26 16:29:00 +01:00
Régis Hanol aa1138ff71
FIX: reindex_search job should work on model with no search data (#11819)
Lots of changes but it's mostly a refactoring.

The interesting part that was fix are the 'load_problem_<model>_ids' methods.
They will now return records with no search data associated so they can be properly indexed for the search.
This "bad" state usually happens after a migration.
2021-01-25 11:23:36 +01:00
Mark VanLandingham 1a7922bea2
FEATURE: Create notification schedule to automatically set do not disturb time (#11665)
This adds a new table UserNotificationSchedules which stores monday-friday start and ends times that each user would like to receive notifications (with a Boolean enabled to remove the use of the schedule). There is then a background job that runs every day and creates do_not_disturb_timings for each user with an enabled notification schedule. The job schedules timings 2 days in advance. The job is designed so that it can be run at any point in time, and it will not create duplicate records.

When a users saves their notification schedule, the schedule processing service will run and schedule do_not_disturb_timings. If the user should be in DND due to their schedule, the user will immediately be put in DND (message bus publishes this state).

The UI for a user's notification schedule is in user -> preferences -> notifications. By default every day is 8am - 5pm when first enabled.
2021-01-20 10:31:52 -06:00
Martin Brennan fb184fed06
DEV: Add created_via column to IncomingEmail (#11751)
This should make it easier to track down how the incoming email was created, which is one of four locations:

The POP3 poller (which picks up reply via email replies)
The admin email controller #handle_mail (which is where hosted mail is sent)
The IMAP sync tool
The group SMTP mailer, which sends emails when replying to IMAP topics, pre-emptively creating IncomingEmail records to avoid double syncing
2021-01-20 13:22:41 +10:00
Martin Brennan 0034cbda8a
DEV: Change Topic Timer from enqueue_at scheduled jobs to incrementally executed jobs (#11698)
Moves the topic timer jobs from being scheduled ahead of time with enqueue_at to a 5 minute scheduled run like bookmark reminders, in a new job called Jobs::EnqueueTopicTimers. Backwards compatibility is maintained by checking if an existing topic timer job is enqueued in sidekiq for the timer, and if it is not running it inside the new job.

The functionality to close/open a topic if it is in the opposite state still remains in the after_save block of TopicTimer, with further commentary, which is used for Open/Close Temporarily.

This also removes the ensure_consistency! functionality of topic timers as it is no longer needed; the new job will always pick up the timers because they are not stored in a fragile state of sidekiq.
2021-01-19 13:30:58 +10:00
Martin Brennan 5710d5d771
FIX: Do not process pop3 mails > 1 week old (#11740)
This adds a safe default to not process pop3 emails when the pop3 polling option is set up that are > 1 week old. This is to avoid the situation where an older mailbox is used, which causes us to go and process all emails in that mailbox, sending out error emails to the senders of emails which cannot be parsed successfully.
2021-01-19 09:49:50 +10:00
Krzysztof Kotlarek 06b7c44593
FEATURE: reason to reject user signup (#11700)
Feature for `Must Approve Users` setup. When a user is rejected, a staff member can optionally set a reason for audit purposes. In addition, feedback email can be sent to the user.

Meta: https://meta.discourse.org/t/account-rejection-email/103112/8
2021-01-15 09:43:26 +11:00
Martin Brennan 2404fa7a23
DEV: Split toggle topic close job (#11679)
Splits the `ToggleTopicClosed` job into two distinct `OpenTopic` and `CloseTopic` jobs to make the code clearer. The old job cannot be deleted yet because of outstanding sidekiq schedules, so a todo has been added to do so later this year.

Also replaced mentions of `topic_status_update` with `topic_timer` in some files, because the `topic_status_update` model is obsolete and replaced by topic timer.

Added some shortcut methods for checking if a topic is open/whether a user can change an open topic.
2021-01-13 08:49:29 +10:00
jbrw b1f32f2f57
DEV: TopicTrackingState calls should happen in the background (#11682)
* DEV: TopicTrackingState calls should happen in the background

It was observed that calling TopicTrackingState on popular topics could result in a large number of calls to redis, resulting in slow response times when posting replies.

These calls should be moved to a background job.

* DEV: PostUpdateTopicTrackingState should execute on default queue
2021-01-11 15:58:27 -05:00
Mark VanLandingham 4601f3be7e
FEATURE: Send notification emails when users leave do not disturb mode (#11643) 2021-01-07 10:49:49 -06:00
Martin Brennan 64ba5b1d21
FIX: Group SMTP email improvements (#11633)
Fixes a rare race condition causing the `Imap::Sync` class to create an incoming email and associated post/topic, which then kicks off the PostAlerter to notify others in the PM about a reply in the topic, but for the OP which is not necessary (because the person emailing the IMAP inbox already knows about the OP). Basically, we should never be sending the group SMTP email for the first post in a topic.

Also in this PR:

* Custom attribute accessors for the to/from/cc addresses on `IncomingEmail`, to parse them from an array to a joined string so the logic for this is only in one place.
* Store extra detail against the `IncomingEmail` created in `GroupSmtpMailer`
* regex test Mail header Reply-To as string instead of Field, which fixes `warning: deprecated Object#=~ is called on Mail::Field; it always returns nil`
* Add DEBUG_IMAP to log all IMAP logs as warnings for easier debugging
* Changed the Rails logging to `ImapSyncLog` in the `GroupSmtpMailer`
2021-01-05 15:32:04 +10:00
Bianca Nenciu 755627caa5
FEATURE: Introduce skip_auto_delete_reply_likes site setting (#11562)
osts from topics with 'auto delete replies timer' with more than
skip_auto_delete_reply_likes likes will no longer be deleted. If 0,
all posts will be deleted.
2020-12-23 16:30:10 +11:00
Roman Rizzi 8a7fe3b276
FIX: Don't enqueue imported users when there're multiple custom fields. (#11559)
My initial implementation didn't consider this case. We should skip imported users if the "imported_id" field is present, even if there're other custom fields.
2020-12-22 14:28:07 -03:00
Krzysztof Kotlarek 6dd3f986b2
FIX: no email when the category is muted and mailing list is on (#11490)
When `mute_all_categories_by_default` setting is enabled we should not send mailing list until category, tag or topic is explicitly watched.
2020-12-16 09:30:21 +11:00
Roman Rizzi c7b9f044a4
FEATURE: Automatically disable slow mode. (#11461)
Staff and TL4 users can decide the slow mode duration. We'll internally set a topic timer to disable it.
2020-12-14 14:06:50 -03:00
Vinoth Kannan eb60fc86dc
PERF: run user merging task in a background job. (#10961)
* PERF: run user merging task in a background job.

Currently, admin page is timing out while merging the users with lots of posts.
2020-12-10 15:52:08 +11:00
Kane York 901a45eeb3
FEATURE: Add likes, flags to user data export (#11439)
This commit is dedicated to https://twitter.com/FiloSottile/status/1335666583126073354 for reminding me that like timestamps are valuable data.

Likes additionally include the topic_id and post_number of the acted post, to aid in analysis. Flag export does not include the disposition by staff.
2020-12-09 15:34:13 -08:00
David Taylor c69bb5d5be
DEV: Always enqueue sidekiq jobs after database transaction commit (#11293)
When jobs are enqueued inside a transaction, it's possible that they will be executed before the necessary data is available in the database. This commit ensures all jobs are enqueued in an ActiveRecord after_commit hook.

One potential downside here is if the job fails to enqueue, the transaction will no longer be aborted. However, the chance of that happening is reasonably low, and the impact is significantly lower than the current issue where jobs are scheduled before their data is ready.
2020-12-08 11:05:01 +11:00
Krzysztof Kotlarek 9c5ee4923b
FEATURE: silently close topic (#11392)
New TopicTimer to silently close topic. It will be used by discourse-solved plugin

Meta: https://meta.discourse.org/t/allow-auto-close-for-solved-to-do-so-silently/169300
2020-12-03 10:43:19 +11:00
Krzysztof Kotlarek b7490e894f FIX: don't error-index category job when missing category
When the category is missing we should not throw an error but just finish the job.
2020-12-01 09:30:53 +08:00
Gerhard Schlager 333f0af0ec
FIX: Cached badge_count isn't updated after backfilling badges (#11281) 2020-11-18 22:01:56 +01:00
Joffrey JAFFEUX de174ef0c4
DEV: cap notifications per run at 300 as stated in comment (#11252) 2020-11-17 09:08:12 +10:00