The job was removed in 6dfe2fbe16 as part of a performance-related refactor.
The issue was that job was already enqueued in sidekiq and now that the file has been deleted, it's generating a lot of `uninitialized constant Jobs::Chat::AutoJoinChannelBatch` errors.
Restoring this file will help clear the sidekiq queue. We'll remove the job in a few months.
Internal ref - t/141563
Chat channels that are linked to a category can be set to automatically join users.
This is handled by subscribing to the following events
- group_destroyed
- user_seen
- user_confirmed_email
- user_added_to_group
- user_removed_from_group
- category_updated
- site_setting_changed (for `chat_allowed_groups`)
As well as a
- hourly background job (`AutoJoinUsers`)
- `CreateCategoryChannel` service
- `UpdateChannel` service
There was however two issues with the current implementation
1. We were triggering a lot of background jobs, mostly because it was decided to batch to auto join/leave into groups of 1000 users, adding a lot of stress to the system
2. We had one "class" (a service or a background job) per "event" and all of them had slightly different ways to select users to join/leave, making it hard to keep everything in sync
This PR "simply" adds two new servicesL `AutoJoinChannels` and `AutoLeaveChannels` that takes care, in an efficient way, of all the cases when users might automatically join a leave a chat channel.
Every other changes come from the fact that we're now always calling either one of those services, depending on the event that happened.
In the making of these classes, a few bugs were encountered and fixed, notably
- A user is only ever able to access chat channels if and only if they're part of a group listed in the `chat_allowed_group` site setting
- A category that has no associated "category groups" is only accessible to staff members (and not "Everyone")
- A silenced user should not be able to automatically join channels
- We should not attempt to automatically join users to deleted chat channels
- There is no need to automatically join users to chat channels that have already more than `max_chat_auto_joined_users` users
Internal - t/135259 & t/70607
* DEV: add specs for auto join/leave channels services
* DEV: less hacky specs
* DEV: no instance variables in specs
This patch replaces the parameters provided to a service through
`params` by the contract object.
That way, it allows better consistency when accessing input params. For
example, if you have a service without a contract, to access a
parameter, you need to use `params[:my_parameter]`. But with a contract,
you do this through `contract.my_parameter`. Now, with this patch,
you’ll be able to access it through `params.my_parameter` or
`params[:my_parameter]`.
Some methods have been added to the contract object to better mimic a
Hash. That way, when accessing/using `params`, you don’t have to think
too much about it:
- `params.my_key` is also accessible through `params[:my_key]`.
- `params.my_key = value` can also be done through `params[:my_key] =
value`.
- `#slice` and `#merge` are available.
- `#to_hash` has been implemented, so the contract object will be
automatically cast as a hash by Ruby depending on the context. For
example, with an AR model, you can do this: `user.update(**params)`.
Currently in services, we don’t make a distinction between input
parameters, options and dependencies.
This can lead to user input modifying the service behavior, whereas it
was not the developer intention.
This patch addresses the issue by changing how data is provided to
services:
- `params` is now used to hold all data coming from outside (typically
user input from a controller) and a contract will take its values from
`params`.
- `options` is a new key to provide options to a service. This typically
allows changing a service behavior at runtime. It is, of course,
totally optional.
- `dependencies` is actually anything else provided to the service (like
`guardian`) and available directly from the context object.
The `service_params` helper in controllers has been updated to reflect
those changes, so most of the existing services didn’t need specific
changes.
The options block has the same DSL as contracts, as it’s also based on
`ActiveModel`. There aren’t any validations, though. Here’s an example:
```ruby
options do
attribute :allow_changing_hidden, :boolean, default: false
end
```
And here’s an example of how to call a service with the new keys:
```ruby
MyService.call(params: { key1: value1, … }, options: { my_option: true }, guardian:, …)
```
Currently, when calling a service with its block form, a `#result`
method is automatically created on the caller object. Even if it never
clashed so far, this could happen.
This patch removes that method, and instead use a more classical way of
doing things: the result object is now provided as an argument to the
main block. This means if we need to access the result object in an
outcome block, it will be done like this from now on:
```ruby
MyService.call(params) do |result|
on_success do
# do something with the result object
do_something(result)
end
end
```
In the same vein, this patch introduces the ability to match keys from
the result object in the outcome blocks, like we already do with step
definitions in a service. For example:
```ruby
on_success do |model:, contract:|
do_something(model, contract)
end
```
Instead of
```ruby
on_success do
do_something(result.model, result.contract)
end
```
On the chat channel settings page, we want to show a single Send push notifications setting instead of the current Desktop notifications and Mobile push notifications settings.
For existing users, use the Mobile push notifications setting value for the new Send push notifications setting.
Makes channel_id and is_direct_message_channel consistent across desktop notifications, which also removes the need to lookup the channel from Chat Notification Manager.
This patch removes the `with_service` helper from the code base.
Instead, we can pass a block with actions directly to the `.call` method
of a service.
This simplifies how to use services:
- use `.call` without a block to run the service and get its result
object.
- use `.call` with a block of actions to run the service and execute
arbitrary code depending on the service outcome.
It also means a service is now “self-contained” and can be used anywhere
without having to include a helper or whatever.
This change introduces a new thread notification level allowing users to get notified when someone replies to the thread.
Users who watch a thread will get a green notification on the chat icon and a user notification (blue). User notifications are consolidated based on thread id to prevent cluttering the original users notification area.
---------
Co-authored-by: Régis Hanol <regis@hanol.fr>
This patch allows using an AR relation as a model in services without
fetching associated records. It will just check if the relation is empty
or not. In the former case, the execution will stop at that point, as
expected.
Prior to this fix we had too logic to detect if a user is active or not:
- idle codepath on the frontend
- online user ids on the backend
The frontend solution is not very reliable, and both solution are just trying to be too smart. Making a lot of people questioning why they receive a notification sometimes and sometimes not. This commit removes all this logic and replaces it with a much more simpler logic:
- you can't receive notifications for channel you are actually watching
- we won't play a sound more than once every 3seconds
Previously services would let you define a high level default `def default_actions_for_service; end` which would define various handlers like `on_success`, after months of usage we consider the cons are superior to the pros here.
Two mains cons:
- people would often not understand where the handling was coming from
- it's easy to miss a case when you write your specs
At the moment, when someone is mentioning a group, or using here or
all mention, we create a chat_mention record per user. What we want
instead is to have special kinds of mentions, so we can create only one
chat_mention record in such cases. This PR implements that.
Note, that such mentions will still have N related notifications, one
notification per a user. We don't expect we'll have performance
problems on the notifications side, but if at some point we do, we
should be able to solve them on the side of notifications
(notifications are handled in jobs, also some little delays with
the notifications are acceptable, so we can make sure notifications
are properly queued, and that processing of every notification is
fast enough to make delays small enough).
The preparation work for this PR was done in fbd24fa, where we make
it possible for one mention to have several related notifications.
A pretty tricky part of this PR is schema and data migration, I've explained
related details inline on the migration files.
This PR is a reworked version of https://github.com/discourse/discourse/pull/24670.
In chat, we need the ability to have several notifications per `chat_mention`.
Currently, we have one_to_one relationship between `chat_mentions` and `notifications`:
d7a09fb08d/plugins/chat/app/models/chat/mention.rb (L9)
We want to have one_to_many relationship. This PR implements that by introducing
a join table between `chat_mentions` and `notifications`.
The main motivation for this is that we want to solve some performance problems
with mentions that we're having now. Let's say a user sends a message with @ all
in a channel with 50 members, we do two things in this case at the moment:
- create 50 chat_mentions
- create 50 notifications
We don't want to change how notifications work in core, but we want to be more
efficient in chat, and create only 1 `chat_mention` which would link to 50 notifications.
Also note, that on the side of notifications, having a lot of notifications is not so
big problem, because notifications processing can be queued.
Apart from improving performance, this change will make the code design better.
Note that I've marked the old `chat_mention.notification_id` column as ignored, but
I'm not deleting it in this PR. We'll delete it later in https://github.com/discourse/discourse/pull/24800.
Mentions and other post processing (like images) are still done asynchronously in the background. This should ensure reloading a channel while the message has not been processed yet doesn’t renders a blank message.
As a followup, we could probably simplify the staged message logic, given we have the new cooked on send.
This commit starts from a simple observation: cooking messages on the hot path can be slow. Especially with a lot of mentions.
To move cooking from the hot path, this commit has made the following changes:
- updating cooked, inserting mentions and notifying user of new mentions has been moved inside the `process_message` job. It happens right after the `Chat::MessageProcessor` run, which is where the cooking happens.
- the similar existing code in `rebake!` has also been moved to rely on the `process_message`job only
- refactored `create_mentions` and `update_mentions` into one single `upsert_mentions` which can be called invariably
- allows services to decide if their job is ran inline or later. It avoids to need to know you have to use `Jobs.run_immediately!` in this case, in tests it will be inline per default
- made various frontend changes to make the chat-channel component lifecycle clearer. we had to handle `did-update @channel` which was super awkward and creating bugs with listeners which the changes of the PR made clear in failing specs
- adds a new `-processed` (and `-not-processed`) class on the chat message, this is made to have a good lifecyle hook in system specs
`Jobs::AutoJoinChannelBatch` was holding a lot of logic which should be in a service. Moreover, this refactoring is the opportunity to address a bug which could cause a duplicate key error.
From now when trying to insert a new membership it won't fail if a membership is already present.
Example error:
```
Job exception: ERROR: duplicate key value violates unique constraint "user_chat_channel_unique_memberships"
DETAIL: Key (user_id, chat_channel_id)=(1, 2) already exists.
Backtrace
rack-mini-profiler-3.1.0/lib/patches/db/pg.rb:110:in `exec'
rack-mini-profiler-3.1.0/lib/patches/db/pg.rb:110:in `async_exec'
(eval):29:in `async_exec'
mini_sql-1.4.0/lib/mini_sql/postgres/connection.rb:209:in `run'
mini_sql-1.4.0/lib/mini_sql/active_record_postgres/connection.rb:38:in `block in run'
mini_sql-1.4.0/lib/mini_sql/active_record_postgres/connection.rb:34:in `block in with_lock'
activesupport-7.0.5.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
activesupport-7.0.5.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
activesupport-7.0.5.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
activesupport-7.0.5.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
mini_sql-1.4.0/lib/mini_sql/active_record_postgres/connection.rb:34:in `with_lock'
mini_sql-1.4.0/lib/mini_sql/active_record_postgres/connection.rb:38:in `run'
mini_sql-1.4.0/lib/mini_sql/postgres/connection.rb:64:in `query_single'
/var/www/discourse/plugins/chat/app/jobs/regular/chat/auto_join_channel_batch.rb:38:in `execute'
```
Note this commit is also using main branch of `shoulda-matchers` as the gem has not been released yet.
Co-authored-by: Loïc Guitaut <5648+Flink@users.noreply.github.com>
Someone who cannot chat is also not able to join chat channels,
so we may not check all the time user.can_chat? && user.can_join_chat_channel?
and just call user.can_join_chat_channel? instead.
Since we create threads in the background regardless of whether
threading is enabled for a channel, we get the unexpected behaviour
of everyone having a lot of unread threads when threading is enabled
for the channel.
To counteract this, when the admin enables threads for a channel
we can just run a high priority background job to mark all threads
as read in the channel for all users, so they are essentially
starting from a clean slate.
This fixes a longstanding TODO to move the contents of the
UpdateUserCountsForChannels job to the ensure_consistency!
method of Chat::Channel, which runs every 15 mins as part of
periodical updates.
This commit also addresses the performance issue of the original,
where we would fetch all channels and do an individual query to
get the count and update the count of each one. Now we do it all
in one query, and only publish the changed channels to the UI.
This commit adds the initial part of thread indicator improvements:
* Show the reply count, last reply date and excerpt,
and the participants of the thread's avatars and
count of additional participants
* Add a participants component for the thread that
can be reused for the list
* Add a query class to get the thread participants
* Live update the thread indicator more consistently
with the last reply and participant details
image image
In subsequent PRs we will cache the participants since
they do not change often, and improve the thread list
further with participants.
This commit also adds a showPresence boolean (default
true) to ChatUserAvatar, since we don't want to show the
online indicator for thread participants.
---------
Co-authored-by: chapoi <charlie@discourse.org>
After this change, in order to join a chat channel, a user needs to be in a group with at least “Reply” permission for the category. If the user only has “See” permission, they are able to preview the channel, but not join it or send messages. The auto-join function also follows this new restriction.
---------
Co-authored-by: Martin Brennan <martin@discourse.org>
* FIX: Link to thread for mentions inside thread
When mentioning a user in a thread, when we send the
notification and display it in the UI we want the URL
of the notification to point to the thread URL to open
the panel, rather than the main channel which is confusing.
For now, we don't have a way to highlight the linked-to message
in the thread, we can revisit this later.
* FIX: Mark mention notifications read when thread opens
Since we have no scrolling/message visibility/thread membership
for now, when a user opens the thread panel we just want to mark
all mention notifications relating to messages in the thread
for the user as read.
Further followup to 24ec06ff85,
where I prevented other chat scheduled jobs from running if
chat was disabled. We should not be running any plugin scheduled
jobs if that plugin is disabled, it can cause unexpected
behaviour.
This was reverted in 38cebd3ed5.
The issue was that I was using Discourse.redis.delete_prefixed
which does a slow redis KEYS lookup, which is not advised in
production. This commit removes that, and also ensures the periodical
thread count update only happens if threading is enabled.
I changed to use a redis INCR/DECR for reply count
cache. This avoids a round trip to redis to GET the current
count, and also avoids multi-process issues, where
if there's two processes trying to increment at the
same time, they may both receive the same value, add one
to it, then both write the same value back.
Then, it's only n+1 instead of n+2.
This also prevents almost all chat scheduled jobs from
running if chat is disabled, the only one remaining is
the message retention job.
This commit introduces a redis cache over the top of the thread
replies_count DB cache, so that we can quickly and accurately
increment/decrement the reply count for all users and not have
to constantly update the database-level count. This is done so
the UI can have a count that is displayed to the users on each
thread indicator, that appears to live update on each chat
message create/trash/recover inside the thread.
This commit also introduces the `Chat::RestoreMessage` service
and moves the restore endpoint into the `Api::ChannelMessages`
controller as part of incremental migrations to move things out
of ChatController.
Finally, this commit refactors `Chat::Publisher` to be less repetitive
with its `MessageBus` sending code.
Similar to 22a55ef0ce,
this commit adds a replies_count to the Chat::Thread
table, which is updated every 15 minutes via PeriodicalUpdates.
This is done so the new thread indicator for the UI can
show the count without intense serializer queries, but
in future we likely want this to update more frequently.
Followup to 0924f874bd,
we migrated Chat::Upload records to UploadReference records
there and have not been making new Chat::Upload records
for some time, we can now delete the model and table.
There are many situations that may cause users to lose permission to
send messages in a chat channel. Until now we have relied on security
checks in `Chat::ChatChannelFetcher` to remove channels which the
user may have a `UserChatChannelMembership` record for but which
they do not have access to.
This commit takes a more proactive approach. Now any of these following
`DiscourseEvent` triggers may cause `UserChatChannelMembership`
records to be deleted:
* `category_updated` - Permissions of the category changed
(i.e. CategoryGroup records changed)
* `user_removed_from_group` - Means the user may not be able to access the
channel based on `GroupUser` or also `chat_allowed_groups`
* `site_setting_changed` - The `chat_allowed_groups` was updated, some
users may no longer be in groups that can access chat.
* `group_destroyed` - Means the user may not be able to access the
channel based on `GroupUser` or also `chat_allowed_groups`
All of these are handled in a distinct service run in a background
job. Users removed are logged via `StaffActionLog` and then we
publish messages on a per-channel basis to users who had their
memberships deleted.
When the user has a channel they are kicked from open, we show
a dialog saying "You no longer have access to this channel".
When they click OK we redirect them either:
* To their first other public channel, if they have any followed
* The chat browse page if they don't
This is to save on tons of requests from kicked out users getting messages
from other channels.
When the user does not have the kicked channel open, we can just
silently yoink it out of their sidebar and turn off subscriptions.
This commit main goal was to comply with Zeitwerk and properly rely on autoloading. To achieve this, most resources have been namespaced under the `Chat` module.
- Given all models are now namespaced with `Chat::` and would change the stored types in DB when using polymorphism or STI (single table inheritance), this commit uses various Rails methods to ensure proper class is loaded and the stored name in DB is unchanged, eg: `Chat::Message` model will be stored as `"ChatMessage"`, and `"ChatMessage"` will correctly load `Chat::Message` model.
- Jobs are now using constants only, eg: `Jobs::Chat::Foo` and should only be enqueued this way
Notes:
- This commit also used this opportunity to limit the number of registered css files in plugin.rb
- `discourse_dev` support has been removed within this commit and will be reintroduced later
<!-- NOTE: All pull requests should have tests (rspec in Ruby, qunit in JavaScript). If your code does not include test coverage, please include an explanation of why it was omitted. -->
`create_notification!` - creates a notification in the database, `send_notifications` sends desktop and mobile notifications. This PR moves some code to decouple these two tasks more explicitly. It only moves code without changing any behavior, and the job is covered with tests (see chat_notify_mentioned_spec).
Before this commit, we created a chat mention record only in case we wanted to send a notification about that mention to the user. Notifications were the only use case for the chat_mention db table. Now we want to use that table for other features, so we have to always create a chat_mention record.
This patch introduces a new `ServiceJob` class allowing the use of
`with_service` in jobs.
This way, it’s easier to use the chat service objects in jobs and
provides the same level of functionality than the one we have in
controllers.
* DEV: Rnemae channel path to just c
Also swap the channel id and channel slug params to be consistent with core.
* linting
* channel_path
* Drop slugify helper and channel route without slug
* Request slug and route models through the channel model if possible
* DEV: Pass messageId as a dynamic segment instead of a query param
* Ensure change is backwards-compatible
* drop query param from oneboxes
* Correctly extract channelId from routes
* Better route organization using siblings for regular and near-message
* Ensures sessions are unique even when using parallelism
* prevents didReceiveAttrs to clear input mid test
* we disable animations in capybara so sometimes the message was barely showing
* adds wait
* ensures finished loading
* is it causing more harm than good?
* this check is slowing things for no reason
* actually target the button
* more resilient select chat message
* apply similar fix to bookmark
* fix
---------
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
We've had the UploadReference table for some time now in core,
but it was added after ChatUpload was and chat was just never
moved over to this new system.
This commit changes all chat code dealing with uploads to create/
update/delete/query UploadReference records instead of ChatUpload
records for consistency. At a later date we will drop the ChatUpload
table, but for now keeping it for data backup.
The migration + post migration are the same, we need both in case
any chat uploads are added/removed during deploy.
This PR removes the limit added to max_users_notified_per_group_mention during #19034 and improve the performance when expanding mentions for large channel or groups by removing some N+1 queries and making the whole process async.
* Fully async chat message notifications
* Remove mention setting limit and get rid of N+1 queries
Note this is a very large PR, and some of it could have been splited, but keeping it one chunk made it to merge conflicts and to revert if necessary. Actual new code logic is also not that much, as most of the changes are removing js tests, adding system specs or moving things around.
To make it possible this commit is doing the following changes:
- converting (and adding new) existing js acceptances tests into system tests. This change was necessary to ensure as little regressions as possible while changing paradigm
- moving away from store. Using glimmer and tracked properties requires to have class objects everywhere and as a result works well with models. However store/adapters are suffering from many bugs and limitations. As a workaround the `chat-api` and `chat-channels-manager` are an answer to this problem by encapsulating backend calls and frontend storage logic; while still using js models.
- dropping `appEvents` as much as possible. Using tracked properties and a better local storage of channel models, allows to be much more reactive and doesn’t require arbitrary manual updates everywhere in the app.
- while working on replacing store, the existing work of a chat api (backend) has been continued to support more cases.
- removing code from the `chat` service to separate concerns, `chat-subscriptions-manager` and `chat-channels-manager`, being the largest examples of where the code has been rewritten/moved.
Future wok:
- improve behavior when closing/deleting a channel, it's already slightly buggy on live, it's rare enough that it's not a big issue, but should be improved
- improve page objects used in chat
- move more endpoints to the API
- finish temporarily skipped tests
- extract more code from the `chat` service
- use glimmer for `chat-messages`
- separate concerns in `chat-live-pane`
- eventually add js tests for `chat-api`, `chat-channels-manager` and `chat-subscriptions-manager`, they are indirectly heavy tested through system tests but it would be nice to at least test the public API
<!-- NOTE: All pull requests should have tests (rspec in Ruby, qunit in JavaScript). If your code does not include test coverage, please include an explanation of why it was omitted. -->
Previously, restricted category chat channel was available for all groups - even `readonly`. From now on, only user who belong to group with `create_post` or `full` permissions can access that chat channel.
Régis Hanol
Loïc Guitaut
David Battersby
Sam
Joffrey JAFFEUX
Jan Cernik
Mark VanLandingham
Andrei Prigorshnev
David Taylor
Martin Brennan
Daniel Waterworth
Roman Rizzi
Krzysztof Kotlarek