This commit is a series of fixes to improve stability of system tests following the use of threadsafe:
* Jobs.run_immediately in before block was causing issues
* During test a js error could be caused by an undefined this.details in chat-live-pane
* Apply the chat composer click trick everywhere when sending a message, it ensures we are not hiding anything with autocomplete
* There was another case not using send_message yet
This PR removes the limit added to max_users_notified_per_group_mention during #19034 and improve the performance when expanding mentions for large channel or groups by removing some N+1 queries and making the whole process async.
* Fully async chat message notifications
* Remove mention setting limit and get rid of N+1 queries
It should fix flakeys we have due to using_session. This commit is also fixing tests which were failing constantly with treadsafe enabled.
A test has also bene skipped as the issue couldn't be found so far.
More info: https://github.com/teamcapybara/capybara#threadsafe-mode
- improves UI by displaying channel status on it's own line
- ensures channel status is correctly updated right after the request on frontend
- adds status on info page
Recent changes surfaced the various issues with this codepath:
- we were not correctly reseting `messageLookup` leading to us trying to scroll to a non existing message in the view
- we were calling markAsRead which would scroll to the bottom, even if we had a target message
- we were not debouncing fetchMessages, which could cause multiple reload of the messages when loading it with a targetMessageId: first fetch from last read and then immediately fetch from targetMessageId
- other naming inconsistencies
- not handling drawer
This commit also adds tests for classic scenarios related to this use case.
I could repro the same failure by doing: `page.driver.browser.network_conditions = { offline: false, latency: 3000, throughput: 0 }`
Wait shouldn't be needed as we wait for selector, but I couldn't find a better solution on this case for now.
We are all in on system specs, so this commit moves all the chat quoting acceptance tests (some of which have been skipped for a while) into system specs.
The way our markdown raw_html hoisting worked, we only
supported one level of hoisting the HTML content. However
when nesting [chat] transcript BBCode we need to allow
for multiple levels of it. This commit changes opts.discourse.hoisted
to be more constant, and the GUID keys that have the hoisted
content are only deleted by unhoistForCooked rather than
the cook function itself, which prematurely deletes them
when they are needed further down the line.
Honestly seems like it's being in some weird loop for
discourse/hashtag_autocomplete_spec.rb for this:
```ruby
within topic_page.post_by_number(2) do
cooked_hashtags = page.all(".hashtag-cooked", count: 2)
expect(cooked_hashtags[0]["outerHTML"]).to eq(<<~HTML.chomp)
<a class=\"hashtag-cooked\" href=\"#{category.url}\" data-type=\"category\" data-slug=\"cool-cat\"><svg class=\"fa d-icon d-icon-folder svg-icon svg-node\"><use href=\"#folder\"></use></svg><span>Cool Category</span></a>
HTML
expect(cooked_hashtags[1]["outerHTML"]).to eq(<<~HTML.chomp)
<a class=\"hashtag-cooked\" href=\"#{tag.url}\" data-type=\"tag\" data-slug=\"cooltag\"><svg class=\"fa d-icon d-icon-tag svg-icon svg-node\"><use href=\"#tag\"></use></svg><span>cooltag</span></a>
HTML
end
```
I see this many times in the full logs with `SELENIUM_VERBOSE_DRIVER_LOGS=1`:
```
COMMAND FindElements {
"using": "css selector",
"value": "#post_2"
}
Followed by:
COMMAND FindChildElements {
"id": "26dfe542-659b-46cc-ac8c-a6c2d9cbdf0a",
"using": "css selector",
"value": ".hashtag-cooked"
}
```
Over and over and over, there are 58 such occurrences. I am beginning to
think `within` is just poison that should be avoided.
Previously, calling `sign_in` would cause the browser to be redirected to `/`, and would cause the Ember app to boot. We would then call `visit()`, causing the app to boot for a second time.
This commit adds a `redirect=false` option to the `/session/username/become` route. This avoids the unnecessary boot of the app, and leads to significantly faster system spec run times.
In local testing, this takes the full system-spec suite for chat from ~6min to ~4min.
- chat-message is not using chat-api yet and the `jsonMode` shouldn't have been added
- correctly error on `getChannel` not found
- adds/correct relevant system tests
Note this is a very large PR, and some of it could have been splited, but keeping it one chunk made it to merge conflicts and to revert if necessary. Actual new code logic is also not that much, as most of the changes are removing js tests, adding system specs or moving things around.
To make it possible this commit is doing the following changes:
- converting (and adding new) existing js acceptances tests into system tests. This change was necessary to ensure as little regressions as possible while changing paradigm
- moving away from store. Using glimmer and tracked properties requires to have class objects everywhere and as a result works well with models. However store/adapters are suffering from many bugs and limitations. As a workaround the `chat-api` and `chat-channels-manager` are an answer to this problem by encapsulating backend calls and frontend storage logic; while still using js models.
- dropping `appEvents` as much as possible. Using tracked properties and a better local storage of channel models, allows to be much more reactive and doesn’t require arbitrary manual updates everywhere in the app.
- while working on replacing store, the existing work of a chat api (backend) has been continued to support more cases.
- removing code from the `chat` service to separate concerns, `chat-subscriptions-manager` and `chat-channels-manager`, being the largest examples of where the code has been rewritten/moved.
Future wok:
- improve behavior when closing/deleting a channel, it's already slightly buggy on live, it's rare enough that it's not a big issue, but should be improved
- improve page objects used in chat
- move more endpoints to the API
- finish temporarily skipped tests
- extract more code from the `chat` service
- use glimmer for `chat-messages`
- separate concerns in `chat-live-pane`
- eventually add js tests for `chat-api`, `chat-channels-manager` and `chat-subscriptions-manager`, they are indirectly heavy tested through system tests but it would be nice to at least test the public API
<!-- NOTE: All pull requests should have tests (rspec in Ruby, qunit in JavaScript). If your code does not include test coverage, please include an explanation of why it was omitted. -->
* UX: added fadeout + hashtag styling
UX: add full name to autocomplete
UX: autocomplete mentions styling
UX: emoji styling user status
UX: autocomplete emoji
* DEV: Move hashtag tag counts into new secondary_text prop
* FIX: Add is-online style to mention users via chat
UX: make is-online avatar styling globally available
* DEV: Fix specs
* DEV: Test fix
Co-authored-by: Martin Brennan <martin@discourse.org>
Follow up to a review in #18937, this commit changes the HashtagAutocompleteService to no longer use class variables to register hashtag data sources or types in context priority order. This is to address multisite concerns, where one site could e.g. have chat disabled and another might not. The filtered plugin registers I added will not be included if the plugin is disabled.
In both ChatMessage#rebake! and in ChatMessageProcessor
when we were calling ChatMessage.cook we were missing the
user_id to cook with, which causes missed hashtag cooks
because of missing permissions.
Previously, restricted category chat channel was available for all groups - even `readonly`. From now on, only user who belong to group with `create_post` or `full` permissions can access that chat channel.
This introduces another "section" of queries to the
hashtag autocomplete search, which returns results for
each type that start with the search term. So now results
will be in this order, and within these sections ordered
by the types in priority order:
1. Exact matches sorted by type
2. "starts with" sorted by type
3. Everything else sorted by type then name within type
There is no need to duplicate check chat messages when they are being
edited but not having their message text changed. This was leading to
a validation error when adding/removing an upload but not changing the
message text.
FEATURE: Chat and Sidebar are now on by default
- Set the sidebar site setting to be enabled by default
- Set the chat site setting to be enabled by default
- Updated existing specs that assumed the original default
- Use a migration to keep old defaults for existing sites
Instead of passing `user` to `guardian.can_chat?`, we
can just use the inner `@user` that is part of the guardian
instance already to determine whether that user can chat,
since this is how it works for all other usages of guardian
even within chat.
Previously with this experimental feature a user would be
able to search for public channels for public categories
using the new #hashtag system even if they couldn't chat.
This commit fixes the hole.
This commit allows us to type # in the UI and present autocomplete
results immediately with the following logic for the topic composer,
and reversed for the chat composer:
* Categories the user can access and has not muted sorted by `topic_count`
* Tags the user can access and has not muted sorted by `topic_count`
* Chat channels the user is a member of sorted by `messages_count`
So in effect, we allow searching for hashtags without a search term.
To do this we add a new `search_without_term` to each data source so
each one can define how it wants to handle this logic.
This new site setting replaces the
`enable_experimental_sidebar_hamburger` and `enable_sidebar` site
settings as the sidebar feature exits the experimental phase.
Note that we're replacing this without depreciation since the previous
site setting was considered experimental.
Internal Ref: /t/86563
* FEATURE: Enforce mention limits for chat messages
The first part of these changes adds a new setting called `max_mentions_per_chat_message`, which skips notifications when the message contains too many mentions. It also respects the `max_users_notified_per_group_mention` setting
and skips notifications if expanding a group mention would exceed it.
We also include a new component to display JIT warning for these limits to the user while composing a message.
* Simplify ignoring/muting filter in chat_notifier
* Post-send warnings for unsent warnings
* Improve pluralization
* Address review feedback
* Fix test
* Address second feedback round
* Third round of feedback
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
This commit adds the messages_count column for ChatChannel messages,
which is the number of not-deleted messages in the channel.
This is not updated every time a message is created or deleted in a
channel, so it should not be displayed in the UI.
It is updated eventually via Jobs::ChatPeriodicalUpdates, which
will have additional functions in future after being introduced
here.
Also update these counts for existing channels in a post migration.
The settings tab of each category channel should now present the option to allow or disallow channel wide mentions: @here and @all.
When disallowed, using these mentions in the channel should have no effect.
There must have been a small loophole that allowed
setting the channel slug in the DB which has led to
conflicts in some cases.
This commit fixes the conflicting chat channel
slugs and then changes the channel slug index
to a unique one in the DB.
This commit adds variousMessageBus.last_ids to serializer payloads
for chat channels and the chat view (for chat live pane) so
we can use those IDs when subscribing to MessageBus channels
from chat.
This allows us to ensure that any messages created between the
server being hit and the UI loaded and subscribing end up being
delivered to the client, rather than just silently dropped.
This commit also fixes an issue where we were subscribing to
the new-messages and new-mentions MessageBus channels multiple
times when following/unfollowing a channel multiple times.
This commit introduce a new API for registering callbacks, which we'll execute when a user gets destroyed, and the `delete_posts` opt is true. The chat plugin registers one callback and queues a job to destroy every message from that user in batches.
* FIX: Unsilence users on chat message flag disagree.
We have an auto silence rule in place for chat message flags, so we need to unsilence users if the flag gets rejected.
Additionally, it also fixes the `disagree_and_restore` action, which wasn't recovering a deleted message.
* Update plugins/chat/spec/models/reviewable_chat_message_spec.rb
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
Only allow maximum of 6000 characters for chat messages when they
are created or edited. A hidden setting can control this limit,
6000 is the default.
There is also a migration here to truncate any existing messages to
6000 characters if the message is already over that and if the
chat_messages table exists. We also set cooked_version to NULL
for those messages so we can identify them for rebake.
Refines the behavior of clicking the chat icon in mobile and when in drawer mode as follows: If chat is open, clicking the icon takes you to the index.
- better handling of drawer state using chat state manager
- removes various float and topic occurrences to use drawer
- ensures user can chat before doing a lot of chat setup
- fixes a bug which was creating presence errors in tests
- removes dead code
* Do not search category name when searching channels to avoid
confusing results
* Overflow text in autocomplete menu with ... if it is too long
* Make autocomplete menu less height
Adds the description as a title="" attribute on the hashtag
autocomplete search items for tags, categories, and channels.
These descriptions can be seen by the user since they are
able to see the results that are returned by the search via
Guardian checks.
Currently when a category is deleted, if it has an associated chat
channel, the latter won’t be deleted automatically.
The fix is quite simple as we were simply missing a `dependent:
:destroy` option on the existing relation.
By doing this, we will:
* Have an open, but safe default People reach `@trust_level_1` pretty
quickly, but `@trust_level_0` is still excluded by default, to limit new
accounts joining and immediately spamming or otherwise abusing channels.
* Make it easier to change the default By keeping `@staff` in the
default, we make it easy for admins to remove `@trust_level_1` and
optionally add additional groups to their liking.
This commit fleshes out and adds functionality for the new `#hashtag` search and
lookup system, still hidden behind the `enable_experimental_hashtag_autocomplete`
feature flag.
**Serverside**
We have two plugin API registration methods that are used to define data sources
(`register_hashtag_data_source`) and hashtag result type priorities depending on
the context (`register_hashtag_type_in_context`). Reading the comments in plugin.rb
should make it clear what these are doing. Reading the `HashtagAutocompleteService`
in full will likely help a lot as well.
Each data source is responsible for providing its own **lookup** and **search**
method that returns hashtag results based on the arguments provided. For example,
the category hashtag data source has to take into account parent categories and
how they relate, and each data source has to define their own icon to use for the
hashtag, and so on.
The `Site` serializer has two new attributes that source data from `HashtagAutocompleteService`.
There is `hashtag_icons` that is just a simple array of all the different icons that
can be used for allowlisting in our markdown pipeline, and there is `hashtag_context_configurations`
that is used to store the type priority orders for each registered context.
When sending emails, we cannot render the SVG icons for hashtags, so
we need to change the HTML hashtags to the normal `#hashtag` text.
**Markdown**
The `hashtag-autocomplete.js` file is where I have added the new `hashtag-autocomplete`
markdown rule, and like all of our rules this is used to cook the raw text on both the clientside
and on the serverside using MiniRacer. Only on the server side do we actually reach out to
the database with the `hashtagLookup` function, on the clientside we just render a plainer
version of the hashtag HTML. Only in the composer preview do we do further lookups based
on this.
This rule is the first one (that I can find) that uses the `currentUser` based on a passed
in `user_id` for guardian checks in markdown rendering code. This is the `last_editor_id`
for both the post and chat message. In some cases we need to cook without a user present,
so the `Discourse.system_user` is used in this case.
**Chat Channels**
This also contains the changes required for chat so that chat channels can be used
as a data source for hashtag searches and lookups. This data source will only be
used when `enable_experimental_hashtag_autocomplete` is `true`, so we don't have
to worry about channel results suddenly turning up.
------
**Known Rough Edges**
- Onebox excerpts will not render the icon svg/use tags, I plan to address that in a follow up PR
- Selecting a hashtag + pressing the Quote button will result in weird behaviour, I plan to address that in a follow up PR
- Mixed hashtag contexts for hashtags without a type suffix will not work correctly, e.g. #ux which is both a category and a channel slug will resolve to a category when used inside a post or within a [chat] transcript in that post. Users can get around this manually by adding the correct suffix, for example ::channel. We may get to this at some point in future
- Icons will not show for the hashtags in emails since SVG support is so terrible in email (this is not likely to be resolved, but still noting for posterity)
- Additional refinements and review fixes wil
This setting limits the number of users in a direct message. 0 means you can only create a direct message with yourself.
Co-authored-by: David McClure <dave@xerotrope.org>
- Note this is also tweaking the UI a little bit as we are now using links/buttons in the header as needed
- It disables the find ideal channel in drawer mode, if loading `/chat` in drawer mode it will either reopen at the last position or just stay on index
The mailer in charge of sending chat summary emails applies a filter to ensure only members of groups listed in the `chat allowed groups` setting receive them. However, when you set it to `everyone`, nobody will be notified because
we treat this group differently and don't create `GroupUser` records for every user on the site.
This commit changes the mailer to skip the filter when the `everyone` ID is in the list.
This commit automatically ensures that category channels
have slugs when they are created or updated based on the
channel name, category name, or existing slug. The behaviour
has been copied from the Category model.
We also include a backfill here with a simplified version
of Slug.for with deduplication to fill the slugs for already
created Category chat channels.
The channel slug is also now used for chat notifications,
and for the UI and navigation for chat. `slugifyChannel`
is still used, but now does the following fallback:
* Uses channel.slug if it is present
* Uses channel.escapedTitle if it is present
* Uses channel.title if it is present
In future we may want to remove this altogether
and always rely on the slug being present, but this
is currently not possible because we are not generating
slugs for DM channels at this point.
Clicking a link of the sidebar will now open the drawer and load the correct channel.
This solution should correctly solve these cases:
closing drawer, clicking sidebar channel, should open the drawer on correct channel
visiting /chat then visiting / and clicking sidebar channel, should open full page chat on correct channel
Currently it’s not possible to delete a category if an associated chat
channel is present even if there are no messages in this channel.
This can lead to annoying situations for our users.
This patch addresses the issue by checking if the channel is empty
instead of just checking if there is a channel.
Follow up to 766bcbc684
Makes ChatMessage.last_editor_id and ChatMessageRevision.user_id
NOT NULL since they are always filled in now and the last commit
had a migration to backfill this data.
Follow up to 766bcbc684
This fixes a gaffe from that commit where I passed in the
guardian to ChatMessageUpdater but then forgot to remove
the old way of setting the guardian and user instance variables
from the chat_message that was passed in.
Also, it moves the ensure_can_edit_message! check from the
controller into ChatMessageUpdater so all the access
checks are in the same place.
This commits makes sure we correctly wait for the end of the transition to reopen the drawer on the correct channel/view. Also fixes a bug when previous URL was `/` and causing a double transition.
Also adds end to end system tests to ensure navigation scenarios are working correctly. This separation will make it easier to implement state in/out from chat.
This commit adds last_editor_id to ChatMessage for parity with Post in
core, as well as adding user_id to the ChatMessageRevision record since
we need to know who is making edits and revisions to messages, in case
in future we want to allow more than just the current user to edit chat
messages. The backfill for data here simply uses the record's creating
user ID, but in future if we allow other people to edit the messages it
will use their ID.
This is a followup of the previous refactor where we created two new
models to handle all the dedicated logic that was present in the
`ChatChannel` model.
For the sake of consistency, `DMChannel` has been renamed to
`DirectMessageChannel` and the previous `DirectMessageChannel` model is
now named `DirectMessage`. This should help reasoning about direct
messages.
Joffrey JAFFEUX
Martin Brennan
Roman Rizzi
Krzysztof Kotlarek
David Taylor
Rafael dos Santos Silva
chapoi
Blake Erickson
Alan Guo Xiang Tan
Gerhard Schlager
Loïc Guitaut
Kris