This commit adds the `add_request_rate_limiter` plugin API which allows plugins to add custom rate limiters on top of the default rate limiters which requests by a user's id or the request's IP address.
Example to add a rate limiter that rate limits all requests from Googlebot under the same rate limit bucket:
```
add_request_rate_limiter(
identifier: :country,
key: ->(request) { "country/#{DiscourseIpInfo.get(request.ip)[:country]}" },
activate_when: ->(request) { DiscourseIpInfo.get(request.ip)[:country].present? },
)
```
When lurking on a Discourse as anonymous, if the sidebar is enabled, and a section contains only secondary links that are not visible to anonymous users, we should not display the "more..." button.
Otherwise it feels broken because clicking on it does nothing, since there are no "visible" links to be shown.
Internal ref t/144716
* wip: return full name in /notifications.json
* DEV: test for full name
* DEV: add test for enable_names=true
* DEV: add notification6, cleanup
* DEV: fix tests
This PR involves cleaning up the codebase from my (@keegangeorge's) todos.
In particular:
- Remove Form Template related todos (these are no longer in the roadmap)
- Remove old left-over AI summarization related code after moving to AI (https://github.com/discourse/discourse-ai/pull/658)
- Update one form template related spec
* Split `shutdown` into two separate methods for better control:
- `shutdown` - signals threads to stop accepting new work
- `wait_for_termination` - waits for threads to finish (with optional timeout)
* Add tracking of busy threads via `@busy_threads` Set
* Make idle_time parameter optional with 30-second default
* Improve thread spawning logic:
- Spawn initial thread immediately when work is posted
- Spawn additional threads when all threads are busy and work is queued
* Fix race condition in work distribution
* Add busy thread count to stats output
* Add test coverage for zero min_threads configuration
This commit makes the ThreadPool more reliable, easier to use, and adds
better visibility into its internal state.
---------
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
This commit introduces a new ThreadPool class that provides efficient worker
thread management for background tasks. Key features include:
- Dynamic scaling from min to max threads based on workload
- Proper database connection management in multisite setup
- Graceful shutdown with task completion
- Robust error handling and logging
- FIFO task processing with a managed queue
- Configurable idle timeout for worker threads
The implementation is thoroughly tested, including stress tests, error
scenarios, and multisite compatibility.
This update adds a ✨ _new_ `<PostList />` component, along with it's child components (`<PostListItem/>` and `<PostListItemDetails />`). This new generic component can be used to show a list of posts.
It can be used like so:
```js
/**
* A component that renders a list of posts
*
* @component PostList
*
* @args {Array<Object>} posts - The array of post objects to display
* @args {Function} fetchMorePosts - A function that fetches more posts. Must return a Promise that resolves to an array of new posts.
* @args {String} emptyText (optional) - Custom text to display when there are no posts
* @args {String|Array} additionalItemClasses (optional) - Additional classes to add to each post list item
* @args {String} titleAriaLabel (optional) - Custom Aria label for the post title
*
*/
```
```hbs
<PostList
@posts={{this.posts}}
@fetchMorePosts={{this.loadMorePosts}}
@emptyText={{i18n "custom_identifier.empty"}}
@additionalItemClasses="custom-class"
/>
```
Setting tab should be added to permalinks so admins do not need to have left `/permalinks`.
A new component called `AreaSetting` was added to avoid duplications and
simplify adding settings to other sections.
This allows plugins to skip the "posted" notifications for watching users, when posts get moved. The specs are kind of wild looking, as this unit tests a private method. This is difficult to isolate otherwise, with lots of trickery needed to make sure that this actually works.
I opted to unit test just this method instead.
There’s currently a bug when using a dedicated class as a policy in
services: if that class delegates its `#call` method (to an underlying
strategy object for example), then an error will be raised saying steps
aren’t allowed to provide default parameters.
This should not happen, and this patch fixes that issue.
This commit reimplements how we monitor Sidekiq processes that are
forked from the Unicorn master process. Prior to this change, we rely on
`Jobs::Heartbeat` to enqueue a `Jobs::RunHeartbeat` job every 3 minutes.
The `Jobs::RunHeartbeat` job then sets a Redis key with a timestamp. In
the Unicorn master process, we then fetch the timestamp that has been set
by the job from Redis every 30 minutes. If the timestamp has not been
updated for more than 30 minutes, we restart the Sidekiq process. The
fundamental flaw with this approach is that it fails to consider
deployments with multiple hosts and multiple Sidekiq processes. A
sidekiq process on a host may be in a bad state but the heartbeat check
will not restart the process because the `Jobs::RunHeartbeat` job is
still being executed by the working Sidekiq processes on other hosts.
In order to properly ensure that stuck Sidekiq processs are restarted,
we now rely on the [Sidekiq::ProcessSet](https://github.com/sidekiq/sidekiq/wiki/API#processes)
API that is supported by Sidekiq. The API provides us with "near real-time (updated every 5 sec)
info about the current set of Sidekiq processes running". The API
provides useful information like the hostname, pid and also when Sidekiq
last did its own heartbeat check. With that information, we can easily
determine if a Sidekiq process needs to be restarted from the Unicorn
master process.
This converts the `<AdminPageHeader />` component and the
`<AdminPageSubheader />` components into new components
that can be used outside of admin, and updates the CSS classes.
Also introduces a `<DPageActionButton />` component and child
components for the header action buttons.
I have to keep the old admin-only components around for
now until plugins are updated, then we can remove it,
and remove the re-exports that are done within
admin-page-action-button.gjs
When freeze_original option is passed to PostMover, and we are moving all posts there is an issue. We attempt to put the small_action right after the last moved post. The issue is when there is an existing small action after the last moved "real" post. We then try to put the moderator post at the same location of the existing small action, which causes an index conflict and the move fails.
This makes sure that we place the moderator post at the verrrrrry end of the topic :)
These controller tests are passing locally and in CI, but are failing the build when run in parallel.
I managed to recreate the failures by running the entire suite with turbo_spec and the right seed locally. After these changes, the parallel suite passes locally as well. 🤞
`new_in_category` was using `first` instead of `limit`
This meant it gets an array and that means that you can not operate on it easily in a modifier.
This ensures we always give the modifier a relation, with the notable exception of suggested topics.
We're changing the default of hide_email_address_taken to true. This is a trade-off we want to make, as it prevents account enumeration with minimal impact on legitimate users. If you forget you have an account and try to sign up again with the same e-mail you'll receive an e-mail letting you know.
Add flag reason filter and improve handling of deleted content in review queue
This commit enhances the review queue with several key improvements:
1. Adds a new "Reason" filter to allow filtering flags by their score type
2. Improves UI for deleted content by:
- Adding visual indication for deleted posts (red background)
- Properly handling deleted content visibility for staff (category mods can not see deleted content)
3. Refactors reviewable score type handling for better code organization
4. Adds tests for trashed topics/posts visibility
This change will help moderators more efficiently manage the review queue by
being able to focus on specific types of flags and better identify deleted
content.
This commit introduces <NotificationsTracking /> which is a wrapper component around <DMenu /> which replaces the select-kit component <TopicNotificationsButton />.
Each tracking case has its dedicated component:
- topic -> `<TopicNotificationsTracking />`
- group -> `<GroupNotificationsTracking />`
- tag -> `<TagNotificationsTracking />`
- category -> `<CategoryNotificationsTracking />`
- chat thread -> `<ThreadNotificationsTracking />`
Previously, theme hbr files were compiled to an IIFE, which would be executed before the app is booted. That is causing silenced deprecations to be printed, because the deprecation-workflow isn't set up when the IIFE is run.
This commit updates the theme compiler so that it matches the ember-cli-based raw-hbs compiler. Templates are output to normal modules, which will then be loaded by the existing `eager-load-raw-templates` initializer. This runs after the app has started booting.
* DEV: add db consistency check for UserEmail
* DEV: add db consistency check for UserAvatar
* DEV: ignore inconsistent data related to user avatars when deciding whether to rebake old posts
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
---------
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
Currently, there are two ways (kind of) for accessing `params` inside a
service:
- when there is no contract or it hasn’t been reached yet, `params` is
just the hash that was provided to the service. To access a key, you
have to use the bracket notation `params[:my_key]`.
- when there is a contract and it has been executed successfully,
`params` now references the contract and the attributes are accessible
using methods (`params.my_key`).
This patch unifies how `params` exposes its attributes. Now, even if
there is no contract at all in a service, `params` will expose its
attributes through methods, that way things are more consistent.
This patch also makes sure there is always a `params` object available
even when no `params` key is provided to the service (this allows a
contract to fail because its attributes are blank instead of having the
service raising an error because it doesn’t find `params` in its context).
This commit adds a new column full_move to the moved_posts table. This is useful to look back at history and determine if a whole topic was moved or partial.
This commit also adds an apply_modifier to skip the creation of the moved posts small action.
This patch aims to improve the steps inspector output:
- The service class name is displayed at the top.
- Next to each step is displayed the time it took to run said step.
- Steps that didn’t run are hidden.
- `#inspect` automatically outputs the error when it is present.
It doesn't make much sense to have the content of a `<details>` in an excerpt so I replaced them with "▶ summary" instead.
That way, they can't be (ab)used in user cards for example.
Reference - https://meta.discourse.org/t/335094
The new name may be too long for the bookmarks.name column and raise an
exception. This changes allows the remapper to truncate the new value to
fit (truncates to 100 characters).
The test was flaky and failing with the following errors:
```
Failure/Error:
klass
.connection
.select_raw(relation.arel) do |result, _|
result.type_map = DB.type_map
result.nfields == 1 ? result.column_values(0) : result.values
end
NoMethodError:
undefined method `select_raw' for nil
./lib/freedom_patches/fast_pluck.rb:60:in `pluck'
./vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/relation/calculations.rb:354:in `pick'
./app/models/web_crawler_request.rb:27:in `request_id'
./app/models/web_crawler_request.rb:31:in `rescue in request_id'
./app/models/web_crawler_request.rb:26:in `request_id'
./app/models/web_crawler_request.rb:19:in `write_cache!'
./app/models/concerns/cached_counting.rb:135:in `block (3 levels) in flush_to_db'
./vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
./vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management.rb:21:in `with_connection'
./app/models/concerns/cached_counting.rb:134:in `block (2 levels) in flush_to_db'
./app/models/concerns/cached_counting.rb:124:in `each'
./app/models/concerns/cached_counting.rb:124:in `block in flush_to_db'
./lib/distributed_mutex.rb:53:in `block in synchronize'
./lib/distributed_mutex.rb:49:in `synchronize'
./lib/distributed_mutex.rb:49:in `synchronize'
./lib/distributed_mutex.rb:34:in `synchronize'
./app/models/concerns/cached_counting.rb:120:in `flush_to_db'
./app/models/concerns/cached_counting.rb:187:in `perform_increment!'
./app/models/web_crawler_request.rb:15:in `increment!'
./lib/middleware/request_tracker.rb:74:in `log_request'
./lib/middleware/request_tracker.rb:409:in `block in log_later'
./lib/scheduler/defer.rb:125:in `block in do_work'
./vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
./vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management.rb:21:in `with_connection'
./lib/scheduler/defer.rb:119:in `do_work'
./lib/scheduler/defer.rb:105:in `block (2 levels) in start_thread'
```
This was due to running the defer thread in an async manner which is
actually no representative of the production environment. It also
revealed a spot in our code base where writes are happening in a GET
request which can cause requests to fail if ActiveRecord is in readonly
mode.
The directory items controller specs that have a search param were not
matching how things worked in production. In a non-test environment the
UserSearch class depends on the `user_search_data` table being
populated, so the tests I corrected now use this table as well to match
reality.
Also added a new test to match the 20 user limit for search results that
currently exists. This 20 user limit is on the line between a bug and a
feature but it is how it is currently working so we should document
that. We have plans to increase this limit and it has been documented
here: https://meta.discourse.org/t/296485
This PR is a no-op and only changes the tests.
Co-authored-by: brrusselburg <25828824+brrusselburg@users.noreply.github.com>
When a post containing an apostrophe (') is being cooked, the apostrophe is being converted to the "typographic" version (’) (because we enable markdown-it's **typographer** mode by default in Discourse)
When you select text that contains such apostrophe and then try to save your fast edit, it fails miserably without any error.
That's because when you select text from the DOM, it uses the cooked version which has the typographic apostrophe.
When you save your fast edit, we fetch the raw version of the post, which has the "regular" apostrophe. Thus doing `raw.replace(selectedText, editedText)` doesn't work because `raw` has the regular apostrophe but `selectedText` has the typographic apostrophe.
Since it's somewhat complicated to handle all typographic characters, we would basically have to reverse the process done in `custom-typographer-replacements.js`, we instead bail out and show the composer when we detect such character in the selection.
Internal ref - t/143836
This reverts commit 766ff723f8.
Ensure that we create the sidekiq log file first before opening it for
logging. This avoids any issue of the log file not being present when we
initialize an instance of the `Logger`.
Alan Guo Xiang Tan
Sam
Régis Hanol
marstall
Keegan George
Martin Brennan
Penar Musaraj
David Taylor
Blake Erickson
Krzysztof Kotlarek
Mark VanLandingham
Jarek Radosz
Loïc Guitaut
Ted Johansson
Natalie Tay
Joffrey JAFFEUX
Kelv
Renato Atilio
Bianca Nenciu
Sérgio Saquetim
Gary Pendergast
Isaac Janzen